This week's book giveaway is in the Other Languages forum.
We're giving away four copies of Functional Reactive Programming and have Stephen Blackheath and Anthony Jones on-line!
See this thread for details.
Win a copy of Functional Reactive Programming this week in the Other Languages forum!
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic

exception while Read very large file > 300 MB

 
Ganesh Sugumaran
Greenhorn
Posts: 10
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi
I want to read a file which has size of more than 300 MB.
While i m executing my code, i m getting
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space

Below is my code

Im getting the exception in the line 2.
Even though i increased the Heap space up to 900 Mb, i m getting this error.
(FYI)
I executed in command prompt like below,
java -Xms128m -Xmx1024m readLargeFile

Kindly give a solution for this. is there any other best way to read large text file ??
I m waiting for your reply..
Thanks in advance
 
William Brogden
Author and all-around good cowpoke
Rancher
Posts: 13074
6
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Since UNICODE characters take two bytes, your 300mb file will take at least 600mb.



Starting with 128mb makes no sense if you know it wont work and you will need the maximum.

What do you have to do with the chrBuff when you get it? Could the work be done with a byte[] instead of char?

Bill
 
Ganesh Sugumaran
Greenhorn
Posts: 10
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Yes. I knew that minimum is not a big consideration.
The reason why im using CharBuffer is , using this im matching a particular string n no of times.
My code is follows

After Execution of above code i m getting a part of CharSequence from the above CharBuffer for another search(again Matching process).
So i thought that, for these process(file read and matching process) if i use CharBuffer then it will be faster as well.
So Guide me How to overcome HeapSpace Memory Problem.
Is there any other way to read and matching process in efficient way ??
i remember you that file size is more.
 
steve souza
Ranch Hand
Posts: 862
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
can you read a certain amount of the data into a buffer (say 10k) and perform a match against each of these buffers? of course a match may span a buffer so you would have to provide special logic for this case
 
Jimmy Clark
Ranch Hand
Posts: 2187
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Is there any other way to read and matching process in efficient way ??


Write the program in KornShell or Perl.
 
Ganesh Sugumaran
Greenhorn
Posts: 10
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thanks for suggestion.

But I've to do in JAVA only.
 
Ganesh Sugumaran
Greenhorn
Posts: 10
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thanks for your reply.
My task is to open a large file in READ&WRITE mode and i need to search some portion of text in that file by searching starting and end point.
Then i need to write that searched area of text to a new file and delete that portion from the original file.

The above process i will do more times.
So I thought that for these process, it will be easy by loading the file into memory by CharBuffer and can search easily by MATCHER class.

Above is my task, then now suggest me some efficient ways.
Note that my file size will be more and using JAVA only i've to do these things.

Thanks in Advance...
 
Amit Ghorpade
Bartender
Posts: 2854
10
Fedora Firefox Browser Java
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
"Ganesh PS " please check your private messages for an important administrative matter. You can check them by clicking the My Private Messages link above.
 
Duc Vo
Ranch Hand
Posts: 254
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Ganesh Sugumaran wrote:Thanks for your reply.
My task is to open a large file in READ&WRITE mode and i need to search some portion of text in that file by searching starting and end point.
Then i need to write that searched area of text to a new file and delete that portion from the original file.

The above process i will do more times.
So I thought that for these process, it will be easy by loading the file into memory by CharBuffer and can search easily by MATCHER class.

Above is my task, then now suggest me some efficient ways.
Note that my file size will be more and using JAVA only i've to do these things.

Thanks in Advance...


Well, how about writing your own search instead of using regex? How complex are your starting and ending points? Are they just regular string? or html tags? or something else?

The whole process should look like:



 
Kees Jan Koster
JavaMonitor Support
Rancher
Posts: 251
5
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Dear Ganesh,

In my experience Java is a dreadful language for the kind of file processing you are tasked with. If you insist on using Java, don't whine about Java using gobs of memory to do this, because it is the wrong tool for the job.

Instead, learn about UNIX shell scripting (notably awk and sed) and you'll find it can be done orders of magnitude faster that Java can do this.
 
It is sorta covered in the JavaRanch Style Guide.
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic