• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

exception while Read very large file > 300 MB

 
Greenhorn
Posts: 10
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi
I want to read a file which has size of more than 300 MB.
While i m executing my code, i m getting

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space


Below is my code

Im getting the exception in the line 2.
Even though i increased the Heap space up to 900 Mb, i m getting this error.
(FYI)
I executed in command prompt like below,

java -Xms128m -Xmx1024m readLargeFile


Kindly give a solution for this. is there any other best way to read large text file ??
I m waiting for your reply..
Thanks in advance
 
Author and all-around good cowpoke
Posts: 13078
6
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Since UNICODE characters take two bytes, your 300mb file will take at least 600mb.



Starting with 128mb makes no sense if you know it wont work and you will need the maximum.

What do you have to do with the chrBuff when you get it? Could the work be done with a byte[] instead of char?

Bill
 
Ganesh Sugumaran
Greenhorn
Posts: 10
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Yes. I knew that minimum is not a big consideration.
The reason why im using CharBuffer is , using this im matching a particular string n no of times.
My code is follows

After Execution of above code i m getting a part of CharSequence from the above CharBuffer for another search(again Matching process).
So i thought that, for these process(file read and matching process) if i use CharBuffer then it will be faster as well.
So Guide me How to overcome HeapSpace Memory Problem.
Is there any other way to read and matching process in efficient way ??
i remember you that file size is more.
 
Ranch Hand
Posts: 862
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
can you read a certain amount of the data into a buffer (say 10k) and perform a match against each of these buffers? of course a match may span a buffer so you would have to provide special logic for this case
 
Ranch Hand
Posts: 2187
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Is there any other way to read and matching process in efficient way ??



Write the program in KornShell or Perl.
 
Ganesh Sugumaran
Greenhorn
Posts: 10
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Thanks for suggestion.

But I've to do in JAVA only.
 
Ganesh Sugumaran
Greenhorn
Posts: 10
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Thanks for your reply.
My task is to open a large file in READ&WRITE mode and i need to search some portion of text in that file by searching starting and end point.
Then i need to write that searched area of text to a new file and delete that portion from the original file.

The above process i will do more times.
So I thought that for these process, it will be easy by loading the file into memory by CharBuffer and can search easily by MATCHER class.

Above is my task, then now suggest me some efficient ways.
Note that my file size will be more and using JAVA only i've to do these things.

Thanks in Advance...
 
Bartender
Posts: 2856
10
Firefox Browser Fedora Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
"Ganesh PS " please check your private messages for an important administrative matter. You can check them by clicking the My Private Messages link above.
 
Ranch Hand
Posts: 254
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Ganesh Sugumaran wrote:Thanks for your reply.
My task is to open a large file in READ&WRITE mode and i need to search some portion of text in that file by searching starting and end point.
Then i need to write that searched area of text to a new file and delete that portion from the original file.

The above process i will do more times.
So I thought that for these process, it will be easy by loading the file into memory by CharBuffer and can search easily by MATCHER class.

Above is my task, then now suggest me some efficient ways.
Note that my file size will be more and using JAVA only i've to do these things.

Thanks in Advance...



Well, how about writing your own search instead of using regex? How complex are your starting and ending points? Are they just regular string? or html tags? or something else?

The whole process should look like:



 
JavaMonitor Support
Posts: 251
5
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Dear Ganesh,

In my experience Java is a dreadful language for the kind of file processing you are tasked with. If you insist on using Java, don't whine about Java using gobs of memory to do this, because it is the wrong tool for the job.

Instead, learn about UNIX shell scripting (notably awk and sed) and you'll find it can be done orders of magnitude faster that Java can do this.
 
With a little knowledge, a cast iron skillet is non-stick and lasts a lifetime.
reply
    Bookmark Topic Watch Topic
  • New Topic