Forums Register Login

Processing a large data file(55Mb)

+Pie Number of slices to send: Send
At one point in my program, I performing calculations on number of rows(~600,000). I use JSP & JAVA. Here are some things I noticed.
1. It's very slow! :-)
2. Initially, the cpu monitor shows a high level of cpu utilisation, but this slowly drops off with a blip every second or so. Basically, things slow down as the process chugs along.
3. I eventually get a java.Lang.OutOfMemory exception after approx 1:00 -1:30 hrs.
I tried running the JVM with heap size set to 256 Mbytes. I still get the outof mem error.
Is there a faster/better way? Any ideas on how I can get rid of the out of mem errors.....???
I tried searching in the forum for similar kind of discussions but..could n't get the solution for it..Pls help...!!!
+Pie Number of slices to send: Send
1. Not really appropriate for JSP forum
2. Something is very inefficient in your processing algorithm.
probably you make a lot of String objects and fail to discard them. Converting bytes to char when reading a String is VERY slow and of course creating lots of objects slows things down too.
3. Since you have the memory to spare apparently, read the whole file into a byte array and process it by moving pointers through it.
Bill

------------------
author of:
See where your hand is? Not there. It's next to this tiny ad:
a bit of art, as a gift, the permaculture playing cards
https://gardener-gift.com


reply
reply
This thread has been viewed 1091 times.
Similar Threads
tomcat & hw configuration
java program problem
Iterating over hashMap in facelets not working
Mulitple fo:page-sequence
jboss5.1 eap,insufficient memory error
More...

All times above are in ranch (not your local) time.
The current ranch time is
Mar 28, 2024 17:03:08.