The soul is dyed the color of its thoughts. Think only on those things that are in line with your principles and can bear the light of day. The content of your character is your choice. Day by day, what you do is who you become. Your integrity is your destiny - it is the light that guides your way. - Heraclitus
SCJP 5.0, SCJD in progress
SCJP 1.4 - SCJP 6 - SCWCD 5 - OCEEJBD 6 - OCEJPAD 6
How To Ask Questions How To Answer Questions
I thought that way I can manage to process all the CSV files with 512MB memory. I guess I was wrong.
SCJP 5.0, SCJD in progress
The soul is dyed the color of its thoughts. Think only on those things that are in line with your principles and can bear the light of day. The content of your character is your choice. Day by day, what you do is who you become. Your integrity is your destiny - it is the light that guides your way. - Heraclitus
The soul is dyed the color of its thoughts. Think only on those things that are in line with your principles and can bear the light of day. The content of your character is your choice. Day by day, what you do is who you become. Your integrity is your destiny - it is the light that guides your way. - Heraclitus
If it does become a limitation, another way to go is to produce a non-redundant, sorted list of all the ids from all files (first pass; very quick), then individually sort just the ids from each file and store just their position (second pass) - since you can use the position in the global id list as the key, it's just an array of integers, so just 30MB per file. Load all those into memory, iterate over the global list and grab the data for each record with random file access, like you were planning to do (third pass). You could actually combine the first and second pass into one, with slightly more work.
Originally posted by TH Lim:
All these memory consumptions were obtained using OS X "taskmanager". I invoked Runtime.getRuntime().freeMemory() to get memory consumed by my application. The values I got are way off the 1 reported by the "taskmanager". How do I calculate the memory consumed programmatically?
Is there any way to determine the JVM memory usage without using any sophisticated memtools?
I am planning to use H2 and been writing some simple tests to determine if it can solve my problem.
Thanks.
[ October 11, 2008: Message edited by: TH Lim ]