• Post Reply Bookmark Topic Watch Topic
  • New Topic

running out of memory - String pool/Thread/ Database/Daemon  RSS feed

 
anthony ward
Greenhorn
Posts: 5
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi,

I have a question with regards to the subject. I have to query a database that contains about 100 million of rows and I have to process some of the String result. The issue I run into is java memory issue heap space. Yes I do have a server and I can increase the memory to 5GB+ and it will work but I want to make sure first the program is efficient.

basically the code looks like
===============================
...
StringBuilder str = new StringBuilder("");
String query = "return 100Million + rows"
ResultSet rs = statement.execute(query)
while(rs.next()){
/**process some of the string
store the data in str.append("...\n");
**/
}

/*** save to file***
....... (close connections)

===================

Now I'm curious I can easily create threads in the rs.next() to do the processing. So if I do this, does the thread have its own Java string pool and heap space so that when it close/destroy the garbage collectors cleans the memory?
Really the main issue is doing the processing of the ResultSet.


Also, I'm building a daemon that should run indefinitely and I want to make sure that it won't run out of memory. So I want to make sure when the threads within it all the java memory such as string pool etc.. gets properly cleaned when destroyed .

Thanks


 
Jelle Klap
Bartender
Posts: 1952
7
Eclipse IDE Java
  • Likes 1
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Now I'm curious I can easily create threads in the rs.next() to do the processing. So if I do this, does the thread have its own Java string pool and heap space so that when it close/destroy the garbage collectors cleans the memory?
Really the main issue is doing the processing of the ResultSet.


No, each thread gets its own stack, but the heap is shared among all threads, and by extension so is the String constant pool - those objects are allocated in the heap's permanent generation.
I don't see how the String constant pool is relevant here by the way, unless you intern() all the String data you get from each row in the resultset.
Also, I don't see multiple threads buying you anything, as the JVM would still run out of heap space. I think a better option would be to not buffer the data in-memoy after processing it, but to write it directly to the file.

 
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!