Bill Gathen

Update (sparked by comment on JDBC cross-post):
The bottom of the memory-usage sawtooth is constant. It's not a textbook memory leak, where something is not being dereferenced and lives until the program dies.
It's the *top* of the sawtooth that I'm concerned with. If the initial allocated heap is 2 meg, it seems to let the objects pile up until just below 2 meg, then run gc. The used memory drops drastically, then starts building back up.
The core problem seems to be that between the time the gc decides to run again and the time it actually starts freeing memory, the main thread has added a couple more objects (running past 2 meg) and has to allocate more space for the heap. Now the allocated heap space is bigger, so it goes longer before gc'ing the next time, with same lag problem increasing the size yet again. Repeat x,000 times and the heap has gotten very large.

The bottom of the sawtooth is constant. It's not a textbook memory leak, where something is not being dereferenced and lives until the program dies.
It's the *top* of the sawtooth that I'm concerned with. If the initial allocated heap is 2 meg, it seems to let the objects pile up until just below 2 meg, then run gc. The used memory drops drastically, then starts building back up.
The core problem seems to be that between the time the gc decides to run again and the time it actually starts freeing memory, the main thread has added a couple more objects (running past 2 meg) and has to allocate more space for the heap. Now the allocated heap space is bigger, so it goes longer before gc'ing the next time, with same lag problem increasing the size yet again. Repeat x,000 times and the heap has gotten very large.

// Simplified for brevity
String qry = "select orig_num from v_sur_universal where inv_id = ?";
String orig_num;
try
{
Connection conn =
ConnectionFactory.getConnection("club", "DBproperties.txt");
PreparedStatement stmt = conn.prepareStatement(qry);
stmt.setInt(1, 427167);
ResultSet rset = stmt.executeQuery();
orig_num = rset.getString(1);
}
catch (SQLException sql) { sql.getMessage(); }

I'm rewriting an app that pulls a dozen data items for about six million rows in an Oracle database. About half of these are strings, and a majority are unique (physical addresses and 10-digit phone numbers).
I've been profiling my memory usage, and the garbage collector gets me back down to a fairly constant size every time it runs, but it gets behind at the threshold, which means it allocates a little extra space before clearing the memory, and the heap gets bigger every cycle.
I posted on the JDBC forum, hoping there was a way to keep strings out of the process, but alternately, is there a way to fiddle with the GC to prevent the reallocate before the collection? Maybe by boosting the priority of the thread?
I'm running System.gc() as often as is rational (by my definition , but the spikes keep getting taller.
Has anyone run into a problem like this before? Am I barking up the wrong tree?
Thanks in advance,
Bill

I'm rewriting an app that pulls a dozen data items for about six million rows in an Oracle database. About half of these are strings, and a majority are unique (physical addresses and 10-digit phone numbers).
I've been profiling my memory usage, and the garbage collector gets me back down to a fairly constant size every time it runs, but it gets behind at the threshold, which means it allocates a little extra space before clearing the memory, and the heap gets bigger every cycle.
Is there a way of pulling them (or streaming them) directly into a StringBuffer, to prevent the string pool from getting enormous from all the unique values? I've been trying getAsciiStream and getCharacterStream but not having much luck.
Alternately, is there a way to fiddle with the GC to prevent the reallocate before the collection?
Has anyone run into a problem like this before? Am I barking up the wrong tree?
Thanks in advance,
Bill