OutOfMemory Using JDK 1.3

[Sanjeev Dhupkar]

All,
This is a very funny problem I am facing. We have an application which processes flat files and extracts information from the file and then applies lot of business rules to convert the data into a desired output. This process does a lot of I/O and Database calls. This application runs perfectly fine and is currently in Production using JDK version 1.2.x
We are in process of migrating this to JDK 1.3.x and this process crashes out while processing a large flat file. This same file processes perfectly fine in the application compiled under JDK 1.2.x
Can anyone help me out on this? Any help is really appreciated.
Thanks & Regards,
Sanjeev Dhupkar

[John Smith]

Can anyone help me out on this? Any help is really appreciated.
Are you closing the resources (such as database connections) after they are used (in the finally block)? Did you try running your app with a -verbosegc flag? Do you have a profiler (such as OptimizeIt)? Are you using the -Xms/Xmx options when you start JVM? How does a stacktrace look like when the error is thrown? Got the corresponding section of code to show?

[Sanjeev Dhupkar]

Hi Eugene
The stack trace looks like this
com.csfb.equity.efs.tcv.rulevalidator.ClientRuleValidatorService - Exception in ClientRuleValidatorService :
java.sql.SQLException: weblogic.rmi.extensions.RemoteRuntimeException - with nested exception:
[java.lang.OutOfMemoryError:
Start server side stack trace:
java.lang.OutOfMemoryError
<<no stack trace available>>
End server side stack trace

Start server side stack trace:
java.lang.OutOfMemoryError:
Start server side stack trace:
java.lang.OutOfMemoryError
<<no stack trace available>>
End server side stack trace

<<no stack trace available>>
End server side stack trace

Start server side stack trace:
java.lang.OutOfMemoryError:
Start server side stack trace:
java.lang.OutOfMemoryError
<<no stack trace available>>
End server side stack trace

Start server side stack trace:
java.lang.OutOfMemoryError:
Start server side stack trace:
java.lang.OutOfMemoryError
<<no stack trace available>>
End server side stack trace

<<no stack trace available>>
End server side stack trace

<<no stack trace available>>
End server side stack trace
]
at weblogic.jdbc.rmi.SerialResultSet.getString(SerialResultSet.java:380)
at com.csfb.equity.efs.tcv.rulevalidator.ClientRuleValidatorService.evaluateResult(ClientRuleValidatorService.java:471)
at com.csfb.equity.efs.tcv.rulevalidator.ClientRuleValidatorService.applyClientRules(ClientRuleValidatorService.java:114)
at com.csfb.equity.efs.tcv.fieldvalidation.FieldValidation.doValidationProcessing(FieldValidation.java:480)
at com.csfb.equity.efs.tcv.fieldvalidation.FieldValidation.validateTrade(FieldValidation.java:387)
at com.csfb.equity.efs.tcv.templatevalidation.TemplateValidationService.insertTrade(TemplateValidationService.java:1807)
at com.csfb.equity.efs.tcv.templatevalidation.TemplateValidationService.validateTrade(TemplateValidationService.java:690)
at com.csfb.equity.efs.tcv.templatevalidation.TemplateValidationService.validateTradeFile(TemplateValidationService.java:385)
at com.csfb.equity.efs.tcv.service.FileValidatorService.checkFileUpdateDatabase(FileValidatorService.java:385)
at com.csfb.equity.efs.tcv.service.FileValidatorService.execute(FileValidatorService.java:64)
at com.csfb.equity.efs.tcv.listener.ListenerThread.processInput(ListenerThread.java:131)
at com.csfb.equity.efs.tcv.listener.ListenerThread.run(ListenerThread.java:58)
at java.lang.Thread.run(Thread.java:484)
2003-11-19 17:12:26,956 [127.0.0.1:1069277602546] ERROR
===============================================
java.sql.SQLException: weblogic.rmi.extensions.RemoteRuntimeException - with nested exception:
[java.lang.OutOfMemoryError:
Start server side stack trace:
java.lang.OutOfMemoryError
<<no stack trace available>>
End server side stack trace
]
at weblogic.jdbc.rmi.SerialPreparedStatement.executeQuery(SerialPreparedStatement.java:47)
2003-11-19 17:09:39,725 [Thread-12] DEBUG com.csfb.equity.efs.tcv.util.Notification - to [email protected]|
at com.csfb.equity.efs.tcv.templatevalidation.TemplateValidationService.insertTrade(TemplateValidationService.java:1768)
at com.csfb.equity.efs.tcv.templatevalidation.TemplateValidationService.validateTrade(TemplateValidationService.java:690)
at com.csfb.equity.efs.tcv.templatevalidation.TemplateValidationService.validateTradeFile(TemplateValidationService.java:385)
at com.csfb.equity.efs.tcv.service.FileValidatorService.checkFileUpdateDatabase(FileValidatorService.java:385)
at com.csfb.equity.efs.tcv.service.FileValidatorService.execute(FileValidatorService.java:64)
at com.csfb.equity.efs.tcv.listener.ListenerThread.processInput(ListenerThread.java:131)
at com.csfb.equity.efs.tcv.listener.ListenerThread.run(ListenerThread.java:58)
at java.lang.Thread.run(Thread.java:484)
================================================
java.sql.SQLException: JZ006: Caught IOException: java.io.IOException: JZ0EM: End of data.

Start server side stack trace:
java.sql.SQLException: JZ006: Caught IOException: java.io.IOException: JZ0EM: End of data.
at com.sybase.jdbc2.jdbc.ErrorMessage.raiseError(ErrorMessage.java:493)
at com.sybase.jdbc2.jdbc.ErrorMessage.raiseErrorCheckDead(ErrorMessage.java:729)
at com.sybase.jdbc2.tds.Tds.handleIOE(Tds.java:3107)
at com.sybase.jdbc2.tds.Tds.nextResult(Tds.java:2110)
at com.sybase.jdbc2.tds.TdsResultSet.nextResult(TdsResultSet.java:510)
at com.sybase.jdbc2.tds.TdsResultSet.next(TdsResultSet.java:296)
at com.sybase.jdbc2.tds.TdsResultSet.close(TdsResultSet.java:392)
at com.sybase.jdbc2.jdbc.SybResultSet.markDead(SybResultSet.java:1507)
at com.sybase.jdbc2.jdbc.SybResultSet.close(SybResultSet.java:1565)
at com.sybase.jdbc2.jdbc.SybResultSet.close(SybResultSet.java:182)
at weblogic.jdbc.pool.ResultSet.close(ResultSet.java:216)
at weblogic.jdbc.rmi.internal.ResultSetImpl.close(ResultSetImpl.java:147)
at weblogic.jdbc.rmi.internal.ResultSetImpl_WLSkel.invoke(Unknown Source)
at weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:441)
at weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:382)
at weblogic.security.service.SecurityServiceManager.runAs(SecurityServiceManager.java:726)
at weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:377)
at weblogic.rmi.internal.BasicExecuteRequest.execute(BasicExecuteRequest.java:30)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:234)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:210)
End server side stack trace

at weblogic.rmi.internal.BasicOutboundRequest.sendReceive(BasicOutboundRequest.java:85)
at weblogic.rmi.internal.BasicRemoteRef.invoke(BasicRemoteRef.java:135)
at weblogic.rmi.internal.ProxyStub.invoke(ProxyStub.java:35)
at $Proxy4.close(Unknown Source)
at weblogic.jdbc.rmi.internal.ResultSetStub.close(ResultSetStub.java:74)
at weblogic.jdbc.rmi.SerialResultSet.close(SerialResultSet.java:95)
at weblogic.jdbc.rmi.SerialResultSet.close(SerialResultSet.java:86)
at com.csfb.equity.efs.tcv.rulevalidator.ClientRuleValidatorService.applyGenericRules(ClientRuleValidatorService.java:300)
at com.csfb.equity.efs.tcv.fieldvalidation.FieldValidation.validateTrade(FieldValidation.java:345)
at com.csfb.equity.efs.tcv.templatevalidation.TemplateValidationService.insertTrade(TemplateValidationService.java:1807)
at com.csfb.equity.efs.tcv.templatevalidation.TemplateValidationService.validateTrade(TemplateValidationService.java:690)
at com.csfb.equity.efs.tcv.templatevalidation.TemplateValidationService.validateTradeFile(TemplateValidationService.java:385)
at com.csfb.equity.efs.tcv.service.FileValidatorService.checkFileUpdateDatabase(FileValidatorService.java:385)
at com.csfb.equity.efs.tcv.service.FileValidatorService.execute(FileValidatorService.java:64)
at com.csfb.equity.efs.tcv.listener.ListenerThread.processInput(ListenerThread.java:131)
at com.csfb.equity.efs.tcv.listener.ListenerThread.run(ListenerThread.java:58)
at java.lang.Thread.run(Thread.java:484)
Exception in FieldValidation:validateTrade() java.sql.SQLException: JZ006: Caught IOException: java.io.IOException: JZ0EM: End of data.

I am sure com.csfb.equity.efs.tcv.rulevalidator.ClientRuleValidatorService - Exception in ClientRuleValidatorService :
java.sql.SQLException: weblogic.rmi.extensions.RemoteRuntimeException - with nested exception:
[java.lang.OutOfMemoryError:
Start server side stack trace:
java.lang.OutOfMemoryError
<<no stack trace available>>
End server side stack trace

Start server side stack trace:
java.lang.OutOfMemoryError:
Start server side stack trace:
java.lang.OutOfMemoryError
<<no stack trace available>>
End server side stack trace

<<no stack trace available>>
End server side stack trace

Start server side stack trace:
java.lang.OutOfMemoryError:
Start server side stack trace:
java.lang.OutOfMemoryError
<<no stack trace available>>
End server side stack trace

Start server side stack trace:
java.lang.OutOfMemoryError:
Start server side stack trace:
java.lang.OutOfMemoryError
<<no stack trace available>>
End server side stack trace

<<no stack trace available>>
End server side stack trace

<<no stack trace available>>
End server side stack trace
]
at weblogic.jdbc.rmi.SerialResultSet.getString(SerialResultSet.java:380)
at com.csfb.equity.efs.tcv.rulevalidator.ClientRuleValidatorService.evaluateResult(ClientRuleValidatorService.java:471)
at com.csfb.equity.efs.tcv.rulevalidator.ClientRuleValidatorService.applyClientRules(ClientRuleValidatorService.java:114)
at com.csfb.equity.efs.tcv.fieldvalidation.FieldValidation.doValidationProcessing(FieldValidation.java:480)
at com.csfb.equity.efs.tcv.fieldvalidation.FieldValidation.validateTrade(FieldValidation.java:387)
at com.csfb.equity.efs.tcv.templatevalidation.TemplateValidationService.insertTrade(TemplateValidationService.java:1807)
at com.csfb.equity.efs.tcv.templatevalidation.TemplateValidationService.validateTrade(TemplateValidationService.java:690)
at com.csfb.equity.efs.tcv.templatevalidation.TemplateValidationService.validateTradeFile(TemplateValidationService.java:385)
at com.csfb.equity.efs.tcv.service.FileValidatorService.checkFileUpdateDatabase(FileValidatorService.java:385)
at com.csfb.equity.efs.tcv.service.FileValidatorService.execute(FileValidatorService.java:64)
at com.csfb.equity.efs.tcv.listener.ListenerThread.processInput(ListenerThread.java:131)
at com.csfb.equity.efs.tcv.listener.ListenerThread.run(ListenerThread.java:58)
at java.lang.Thread.run(Thread.java:484)
2003-11-19 17:12:26,956 [127.0.0.1:1069277602546] ERROR
===============================================
java.sql.SQLException: weblogic.rmi.extensions.RemoteRuntimeException - with nested exception:
[java.lang.OutOfMemoryError:
Start server side stack trace:
java.lang.OutOfMemoryError
<<no stack trace available>>
End server side stack trace
]
at weblogic.jdbc.rmi.SerialPreparedStatement.executeQuery(SerialPreparedStatement.java:47)
2003-11-19 17:09:39,725 [Thread-12] DEBUG com.csfb.equity.efs.tcv.util.Notification - to [email protected]|
at com.csfb.equity.efs.tcv.templatevalidation.TemplateValidationService.insertTrade(TemplateValidationService.java:1768)
at com.csfb.equity.efs.tcv.templatevalidation.TemplateValidationService.validateTrade(TemplateValidationService.java:690)
at com.csfb.equity.efs.tcv.templatevalidation.TemplateValidationService.validateTradeFile(TemplateValidationService.java:385)
at com.csfb.equity.efs.tcv.service.FileValidatorService.checkFileUpdateDatabase(FileValidatorService.java:385)
at com.csfb.equity.efs.tcv.service.FileValidatorService.execute(FileValidatorService.java:64)
at com.csfb.equity.efs.tcv.listener.ListenerThread.processInput(ListenerThread.java:131)
at com.csfb.equity.efs.tcv.listener.ListenerThread.run(ListenerThread.java:58)
at java.lang.Thread.run(Thread.java:484)
================================================
java.sql.SQLException: JZ006: Caught IOException: java.io.IOException: JZ0EM: End of data.

Start server side stack trace:
java.sql.SQLException: JZ006: Caught IOException: java.io.IOException: JZ0EM: End of data.
at com.sybase.jdbc2.jdbc.ErrorMessage.raiseError(ErrorMessage.java:493)
at com.sybase.jdbc2.jdbc.ErrorMessage.raiseErrorCheckDead(ErrorMessage.java:729)
at com.sybase.jdbc2.tds.Tds.handleIOE(Tds.java:3107)
at com.sybase.jdbc2.tds.Tds.nextResult(Tds.java:2110)
at com.sybase.jdbc2.tds.TdsResultSet.nextResult(TdsResultSet.java:510)
at com.sybase.jdbc2.tds.TdsResultSet.next(TdsResultSet.java:296)
at com.sybase.jdbc2.tds.TdsResultSet.close(TdsResultSet.java:392)
at com.sybase.jdbc2.jdbc.SybResultSet.markDead(SybResultSet.java:1507)
at com.sybase.jdbc2.jdbc.SybResultSet.close(SybResultSet.java:1565)
at com.sybase.jdbc2.jdbc.SybResultSet.close(SybResultSet.java:182)
at weblogic.jdbc.pool.ResultSet.close(ResultSet.java:216)
at weblogic.jdbc.rmi.internal.ResultSetImpl.close(ResultSetImpl.java:147)
at weblogic.jdbc.rmi.internal.ResultSetImpl_WLSkel.invoke(Unknown Source)
at weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:441)
at weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:382)
at weblogic.security.service.SecurityServiceManager.runAs(SecurityServiceManager.java:726)
at weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:377)
at weblogic.rmi.internal.BasicExecuteRequest.execute(BasicExecuteRequest.java:30)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:234)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:210)
End server side stack trace

at weblogic.rmi.internal.BasicOutboundRequest.sendReceive(BasicOutboundRequest.java:85)
at weblogic.rmi.internal.BasicRemoteRef.invoke(BasicRemoteRef.java:135)
at weblogic.rmi.internal.ProxyStub.invoke(ProxyStub.java:35)
at $Proxy4.close(Unknown Source)
at weblogic.jdbc.rmi.internal.ResultSetStub.close(ResultSetStub.java:74)
at weblogic.jdbc.rmi.SerialResultSet.close(SerialResultSet.java:95)
at weblogic.jdbc.rmi.SerialResultSet.close(SerialResultSet.java:86)
at com.csfb.equity.efs.tcv.rulevalidator.ClientRuleValidatorService.applyGenericRules(ClientRuleValidatorService.java:300)
at com.csfb.equity.efs.tcv.fieldvalidation.FieldValidation.validateTrade(FieldValidation.java:345)
at com.csfb.equity.efs.tcv.templatevalidation.TemplateValidationService.insertTrade(TemplateValidationService.java:1807)
at com.csfb.equity.efs.tcv.templatevalidation.TemplateValidationService.validateTrade(TemplateValidationService.java:690)
at com.csfb.equity.efs.tcv.templatevalidation.TemplateValidationService.validateTradeFile(TemplateValidationService.java:385)
at com.csfb.equity.efs.tcv.service.FileValidatorService.checkFileUpdateDatabase(FileValidatorService.java:385)
at com.csfb.equity.efs.tcv.service.FileValidatorService.execute(FileValidatorService.java:64)
at com.csfb.equity.efs.tcv.listener.ListenerThread.processInput(ListenerThread.java:131)
at com.csfb.equity.efs.tcv.listener.ListenerThread.run(ListenerThread.java:58)
at java.lang.Thread.run(Thread.java:484)
Exception in FieldValidation:validateTrade() java.sql.SQLException: JZ006: Caught IOException: java.io.IOException: JZ0EM: End of data.

I am sure we have used the finally block to close resources. But will cross-check again.

[John Smith]

java.sql.SQLException: JZ006: Caught IOException: java.io.IOException: JZ0EM: End of data.
at com.sybase.jdbc2.jdbc.ErrorMessage.raiseError(ErrorMessage.java:493)
at com.sybase.jdbc2.jdbc.ErrorMessage.raiseErrorCheckDead(ErrorMessage.java:729)
at com.sybase.jdbc2.tds.Tds.handleIOE(Tds.java:3107)
at com.sybase.jdbc2.tds.Tds.nextResult(Tds.java:2110)
at com.sybase.jdbc2.tds.TdsResultSet.nextResult(TdsResultSet.java:510)
at com.sybase.jdbc2.tds.TdsResultSet.next(TdsResultSet.java:296)
at com.sybase.jdbc2.tds.TdsResultSet.close(TdsResultSet.java:392)
at com.sybase.jdbc2.jdbc.SybResultSet.markDead(SybResultSet.java:1507)
at com.sybase.jdbc2.jdbc.SybResultSet.close(SybResultSet.java:1565)
at com.sybase.jdbc2.jdbc.SybResultSet.close(SybResultSet.java:182)
at weblogic.jdbc.pool.ResultSet.close(ResultSet.java:216)
at weblogic.jdbc.rmi.internal.ResultSetImpl.close(ResultSetImpl.java:147)
at weblogic.jdbc.rmi.internal.ResultSetImpl_WLSkel.invoke(Unknown Source)
at weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:441)
at weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:382)
at weblogic.security.service.SecurityServiceManager.runAs(SecurityServiceManager.java:726)
at weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:377)
at weblogic.rmi.internal.BasicExecuteRequest.execute(BasicExecuteRequest.java:30)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:234)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:210)

Well, here is your problem: as the stack trace indicates, weblogic attempts to close the resultset, but never succeeds, -- an SQLException is raised. I would suggest that you concentrate on the source for that exception, and I would not be surprised if once you fix it, your memory leak will go away.
How does your ClientRuleValidatorService.java look like? Looks like it may be the originator of the problem.

[Chinmay Bajikar]

Sanjeev,
Here are couple of my experiances which were causing similar problems in my appln.
1)Having global PreparedStatements,which will be used throughout the application lifetime.These statements cannot be closed.For such statements we need to close their ResultSets.
2)While doing the above thing for a non-global PreparedStatement a careless subordinate put in a statement.close() under an if, and had a ResultSet.close() after that.If the code went through the If, an exception would occur.
3)Management of transaction using setAutoCommit().This might give a problem if you are using a connection pool.
Thanks,
Chinmay