• Post Reply Bookmark Topic Watch Topic
  • New Topic

Java Memory Issue

 
Gandhi M Subramanian
Greenhorn
Posts: 3
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi,

How System Works :

1. Fetching the data from the database (Max of 300000 Records) and storing it List.
2. Iterating thro the List and conveting them into arranged data(application specific grouping)
3. Convert the whole data into a report.
( At this time the system works fine).

Problem Area:
1. As per future requirement i would get max of (1.1 Millon Record)
2. Storing that into List and processing from the list blew up with OutOfMemorryError.

Any Solution:

One Solution i thought of is
1. connect to database get all the records in a single query and have it in resultset and iterate throught resultset to get a speicfied chunk of data of resultset and iterate back again until all the records are fetched from the resultset.

Anything wrong in my solution that i have thoguht about , if you guys could provide me solutions to this problem that would be of great help to me..

Thanks in Advance
 
Scott Selikoff
author
Bartender
Posts: 4033
18
Eclipse IDE Flex Google Web Toolkit
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Sounds fine to me, you might also consider using a stored procedure if possible to perform some of your data grouping. You could also increase memory of the application using the -Xmx switch such as -Xmx384M for 384 megabytes.
 
JuanP barbancho
Ranch Hand
Posts: 52
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
HI,

Download all record in a file, make a program that read the file and make your process.

You could too using a very complex SQL query with Pararell process.

Thanks you
 
Ilja Preuss
author
Sheriff
Posts: 14112
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Originally posted by JuanP barbancho:

Download all record in a file, make a program that read the file and make your process.


How is that going to help?
 
Vinnie Jenks
Ranch Hand
Posts: 207
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Pagination might be useful here.

Break your records up into sets of 100, 1000, whatever...and only display that chunk in a page at one time...so you'd only really be pulling a small block at once in a query. When you move to the next page, grab the next block of records.
 
Gandhi M Subramanian
Greenhorn
Posts: 3
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thanks a lot for your suggestions and help guys.. i have put in the code sample i did for the chunk fetch and writing it to file. ( please check the Handle Huge Data subject in this forum. appreciate your help.

Gandhi
 
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!