• Post Reply Bookmark Topic Watch Topic
  • New Topic

Java heap size, outofmemory  RSS feed

 
xy zhang
Greenhorn
Posts: 14
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
We have a program which does the following stuff:

use multi threads to fetch web document(url) from the internet

1, put document in documentdb

2, index documents in batch (count = 100) and put them in indexdb

3, repeat 1 and 2

usu. i assign 10 threads to do the fetching, and then assign a website to crawl. to small website, it works well.

but to a big website, the threads will be interrupted due to the famous Java heap size, outofmemory issue. I ever tried to set the -Xmx, but it seems question can not be solved.

Give me some suggestion?

Thanks!
 
Olaf Kummer
Greenhorn
Posts: 11
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Originally posted by xy zhang:
but to a big website, the threads will be interrupted due to the famous Java heap size, outofmemory issue. I ever tried to set the -Xmx, but it seems question can not be solved.

This is probably not a concurrency issue. Have you tried to reduce the number of threads to 1?

How does the application react to increasing the heap size? Does it take linearly longer until the application fills up? Have you searched for possible memory leaks like global hash maps, thread locals, or job lists?

You may want to run a memory profiler like Optimizeit.

Olaf

[ August 15, 2005: Message edited by: Olaf Kummer ]
[ August 15, 2005: Message edited by: Olaf Kummer ]
 
David Ulicny
Ranch Hand
Posts: 724
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
What is the size of your physical RAM in your computer? How many takes the JVM? Maybe you just don't have enough memory for running 10 threads.
 
David Ulicny
Ranch Hand
Posts: 724
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I suppose xy is not your real name. Read about naming policy.
 
xy zhang
Greenhorn
Posts: 14
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thanks, Olaf Kummer

I will think about what you suggested.
 
xy zhang
Greenhorn
Posts: 14
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
David Ulicny
My phisycal mem is 512M

And xy is the abbreviation of my name, or the first letter of my name.
 
It is sorta covered in the JavaRanch Style Guide.
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!