• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

Java heap size, outofmemory

 
Greenhorn
Posts: 14
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
We have a program which does the following stuff:

use multi threads to fetch web document(url) from the internet

1, put document in documentdb

2, index documents in batch (count = 100) and put them in indexdb

3, repeat 1 and 2

usu. i assign 10 threads to do the fetching, and then assign a website to crawl. to small website, it works well.

but to a big website, the threads will be interrupted due to the famous Java heap size, outofmemory issue. I ever tried to set the -Xmx, but it seems question can not be solved.

Give me some suggestion?

Thanks!
 
Greenhorn
Posts: 11
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Originally posted by xy zhang:
but to a big website, the threads will be interrupted due to the famous Java heap size, outofmemory issue. I ever tried to set the -Xmx, but it seems question can not be solved.


This is probably not a concurrency issue. Have you tried to reduce the number of threads to 1?

How does the application react to increasing the heap size? Does it take linearly longer until the application fills up? Have you searched for possible memory leaks like global hash maps, thread locals, or job lists?

You may want to run a memory profiler like Optimizeit.

Olaf

[ August 15, 2005: Message edited by: Olaf Kummer ]
[ August 15, 2005: Message edited by: Olaf Kummer ]
 
Ranch Hand
Posts: 724
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
What is the size of your physical RAM in your computer? How many takes the JVM? Maybe you just don't have enough memory for running 10 threads.
 
David Ulicny
Ranch Hand
Posts: 724
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I suppose xy is not your real name. Read about naming policy.
 
xy zhang
Greenhorn
Posts: 14
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Thanks, Olaf Kummer

I will think about what you suggested.
 
xy zhang
Greenhorn
Posts: 14
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
David Ulicny
My phisycal mem is 512M

And xy is the abbreviation of my name, or the first letter of my name.
 
With a little knowledge, a cast iron skillet is non-stick and lasts a lifetime.
reply
    Bookmark Topic Watch Topic
  • New Topic