Multi-Thread Performance

[Ultra Powerz]

Hi ALl:
I wrote a Multi-Thread program to fetch web data.
and I need it run 24 hr a day. But there always
be problems:
1.It will make CPU usage up to 100%, until the
page is completely downloaded.
2.Memory is never enough, the usage keep growing
as long as the program is running, and finally
get an outOfMemory Exception.
I am using J2SE 1.4.0_01 on Windows 2000 Server.
It starts with 5 threads to get web page, 1 thread
to extract URL link in every downloaded page and
another thread to write to Database.When I stop
the Thread, the usage of memory stop growing, but
never going down unless I quit the program.
Any help will be appreciated.
Thanks.

[El Roy Cabildo]

i have encountered that problem too..but it was resolved that by making my code more precise..Just look at your code that would make your thread continously make a loop, change it by sleeping the threads or using wait() and notify().

[Ultra Powerz]

Thanks...
Yes, I have made all threads sleeping after every
page done. It can slow down the usage of CPU, but
not memory. So I still get an outOfMemory Exception if it run for a long time.

[kapil munjal]

OutofMemory error comes when all the jvm memory is consumed. Look into your code and see if there are any objects which are not been nullify.

[Ilja Preuss]

Hello "Ultra Powerz",
welcome to JavaRanch!

We're a friendly group with not many rules, but we do require members to have valid display names.

Display names must be two words: your first name, a space, then your last name. Fictitious names are not allowed.

Please edit your profile and correct your display name.
Thanks, and have fun!

[Paul Fouxson]

Two suggestions:
1: Make sure there are no references to the objects you no longer need. For example, if you store a reference in a container, the memory will not be released until there are no more references to a particular object.
2: Once you reset all of the references, invoke:
System.gc() or Runtime.getRuntime().gc(). This should suggest JVM to kick off a garbage collector.

[Guy Allard]

I'm responsible for several highly multithreaded applications at work.

When work arrives (batches) the apps typically spawn 20-40 threads, each of which is making HTTP (SOAP really) invocations. This happens 2-3 times in a 24 hour period.

These apps will run for *weeks* without need for a restart.

And they make no System calls to suggest garbage implementation.

I think you have a memory leak bug - just because there is a GC component in the runtime does not guarantee your code is right.

Guy

[Stan James]

To the 100% CPU "problem" ... why would you want a program that uses less? Would you want it to run longer than necessary? Well, maybe if you want to do anything else at the same time. I have a little download program that runs up to 100% CPU (3.1 ghz desktop PC) with about 4 threads. I gave some thought to interrupting it so I could browse the Ranch at the same time but decided to just let it have the machine for as long as it takes. Actually Windows is good enough to let me do other things, just slightly slower than usual.

[Ernest Friedman-Hill]

All you guys, starting with Kapil, are responding to a four year old thread. Let it go, eh?

[Stefan Wagner]

Exhumition party?
Let's wait for halloween ...