Win a copy of Kotlin in Action this week in the Kotlin forum!
  • Post Reply Bookmark Topic Watch Topic
  • New Topic

application hanging but no heap exhaustion  RSS feed

 
luuk citroen wty
Greenhorn
Posts: 13
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator


hallo. maximum heap size is set at 1.5GB and consumption of memory at peak load is about 1.1GB. when it reaches 1.1GB, application starts to hang. what could be the problem? shouldn't it be hanging at the point where memory is about equal to the max heap setting? no heap dumps were generated. is this due to server hardware or something? already got the garbage collection data and nothing seemed unusual.
 
Steve Luke
Bartender
Posts: 4181
22
IntelliJ IDE Java Python
  • Likes 1
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
There is not enough information here to figure out why a hang occurs. Hanging applications can be caused by many things, the first thing to do is define exactly what 'hanging' means. Does the GUI become unresponsive but other parts of the application continue to work? Does the entire application pause temporarily, or does it pause forever?

Then you need to figure out why, based on the symptoms. Is it putting long tasks on the GUI thread, race conditions, long garbage collection cycles, deadlock, infinite loops... You will need to look at thread dumps but what you will be looking for depends a lot on what the symptom is.
 
luuk citroen wty
Greenhorn
Posts: 13
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Garbage collection data:

Overall Garbage Collection overhead: 3.34% (AKA the Total Garbage collection pause time / Total Elapsed time)
Maximum Garbage collection overhead: 100%
Total Garbage collection duration: 3.12% of the elapsed time
Total Garbage collection interval: 96.88% of the elapsed time

From Core Dump: 59% of all threads are blocked


Nothing seems unusual. What could be wrong?

 
William Brogden
Author and all-around good cowpoke
Rancher
Posts: 13078
6
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
From Core Dump: 59% of all threads are blocked

Nothing seems unusual. What could be wrong?


Really? blocked threads is normal? What are they blocked on?

You still have not explained exactly what "application hanging" means. Like Steve said what do you mean?... not accepting input? accepting input but doing nothing? We see lots of problems described as "hanging" You have not even said anything about the interface / operating system/ architecture.

Bill
 
luuk citroen wty
Greenhorn
Posts: 13
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Well i figured that since heap consumption is already near its limit, threads would be naturally blocked as there is no more memory available for them so they have to wait.... Unless that understanding of mine is incorrect

OS is AIX.

Hanging meaning any action you do in the application takes too long to load. You have to restart app server instance to have it back to normal again
 
William Brogden
Author and all-around good cowpoke
Rancher
Posts: 13078
6
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hanging meaning any action you do in the application takes too long to load. You have to restart app server instance to have it back to normal again


AHA so this is a client - server situation - pity you did not feel it necessary to describe the architecture earlier. This "server" - is it an HTTP servlet server like Tomcat or some other kind of server.

My crystal ball is in the shop for the 10,000 vision checkup - you must realize that we won't magically know what kind of a system you are dealing with unless you tell us.

I am going to take a SWAG (Scientific Wild Ass Guess) and say your application is "hung" due to the unavailability of some resource, NOT memory. Are those Threads trying to connect to a database? When this happened to me it was due to threads waiting for a service that had not been started.

If this is a Tomcat server we are talking about, can you run the Management App? That helped in my situation.

Bill
 
luuk citroen wty
Greenhorn
Posts: 13
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Sorry my bad, there's a correction in my previous post. I meant waiting threads not blocked threads.

@William: web server is IBM HTTP server and app server is Websphere 6.1 . Architecture? hmmm, not sure about that. Not sure if its load balanced, failover , clustered or what. ill check on that.
 
William Brogden
Author and all-around good cowpoke
Rancher
Posts: 13078
6
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
OK, on to the next question. your initial statement:


consumption of memory at peak load is about 1.1GB. when it reaches 1.1GB, application starts to hang.



Describes load in terms of memory used, but does not say how this load is generated. Are you running some sort of artificial load test simulation or are these real users?

How many users/requests does it take to get to the point where the application "hangs"?

That seems to me like quite a lot of memory - what does this application do that takes so much memory?

Are database operations involved?

The question Steve asked is still open - what exactly does "hanging" mean? Possibilities include:

1. Each request takes progressively longer and longer to respond until users just give up.
2. Seems normal until a request is simply never responded to

Will the server respond to requests on other apps when this critical one is "hung" - for example simple HTML page requests? This is a critical distinction.

What is the CPU utilization pattern here? Does a "hung" application still consume CPU cycles?

Bill

 
luuk citroen wty
Greenhorn
Posts: 13
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Glad you mentioned the cpu utilization, I was just about that ask that.

Based on data, CPU utilization nears 100% when the "hang" occurs. What I'd like to ask is, can high cpu utilization (due to other running jobs in the server) cause threads in the app server to be in wait mode? or is it vice versa where accumulation of waiting threads can start to cause high cpu utilization?


To answer your question:

1. This occurs in production during peak load where 65% are transactions done and 35% report generation. Both has connection to the database and web services are being called. Not sure about that number of concurrent users.
 
William Brogden
Author and all-around good cowpoke
Rancher
Posts: 13078
6
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
You are still not answering all of my questions - this would go much faster if you would READ the questions thoroughly and not jump to conclusions.

A. which type of hang? 1 or 2 ?
B. are other apps affected? can simple HTML requests run when your app is "hung"?

The CPU utilization is due to running Threads, waiting Threads do not use significant CPU time.

Other apps on the server will slow your app but not force your Threads to wait.

Since both database and web service connections are being made there are many many possibilities for resources not being cleaned up - are you using some sort of database connection pool? Did you write the web service connection portion of this application?

Surely Websphere has something like the Tomcat management app that will let you look at the state of user requests.

Bill
 
luuk citroen wty
Greenhorn
Posts: 13
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Type of Hang:

. Each request takes progressively longer and longer to respond until users just give up.

are other apps affected?

yes


can simple HTML requests run when your app is "hung"?


not sure. we'll take a look at this


are you using some sort of database connection pool?

Yes


Did you write the web service connection portion of this application?


Our application just consumes a web service from another app. A library was given to us to connect to the web service, we did not write the ws connection portion
 
William Brogden
Author and all-around good cowpoke
Rancher
Posts: 13078
6
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
So - high CPU use, high memory use - system gets slower and slower.

If this was my problem I would be inserting logging statements to monitor each request, reporting memory use at start and end of request, and time elapsed per request.

Time consumed in each DB call would also be of interest.

See java.lang.System.currentTimeMillis and java.lang.Runtime maxMemory and freeMemory

How many simultaneous users are we talking about here?

Bill
 
Chris Hurst
Ranch Hand
Posts: 443
3
C++ Eclipse IDE Java
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
- What were your minor and full(??) GC's like ? Have you tried GCViewer its free. Do you have Xmx set to the same value as Xms ?

- Have you eliminated deadlock in your threads ?.. you can do this from the command line against your application.

- Have you eliminated the OS - swap file ? (which OS ?)


It will 'hang' before max memory is used in that the GC times will get longer and longer till they hit a threshold then the JVM should give up and dump if you have enabled it however that would not be expected given the usage you appear to be reporting.
 
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!