jim terry

Ranch Hand
+ Follow
since Nov 18, 2018
Cows and Likes
Cows
Total received
0
In last 30 days
0
Total given
0
Likes
Total received
2
Received in last 30 days
0
Total given
12
Given in last 30 days
0
Forums and Threads
Scavenger Hunt
expand Ranch Hand Scavenger Hunt
expand Greenhorn Scavenger Hunt

Recent posts by jim terry

Learn how to troubleshoot OutOfMemoryError using the Eclipse MAT tool. You will learn best practices, cool tips & tricks to debug memory problems effectively. Watch this video to know more!

In the series of chaos engineering articles, we have been learning to simulate various performance problems. In this post, let’s discuss how to simulate thread leaks. ‘java.lang.OutOfMemoryError: unable to create new native thread’ will be thrown when more threads are created than the memory capacity of the device. When this error is thrown, it will disrupt the application’s availability.

Sample Program
Here is a sample program from the open source BuggyApp application, which keeps creating an infinite number of threads.


You can notice that the sample program contains the ‘ThreadLeakDemo’ class. This class has start() method. In this method, ‘ForeverThread’ is created an infinite number of times because of the ‘while (true)’ loop.

In ‘ForeverThread’ class there is the run() method.  In this method, thread is put to continuous sleep i.e. thread is repeatedly sleeping for 10 minutes again and again. This will keep the ‘ForeverThread’ alive always without doing any activity. A thread will die only if it exits the run() method. In this sample program run() method will never exit because of the never-ending sleep.

Since ‘ThreadLeakDemo’ class keeps creating ‘ForeverThread’ infinitely and they never exit. Thus very soon, several thousands of ‘ForeverThread’ will be created. It will saturate memory capacity, ultimately resulting in ‘java.lang.OutOfMemoryError: unable to create new native thread’ problem.

How to diagnose ‘java.lang.OutOfMemoryError: unable to create new native thread’?
You can diagnose ‘OutOfMemoryError: unable to create new native thread’ problem either through a manual or automated approach.

Manual approach

In the manual approach, you need to capture thread dumps as the first step. A thread dump shows all the threads that are in memory and their code execution path. You can capture thread dump using one of the 8 options mentioned here. But an important criteria is: You need to capture thread dump right when the problem is happening. As thread leaks cause production outage, your support/SRE team might restart the application before thread dumps are captured. If thread dumps are captured after the application is restarted, you won’t be able to identify the leaking threads.  Even if thread dumps are captured at the right point in time, you need to import the thread dumps from production servers to your local machine. Then you need to use thread dump analysis tools like fastThread, Samurai to analyze the thread dumps to identify the problem.

Automated approach

You can use root cause analysis tools like yCrash – which automatically captures application-level data (thread dump, heap dump, Garbage Collection log) and system-level data (netstat, vmstat, iostat, top, top -H, dmesg,…). Besides capturing the data automatically, it marries application-level data and system-level data generates an instant root cause analysis report. Below is the report generated by the yCrash tool when the above sample program is executed:



Fig:  yCrash reporting 12,000+ are created and they can cause ‘OutOfMemoryError: unable to create new native thread’


Fig:  yCrash reporting the line of code in which 12,000+ threads are stuck

From the report, you can notice that yCrash points out that 12,000+ threads are created, and they have the potential to cause ‘OutOfMemoryError: unable to create new native thread’ problem. Besides the thread count, the tool is also reporting the line of code, i.e. ‘com.buggyapp.threadleak.ForeverThread.run(ForeverThread.java:12)‘ in which all the 12,000+ threads are stuck. Equipped with this information, one can quickly go ahead and fix the problematic code.

I was surprised to witness the Java Executor behavior – which caused “java.lang.OutOfMemoryError: unable to create new native thread” in our application. I would like to share my surprise (i.e. problem) and resolution to it.

In order to explain the problem better, I created the following example:

Above program creates 5 workers threads in the runJobs() method and executes 10 DummyJobs in parallel. In order to do so, it uses ThreadPoolExecutor.

In the “runJobs()” method, one would expect ThreadPoolExecutor instance & the worker threads created by that executor would be ready for garbage collection when:

1. “runJobs()” method is completed. As ‘executor’ is a local variable, it should be made available for garbage collection after the execution of the method. As variable has become out of scope.
2. All jobs that were dropped to the Executor were executed.

However surprisingly – even though both of the conditions are meet, still worker threads in the ThreadPoolExecutor instance aren’t getting garbage collected. Whenever “runJobs()” method is called, 5 new worker threads gets created and remains in the memory. If “runJobs()” is called few hundred times, several hundreds of worker threads are created. This causes memory leak in the application.

How to fix this problem?

On line number 37, one would need to invoke “shutdown()” API. So that all the worker threads would be explicitly destroyed after execution of the jobs. So revised runJobs() method would look like:


What triggers this Memory Leak?

Apparently worker threads were put on to wait state by the Executor. Following is the excerpt from the thread dump of the program. Excerpt shows the stack trace of one of the worker thread in the Executor.

You can see the line “at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)". This will park the threads thus making them not to die down. Only when shutdown() API is invoked it would unpark the threads and they would die down.



1 month ago
In this post, we are going to discuss how to find your Java application process Id quickly. For certain monitoring tools like yCrash, you need to pass your application process Id as input. If you want to look for a more detailed post with several different options to find your application’s process Id, you can refer to this blog post.

Finding Java application Process Id in Linux
On any Linux/Unix flavour of Operating system issue the command:



Above command will display all the Java processes running on this machine and their arguments, process Id, and user who launched it. When I issued the above command following was the output in my AWS EC2 Linux instance:



Fig: ‘ps’ command displaying all Java processes running on Linux machine

Red color highlight in the above figure indicates the process Ids of all Java processes running on this EC2 instance. From here, you can get hold of your application’s process Id

Finding Java application Process Id in Windows
‘jps’ – Java Virtual Machine Process Status Tool is packaged in JDK. This tool will display all the Java processes that are running on that machine. Below are the steps to invoke ‘jps’ command

1. open command prompt.

2. cd to ‘bin’ folder, where JDK is installed

3. Issue ‘jps’ command

Example:


When above command was issued, following was the output:



Fig: ‘jps’ command displaying all Java processes running on windows machine

Red color highlight in the above figure indicates the process Ids of all Java processes running on this windows instance. From here, you can get hold of your application’s process Id

Note: Unlike ‘ps’ command in Linux (example given above), you will not see all the Java process arguments. One shortcoming in this approach is ‘jps’ will show only the java process’s first command. You can see all the java process arguments, only when the ‘ps’ command is issued.