Tips and Tricks for Thread Dumps

Tip 1#: To find out the number of threads spawned by the JVM, run the following command: ps -eLF
This command will also print a column called 'LWP ID' (light-weight process ID) that prints the thread-id and the CPU utilization of that thread. This same thread-id can be correlated in the thread-dump obtained from the JVM.

Tip 2#: The thread-dump can be obtained by using the following command: jstack PID
If you are using Spring Boot framework, then we can use the Actuator URLs to download the thread-dump.

Tip 3#: This thread-dump file can be uploaded to a cool online tool : http://fastthread.io/ which gives a beautiful report on the threads running inside the JVM that can be analyzed.

Ruminating on Thread Pool sizes and names

Recently we were analyzing the thread-dumps of some JVMs in production and found a large number of threads created in certain thread-pools. Since the name of the thread-pool was generic (e.g. thread-pool-3, etc.) it was very difficult to diagnose the code that spawned the thread-pool.

The following code snippets should help developers in properly naming their thread-pools and also limiting the number of threads. In a cloud environment, the number of default threads in a pool will vary based on the CPU's available - for e.g. in the absence of a thread-pool size, the thread pool may grow to hundreds of threads.

We had seen this happen for the RabbitMQ java client. The default RabbitMQ java client uses a thread pool for callback messages (ACK) and the size of this thread-pool depends on the number of CPUs. Since the JVM is not aware of the docker config, all the processors on the host machine are considered and a large thread pool is created.

More details available here - https://github.com/logstash-plugins/logstash-input-rabbitmq/issues/93