Recently we were analyzing the thread-dumps of some JVMs in production and found a large number of threads created in certain thread-pools. Since the name of the thread-pool was generic (e.g. thread-pool-3, etc.) it was very difficult to diagnose the code that spawned the thread-pool.
The following code snippets should help developers in properly naming their thread-pools and also limiting the number of threads. In a cloud environment, the number of default threads in a pool will vary based on the CPU's available - for e.g. in the absence of a thread-pool size, the thread pool may grow to hundreds of threads.
We had seen this happen for the RabbitMQ java client. The default RabbitMQ java client uses a thread pool for callback messages (ACK) and the size of this thread-pool depends on the number of CPUs. Since the JVM is not aware of the docker config, all the processors on the host machine are considered and a large thread pool is created.
More details available here - https://github.com/logstash-plugins/logstash-input-rabbitmq/issues/93
The following code snippets should help developers in properly naming their thread-pools and also limiting the number of threads. In a cloud environment, the number of default threads in a pool will vary based on the CPU's available - for e.g. in the absence of a thread-pool size, the thread pool may grow to hundreds of threads.
We had seen this happen for the RabbitMQ java client. The default RabbitMQ java client uses a thread pool for callback messages (ACK) and the size of this thread-pool depends on the number of CPUs. Since the JVM is not aware of the docker config, all the processors on the host machine are considered and a large thread pool is created.
More details available here - https://github.com/logstash-plugins/logstash-input-rabbitmq/issues/93
No comments:
Post a Comment