So we have created a new Netty4 server and are generating load against it. The number of host threads is growing as a function of the TPS to the server. Moreover once the load test is over, the host thread count is not going down (suggesting there is a thread leak of some kind).
I took an hprof dump using jstack and hooked it up to JProfiler. In the thread view the lions share of the threads are of the format. (Note this is a thread dump hours after the load test and is one instance of it, there are thousands of these being stuck in ).
Thread dump at 26013:42.622.361
Thread group "main":
Thread "I/O dispatcher 18120":
at sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int)
at sun.nio.ch.EPollArrayWrapper.poll(long) (line: 269)
at sun.nio.ch.EPollSelectorImpl.doSelect(long) (line: 93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(long) (line: 86)
at sun.nio.ch.SelectorImpl.select(long) (line: 97)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute() (line: 255)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(org.apache.http.nio.reactor.IOEventDispatch) (line: 104) at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run() (line: 588 at java.lang.Thread.run() (line: 748)
I am not sure how to proceed from here further (for instance I have no idea what executor service IO dispatcher is supposed to reference).
This question is for general guidance on how to attack thread leak problems like this.
So it turns out the IO thread dispatcher is part of the Apache HTTP client.
We had to keep a cache of apache http clients (to be more pedantic, aws elastic search clients since they are setup per request credentials https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-configuration-samples.html). Anytime we saw a new request we acquired from the client from this cache.
Now our load test was throwing many requests with different credentials. The cache kept getting larger and larger faster than the clients expired from the cache. Each client used up more and more threads until we ran out of them.
Related
I am doing an upgrade from JBPM 3 to 7. The Process Instances runs on a different thread, Sometimes the UI thread needs to access the process Instance via the KIESession. If I try to execute an operation for example sending a signal the UI is blocked until the process instances finish.
I noticed this also happens if I try to about the process Instance, get the status of the the ProcessInstance and get a variable from the process Instance.
I looked further into it and the PersistableRunner.execute() is synchronized .
FYI I am using the per-process Instance strategy.
Is there a way to get around this issue?
A snippet of the thread DUMP:
java.lang.Thread.State: BLOCKED
waiting for JBPM-Processor-5692548#955268 to release lock on <0xb176> (a org.drools.persistence.PersistableRunner)
at org.drools.persistence.PersistableRunner.execute(PersistableRunner.java:400)
at org.drools.persistence.PersistableRunner.execute(PersistableRunner.java:68)
I tried using the singleton strategy
I would like to be able to access a running KieSession from a different threat without being blocked.
I have a web server that is run by a fleet of 50 hosts. One request to the server can result in 10 subsequent network calls, all of which can be done in parallel.
My idea is to create an executor service thread pool with 10 threads, so that each host can make the network calls in parallel.
However, there seems to be a problem with this. What if I get 1000 requests at once? And suppose a single host is tasked with 20 requests at the same time? Does this mean that the host will only have 10 threads available, and thus all 20 requests will compete with each other for the 10 threads? This seems WORSE than without thread pooling, in which case each request lives on its own thread and there's effectively 20 threads running at once.
Thus, it appears as if executor service is very dangerous in this situation, and has potential to actually make my application slower when in spiky volume. Am I understanding the situation correctly? If so, what is a way to solve it? Should I have each request CREATE the 10 threads manually, rather than attempting to share from a pool and introduce that entanglement between different requests?
You seem to be conflating thread pooling to mean easier thread creation. But, its primary aim to reduce the thread requirements of an application, because threads get reused. So, if the first request ends up starting 10 threads, when the second request comes in, some of them may be available to be reused. So, the second request may not end up creating another 10 additional threads, but maybe 5. And the third request may not create any new thread at all. So, based on this, at a time, your service may need a thread-pool with only 15 threads. The advantage of the thread-pool in this case is these 15 threads will get created shortly after the requests start coming in, and will get reused till it dies, and your application, its runtime, and the underlying OS will not waste time creating and destroying threads, allocating stacks for them, etc.
Using varnishstat, the metric 'sess_herd' is increasing a lot, during trafic and it seems that I've maybe reached some a limit (300 sess_herd / s)
I think, I got no backend issue (all busy, unhealthy, retry, failed at 0).
Backend_req/Client_req is around 150 req/s.
Right now, our Varnish isn't caching at all, it is just "proxying" to our backend server. So the "pass" rates is about 150 req/s
What could explain such a sess_herd ?
Session_herd
Regards
Olivier
Session herding is a counter that indicates when an ongoing session (TCP-connection) is handed off the worker thread and to a waiter that keeps it while the client thinks.
By default a connection get to keep its worker thread for 50ms (timeout_linger parameter in 4.1) before this happens.
Since networks and clients are slow, a worker thread can in that way serve a whole lot of clients. This reduces the number of running threads needed.
In practice this happens after a response has been sent and while waiting for another request on the reused connection.
I'm using Managed Executor Service to implement a process manager which will process tasks in the background upon receiving an JMS message event. Normally, there will be a small number of tasks running (maybe 10 max) but what if something happens and my application starts getting hundred of JMS message events. How do I handle such event?
My thought is to limit the number of threads if possible and save all the other messages to database and will be run when thread available. Thanks in advance.
My thought is to limit the number of threads if possible and save all the other messages to database and will be run when thread available.
The detailed answer to this question depends on which Java EE app server you choose to run on, since they all have slightly different configuration.
Any Java EE app server will allow you to configure the thread pool size of your Managed Executor Service (MES), this is the number of worker threads for your thread pool.
Say you have a 10 worker threads, and you get flooded with 100 requests all at once, the MES will keep a queue of requests that are backlogged, and the worker threads will take work off the queue whenever they finish work until the queue is empty.
Now, it's fine if work goes to the queue sometimes but if overall your work queue increases more quickly than your worker threads can take work off the queue, you will run into problems. The solution to this is to increase your thread pool size otherwise the backlog will get overrun and your server will run out of memory.
what if something happens and my application starts getting hundred of JMS message events. How do I handle such event?
If the load on your server will be so sporadic that tasks need to be saved to a database, it seems that the best approach would be to either:
increase thread pool size
have the server immediately reject incoming tasks when the task backlog queue is full
have clients do a blocking wait for the server task queue to be not full (I would only advise this option if client task submission is in no way connected to user experience)
we have a mule app with HTTP inbound endpoint and I'm trying to figure out how to control the thread count under load. As an experiment I have added the following configuration:
<core:configuration>
<core:default-threading-profile doThreading="false" maxThreadsActive="500" poolExhaustedAction="RUN"/>
</core:configuration>
Under load I'm seeing the thread count peak at over 1000 threads. Am not sure why this is the case give the maxThreadsActive setting and the doThreading="false". Reading about poolExhaustedAction="RUN", I would expect the listener thread to block while processing inbound requests rather than spawn new ones, and finally reject the connection if its backlog queue is full. I never see rejected client connections.
Does Mule maintain a separate thread pool for each inbound endpoint in the app (sorry if this is in the documentation)? Even if so, don't think it helps explain what I'm seeing.
Any help appreciated. We are running a number of mule apps in one container and I'd like to control the total number of threads.
Thanks, Alfie.
Clearly the doThreading attribute on default-threading-profile is not enough to control Mule threading as a whole nor limit with a global cap the specific threading behaviour of transports. I reckon you're getting 500 threads for the HTTP message receiver pool and 500 for the VM message dispatcher pool.
I strongly suggest you reading about tuning Mule: http://www.mulesoft.org/documentation/display/current/Tuning+Performance
My gut feel is that you need to
configure threading on each transport (VM, HTTP), strictly specifying the pool size for receivers and dispatchers,
select flow processing strategies that prevent Mule from spawning new threads (i.e. use synchronous to hog the receiver threads),
select exchange patterns that also prevent Mule from spawning new threads (i.e. use request-response to piggyback the current execution thread).