Problem first:
My service reponse time increases while the concurrency increases. I found 1000+ threads blocks on the step that tomcat response to client. But I don't know how to solve it.
Background:
My service is a Spring boot project which embedded a tomcat. (tomcat-embed-core-8.5.23)
My experiment is to use JMeter as client, create 2000 threads to send request to my service for 10 round (totally 20000 requests). And after my service there is a stub services to response in 2 seconds delay for each request. (i.e. JMeter (2000 thread) => My Service => Stub Server )
experiment data
In My service, I set the “server.tomcat.max-threads=2000”. So there are 2000+ threads named "http-nio-8080*”
The result is, the max response time are >30 s. This is copied from CLI,
summary = 20000 in 00:01:23 = 241.7/s Avg: 6890 Min: 2288 Max: 31158 Err: 0 (0.00%)
The Min response time (2288ms) is as expected (as there are 2000ms cost in stub server). But the Max response time (31158 ms) is too long. I analyzed the jstack log. There were 1121 threads blocked on the same lock. (I attached one of the stack content. And I think it’s the time point of My service response to JMeter. But unfortunately there is a SynchronizedStack blocking the 1000+ threads)
I have evidence that the lock is the real killer of the response time. In another jstack log 9 seconds later, there were still 1058 threads lock on the same lock.
"http-nio-8080-exec-3988" #4038 daemon prio=5 os_prio=0 tid=0x00007fdec045c800 nid=0x1050 waiting for monitor entry [0x00007fdb6c94e000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.tomcat.util.collections.SynchronizedStack.pop(SynchronizedStack.java:75)
- waiting to lock <0x00000000dac7afd0> (a org.apache.tomcat.util.collections.SynchronizedStack)
at org.apache.tomcat.util.net.NioBlockingSelector.write(NioBlockingSelector.java:89)
at org.apache.tomcat.util.net.NioSelectorPool.write(NioSelectorPool.java:157)
at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.doWrite(NioEndpoint.java:1267)
at org.apache.tomcat.util.net.SocketWrapperBase.doWrite(SocketWrapperBase.java:670)
at org.apache.tomcat.util.net.SocketWrapperBase.flushBlocking(SocketWrapperBase.java:607)
at org.apache.tomcat.util.net.SocketWrapperBase.flush(SocketWrapperBase.java:597)
at org.apache.coyote.http11.Http11OutputBuffer.flushBuffer(Http11OutputBuffer.java:581)
at org.apache.coyote.http11.Http11OutputBuffer.flush(Http11OutputBuffer.java:272)
at org.apache.coyote.http11.Http11Processor.flush(Http11Processor.java:1560)
at org.apache.coyote.AbstractProcessor.action(AbstractProcessor.java:283)
at org.apache.coyote.Response.action(Response.java:173)
at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:317)
at org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:284)
at org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java:118)
at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:297)
at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141)
- locked <0x00000000fe000000> (a java.io.OutputStreamWriter)
at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229)
at org.springframework.util.StreamUtils.copy(StreamUtils.java:119)
at org.springframework.http.converter.StringHttpMessageConverter.writeInternal(StringHttpMessageConverter.java:106)
at org.springframework.http.converter.StringHttpMessageConverter.writeInternal(StringHttpMessageConverter.java:41)
at org.springframework.http.converter.AbstractHttpMessageConverter.write(AbstractHttpMessageConverter.java:227)
at org.springframework.web.servlet.mvc.method.annotation.AbstractMessageConverterMethodProcessor.writeWithMessageConverters(AbstractMessageConverterMethodProcessor.java:247)
at org.springframework.web.servlet.mvc.method.annotation.RequestResponseBodyMethodProcessor.handleReturnValue(RequestResponseBodyMethodProcessor.java:174)
at org.springframework.web.method.support.HandlerMethodReturnValueHandlerComposite.handleReturnValue(HandlerMethodReturnValueHandlerComposite.java:81)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:113)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:661)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:110)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:108)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:106)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:478)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
- locked <0x00000000e1a59188> (a org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- <0x00000000df55e210> (a java.util.concurrent.ThreadPoolExecutor$Worker)
You mistakenly allow tomcat to overcommit.
First thing first, limit the tomcat executor threads to about 100 or 200 max.
Practically you should never have more than 200 threads in any jvm, unless you really know what you are doing and have 99% of them waiting (not blocked monitor entry).
Then, limit the tcp SYN queue length (the serversocket accept queue) to make sure that you don't accept all jmeter connections when too many are made.
Start with severely limited threads and increase the number of thread only as long as you have idle on both network bandwidth and CPU, and even then, as long as the responsetime doesn't increase.
Related
We are facing issue in our production environment with multiple blocking threads and while checking the thread dumps have found the below stack trace where in 14 blocking threads were present with same description. Any help with the below is much appreciated.
We are using weblogic 10.3.6.0 version and JDK 1.7.0_80.
ExecuteThread: '387' is blocked because ExecuteThread: '159' is already holding the lock and is long-running.
======================
"[ACTIVE] ExecuteThread: '387' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00002ac72d4f4000 nid=0x530d waiting for monitor entry [0x00002ac767c38000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.springframework.beans.factory.support.AbstractBeanFactory.getMergedBeanDefinition(AbstractBeanFactory.java:1030)
- waiting to lock <0x000000051ea30728> (a java.util.concurrent.ConcurrentHashMap)
at org.springframework.beans.factory.support.AbstractBeanFactory.getMergedBeanDefinition(AbstractBeanFactory.java:1013)
at org.springframework.beans.factory.support.AbstractBeanFactory.getMergedLocalBeanDefinition(AbstractBeanFactory.java:999)
at org.springframework.beans.factory.support.AbstractBeanFactory.isTypeMatch(AbstractBeanFactory.java:433)
..
"[ACTIVE] ExecuteThread: '159' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00002ac7491f5800 nid=0x4781 runnable [0x00002ac7540fd000]
java.lang.Thread.State: RUNNABLE
at org.springframework.beans.factory.support.AbstractBeanFactory.getMergedBeanDefinition(AbstractBeanFactory.java:1030)
- locked <0x000000051ea30728> (a java.util.concurrent.ConcurrentHashMap)
at org.springframework.beans.factory.support.AbstractBeanFactory.getMergedBeanDefinition(AbstractBeanFactory.java:1013)
at org.springframework.beans.factory.support.AbstractBeanFactory.getMergedLocalBeanDefinition(AbstractBeanFactory.java:999)
..
at com.opensymphony.xwork2.spring.SpringObjectFactory.buildBean(SpringObjectFactory.java:151)
[ACTIVE] ExecuteThread: '359' for queue: 'weblogic.kernel.Default (self-tuning)'
[ACTIVE] ExecuteThread: '359' for queue: 'weblogic.kernel.Default (self-tuning)'
Stack Trace is:
java.lang.Thread.State: RUNNABLE
at oracle.jdbc.driver.OracleResultSetImpl.getString(OracleResultSetImpl.java:2818)
- locked <0x0000000566efc058> (a oracle.jdbc.driver.T4CConnection)
at oracle.jdbc.driver.OracleResultSet.getString(OracleResultSet.java:1215)
at weblogic.jdbc.wrapper.ResultSet_oracle_jdbc_driver_OracleResultSetImpl.getString(Unknown Source)
at com.ibatis.sqlmap.engine.type.StringTypeHandler.getResult(StringTypeHandler.java:35)
at com.ibatis.sqlmap.engine.mapping.result.ResultMap.getPrimitiveResultMappingValue(ResultMap.java:619)
at com.ibatis.sqlmap.engine.mapping.result.ResultMap.getResults(ResultMap.java:345)
at com.ibatis.sqlmap.engine.mapping.result.AutoResultMap.getResults(AutoResultMap.java:47)
- locked <0x00000005205e0968> (a com.ibatis.sqlmap.engine.mapping.result.AutoResultMap)
at com.ibatis.sqlmap.engine.execution.SqlExecutor.handleResults(SqlExecutor.java:384)
at com.ibatis.sqlmap.engine.execution.SqlExecutor.handleMultipleResults(SqlExecutor.java:300)
at com.ibatis.sqlmap.engine.execution.SqlExecutor.executeQuery(SqlExecutor.java:189)
at com.ibatis.sqlmap.engine.mapping.statement.MappedStatement.sqlExecuteQuery(MappedStatement.java:221)
at com.ibatis.sqlmap.engine.mapping.statement.MappedStatement.executeQueryWithCallback(MappedStatement.java:189)
at com.ibatis.sqlmap.engine.mapping.statement.MappedStatement.executeQueryForList(MappedStatement.java:139)
at com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.queryForList(SqlMapExecutorDelegate.java:567)
at com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.queryForList(SqlMapExecutorDelegate.java:541)
at com.ibatis.sqlmap.engine.impl.SqlMapSessionImpl.queryForList(SqlMapSessionImpl.java:118)
at org.springframework.orm.ibatis.SqlMapClientTemplate$3.doInSqlMapClient(SqlMapClientTemplate.java:298)
at org.springframework.orm.ibatis.SqlMapClientTemplate.execute(SqlMapClientTemplate.java:209)
at org.springframework.orm.ibatis.SqlMapClientTemplate.executeWithListResult(SqlMapClientTemplate.java:249)
at org.springframework.orm.ibatis.SqlMapClientTemplate.queryForList(SqlMapClientTemplate.java:296)
at org.springframework.orm.ibatis.SqlMapClientTemplate.queryForList(SqlMapClientTemplate.java:290)
at com.dms.common.masters.daos.HomeDaoImpl.populateDropDowns(Unknown Source)
at com.dms.common.masters.daos.HomeDaoImpl$$FastClassByCGLIB$$dda6d555.invoke(<generated>)
at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:191)
at org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:696)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:106)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:631)
at com.dms.common.masters.daos.HomeDaoImpl$$EnhancerByCGLIB$$7cce49ec.populateDropDowns(<generated>)
at com.dms.common.masters.business.HomeBusinessImpl.populateDropDowns(Unknown Source)
at com.dms.common.masters.service.HomeServiceImpl.populateDropDowns(Unknown Source)
at com.dms.common.masters.action.HomeAction.menu(Unknown Source)
at sun.reflect.GeneratedMethodAccessor54415.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.opensymphony.xwork2.DefaultActionInvocation.invokeAction(DefaultActionInvocation.java:450)
at com.opensymphony.xwork2.DefaultActionInvocation.invokeActionOnly(DefaultActionInvocation.java:289)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:252)
at org.apache.struts2.interceptor.MultiselectInterceptor.intercept(MultiselectInterceptor.java:73)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:246)
at com.opensymphony.xwork2.interceptor.ConversionErrorInterceptor.intercept(ConversionErrorInterceptor.java:138)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:246)
at com.opensymphony.xwork2.interceptor.ParametersInterceptor.doIntercept(ParametersInterceptor.java:254)
at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:98)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:246)
at com.opensymphony.xwork2.interceptor.StaticParametersInterceptor.intercept(StaticParametersInterceptor.java:191)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:246)
at org.apache.struts2.interceptor.CheckboxInterceptor.intercept(CheckboxInterceptor.java:91)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:246)
at org.apache.struts2.interceptor.FileUploadInterceptor.intercept(FileUploadInterceptor.java:252)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:246)
at com.opensymphony.xwork2.interceptor.ModelDrivenInterceptor.intercept(ModelDrivenInterceptor.java:100)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:246)
at com.opensymphony.xwork2.interceptor.ScopedModelDrivenInterceptor.intercept(ScopedModelDrivenInterceptor.java:141)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:246)
at com.opensymphony.xwork2.interceptor.ChainingInterceptor.intercept(ChainingInterceptor.java:145)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:246)
at com.opensymphony.xwork2.interceptor.I18nInterceptor.intercept(I18nInterceptor.java:139)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:246)
at com.opensymphony.xwork2.interceptor.PrepareInterceptor.doIntercept(PrepareInterceptor.java:171)
at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:98)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:246)
at org.apache.struts2.interceptor.ServletConfigInterceptor.intercept(ServletConfigInterceptor.java:164)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:246)
at com.opensymphony.xwork2.interceptor.AliasInterceptor.intercept(AliasInterceptor.java:193)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:246)
at com.opensymphony.xwork2.interceptor.ExceptionMappingInterceptor.intercept(ExceptionMappingInterceptor.java:189)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:246)
at com.utils.LoginInterceptor.intercept(Unknown Source)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:246)
at org.apache.struts2.impl.StrutsActionProxy.execute(StrutsActionProxy.java:54)
at org.apache.struts2.dispatcher.Dispatcher.serviceAction(Dispatcher.java:562)
at org.apache.struts2.dispatcher.ng.ExecuteOperations.executeAction(ExecuteOperations.java:77)
at org.apache.struts2.dispatcher.ng.filter.StrutsPrepareAndExecuteFilter.doFilter(StrutsPrepareAndExecuteFilter.java:99)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:60)
at com.opensymphony.module.sitemesh.filter.PageFilter.parsePage(PageFilter.java:118)
at com.opensymphony.module.sitemesh.filter.PageFilter.doFilter(PageFilter.java:52)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:60)
at com.utils.SetResponseBufferFilter.doFilter(Unknown Source)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:60)
at oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:119)
at java.security.AccessController.doPrivileged(Native Method)
at oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:324)
at oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:460)
at oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:103)
at oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:171)
at oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:71)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:60)
at oracle.dms.servlet.DMSServletFilter.doFilter(DMSServletFilter.java:163)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:60)
at weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:27)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:60)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3748)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3714)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2283)
at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2182)
at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1499)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:263)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
Locked ownable synchronizers:
- None
Blocked threads are not necessarily a concern, in java when handling concurrent requests and using singleton classes or hashmaps etc its natural to have some blocking as a means to achieve sanity in data...
When to take thread dumps -
When weblogic server is reporting stuck threads, or you are getting general slowness in response (though many other factors contribute to this)
How to take thread dumps -
Always take 3-4 thread dumps spaced 1-2 secs apart. Looking at 1 momentary thread dump usually does not provide the complete picture.. If you notice the same threads are in blocked state in almost all 3-4 dumps, check which thread is holding the lock and what it is doing, is the thread holding the lock doing the same task in all 4 dumps, if so you got the clue on what to troubleshoot..
In the thread dump you gave..
ExecuteThread: '387' is waiting on
waiting to lock <0x000000051ea30728>
and ExecuteThread: '159' is holding locked <0x000000051ea30728> but 159 is in Runnable state, meaning its not stuck and doing its normal operations..
had you taken several thread dumps and saw 159 holding the lock for far too long you need to consider tuning your application and think about a workaround if possible around this lock.
The problem seems to be related to the lock on com.ibatis.sqlmap.engine.mapping.result.AutoResultMap
suggesting that part of the code is synchronized, meaning that ONLY one thread at a time will be executed.
If you collect a thread dump, you'll see several threads are waiting for the lock and only one is executing something like:
"[ACTIVE] ExecuteThread: '339' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=REMOVED nid=REMOVED runnable [REMOVED]
java.lang.Thread.State: RUNNABLE
at weblogic.jdbc.wrapper.ResultSet_oracle_jdbc_driver_OracleResultSetImpl.wasNull(Unknown Source)
at com.ibatis.sqlmap.engine.type.StringTypeHandler.getResult(StringTypeHandler.java:36)
at com.ibatis.sqlmap.engine.mapping.result.ResultMap.getPrimitiveResultMappingValue(ResultMap.java:619)
at com.ibatis.sqlmap.engine.mapping.result.ResultMap.getResults(ResultMap.java:345)
at com.ibatis.sqlmap.engine.mapping.result.AutoResultMap.getResults(AutoResultMap.java:47)
- locked <0x00000004544b7328> (a com.ibatis.sqlmap.engine.mapping.result.AutoResultMap)
at com.ibatis.sqlmap.engine.execution.SqlExecutor.handleResults(SqlExecutor.java:384)
at com.ibatis.sqlmap.engine.execution.SqlExecutor.handleMultipleResults(SqlExecutor.java:300)
...
That leads us to another question, does IBatis support multithread? Based on the following link the answer is no:
Mybatis does not work in multithreaded programming
In standard usage of MyBatis core module, the SqlSession and mapper object does not support the thread-safe. This is design. The MyBatis provide the SqlSessionManager for supporting the thread-safe as optional component.
https://github.com/mybatis/mybatis-3/issues/1529
We have a spark streaming application, for each interval it will read a hive table and do some aggregation, usually it's very fast and can finish in 20 seconds. However after run several hours, one executor will be very slow, while other executor is fast as normal, I check the slow server: disk, network has no problem and there is other executors of other application on same server, they are all good.
The CPU usage of slow executor is 100%, and I get threaddump of slow executor, it is reading HDFS, stack below. The data size reading is 128M, it should be done in several seconds, but in this slow executor it takes 90+ seconds.
Any ideas? Thanks in advance.
2017-06-15 15:04:04
"Executor task launch worker-172" daemon prio=10 tid=0x00007fb308001000 nid=0x3966 runnable [0x00007fb338a4c000]
java.lang.Thread.State: RUNNABLE
at sun.nio.cs.UTF_8$Decoder.decodeArrayLoop(UTF_8.java:211)
at sun.nio.cs.UTF_8$Decoder.decodeLoop(UTF_8.java:354)
at java.nio.charset.CharsetDecoder.decode(CharsetDecoder.java:561)
at java.nio.charset.CharsetDecoder.decode(CharsetDecoder.java:783)
at org.apache.hadoop.io.Text.decode(Text.java:405)
at org.apache.hadoop.io.Text.decode(Text.java:382)
at org.apache.hadoop.io.Text.toString(Text.java:280)
at org.apache.spark.sql.hive.HiveInspectors$class.unwrap(HiveInspectors.scala:322)
at org.apache.spark.sql.hive.HadoopTableReader$.unwrap(TableReader.scala:315)
at org.apache.spark.sql.hive.HadoopTableReader$$anonfun$14$$anonfun$apply$14.apply(TableReader.scala:401)
at org.apache.spark.sql.hive.HadoopTableReader$$anonfun$14$$anonfun$apply$14.apply(TableReader.scala:401)
at org.apache.spark.sql.hive.HadoopTableReader$$anonfun$fillObject$2.apply(TableReader.scala:416)
at org.apache.spark.sql.hive.HadoopTableReader$$anonfun$fillObject$2.apply(TableReader.scala:408)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
at scala.collection.Iterator$$anon$14.hasNext(Iterator.scala:389)
at org.apache.spark.sql.execution.aggregate.TungstenAggregationIterator.processInputs(TungstenAggregationIterator.scala:511)
at org.apache.spark.sql.execution.aggregate.TungstenAggregationIterator.<init>(TungstenAggregationIterator.scala:686)
2017-06-15 15:04:07
"Executor task launch worker-172" daemon prio=10 tid=0x00007fb308001000 nid=0x3966 runnable [0x00007fb338a4c000]
java.lang.Thread.State: RUNNABLE
at sun.misc.Unsafe.getByte(Native Method)
at java.nio.DirectByteBuffer.get(DirectByteBuffer.java:249)
at java.nio.ByteBuffer.get(ByteBuffer.java:673)
at java.nio.DirectByteBuffer.get(DirectByteBuffer.java:279)
at java.nio.ByteBuffer.get(ByteBuffer.java:694)
at org.apache.hadoop.util.DataChecksum.verifyChunkedSums(DataChecksum.java:304)
at org.apache.hadoop.hdfs.RemoteBlockReader2.readNextPacket(RemoteBlockReader2.java:203)
at org.apache.hadoop.hdfs.RemoteBlockReader2.read(RemoteBlockReader2.java:138)
- locked <0x00000007bed42868> (a org.apache.hadoop.hdfs.RemoteBlockReader2)
at org.apache.hadoop.hdfs.DFSInputStream$ByteArrayStrategy.doRead(DFSInputStream.java:683)
at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:739)
- locked <0x00000007bed42808> (a org.apache.hadoop.hdfs.DFSInputStream)
at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:796)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:837)
- locked <0x00000007bed42808> (a org.apache.hadoop.hdfs.DFSInputStream)
at java.io.DataInputStream.read(DataInputStream.java:100)
at org.apache.hadoop.util.LineReader.fillBuffer(LineReader.java:180)
at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:216)
at org.apache.hadoop.util.LineReader.readLine(LineReader.java:174)
at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:209)
- locked <0x00000007bed41c58> (a org.apache.hadoop.mapred.LineRecordReader)
at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:47)
at org.apache.spark.rdd.HadoopRDD$anon$1.getNext(HadoopRDD.scala:249)
at org.apache.spark.rdd.HadoopRDD$anon$1.getNext(HadoopRDD.scala:211)
at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73)
at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
at scala.collection.Iterator$anon$11.hasNext(Iterator.scala:327)
at scala.collection.Iterator$anon$11.hasNext(Iterator.scala:327)
at scala.collection.Iterator$anon$14.hasNext(Iterator.scala:388)
at org.apache.spark.sql.execution.aggregate.TungstenAggregationIterator.processInputs(TungstenAggregationIterator.scala:511)
at org.apache.spark.sql.execution.aggregate.TungstenAggregationIterator.<init>(TungstenAggregationIterator.scala:686)
at org.apache.spark.sql.execution.aggregate.TungstenAggregate$anonfun$doExecute$1$anonfun$2.apply(TungstenAggregate.scala:95)
When looking through some logs after a glassfish server was being unresponsive, I came upon this error :
{{[#|2016-06-02T08:37:51.737+0200|WARNING|oracle-
glassfish3.1.2|org.restlet.Component.ServiceDispatcher|_ThreadID=165;_ThreadName=http-thread-pool-37860(5);|Exception or error caught in status service
javax.ejb.EJBException
at com.sun.ejb.containers.BaseContainer.processSystemException(BaseContainer.java:5236)
at com.sun.ejb.containers.BaseContainer.checkExceptionNoTx(BaseContainer.java:5065)
at com.sun.ejb.containers.BaseContainer.postInvokeTx(BaseContainer.java:4900)
at com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2046)
at com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:1995)
at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:222)
at com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:89)
at com.sun.proxy.$Proxy228.getKpiValuesFlat(Unknown Source)
at com.business.renderer.TopSuppliers.getData(TopSuppliers.java:174)
at com.business.renderer.TopSuppliers.getDataAsJSONTopSuppliers.java:179)
at com.business.renderer.TopSuppliers.render(TopSuppliers.java:107)
at com.ui.renderer.engine.AbstractRenderingEngine.render(AbstractRenderingEngine.java:140)
at com.services.filter.ContainerAdapter.represent(ContainerAdapter.java:311)
at org.restlet.resource.Resource.getRepresentation(Resource.java:259)
at org.restlet.resource.Resource.handleGet(Resource.java:425)
at org.restlet.resource.Finder.handle(Finder.java:470)
at org.restlet.routing.Filter.doHandle(Filter.java:156)
at org.restlet.routing.Filter.handle(Filter.java:203)
at org.restlet.routing.Router.doHandle(Router.java:497)
at org.restlet.routing.Router.handle(Router.java:737)
at com.servlets.ServiceDispatcher$1.handle(ServiceDispatcher.java:55)
at org.restlet.routing.Filter.doHandle(Filter.java:156)
at org.restlet.routing.Filter.handle(Filter.java:203)
at org.restlet.routing.Filter.doHandle(Filter.java:156)
at org.restlet.routing.Filter.handle(Filter.java:203)
at org.restlet.routing.Filter.doHandle(Filter.java:156)
at org.restlet.engine.application.StatusFilter.doHandle(StatusFilter.java:151)
at org.restlet.routing.Filter.handle(Filter.java:203)
at org.restlet.routing.Filter.doHandle(Filter.java:156)
at org.restlet.routing.Filter.handle(Filter.java:203)
at org.restlet.engine.ChainHelper.handle(ChainHelper.java:111)
at org.restlet.engine.application.ApplicationHelper.handle(ApplicationHelper.java:72)
at org.restlet.Application.handle(Application.java:388)
at org.restlet.routing.Filter.doHandle(Filter.java:156)
at org.restlet.routing.Filter.handle(Filter.java:203)
at org.restlet.routing.Router.doHandle(Router.java:497)
at org.restlet.routing.Router.handle(Router.java:737)
at org.restlet.routing.Filter.doHandle(Filter.java:156)
at org.restlet.routing.Filter.handle(Filter.java:203)
at org.restlet.routing.Router.doHandle(Router.java:497)
at org.restlet.routing.Router.handle(Router.java:737)
at org.restlet.routing.Filter.doHandle(Filter.java:156)
at org.restlet.routing.Filter.handle(Filter.java:203)
at org.restlet.engine.ChainHelper.handle(ChainHelper.java:111)
at org.restlet.Component.handle(Component.java:387)
at org.restlet.Server.handle(Server.java:488)
at org.restlet.engine.ServerHelper.handle(ServerHelper.java:71)
at org.restlet.engine.http.HttpServerHelper.handle(HttpServerHelper.java:150)
at org.restlet.ext.servlet.ServerServlet.service(ServerServlet.java:1037)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:770)
at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1554)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:339)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at com.jamonapi.JAMonFilter.doFilter(JAMonFilter.java:59)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:278)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:655)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:595)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:161)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:655)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:595)
at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:334)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:230)
at com.sun.enterprise.v3.services.impl.ContainerMapper$AdapterCallable.call(ContainerMapper.java:311)
at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:189)
at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:850)
at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:747)
at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1032)
at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:231)
at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
at java.lang.Thread.run(Thread.java:745)
Caused by: Exception [EclipseLink-2004] (Eclipse Persistence Services - 2.3.4.v20151027-346465e): org.eclipse.persistence.exceptions.ConcurrencyException
Exception Description: A signal was attempted before wait() on ConcurrencyManager. This normally means that an attempt was made to
commit or rollback a transaction before it was started, or to rollback a transaction twice.
at org.eclipse.persistence.exceptions.ConcurrencyException.signalAttemptedBeforeWait(ConcurrencyException.java:84)
at org.eclipse.persistence.internal.helper.ConcurrencyManager.releaseReadLock(ConcurrencyManager.java:489)
at org.eclipse.persistence.internal.identitymaps.CacheKey.releaseReadLock(CacheKey.java:386)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.cloneAndRegisterObject(UnitOfWorkImpl.java:1018)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.cloneAndRegisterObject(UnitOfWorkImpl.java:929)
at org.eclipse.persistence.internal.sessions.UnitOfWorkIdentityMapAccessor.getAndCloneCacheKeyFromParent(UnitOfWorkIdentityMapAccessor.java:181)
at org.eclipse.persistence.internal.sessions.UnitOfWorkIdentityMapAccessor.getFromIdentityMap(UnitOfWorkIdentityMapAccessor.java:120)
at org.eclipse.persistence.internal.sessions.IdentityMapAccessor.getFromIdentityMap(IdentityMapAccessor.java:380)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.registerExistingObject(UnitOfWorkImpl.java:3899)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.registerExistingObject(UnitOfWorkImpl.java:3854)
at org.eclipse.persistence.mappings.CollectionMapping.buildElementUnitOfWorkClone(CollectionMapping.java:267)
at org.eclipse.persistence.mappings.CollectionMapping.buildElementClone(CollectionMapping.java:279)
at org.eclipse.persistence.internal.queries.ContainerPolicy.addNextValueFromIteratorInto(ContainerPolicy.java:213)
at org.eclipse.persistence.mappings.CollectionMapping.buildCloneForPartObject(CollectionMapping.java:205)
at org.eclipse.persistence.internal.indirection.UnitOfWorkQueryValueHolder.buildCloneFor(UnitOfWorkQueryValueHolder.java:51)
at org.eclipse.persistence.internal.indirection.UnitOfWorkValueHolder.instantiateImpl(UnitOfWorkValueHolder.java:161)
at org.eclipse.persistence.internal.indirection.UnitOfWorkValueHolder.instantiate(UnitOfWorkValueHolder.java:222)
at org.eclipse.persistence.internal.indirection.DatabaseValueHolder.getValue(DatabaseValueHolder.java:88)
at org.eclipse.persistence.indirection.IndirectList.buildDelegate(IndirectList.java:244)
at org.eclipse.persistence.indirection.IndirectList.getDelegate(IndirectList.java:414)
at org.eclipse.persistence.indirection.IndirectList$1.<init>(IndirectList.java:542)
at org.eclipse.persistence.indirection.IndirectList.listIterator(IndirectList.java:541)
at org.eclipse.persistence.indirection.IndirectList.iterator(IndirectList.java:505)
at com.business.frontend.facade.impl.FrontEndBFBean.combineAllContainerKpis(FrontEndBFBean.java:515)
at com.business.frontend.facade.impl.FrontEndBFBean.getKpiValuesFlat(FrontEndBFBean.java:405)
at sun.reflect.GeneratedMethodAccessor237.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.glassfish.ejb.security.application.EJBSecurityManager.runMethod(EJBSecurityManager.java:1052)
at org.glassfish.ejb.security.application.EJBSecurityManager.invoke(EJBSecurityManager.java:1124)
at com.sun.ejb.containers.BaseContainer.invokeBeanMethod(BaseContainer.java:5409)
at com.sun.ejb.EjbInvocation.invokeBeanMethod(EjbInvocation.java:619)
at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:800)
at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:571)
at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.doAround(SystemInterceptorProxy.java:162)
at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.aroundInvoke(SystemInterceptorProxy.java:144)
at sun.reflect.GeneratedMethodAccessor170.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:861)
at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:800)
at com.sun.ejb.containers.interceptors.InterceptorManager.intercept(InterceptorManager.java:370)
at com.sun.ejb.containers.BaseContainer.__intercept(BaseContainer.java:5381)
at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:5369)
at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:214)
... 75 more
On further inspection I notised the lines :
_ThreadID=165;_ThreadName=http-thread-pool-37860(5);
and
Exception [EclipseLink-2004] (Eclipse Persistence Services - 2.3.4.v20151027-346465e)
My initial thought were that that the HTTP thread timeouted waiting for the DB call to complete, and then the HTTP thread interrupted the EclipseLink leaving behind a ConcurrencyManager in an unexpected state (with a DeferredLock not in fact released after use). I made a small app, testing my hypothesis by tweking the server settings and making long calls to the DB and terminating the http call before the DB call was ready, but this doesn't seem to reproduce the problem. There is also the fact that the method called when the exception occurs does not use any kind of JPA(ORM), the call uses JDBC. Most of the application uses EclipseLink and JPA, but not this part. Also it is not uncommon for the application to be under heavy database usage.
So I was thinking, what can be the reason behind this error appart from the descriptive (This normally means that an attempt was made to
commit or rollback a transaction before it was started, or to rollback a transaction twice), I'm pretty sure the code is not doing such a thing.
So it must be some lazy loading conflict, but I have no idea what conditions can cause this problem.
I have no idea where to investigate from this point forward. Any ideas are well welcomed :)
In this error case, the exact reason for the concurrency exception isn't known - we can only see the one thread accessing the context, by triggering a lazy relationship on an entity. How this entity was read in and what happened in other threads that might also be using this EntityManager the entity was read from (directly or indirectly by accessing entities read from it) requires tracking down, but can usually be avoided.
EntityManagers are not meant to be accessed by multiple threads, and so do not handle concurrent access. Entities read in from EntityManagers have many hooks to the context they were read from; these hooks allow for lazy relationships, change tracking etc to occur transparent to the application. This has the consequence though that these entities too cannot be concurrently accessed - they should not be cached or passed around by the application without careful consideration. In many cases, applications might be better off letting JPA handle caching, and access entities as needed from a context.
can any one help me to understand the below thread blocks in netty4.0.15
nioEventLoopGroup-5-13 [BLOCKED] CPU time: 38s
sun.nio.ch.SelectorImpl.select(long)
io.netty.channel.nio.NioEventLoop.select()
io.netty.channel.nio.NioEventLoop.run()
io.netty.util.concurrent.SingleThreadEventExecutor$2.run()
java.lang.Thread.run()
nioEventLoopGroup-5-2 [BLOCKED] CPU time: 40s
java.lang.Class.isAnnotationPresent(Class)
io.netty.channel.ChannelHandlerAdapter.isSharable()
io.netty.channel.DefaultChannelPipeline.checkMultiplicity(ChannelHandlerContext)
io.netty.channel.DefaultChannelPipeline.addLast0(String, DefaultChannelHandlerContext)
io.netty.channel.DefaultChannelPipeline.addLast(EventExecutorGroup, String, ChannelHandler)
io.netty.channel.DefaultChannelPipeline.addLast(String, ChannelHandler)
org.zestadz.adsearch.server.impl.KomliAdServerInitializer.initChannel(SocketChannel)
org.zestadz.adsearch.server.impl.KomliAdServerInitializer.initChannel(Channel)
io.netty.channel.ChannelInitializer.channelRegistered(ChannelHandlerContext)
io.netty.channel.DefaultChannelHandlerContext.invokeChannelRegistered()
io.netty.channel.DefaultChannelHandlerContext.fireChannelRegistered()
io.netty.channel.DefaultChannelPipeline.fireChannelRegistered()
io.netty.channel.AbstractChannel$AbstractUnsafe.register0(ChannelPromise)
io.netty.channel.AbstractChannel$AbstractUnsafe.access$100(AbstractChannel$AbstractUnsafe, ChannelPromise)
io.netty.channel.AbstractChannel$AbstractUnsafe$1.run()
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(long)
io.netty.channel.nio.NioEventLoop.run()
io.netty.util.concurrent.SingleThreadEventExecutor$2.run()
java.lang.Thread.run()
blocked threads i took it from yourkit profiler
This is normal for a worker to block on select() waiting for incoming messages
After a load test in my application, my application server CPU is hogging at 50% always even though nothing is running in the system after the test. When I look at the thread dump, I can see at least 24 BLOCKED threads in the dump. Most of them are related to spring framework, mule framework and so on....nothing specific to my application.
Please see below for a few stack traces of the thread dump.
"ajp-10.40.44.47-8009-116" daemon prio=6 tid=0x000000000cdce800 nid=0x7ac waiting for monitor entry [0x000000003cf1e000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.lang.ClassLoader.loadClass(ClassLoader.java:291)
- waiting to lock <0x00000000c274dab8> (a org.jboss.web.tomcat.service.WebCtxLoader$ENCLoader)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at org.mule.util.ClassUtils.loadClass(ClassUtils.java:340)
at org.mule.util.ClassUtils.instanciateClass(ClassUtils.java:437)
at org.mule.transport.service.DefaultTransportServiceDescriptor.createEndpointURIBuilder(DefaultTransportServiceDescriptor.java:448)
at org.mule.endpoint.MuleEndpointURI.initialise(MuleEndpointURI.java:222)
at org.mule.transport.servlet.MuleReceiverServlet.getReceiverForURI(MuleReceiverServlet.java:308)
at org.mule.transport.servlet.MuleReceiverServlet.doAllMethods(MuleReceiverServlet.java:241)
at org.mule.transport.servlet.MuleReceiverServlet.doPost(MuleReceiverServlet.java:198)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
at org.mule.transport.servlet.MuleReceiverServlet.service(MuleReceiverServlet.java:180)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
"financials.pctogl.staging.partitioned.jms.taskExecutor-9" prio=6 tid=0x0000000009b8f000 nid=0x1730 waiting for monitor entry [0x000000002cf1f000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.springframework.integration.util.ErrorHandlingTaskExecutor$1.run(ErrorHandlingTaskExecutor.java:52)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
"[powercomp].iip.flow.drools.service.stage1.01" prio=6 tid=0x00000000063af800 nid=0x1608 waiting for monitor entry [0x000000002979f000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.mule.work.WorkerContext.run(WorkerContext.java:315)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
"task-scheduler-10" prio=6 tid=0x000000000d075000 nid=0xcf0 waiting for monitor entry [0x000000002649f000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.springframework.integration.channel.MessagePublishingErrorHandler.handleError(MessagePublishingErrorHandler.java:83)
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:59)
at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:81)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
"UDP ucast,10.40.44.47:61710" prio=6 tid=0x0000000009b82800 nid=0xeb0 waiting for monitor entry [0x000000001771e000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.log4j.Category.callAppenders(Category.java:201)
- waiting to lock <0x00000000a197e360> (a org.apache.log4j.spi.RootLogger)
at org.apache.log4j.Category.forcedLog(Category.java:388)
at org.apache.log4j.Category.log(Category.java:853)
at sun.reflect.GeneratedMethodAccessor294.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.commons.logging.impl.Log4jProxy.log(Log4jProxy.java:301)
at org.apache.commons.logging.impl.Log4jProxy.error(Log4jProxy.java:283)
at org.apache.commons.logging.impl.Log4JLogger.error(Log4JLogger.java:208)
at org.jgroups.protocols.TP.down(TP.java:1205)
at org.jgroups.protocols.TP$ProtocolAdapter.down(TP.java:2316)
at org.jgroups.protocols.Discovery.down(Discovery.java:374)
at org.jgroups.protocols.MERGE2.down(MERGE2.java:175)
at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:362)
at org.jgroups.protocols.FD.sendHeartbeatResponse(FD.java:325)
at org.jgroups.protocols.FD.up(FD.java:235)
at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:309)
at org.jgroups.protocols.MERGE2.up(MERGE2.java:144)
at org.jgroups.protocols.Discovery.up(Discovery.java:264)
at org.jgroups.protocols.PING.up(PING.java:273)
at org.jgroups.protocols.TP$ProtocolAdapter.up(TP.java:2327)
at org.jgroups.protocols.TP.passMessageUp(TP.java:1261)
at org.jgroups.protocols.TP.access$100(TP.java:49)
at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1838)
at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1817)
at java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:1746)
at org.jgroups.util.ShutdownRejectedExecutionHandler.rejectedExecution(ShutdownRejectedExecutionHandler.java:39)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at org.jgroups.protocols.TP.dispatchToThreadPool(TP.java:1366)
at org.jgroups.protocols.TP.receive(TP.java:1333)
at org.jgroups.protocols.UDP$UcastReceiver.run(UDP.java:960)
at java.lang.Thread.run(Thread.java:662)