Threads got BLOCKED when application threads heavily tried to log. Has this been encountered before. Does Async logger solve problems around this. Below is the thread dump indicating application threads getting BLOCKED when it was trying to heavily log exceptions. Multiple threads got into a BLOCKED state
"app[LARLAca][TG][T#7]" #55 daemon prio=5 os_prio=0 tid=0x00007f2de800b800 nid=0x2fee runnable [0x00007f2e048ee000]
java.lang.Thread.State: RUNNABLE
at java.util.zip.ZipFile.getEntry(Native Method)
at java.util.zip.ZipFile.getEntry(ZipFile.java:316)
- locked <0x00007f2f5c2e1310> (a java.util.jar.JarFile)
at java.util.jar.JarFile.getEntry(JarFile.java:240)
at java.util.jar.JarFile.getJarEntry(JarFile.java:223)
at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:1054)
at sun.misc.URLClassPath.getResource(URLClassPath.java:249)
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
- locked <0x00007f2f5e3c4d30> (a java.lang.Object)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
at org.apache.logging.log4j.core.util.Loader.loadClass(Loader.java:240)
at org.apache.logging.log4j.core.impl.ThrowableProxy.loadClass(ThrowableProxy.java:633)
at org.apache.logging.log4j.core.impl.ThrowableProxy.loadClass(ThrowableProxy.java:624)
at org.apache.logging.log4j.core.impl.ThrowableProxy.toExtendedStackTrace(ThrowableProxy.java:738)
at org.apache.logging.log4j.core.impl.ThrowableProxy.<init>(ThrowableProxy.java:138)
at org.apache.logging.log4j.core.impl.ThrowableProxy.<init>(ThrowableProxy.java:122)
at org.apache.logging.log4j.core.impl.Log4jLogEvent.getThrownProxy(Log4jLogEvent.java:605)
at org.apache.logging.log4j.core.pattern.ExtendedThrowablePatternConverter.format(ExtendedThrowablePatternConverter.java:64)
at org.apache.logging.log4j.core.pattern.PatternFormatter.format(PatternFormatter.java:38)
at org.apache.logging.log4j.core.layout.PatternLayout$PatternSerializer.toSerializable(PatternLayout.java:334)
at org.apache.logging.log4j.core.layout.PatternLayout.toText(PatternLayout.java:233)
at org.apache.logging.log4j.core.layout.PatternLayout.encode(PatternLayout.java:218)
at org.apache.logging.log4j.core.layout.PatternLayout.encode(PatternLayout.java:58)
at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(AbstractOutputStreamAppender.java:177)
at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutputStreamAppender.java:170)
at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputStreamAppender.java:161)
at org.apache.logging.log4j.core.appender.RollingFileAppender.append(RollingFileAppender.java:309)
at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:156)
at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:129)
at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:120)
at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84)
at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:464)
at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:448)
at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:431)
at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:406)
at org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:63)
at org.apache.logging.log4j.core.Logger.logMessage(Logger.java:146)
at org.apache.logging.log4j.spi.AbstractLogger.tryLogMessage(AbstractLogger.java:2170)
at org.apache.logging.log4j.spi.AbstractLogger.logMessageTrackRecursion(AbstractLogger.java:2125)
at org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2108)
at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:1991)
at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1835)
at org.apache.logging.log4j.spi.AbstractLogger.warn(AbstractLogger.java:2684)
"app[LARLAca][TG][T#1]" #44 daemon prio=5 os_prio=0 tid=0x00007f36fe6b3000 nid=0x2fe3 waiting for monitor entry [0x00007f2e06bfb000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.lang.ClassLoader.loadClass(ClassLoader.java:399)
- waiting to lock <0x00007f2f5e3c4d30> (a java.lang.Object)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at org.apache.logging.log4j.util.LoaderUtil.loadClass(LoaderUtil.java:168)
at org.apache.logging.log4j.core.impl.ThrowableProxy.loadClass(ThrowableProxy.java:622)
at org.apache.logging.log4j.core.impl.ThrowableProxy.toExtendedStackTrace(ThrowableProxy.java:738)
at org.apache.logging.log4j.core.impl.ThrowableProxy.<init>(ThrowableProxy.java:138)
at org.apache.logging.log4j.core.impl.ThrowableProxy.<init>(ThrowableProxy.java:122)
at org.apache.logging.log4j.core.impl.Log4jLogEvent.getThrownProxy(Log4jLogEvent.java:605)
at org.apache.logging.log4j.core.pattern.ExtendedThrowablePatternConverter.format(ExtendedThrowablePatternConverter.java:64)
at org.apache.logging.log4j.core.pattern.PatternFormatter.format(PatternFormatter.java:38)
at org.apache.logging.log4j.core.layout.PatternLayout$PatternSerializer.toSerializable(PatternLayout.java:334)
at org.apache.logging.log4j.core.layout.PatternLayout.toText(PatternLayout.java:233)
at org.apache.logging.log4j.core.layout.PatternLayout.encode(PatternLayout.java:218)
at org.apache.logging.log4j.core.layout.PatternLayout.encode(PatternLayout.java:58)
at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(AbstractOutputStreamAppender.java:177)
at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutputStreamAppender.java:170)
at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputStreamAppender.java:161)
at org.apache.logging.log4j.core.appender.RollingFileAppender.append(RollingFileAppender.java:309)
at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:156)
at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:129)
at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:120)
at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84)
at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:464)
at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:448)
at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:431)
at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:406)
at org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:63)
at org.apache.logging.log4j.core.Logger.logMessage(Logger.java:146)
at org.apache.logging.log4j.spi.AbstractLogger.tryLogMessage(AbstractLogger.java:2170)
at org.apache.logging.log4j.spi.AbstractLogger.logMessageTrackRecursion(AbstractLogger.java:2125)
at org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2108)
at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:1991)
at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1835)
at org.apache.logging.log4j.spi.AbstractLogger.warn(AbstractLogger.java:2684)
I found a solution. using a %ex in the pattern layout prevent the ThrowableProxy class loading thus saving the thread from getting BLOCKED
Related
I use spring integration imap-idle-channel-adapter to receive email, however seems thread stuck during email receiving, while we are trying to stop the adapter, we found there is a thread stuck issue.
Could you advise how to solve the problem ? we did set timeout, seems it doesn't work.
it's so strange why timeout also doestn't work
<util:properties id="exchangeJavaMailProperties">
<prop key="mail.imaps.socketFactory.class">javax.net.ssl.SSLSocketFactory</prop>
<prop key="mail.imap.starttls.enable">false</prop>
<prop key="mail.imaps.socketFactory.fallback">false</prop>
<prop key="mail.store.protocol">imaps</prop>
<prop key="mail.imaps.auth.plain.disable">true</prop>
<prop key="mail.debug">true</prop>
<prop key="mail.imaps.timeout">300000</prop>
<prop key="mail.imap.timeout">300000</prop>
<prop key="mail.imap.connectiontimeout">300000</prop>
<prop key="mail.imaps.connectiontimeout">300000</prop>
</util:properties>
"task-scheduler-7" #942 prio=5 os_prio=0 tid=0x00000000029a9000 nid=0x51a runnable [0x00007f8a97835000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at com.sun.mail.util.TraceInputStream.read(TraceInputStream.java:124)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
- locked <0x0000000086280c00> (a java.io.BufferedInputStream)
at com.sun.mail.iap.ResponseInputStream.readResponse(ResponseInputStream.java:103)
at com.sun.mail.iap.Response.<init>(Response.java:114)
at com.sun.mail.imap.protocol.IMAPResponse.<init>(IMAPResponse.java:60)
at com.sun.mail.imap.protocol.IMAPProtocol.readResponse(IMAPProtocol.java:390)
at com.sun.mail.iap.Protocol.command(Protocol.java:354)
- locked <0x0000000086280c38> (a com.sun.mail.imap.protocol.IMAPProtocol)
at com.sun.mail.imap.protocol.IMAPProtocol.fetch(IMAPProtocol.java:2113)
at com.sun.mail.imap.protocol.IMAPProtocol.fetch(IMAPProtocol.java:2105)
at com.sun.mail.imap.protocol.IMAPProtocol.fetchSectionBody(IMAPProtocol.java:1818)
at com.sun.mail.imap.protocol.IMAPProtocol.fetchBody(IMAPProtocol.java:1801)
at com.sun.mail.imap.protocol.IMAPProtocol.peekBody(IMAPProtocol.java:1774)
at com.sun.mail.imap.IMAPInputStream.fill(IMAPInputStream.java:154)
- locked <0x0000000086280cb8> (a java.lang.Object)
at com.sun.mail.imap.IMAPInputStream.read(IMAPInputStream.java:218)
- locked <0x0000000086280cc8> (a com.sun.mail.imap.IMAPInputStream)
at com.sun.mail.imap.IMAPInputStream.read(IMAPInputStream.java:244)
at com.sun.mail.imap.IMAPMessage.writeTo(IMAPMessage.java:849)
at javax.mail.internet.MimeMessage.<init>(MimeMessage.java:245)
at org.springframework.integration.mail.AbstractMailReceiver$IntegrationMimeMessage.<init>(AbstractMailReceiver.java:557)
at org.springframework.integration.mail.AbstractMailReceiver$IntegrationMimeMessage.<init>(AbstractMailReceiver.java:552)
at org.springframework.integration.mail.AbstractMailReceiver.postProcessFilteredMessages(AbstractMailReceiver.java:415)
at org.springframework.integration.mail.AbstractMailReceiver.receive(AbstractMailReceiver.java:342)
- locked <0x0000000086284e98> (a java.lang.Object)
at org.springframework.integration.mail.ImapIdleChannelAdapter$IdleTask.run(ImapIdleChannelAdapter.java:273)
at org.springframework.integration.mail.ImapIdleChannelAdapter$ReceivingTask.run(ImapIdleChannelAdapter.java:241)
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:81)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"[STUCK] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'" #31 daemon prio=1 os_prio=0 tid=0x00007f8ac021a000 nid=0x183 waiting for monitor entry [0x00007f8aea5d2000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.springframework.integration.mail.AbstractMailReceiver.destroy(AbstractMailReceiver.java:519)
- waiting to lock <0x0000000086284e98> (a java.lang.Object)
at org.springframework.integration.mail.ImapIdleChannelAdapter.doStop(ImapIdleChannelAdapter.java:170)
I think the fix for this issue is here: https://github.com/spring-projects/spring-integration/commit/6d0757a08a75828eea408e68a9284cebb6e602fd.
So, you need to consider to upgrade your project to the latest Spring Integration: https://spring.io/projects/spring-integration#learn
Even if the fix was not exactly for mentioned dead lock, it looks like cancelling the ping should help to prevent waiting for task to finish.
I have some unit test cases written for spring integration flows. On running the individual test cases from IDE no issues, but on running all test cases from gradle build some test cases are failing.
I found a QC on related issue:
https://jira.spring.io/browse/XD-3709
If any one knows solution for this, please post.
Unit Tests with below annotations:
#RunWith(SpringRunner.class)
#Import(AlarmAPIApplication.class)
#PropertySource("classpath:application.properties ")
#FixMethodOrder
public class HostRegistrationFlowTest {
#Value("${em.host.registration.url}")
String hostRegistrationUrl;
#Autowired
RestTemplate restTemplate;
#Autowired
SubscribableChannel registerHostInputChannel;
#Test
public void testRegisterHost() {
Error on running from gradle script:
java.lang.IllegalStateException: Failed to load ApplicationContext
at org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate.loadContext(DefaultCacheAwareContextLoaderDelegate.java:124)
at org.springframework.test.context.support.DefaultTestContext.getApplicationContext(DefaultTestContext.java:83)
at org.springframework.test.context.support.DependencyInjectionTestExecutionListener.injectDependencies(DependencyInjectionTestExecutionListener.java:117)
at org.springframework.test.context.support.DependencyInjectionTestExecutionListener.prepareTestInstance(DependencyInjectionTestExecutionListener.java:83)
at org.springframework.test.context.TestContextManager.prepareTestInstance(TestContextManager.java:230)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.createTest(SpringJUnit4ClassRunner.java:228)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner$1.runReflectiveCall(SpringJUnit4ClassRunner.java:287)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.methodBlock(SpringJUnit4ClassRunner.java:289)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:247)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:94)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61)
at org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:70)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:191)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:86)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:49)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:69)
at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:48)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at org.gradle.messaging.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
at org.gradle.messaging.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:105)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at org.gradle.messaging.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:355)
at org.gradle.internal.concurrent.DefaultExecutorFactory$StoppableExecutorImpl$1.run(DefaultExecutorFactory.java:64)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.springframework.jmx.export.UnableToRegisterMBeanException: Unable to register MBean [org.springframework.integration.router.MethodInvokingRouter#0] with key 'org.springframework.integration.router.MethodInvokingRouter#0'; nested exception is javax.management.InstanceAlreadyExistsException: org.springframework.integration.router:name=org.springframework.integration.router.MethodInvokingRouter#0,type=MethodInvokingRouter
at org.springframework.jmx.export.MBeanExporter.registerBeanNameOrInstance(MBeanExporter.java:628)
at org.springframework.jmx.export.MBeanExporter.registerBeans(MBeanExporter.java:550)
at org.springframework.jmx.export.MBeanExporter.afterSingletonsInstantiated(MBeanExporter.java:432)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:781)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:866)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:542)
at org.springframework.test.context.support.AbstractGenericContextLoader.loadContext(AbstractGenericContextLoader.java:128)
at org.springframework.test.context.support.AbstractGenericContextLoader.loadContext(AbstractGenericContextLoader.java:60)
at org.springframework.test.context.support.AbstractDelegatingSmartContextLoader.delegateLoading(AbstractDelegatingSmartContextLoader.java:108)
at org.springframework.test.context.support.AbstractDelegatingSmartContextLoader.loadContext(AbstractDelegatingSmartContextLoader.java:259)
at org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate.loadContextInternal(DefaultCacheAwareContextLoaderDelegate.java:98)
at org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate.loadContext(DefaultCacheAwareContextLoaderDelegate.java:116)
... 44 more
Caused by: javax.management.InstanceAlreadyExistsException: org.springframework.integration.router:name=org.springframework.integration.router.MethodInvokingRouter#0,type=MethodInvokingRouter
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at org.springframework.jmx.support.MBeanRegistrationSupport.doRegister(MBeanRegistrationSupport.java:195)
at org.springframework.jmx.export.MBeanExporter.registerBeanInstance(MBeanExporter.java:682)
at org.springframework.jmx.export.MBeanExporter.registerBeanNameOrInstance(MBeanExporter.java:618)
... 55 more
javax.management.InstanceAlreadyExistsException: org.springframework.integration.router:name=org.springframework.integration.router.MethodInvokingRouter#0,type=MethodInvokingRouter
You have to use #DirtiesContext on the test class level to close all the application contexts after performing their tests. Otherwise the application contexts, created during test class initialization, aren't closed to chase the caching goal in between different test classes.
In your case one of the test classes initialize application context and register MBeans into the JMX. But not closing the application context cause the next one to try to register the MBean again. Just because you use some common configuration for different test classes.
I'm developing a big java fx application with http communication, sockets etc.. The content of the messages update the GUI-elements. I am aware of the fact that I need to call Platform.runlater() whenever I want to update a javafx element from a normal thread. I am 99% sure, I have this everywhere. But the application is quite big and not everything is done by myself.
To give some more information, there is a TimerTask every 100ms which creates an Image from a byte-array. The call for imageView.setImage(image) is done in Platform.runlater().
However, I get the following error (using java 1.8.0.0_102):
2017-01-20T05:03:47,030 FATAL [JavaFX Application Thread] uncaughtException - Thread[JavaFX Application Thread,5,main]
java.lang.NullPointerException
at javafx.scene.Scene$ScenePulseListener.synchronizeSceneNodes(Unknown Source) ~[jfxrt.jar:?]
at javafx.scene.Scene$ScenePulseListener.pulse(Unknown Source) ~[jfxrt.jar:?]
at com.sun.javafx.tk.Toolkit.lambda$runPulse$30(Unknown Source) ~[jfxrt.jar:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_77]
at com.sun.javafx.tk.Toolkit.runPulse(Unknown Source) ~[jfxrt.jar:?]
at com.sun.javafx.tk.Toolkit.firePulse(Unknown Source) ~[jfxrt.jar:?]
at com.sun.javafx.tk.quantum.QuantumToolkit.pulse(Unknown Source) ~[jfxrt.jar:?]
at com.sun.javafx.tk.quantum.QuantumToolkit.pulse(Unknown Source) ~[jfxrt.jar:?]
at com.sun.javafx.tk.quantum.QuantumToolkit.lambda$runToolkit$404(Unknown Source) ~[jfxrt.jar:?]
at com.sun.glass.ui.InvokeLaterDispatcher$Future.run(Unknown Source) ~[jfxrt.jar:?]
at com.sun.glass.ui.win.WinApplication._runLoop(Native Method) ~[jfxrt.jar:?]
at com.sun.glass.ui.win.WinApplication.lambda$null$148(Unknown Source) ~[jfxrt.jar:?]
at java.lang.Thread.run(Unknown Source) [?:1.8.0_77]
Any suggestions how to find the source of this error? As I mentioned above I checked all threads that nothing fx related is called on an fx thread. What I am not sure, are you allowed to load an Image (a Java FX Image) on a non fx thread? I tested it in a small sample application and it gave no error..
Edit
I found this question: same error, but I tried to force the scene update with no luck.
Thanks for any ideas!
Our benchmarking uncovered a bootleneck in Apache Tomcat 7.0.59. When the server reaches its performance limit, most of its threads are locked by ClassLoader.
Stack traces of blocked threads looks like this example:
"http-bio-4504-exec-500" Id=2335 BLOCKED on java.util.jar.JarFile#464f9f8 owned by "[1432628598653] POST /services/signin HTTP/1.0" Id=1990
at java.util.zip.ZipFile.getEntry(ZipFile.java:304)
- blocked on java.util.jar.JarFile#464f9f8
at java.util.jar.JarFile.getEntry(JarFile.java:226)
at java.util.jar.JarFile.getJarEntry(JarFile.java:209)
at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:840)
at sun.misc.URLClassPath$JarLoader.findResource(URLClassPath.java:818)
at sun.misc.URLClassPath$1.next(URLClassPath.java:226)
at sun.misc.URLClassPath$1.hasMoreElements(URLClassPath.java:236)
at java.net.URLClassLoader$3$1.run(URLClassLoader.java:583)
at java.net.URLClassLoader$3$1.run(URLClassLoader.java:581)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader$3.next(URLClassLoader.java:580)
at java.net.URLClassLoader$3.hasMoreElements(URLClassLoader.java:605)
at sun.misc.CompoundEnumeration.next(CompoundEnumeration.java:45)
at sun.misc.CompoundEnumeration.hasMoreElements(CompoundEnumeration.java:54)
at sun.misc.CompoundEnumeration.next(CompoundEnumeration.java:45)
at sun.misc.CompoundEnumeration.hasMoreElements(CompoundEnumeration.java:54)
at com.sun.xml.ws.util.ServiceFinder$LazyIterator.hasNext(ServiceFinder.java:443)
at com.sun.xml.ws.util.ServiceFinder$CompositeIterator.hasNext(ServiceFinder.java:390)
at com.sun.xml.ws.api.message.saaj.SAAJFactory.getMessageFactory(SAAJFactory.java:96)
at com.sun.xml.ws.api.SOAPVersion.getMessageFactory(SOAPVersion.java:221)
at com.sun.xml.ws.api.message.saaj.SAAJFactory.readAsSOAPMessage(SAAJFactory.java:275)
at com.sun.xml.ws.api.message.saaj.SAAJFactory.readAsSAAJ(SAAJFactory.java:205)
at com.sun.xml.ws.api.message.saaj.SAAJFactory.read(SAAJFactory.java:194)
at com.sun.xml.ws.message.AbstractMessageImpl.toSAAJ(AbstractMessageImpl.java:199)
at com.sun.xml.ws.api.message.MessageWrapper.readAsSOAPMessage(MessageWrapper.java:160)
at com.sun.xml.ws.handler.SOAPMessageContextImpl.getMessage(SOAPMessageContextImpl.java:86)
and the blocker thread is running at the same place
"[1432628598653] POST /services/signin HTTP/1.0" Id=1990 RUNNABLE
at java.util.zip.ZipFile.getEntry(ZipFile.java:304)
- locked java.util.jar.JarFile#464f9f8
at java.util.jar.JarFile.getEntry(JarFile.java:226)
at java.util.jar.JarFile.getJarEntry(JarFile.java:209)
at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:840)
at sun.misc.URLClassPath$JarLoader.findResource(URLClassPath.java:818)
at sun.misc.URLClassPath$1.next(URLClassPath.java:226)
at sun.misc.URLClassPath$1.hasMoreElements(URLClassPath.java:236)
at java.net.URLClassLoader$3$1.run(URLClassLoader.java:583)
at java.net.URLClassLoader$3$1.run(URLClassLoader.java:581)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader$3.next(URLClassLoader.java:580)
at java.net.URLClassLoader$3.hasMoreElements(URLClassLoader.java:605)
at sun.misc.CompoundEnumeration.next(CompoundEnumeration.java:45)
at sun.misc.CompoundEnumeration.hasMoreElements(CompoundEnumeration.java:54)
at sun.misc.CompoundEnumeration.next(CompoundEnumeration.java:45)
at sun.misc.CompoundEnumeration.hasMoreElements(CompoundEnumeration.java:54)
at com.sun.xml.ws.util.ServiceFinder$LazyIterator.hasNext(ServiceFinder.java:443)
at com.sun.xml.ws.util.ServiceFinder$CompositeIterator.hasNext(ServiceFinder.java:390)
at com.sun.xml.ws.api.message.saaj.SAAJFactory.read(SAAJFactory.java:190)
at com.sun.xml.ws.message.AbstractMessageImpl.toSAAJ(AbstractMessageImpl.java:199)
at com.sun.xml.ws.api.message.MessageWrapper.readAsSOAPMessage(MessageWrapper.java:160)
at com.sun.xml.ws.handler.SOAPMessageContextImpl.getMessage(SOAPMessageContextImpl.java:86)
Do our applications do something in a wrong way? Is there a workaround to avoid blocking in classloaders?
The problem was fixed completely with the following hack:
I added -Djaxp.debug=true to JVM args.
Then I found in logs all classes that JAX-WS was looking for.
I added system property for every mentioned class, and it made ServiceFinder to omit searching for service.properties files in classpath.
My final JVM args looked like:
-Djavax.xml.soap.MetaFactory=com.sun.xml.internal.messaging.saaj.soap.SAAJMetaFactoryImpl
-Djavax.xml.datatype.DatatypeFactory=com.sun.org.apache.xerces.internal.jaxp.datatype.DatatypeFactoryImpl
-Djavax.xml.parsers.DocumentBuilderFactory=com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl
-Djavax.xml.parsers.SAXParserFactory=com.sun.org.apache.xerces.internal.jaxp.SAXParserFactoryImpl
-Djavax.xml.stream.XMLInputFactory=com.ctc.wstx.stax.WstxInputFactory
-Djavax.xml.stream.XMLOutputFactory=com.ctc.wstx.stax.WstxOutputFactory
-Djavax.xml.transform.TransformerFactory=com.sun.org.apache.xalan.internal.xsltc.trax.TransformerFactoryImpl
I am using Groovy 1.7.8. and have written a Groovy DSL which I execute concurrently on different domain objects.
Of late, I have started facing deadlocks under heavy concurrency when DSLs are compiling/executing concurrently.
I am compiling/executing Groovy DSL at runtime using:
Script s = compileScript(dsl)
def binding = new Binding()
binding.setVariable("domainObj", domainObj)
s.setBinding(binding)
s.run()
def Script compileScript(String dsl) {
def scriptText = getScriptText(dsl)
def conf = new CompilerConfiguration()
conf.setScriptBaseClass(GroovyViewExecutionBase.class.getName())
new GroovyShell(conf).parse(scriptText)
}
Below is the thread dump of deadlocked threads, looking at the thread dumps I feel that Groovy is deadlocking internally when we concurrently compile/execute scripts at runtime.
Deadlock-Participant-1:
at java.beans.PropertyDescriptor.getReadMethod(PropertyDescriptor.java:158)
- waiting to lock <0x00007f41a3363960> (a java.beans.PropertyDescriptor)
at java.beans.Introspector.processPropertyDescriptors(Introspector.java:683)
at java.beans.Introspector.getTargetPropertyInfo(Introspector.java:615)
at java.beans.Introspector.getBeanInfo(Introspector.java:407)
at java.beans.Introspector.getBeanInfo(Introspector.java:164)
- locked <0x00007f41a3358c28> (a java.lang.Object)
at groovy.lang.MetaClassImpl$15.run(MetaClassImpl.java:2940)
at java.security.AccessController.doPrivileged(Native Method)
at groovy.lang.MetaClassImpl.addProperties(MetaClassImpl.java:2938)
at groovy.lang.MetaClassImpl.initialize(MetaClassImpl.java:2921)
- locked <0x00007f429f011268> (a groovy.lang.ExpandoMetaClass)
at groovy.lang.ExpandoMetaClass.initialize(ExpandoMetaClass.java:463)
- locked <0x00007f429f011268> (a groovy.lang.ExpandoMetaClass)
at org.codehaus.groovy.reflection.ClassInfo.getMetaClassUnderLock(ClassInfo.java:166)
at org.codehaus.groovy.reflection.ClassInfo.getMetaClass(ClassInfo.java:182)
at org.codehaus.groovy.runtime.metaclass.MetaClassRegistryImpl.getMetaClass(MetaClassRegistryImpl.java:210)
at org.codehaus.groovy.runtime.InvokerHelper.getMetaClass(InvokerHelper.java:751)
at groovy.lang.GroovyObjectSupport.<init>(GroovyObjectSupport.java:32)
at groovy.lang.Script.<init>(Script.java:40)
at groovy.lang.Script.<init>(Script.java:37)
at flipkart.cms.views.core.GroovyViewExecutionBase.<init>(GroovyViewExecutionBase.groovy)
at Script1.<init>(Script1.groovy)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at java.lang.Class.newInstance0(Class.java:355)
at java.lang.Class.newInstance(Class.java:308)
at org.codehaus.groovy.runtime.InvokerHelper.createScript(InvokerHelper.java:408)
at groovy.lang.GroovyShell.parse(GroovyShell.java:743)
at groovy.lang.GroovyShell.parse(GroovyShell.java:770)
at groovy.lang.GroovyShell.parse(GroovyShell.java:761)
at groovy.lang.GroovyShell$parse.call(Unknown Source)
at cms.views.core.GroovyDSLViewComputer.compileScript(GroovyDSLViewComputer.groovy:85)
==================
Deadlock-Participant-2:
at java.beans.Introspector.getPublicDeclaredMethods(Introspector.java:1277)
- waiting to lock <0x00007f41a3358c28> (a java.lang.Object)
at java.beans.Introspector.internalFindMethod(Introspector.java:1312)
at java.beans.Introspector.findMethod(Introspector.java:1383)
at java.beans.Introspector.findMethod(Introspector.java:1363)
at java.beans.PropertyDescriptor.getReadMethod(PropertyDescriptor.java:179)
- locked <0x00007f41a3363960> (a java.beans.PropertyDescriptor)
at groovy.lang.MetaClassImpl.applyPropertyDescriptors(MetaClassImpl.java:2215)
at groovy.lang.MetaClassImpl.setupProperties(MetaClassImpl.java:1995)
at groovy.lang.MetaClassImpl.addProperties(MetaClassImpl.java:2950)
at groovy.lang.MetaClassImpl.initialize(MetaClassImpl.java:2921)
- locked <0x00007f42a12972b8> (a groovy.lang.ExpandoMetaClass)
at groovy.lang.ExpandoMetaClass.initialize(ExpandoMetaClass.java:463)
- locked <0x00007f42a12972b8> (a groovy.lang.ExpandoMetaClass)
at org.codehaus.groovy.runtime.HandleMetaClass.replaceDelegate(HandleMetaClass.java:66)
at org.codehaus.groovy.runtime.HandleMetaClass.setProperty(HandleMetaClass.java:91)
at org.codehaus.groovy.runtime.InvokerHelper.setProperty(InvokerHelper.java:179)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.setProperty(ScriptBytecodeAdapter.java:483)
at flipkart.cms.views.core.GroovyViewExecutionBase.setupAttrProperties(GroovyViewExecutionBase.groovy:60)
at flipkart.cms.views.core.GroovyViewExecutionBase$setupAttrProperties$0.callCurrent(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:44)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:141)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:149)
at cms.views.core.GroovyViewExecutionBase.computeView(GroovyViewExecutionBase.groovy:32)
==================
Any ideas how I can fix/workaround this?
Posted this on groovy-user forum and looks like the deadlocks are caused because of synchronization issues in java 6's java.beans.Introspector class which apparently have been fixed in Java 7. Also received a backport of these classes for java 6 and am testing the backported classes to reproduce the deadlocks.
More details here:
http://groovy.329449.n5.nabble.com/Groovy-deadlocks-when-executing-DSL-from-GroovyShell-td5709889.html