Our benchmarking uncovered a bootleneck in Apache Tomcat 7.0.59. When the server reaches its performance limit, most of its threads are locked by ClassLoader.
Stack traces of blocked threads looks like this example:
"http-bio-4504-exec-500" Id=2335 BLOCKED on java.util.jar.JarFile#464f9f8 owned by "[1432628598653] POST /services/signin HTTP/1.0" Id=1990
at java.util.zip.ZipFile.getEntry(ZipFile.java:304)
- blocked on java.util.jar.JarFile#464f9f8
at java.util.jar.JarFile.getEntry(JarFile.java:226)
at java.util.jar.JarFile.getJarEntry(JarFile.java:209)
at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:840)
at sun.misc.URLClassPath$JarLoader.findResource(URLClassPath.java:818)
at sun.misc.URLClassPath$1.next(URLClassPath.java:226)
at sun.misc.URLClassPath$1.hasMoreElements(URLClassPath.java:236)
at java.net.URLClassLoader$3$1.run(URLClassLoader.java:583)
at java.net.URLClassLoader$3$1.run(URLClassLoader.java:581)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader$3.next(URLClassLoader.java:580)
at java.net.URLClassLoader$3.hasMoreElements(URLClassLoader.java:605)
at sun.misc.CompoundEnumeration.next(CompoundEnumeration.java:45)
at sun.misc.CompoundEnumeration.hasMoreElements(CompoundEnumeration.java:54)
at sun.misc.CompoundEnumeration.next(CompoundEnumeration.java:45)
at sun.misc.CompoundEnumeration.hasMoreElements(CompoundEnumeration.java:54)
at com.sun.xml.ws.util.ServiceFinder$LazyIterator.hasNext(ServiceFinder.java:443)
at com.sun.xml.ws.util.ServiceFinder$CompositeIterator.hasNext(ServiceFinder.java:390)
at com.sun.xml.ws.api.message.saaj.SAAJFactory.getMessageFactory(SAAJFactory.java:96)
at com.sun.xml.ws.api.SOAPVersion.getMessageFactory(SOAPVersion.java:221)
at com.sun.xml.ws.api.message.saaj.SAAJFactory.readAsSOAPMessage(SAAJFactory.java:275)
at com.sun.xml.ws.api.message.saaj.SAAJFactory.readAsSAAJ(SAAJFactory.java:205)
at com.sun.xml.ws.api.message.saaj.SAAJFactory.read(SAAJFactory.java:194)
at com.sun.xml.ws.message.AbstractMessageImpl.toSAAJ(AbstractMessageImpl.java:199)
at com.sun.xml.ws.api.message.MessageWrapper.readAsSOAPMessage(MessageWrapper.java:160)
at com.sun.xml.ws.handler.SOAPMessageContextImpl.getMessage(SOAPMessageContextImpl.java:86)
and the blocker thread is running at the same place
"[1432628598653] POST /services/signin HTTP/1.0" Id=1990 RUNNABLE
at java.util.zip.ZipFile.getEntry(ZipFile.java:304)
- locked java.util.jar.JarFile#464f9f8
at java.util.jar.JarFile.getEntry(JarFile.java:226)
at java.util.jar.JarFile.getJarEntry(JarFile.java:209)
at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:840)
at sun.misc.URLClassPath$JarLoader.findResource(URLClassPath.java:818)
at sun.misc.URLClassPath$1.next(URLClassPath.java:226)
at sun.misc.URLClassPath$1.hasMoreElements(URLClassPath.java:236)
at java.net.URLClassLoader$3$1.run(URLClassLoader.java:583)
at java.net.URLClassLoader$3$1.run(URLClassLoader.java:581)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader$3.next(URLClassLoader.java:580)
at java.net.URLClassLoader$3.hasMoreElements(URLClassLoader.java:605)
at sun.misc.CompoundEnumeration.next(CompoundEnumeration.java:45)
at sun.misc.CompoundEnumeration.hasMoreElements(CompoundEnumeration.java:54)
at sun.misc.CompoundEnumeration.next(CompoundEnumeration.java:45)
at sun.misc.CompoundEnumeration.hasMoreElements(CompoundEnumeration.java:54)
at com.sun.xml.ws.util.ServiceFinder$LazyIterator.hasNext(ServiceFinder.java:443)
at com.sun.xml.ws.util.ServiceFinder$CompositeIterator.hasNext(ServiceFinder.java:390)
at com.sun.xml.ws.api.message.saaj.SAAJFactory.read(SAAJFactory.java:190)
at com.sun.xml.ws.message.AbstractMessageImpl.toSAAJ(AbstractMessageImpl.java:199)
at com.sun.xml.ws.api.message.MessageWrapper.readAsSOAPMessage(MessageWrapper.java:160)
at com.sun.xml.ws.handler.SOAPMessageContextImpl.getMessage(SOAPMessageContextImpl.java:86)
Do our applications do something in a wrong way? Is there a workaround to avoid blocking in classloaders?
The problem was fixed completely with the following hack:
I added -Djaxp.debug=true to JVM args.
Then I found in logs all classes that JAX-WS was looking for.
I added system property for every mentioned class, and it made ServiceFinder to omit searching for service.properties files in classpath.
My final JVM args looked like:
-Djavax.xml.soap.MetaFactory=com.sun.xml.internal.messaging.saaj.soap.SAAJMetaFactoryImpl
-Djavax.xml.datatype.DatatypeFactory=com.sun.org.apache.xerces.internal.jaxp.datatype.DatatypeFactoryImpl
-Djavax.xml.parsers.DocumentBuilderFactory=com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl
-Djavax.xml.parsers.SAXParserFactory=com.sun.org.apache.xerces.internal.jaxp.SAXParserFactoryImpl
-Djavax.xml.stream.XMLInputFactory=com.ctc.wstx.stax.WstxInputFactory
-Djavax.xml.stream.XMLOutputFactory=com.ctc.wstx.stax.WstxOutputFactory
-Djavax.xml.transform.TransformerFactory=com.sun.org.apache.xalan.internal.xsltc.trax.TransformerFactoryImpl
Related
Context
This basically relates directly to this other question.
Essentially, our PUT endpoint needs to be more performant. Our approach is now to DELETE all relevant entries and then INSERT the new ones. This endpoint is only called once every few minutes, so there is no actual potential concurrent requests, but it does need to run fast.
I've already done other performance-related improvements like using JPQL for bulk queries, activating EclipseLink's batch-writing, and properly indexing the database.
At this point, the only other improvement I can think of is to run in parallel the calls which aren't related. Here is a screenshot of an execution of our endpoint as seen through Dynatrace PurePath:
Analysis
There are 9 calls to the DB which could be turned into 2 sets of calls:
All of the SELECT and DELETE calls could be started simultaneously and shouldn't step on each others' toes since they relate to different tables.
All of the INSERT calls could be started simultaneously and shouldn't step on each others' toes since they relate to different tables.
The 3x Update calls simply don't exist anymore so we can forget about them. Thus, it seems like this endpoint could potentially be twice as fast if this works as intended.
Difficulty
Although to be confirmed, it is my understanding that:
EntityManager is not thread-safe
A JDBC connection cannot be shared by multiple threads
Thus each thread needs to operate on a different connection
Thus, to allow rollback, those threads should wait until all requests are confirmed to be successful before committing after being done with their individual work
Additionally, #Async methods which return void are harder to control in terms of error handling
Question
How do we properly set up a Spring-MVC endpoint to issue database calls in different threads, and then use the returned CompletableFutures to get proper flow control by ensuring the queries ran successfully (i.e. completed without Exceptions).
The execution needs to be able to:
Await the first set of CompletableFutures completion before proceeding with the next set of requests
Allow rollback in case of error from any of the requests
Current state and problem
I'm trying to achieve this goal by using Spring's #Async, but I'm having trouble understanding how I should do this correctly.
The way I've done it currently throws a SQLTransientConnectionException after the 30s timeout configured for the HikariPool (which contains 8 threads):
javax.persistence.PersistenceException: java.lang.reflect.UndeclaredThrowableException
at org.eclipse.persistence.internal.jpa.QueryImpl.getResultList(QueryImpl.java:493)
at jdk.internal.reflect.GeneratedMethodAccessor236.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.orm.jpa.SharedEntityManagerCreator$DeferredQueryInvocationHandler.invoke(SharedEntityManagerCreator.java:406)
at com.sun.proxy.$Proxy280.getResultList(Unknown Source)
at org.springframework.data.jpa.repository.query.JpaQueryExecution$CollectionExecution.doExecute(JpaQueryExecution.java:126)
at org.springframework.data.jpa.repository.query.JpaQueryExecution.execute(JpaQueryExecution.java:88)
at org.springframework.data.jpa.repository.query.AbstractJpaQuery.doExecute(AbstractJpaQuery.java:155)
at org.springframework.data.jpa.repository.query.AbstractJpaQuery.execute(AbstractJpaQuery.java:143)
at org.springframework.data.repository.core.support.RepositoryMethodInvoker.doInvoke(RepositoryMethodInvoker.java:137)
at org.springframework.data.repository.core.support.RepositoryMethodInvoker.invoke(RepositoryMethodInvoker.java:121)
at org.springframework.data.repository.core.support.QueryExecutorMethodInterceptor.doInvoke(QueryExecutorMethodInterceptor.java:152)
at org.springframework.data.repository.core.support.QueryExecutorMethodInterceptor.invoke(QueryExecutorMethodInterceptor.java:131)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:137)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.data.jpa.repository.support.CrudMethodMetadataPostProcessor$CrudMethodMetadataPopulatingMethodInterceptor.invoke(CrudMethodMetadataPostProcessor.java:145)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy187.getAllSitesWithConcatenatedIds(Unknown Source)
...
Caused by: java.lang.reflect.UndeclaredThrowableException: null
at com.sun.proxy.$Proxy127.getConnection(Unknown Source)
at org.eclipse.persistence.sessions.JNDIConnector.connect(JNDIConnector.java:138)
at org.eclipse.persistence.sessions.DatasourceLogin.connectToDatasource(DatasourceLogin.java:172)
at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.connectInternal(DatasourceAccessor.java:348)
at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.connectInternal(DatabaseAccessor.java:316)
at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.reconnect(DatasourceAccessor.java:583)
at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.reconnect(DatabaseAccessor.java:1665)
at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.incrementCallCount(DatasourceAccessor.java:323)
at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:622)
at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeCall(DatabaseAccessor.java:567)
at org.eclipse.persistence.internal.sessions.AbstractSession.basicExecuteCall(AbstractSession.java:2099)
at org.eclipse.persistence.sessions.server.ServerSession.executeCall(ServerSession.java:603)
at org.eclipse.persistence.sessions.server.ClientSession.executeCall(ClientSession.java:265)
at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:275)
at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:261)
at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeSelectCall(DatasourceCallQueryMechanism.java:332)
at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.selectAllRows(DatasourceCallQueryMechanism.java:744)
at org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectAllRowsFromTable(ExpressionQueryMechanism.java:2759)
at org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectAllRows(ExpressionQueryMechanism.java:2712)
at org.eclipse.persistence.queries.ReadAllQuery.executeObjectLevelReadQuery(ReadAllQuery.java:584)
at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1232)
at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:911)
at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1191)
at org.eclipse.persistence.queries.ReadAllQuery.execute(ReadAllQuery.java:485)
at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:1279)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2983)
at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1898)
at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1880)
at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1845)
at org.eclipse.persistence.internal.jpa.QueryImpl.executeReadQuery(QueryImpl.java:262)
at org.eclipse.persistence.internal.jpa.QueryImpl.getResultList(QueryImpl.java:482)
... 127 common frames omitted
Caused by: java.lang.reflect.InvocationTargetException: null
at jdk.internal.reflect.GeneratedMethodAccessor233.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.foo.datasource.SchemaBasedDataSourceInvocationHandler.invoke(SchemaBasedDataSourceInvocationHandler.java:27)
... 158 common frames omitted
Caused by: java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30006ms.
at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:695)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:197)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:162)
at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:128)
at com.foo.datasource.DataSourceSelector.getConnection(DataSourceSelector.java:64)
... 162 common frames omitted
It's also worth noting that Hikari also complains:
11:50:21.0800 - DEBUG TenantId[] UserId[] TraceId[] Thread[HikariPool-1 housekeeper]
Logger[com.zaxxer.hikari.pool.HikariPool]
Msg[HikariPool-1 - Pool stats (total=8, active=8, idle=0, waiting=1)]
11:50:21.0800 - DEBUG TenantId[] UserId[] TraceId[] Thread[HikariPool-1 housekeeper]
Logger[com.zaxxer.hikari.pool.HikariPool]
Msg[HikariPool-1 - Fill pool skipped, pool is at sufficient level.]
11:50:35.0582 - DEBUG TenantId[-1] UserId[] TraceId[] Thread[pool-6-thread-1]
Logger[com.zaxxer.hikari.pool.HikariPool]
Msg[HikariPool-1 - Timeout failure stats (total=8, active=8, idle=0, waiting=0)]
It seems pretty obvious that the spawned threads are having trouble managing the connections. They either aren't getting any, or are not releasing them back to the pool properly. (Although in any case, even if I fix this, I'm not entirely certain the whole "rollback everything if an error occurs" is achievable?)
Here is an attempt at abstracting the current code's state (CF are CompletableFuture, --> are calls to other classes, <-- are the returned type, ### is a blocking operation):
#RestController
--> #Service "Service 1"
--> #Transactional #Service "Service for First Select"
--> #Repository #Async "Table A"
<-- CF
<-- CF
<-- CF#thenApply
--> #Service "Service 1"
--> #Transactional #Service "Service for Second Select"
--> #Repository #Async "Table B"
<-- CF
<-- CF
<-- CF#thenApply
--> #Transactional #Service "Service 2"
--> #Repository #Async "Delete from Table X"
<-- void
--> #Repository #Async "Delete from Table Y"
<-- void
--> #Repository #Async "Delete from Table Z"
<-- void
### Need to wait for ALL (CF and void) of the previous #Async calls to be done
--> #Repository #Async "Insert into Table X"
<-- void
--> #Repository #Async "Insert into Table Y"
<-- void
--> #Repository #Async "Insert into Table Z"
<-- void
<-- void
Other relevant threads
I've read through these to get some insight:
https://stackoverflow.com/a/52177959/9768291
https://stackoverflow.com/a/24917195/9768291
https://stackoverflow.com/a/29349819/9768291
https://stackoverflow.com/a/47352233/9768291
https://dzone.com/articles/spring-async-and-transaction
https://dzone.com/articles/spring-transaction-management-over-multiple-thread-1
Sometimes when I stop my Liferay module (for instance when I put a new version of its JAR in deploy/) the module refuses to stop.
While the module should go to "Resolved" state, it stays in the "Stopping" state forever:
Usually it is due to a thread not terminated somewhere, or a network connection not properly closed, and it is often a pain to investigate.
My question: How to find out more efficiently what this Liferay module's problem is?
What I tried:
In Gogo Shell diag <module id> does not seem to provide any valuable information about why the module refuses to leave the "Stopping" state.
jstack outputs thousands of lines, the vast majority of which is outside of the Liferay module in question. If there was a way to show jstack information for only my module that would be wonderful.
First, find the PID of your webapp server:
ps aux | grep tomcat
Adapt the command if you are running another server than tomcat, or if you have several instances running.
Then, dump all threads of that server to a file:
jstack 12345 > jstack.txt
Where 12345 is the PID you found in the first step.
Then, look at the source code of your bundle, and find the service activator. It typically looks like this:
package fr.free.nrw;
[import section]
public class ServiceActivator implements BundleActivator {
private ServiceRegistration registration;
#Override
public void start(BundleContext context) throws Exception {
registration = context.registerService(
MyService.class.getName(), new MyServiceImpl(), null);
}
#Override
public void stop(BundleContext context) throws Exception {
registration.unregister();
}
}
Take note of:
the namespace,
the class name,
the stopping method name.
For instance in the example above they are fr.free.nrw, ServiceActivator and stop, and from these three get the full name fr.free.nrw.ServiceActivator.stop.
Now open jstack.txt and search for the full name. Even though the file is thousands of lines long, there will most probably only be one hit, and that is the problematic thread:
at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.unregister(ServiceRegistrationImpl.java:222)
at fr.free.nrw.ServiceActivator.stop(ServiceActivator.java:30)
at org.eclipse.osgi.internal.framework.BundleContextImpl$4.run(BundleContextImpl.java:830)
at org.eclipse.osgi.internal.framework.BundleContextImpl$4.run(BundleContextImpl.java:1)
at java.security.AccessController.doPrivileged(Native Method)
at org.eclipse.osgi.internal.framework.BundleContextImpl.stop(BundleContextImpl.java:823)
In this file, go up to the beginning of the paragraph, which will be something like this:
"fileinstall-/home/nico/p/liferay/osgi/modules" #37 daemon prio=5 os_prio=0 tid=0x00007f39480e3000 nid=0x384f waiting on condition [0x00007f395d169000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000eb8defb8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
With this information in hand, you will known what kind of threading problem is going on, and you will be able to solve it with usual Java thread debugging techniques (1 2).
The Activator you shared should never block on the stop method. So I doubt it can cause the behavior you described.
I'm using Fiji on linux (inside a virtual machine) from the command line.
Specifically I have a python script that calls many fiji macros.
Each time fiji loads it takes about a minute, making it the bottleneck of the process and slowing me considerably.
I wonder if this is normal. Could fiji perhaps be made to load more quickly?
Fiji also outputs a lengthy error message when it is called. I wonder if it is related to the lengthy time it takes to load.
The command I use to call it (for example)
//shared_directory/projects/Fiji.app/ImageJ-linux64 --headless
-macro //shared_directory/projects/retracted/fiji_scripts/patch_cropper.txt
The error message is as follows
java.awt.HeadlessException
at java.awt.GraphicsEnvironment.checkHeadless(GraphicsEnvironment.java:204)
at java.awt.Window.<init>(Window.java:536)
at java.awt.Frame.<init>(Frame.java:420)
at ij.plugin.frame.PlugInFrame.<init>(PlugInFrame.java:13)
at ij.plugin.frame.RoiManager.<init>(RoiManager.java:94)
at ij.macro.Interpreter.getBatchModeRoiManager(Interpreter.java:1875)
at ij.plugin.frame.RoiManager.getInstance(RoiManager.java:1795)
at ij.ImagePlus.deleteRoi(ImagePlus.java:1735)
at ij.ImagePlus.close(ImagePlus.java:391)
at ij.plugin.Commands.closeImage(Commands.java:136)
at ij.plugin.Commands.close(Commands.java:83)
at ij.plugin.Commands.run(Commands.java:29)
at ij.IJ.runPlugIn(IJ.java:187)
at ij.Executer.runCommand(Executer.java:137)
at ij.Executer.run(Executer.java:66)
at ij.IJ.run(IJ.java:297)
at ij.IJ.run(IJ.java:272)
at ij.macro.Functions.doRun(Functions.java:603)
at ij.macro.Functions.doFunction(Functions.java:96)
at ij.macro.Interpreter.doStatement(Interpreter.java:230)
at ij.macro.Interpreter.doStatements(Interpreter.java:218)
at ij.macro.Interpreter.run(Interpreter.java:115)
at ij.macro.Interpreter.run(Interpreter.java:85)
at ij.macro.Interpreter.run(Interpreter.java:96)
at ij.plugin.Macro_Runner.runMacro(Macro_Runner.java:155)
at ij.plugin.Macro_Runner.runMacroFile(Macro_Runner.java:139)
at ij.IJ.runMacroFile(IJ.java:148)
at net.imagej.legacy.IJ1Helper$4.call(IJ1Helper.java:1056)
at net.imagej.legacy.IJ1Helper$4.call(IJ1Helper.java:1052)
at net.imagej.legacy.IJ1Helper.runMacroFriendly(IJ1Helper.java:986)
at net.imagej.legacy.IJ1Helper.runMacroFile(IJ1Helper.java:1052)
at net.imagej.legacy.LegacyCommandline$Macro.handle(LegacyCommandline.java:188)
at org.scijava.console.DefaultConsoleService.processArgs(DefaultConsoleService.java:102)
at net.imagej.legacy.LegacyConsoleService.processArgs(LegacyConsoleService.java:81)
at org.scijava.AbstractGateway.launch(AbstractGateway.java:95)
at net.imagej.Main.launch(Main.java:62)
at net.imagej.Main.main(Main.java:68)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at net.imagej.launcher.ClassLauncher.launch(ClassLauncher.java:279)
at net.imagej.launcher.ClassLauncher.run(ClassLauncher.java:186)
at net.imagej.launcher.ClassLauncher.main(ClassLauncher.java:77)
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=128m; support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: Using incremental CMS is deprecated and will likely be removed in a future release
java.lang.IllegalArgumentException: Cannot handle app name in ij.gui.YesNoCancelDialog's public <init>(java.awt.Frame parent, java.lang.String title, java.lang.String msg)
at net.imagej.patcher.CodeHacker.replaceAppNameInCall(CodeHacker.java:446)
at net.imagej.patcher.LegacyExtensions.insertAppNameHooks(LegacyExtensions.java:419)
at net.imagej.patcher.LegacyExtensions.injectHooks(LegacyExtensions.java:291)
at net.imagej.patcher.LegacyInjector.inject(LegacyInjector.java:308)
at net.imagej.patcher.LegacyInjector.injectHooks(LegacyInjector.java:109)
at net.imagej.patcher.LegacyEnvironment.initialize(LegacyEnvironment.java:101)
at net.imagej.patcher.LegacyEnvironment.applyPatches(LegacyEnvironment.java:495)
at net.imagej.patcher.LegacyInjector.preinit(LegacyInjector.java:397)
at net.imagej.patcher.LegacyInjector.preinit(LegacyInjector.java:376)
at net.imagej.legacy.LegacyService.<clinit>(LegacyService.java:134)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at java.lang.Class.newInstance(Class.java:442)
at org.scijava.service.ServiceHelper.createServiceRecursively(ServiceHelper.java:302)
at org.scijava.service.ServiceHelper.createExactService(ServiceHelper.java:269)
at org.scijava.service.ServiceHelper.loadService(ServiceHelper.java:231)
at org.scijava.service.ServiceHelper.createServiceRecursively(ServiceHelper.java:340)
at org.scijava.service.ServiceHelper.createExactService(ServiceHelper.java:269)
at org.scijava.service.ServiceHelper.loadService(ServiceHelper.java:231)
at org.scijava.service.ServiceHelper.loadService(ServiceHelper.java:194)
at org.scijava.service.ServiceHelper.loadServices(ServiceHelper.java:166)
at org.scijava.Context.<init>(Context.java:278)
at org.scijava.Context.<init>(Context.java:234)
at org.scijava.Context.<init>(Context.java:174)
at org.scijava.Context.<init>(Context.java:160)
at net.imagej.ImageJ.<init>(ImageJ.java:77)
at net.imagej.Main.launch(Main.java:61)
at net.imagej.Main.main(Main.java:68)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at net.imagej.launcher.ClassLauncher.launch(ClassLauncher.java:279)
at net.imagej.launcher.ClassLauncher.run(ClassLauncher.java:186)
at net.imagej.launcher.ClassLauncher.main(ClassLauncher.java:77)
Caused by: javassist.CannotCompileException: No code replaced!
at net.imagej.patcher.CodeHacker$EagerExprEditor.instrument(CodeHacker.java:1280)
at net.imagej.patcher.CodeHacker.replaceAppNameInCall(CodeHacker.java:443)
... 36 more
Thanks!
I wonder if this is normal. Could fiji perhaps be made to load more quickly?
I don't think this is intended behavior. As this issue is very specific to ImageJ/Fiji, you should better raise this point on the ImageJ forum.
java.lang.IllegalArgumentException: Cannot handle app name in ij.gui.YesNoCancelDialog's public <init>(java.awt.Frame parent, java.lang.String title, java.lang.String msg)
This is likely due to ij1-patcher not handling ij.gui.YesNoCancelDialog to make it independent on the Java AWT framework (which interferes with the goal of running things headless).
You are welcome to submit a patch to the ij1-patcher project to make it deal with YesNoCancelDialog in a similar way as it currently does with GenericDialog, see these lines.
As a workaround, simply use a macro that doesn't use YesNoCancelDialog, i.e. does not run the getBoolean() macro function.
EDIT: as also noted in the ImageJ forum topic, a patch was submitted here.
Until this gets merged and released, you can work around the issue by downgrading ij.jar using Help > Update ImageJ...
I want to execute instance of supertype AbstractEntryProcessor (lets say SomeEntryProcessor) with IMap.executeOnKey method.
Server side ClassLoader doesn't have this class (SomeEntryProcessor).
So it expectedly fails with:
java.lang.ClassNotFoundException: com.package.SomeEntryProcessor
com.hazelcast.nio.serialization.HazelcastSerializationException: java.lang.ClassNotFoundException: com.package.SomeEntryProcessor
at com.hazelcast.nio.serialization.DefaultSerializers$ObjectSerializer.read(DefaultSerializers.java:190) ~[hazelcast-3.2.3.jar:3.2.3]
at com.hazelcast.nio.serialization.StreamSerializerAdapter.read(StreamSerializerAdapter.java:40) ~[hazelcast-3.2.3.jar:3.2.3]
at com.hazelcast.nio.serialization.SerializationServiceImpl.readObject(SerializationServiceImpl.java:279) ~[hazelcast-3.2.3.jar:3.2.3]
at com.hazelcast.nio.serialization.ByteArrayObjectDataInput.readObject(ByteArrayObjectDataInput.java:433) ~[hazelcast-3.2.3.jar:3.2.3]
at com.hazelcast.map.client.MapExecuteOnKeyRequest.read(MapExecuteOnKeyRequest.java:88) ~[hazelcast-3.2.3.jar:3.2.3]
But after this executeOnKey hangs forever.
I believe this happens due to infinite wait in method
ClientCallFuture.get(long timeout, TimeUnit unit) {
...
this.wait(waitMillis); // line 103 in hazelcast 3.2.3
}
Has anyone seen the same?
Created an issue https://github.com/hazelcast/hazelcast/issues/3842, but no response received
Sorry somehow missed the issue. Can you try a 3.3.x release? I guess the issue is fixed on those versions.
I am using Groovy 1.7.8. and have written a Groovy DSL which I execute concurrently on different domain objects.
Of late, I have started facing deadlocks under heavy concurrency when DSLs are compiling/executing concurrently.
I am compiling/executing Groovy DSL at runtime using:
Script s = compileScript(dsl)
def binding = new Binding()
binding.setVariable("domainObj", domainObj)
s.setBinding(binding)
s.run()
def Script compileScript(String dsl) {
def scriptText = getScriptText(dsl)
def conf = new CompilerConfiguration()
conf.setScriptBaseClass(GroovyViewExecutionBase.class.getName())
new GroovyShell(conf).parse(scriptText)
}
Below is the thread dump of deadlocked threads, looking at the thread dumps I feel that Groovy is deadlocking internally when we concurrently compile/execute scripts at runtime.
Deadlock-Participant-1:
at java.beans.PropertyDescriptor.getReadMethod(PropertyDescriptor.java:158)
- waiting to lock <0x00007f41a3363960> (a java.beans.PropertyDescriptor)
at java.beans.Introspector.processPropertyDescriptors(Introspector.java:683)
at java.beans.Introspector.getTargetPropertyInfo(Introspector.java:615)
at java.beans.Introspector.getBeanInfo(Introspector.java:407)
at java.beans.Introspector.getBeanInfo(Introspector.java:164)
- locked <0x00007f41a3358c28> (a java.lang.Object)
at groovy.lang.MetaClassImpl$15.run(MetaClassImpl.java:2940)
at java.security.AccessController.doPrivileged(Native Method)
at groovy.lang.MetaClassImpl.addProperties(MetaClassImpl.java:2938)
at groovy.lang.MetaClassImpl.initialize(MetaClassImpl.java:2921)
- locked <0x00007f429f011268> (a groovy.lang.ExpandoMetaClass)
at groovy.lang.ExpandoMetaClass.initialize(ExpandoMetaClass.java:463)
- locked <0x00007f429f011268> (a groovy.lang.ExpandoMetaClass)
at org.codehaus.groovy.reflection.ClassInfo.getMetaClassUnderLock(ClassInfo.java:166)
at org.codehaus.groovy.reflection.ClassInfo.getMetaClass(ClassInfo.java:182)
at org.codehaus.groovy.runtime.metaclass.MetaClassRegistryImpl.getMetaClass(MetaClassRegistryImpl.java:210)
at org.codehaus.groovy.runtime.InvokerHelper.getMetaClass(InvokerHelper.java:751)
at groovy.lang.GroovyObjectSupport.<init>(GroovyObjectSupport.java:32)
at groovy.lang.Script.<init>(Script.java:40)
at groovy.lang.Script.<init>(Script.java:37)
at flipkart.cms.views.core.GroovyViewExecutionBase.<init>(GroovyViewExecutionBase.groovy)
at Script1.<init>(Script1.groovy)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at java.lang.Class.newInstance0(Class.java:355)
at java.lang.Class.newInstance(Class.java:308)
at org.codehaus.groovy.runtime.InvokerHelper.createScript(InvokerHelper.java:408)
at groovy.lang.GroovyShell.parse(GroovyShell.java:743)
at groovy.lang.GroovyShell.parse(GroovyShell.java:770)
at groovy.lang.GroovyShell.parse(GroovyShell.java:761)
at groovy.lang.GroovyShell$parse.call(Unknown Source)
at cms.views.core.GroovyDSLViewComputer.compileScript(GroovyDSLViewComputer.groovy:85)
==================
Deadlock-Participant-2:
at java.beans.Introspector.getPublicDeclaredMethods(Introspector.java:1277)
- waiting to lock <0x00007f41a3358c28> (a java.lang.Object)
at java.beans.Introspector.internalFindMethod(Introspector.java:1312)
at java.beans.Introspector.findMethod(Introspector.java:1383)
at java.beans.Introspector.findMethod(Introspector.java:1363)
at java.beans.PropertyDescriptor.getReadMethod(PropertyDescriptor.java:179)
- locked <0x00007f41a3363960> (a java.beans.PropertyDescriptor)
at groovy.lang.MetaClassImpl.applyPropertyDescriptors(MetaClassImpl.java:2215)
at groovy.lang.MetaClassImpl.setupProperties(MetaClassImpl.java:1995)
at groovy.lang.MetaClassImpl.addProperties(MetaClassImpl.java:2950)
at groovy.lang.MetaClassImpl.initialize(MetaClassImpl.java:2921)
- locked <0x00007f42a12972b8> (a groovy.lang.ExpandoMetaClass)
at groovy.lang.ExpandoMetaClass.initialize(ExpandoMetaClass.java:463)
- locked <0x00007f42a12972b8> (a groovy.lang.ExpandoMetaClass)
at org.codehaus.groovy.runtime.HandleMetaClass.replaceDelegate(HandleMetaClass.java:66)
at org.codehaus.groovy.runtime.HandleMetaClass.setProperty(HandleMetaClass.java:91)
at org.codehaus.groovy.runtime.InvokerHelper.setProperty(InvokerHelper.java:179)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.setProperty(ScriptBytecodeAdapter.java:483)
at flipkart.cms.views.core.GroovyViewExecutionBase.setupAttrProperties(GroovyViewExecutionBase.groovy:60)
at flipkart.cms.views.core.GroovyViewExecutionBase$setupAttrProperties$0.callCurrent(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:44)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:141)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:149)
at cms.views.core.GroovyViewExecutionBase.computeView(GroovyViewExecutionBase.groovy:32)
==================
Any ideas how I can fix/workaround this?
Posted this on groovy-user forum and looks like the deadlocks are caused because of synchronization issues in java 6's java.beans.Introspector class which apparently have been fixed in Java 7. Also received a backport of these classes for java 6 and am testing the backported classes to reproduce the deadlocks.
More details here:
http://groovy.329449.n5.nabble.com/Groovy-deadlocks-when-executing-DSL-from-GroovyShell-td5709889.html