Context
This basically relates directly to this other question.
Essentially, our PUT endpoint needs to be more performant. Our approach is now to DELETE all relevant entries and then INSERT the new ones. This endpoint is only called once every few minutes, so there is no actual potential concurrent requests, but it does need to run fast.
I've already done other performance-related improvements like using JPQL for bulk queries, activating EclipseLink's batch-writing, and properly indexing the database.
At this point, the only other improvement I can think of is to run in parallel the calls which aren't related. Here is a screenshot of an execution of our endpoint as seen through Dynatrace PurePath:
Analysis
There are 9 calls to the DB which could be turned into 2 sets of calls:
All of the SELECT and DELETE calls could be started simultaneously and shouldn't step on each others' toes since they relate to different tables.
All of the INSERT calls could be started simultaneously and shouldn't step on each others' toes since they relate to different tables.
The 3x Update calls simply don't exist anymore so we can forget about them. Thus, it seems like this endpoint could potentially be twice as fast if this works as intended.
Difficulty
Although to be confirmed, it is my understanding that:
EntityManager is not thread-safe
A JDBC connection cannot be shared by multiple threads
Thus each thread needs to operate on a different connection
Thus, to allow rollback, those threads should wait until all requests are confirmed to be successful before committing after being done with their individual work
Additionally, #Async methods which return void are harder to control in terms of error handling
Question
How do we properly set up a Spring-MVC endpoint to issue database calls in different threads, and then use the returned CompletableFutures to get proper flow control by ensuring the queries ran successfully (i.e. completed without Exceptions).
The execution needs to be able to:
Await the first set of CompletableFutures completion before proceeding with the next set of requests
Allow rollback in case of error from any of the requests
Current state and problem
I'm trying to achieve this goal by using Spring's #Async, but I'm having trouble understanding how I should do this correctly.
The way I've done it currently throws a SQLTransientConnectionException after the 30s timeout configured for the HikariPool (which contains 8 threads):
javax.persistence.PersistenceException: java.lang.reflect.UndeclaredThrowableException
at org.eclipse.persistence.internal.jpa.QueryImpl.getResultList(QueryImpl.java:493)
at jdk.internal.reflect.GeneratedMethodAccessor236.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.orm.jpa.SharedEntityManagerCreator$DeferredQueryInvocationHandler.invoke(SharedEntityManagerCreator.java:406)
at com.sun.proxy.$Proxy280.getResultList(Unknown Source)
at org.springframework.data.jpa.repository.query.JpaQueryExecution$CollectionExecution.doExecute(JpaQueryExecution.java:126)
at org.springframework.data.jpa.repository.query.JpaQueryExecution.execute(JpaQueryExecution.java:88)
at org.springframework.data.jpa.repository.query.AbstractJpaQuery.doExecute(AbstractJpaQuery.java:155)
at org.springframework.data.jpa.repository.query.AbstractJpaQuery.execute(AbstractJpaQuery.java:143)
at org.springframework.data.repository.core.support.RepositoryMethodInvoker.doInvoke(RepositoryMethodInvoker.java:137)
at org.springframework.data.repository.core.support.RepositoryMethodInvoker.invoke(RepositoryMethodInvoker.java:121)
at org.springframework.data.repository.core.support.QueryExecutorMethodInterceptor.doInvoke(QueryExecutorMethodInterceptor.java:152)
at org.springframework.data.repository.core.support.QueryExecutorMethodInterceptor.invoke(QueryExecutorMethodInterceptor.java:131)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:137)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.data.jpa.repository.support.CrudMethodMetadataPostProcessor$CrudMethodMetadataPopulatingMethodInterceptor.invoke(CrudMethodMetadataPostProcessor.java:145)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy187.getAllSitesWithConcatenatedIds(Unknown Source)
...
Caused by: java.lang.reflect.UndeclaredThrowableException: null
at com.sun.proxy.$Proxy127.getConnection(Unknown Source)
at org.eclipse.persistence.sessions.JNDIConnector.connect(JNDIConnector.java:138)
at org.eclipse.persistence.sessions.DatasourceLogin.connectToDatasource(DatasourceLogin.java:172)
at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.connectInternal(DatasourceAccessor.java:348)
at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.connectInternal(DatabaseAccessor.java:316)
at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.reconnect(DatasourceAccessor.java:583)
at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.reconnect(DatabaseAccessor.java:1665)
at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.incrementCallCount(DatasourceAccessor.java:323)
at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:622)
at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeCall(DatabaseAccessor.java:567)
at org.eclipse.persistence.internal.sessions.AbstractSession.basicExecuteCall(AbstractSession.java:2099)
at org.eclipse.persistence.sessions.server.ServerSession.executeCall(ServerSession.java:603)
at org.eclipse.persistence.sessions.server.ClientSession.executeCall(ClientSession.java:265)
at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:275)
at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:261)
at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeSelectCall(DatasourceCallQueryMechanism.java:332)
at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.selectAllRows(DatasourceCallQueryMechanism.java:744)
at org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectAllRowsFromTable(ExpressionQueryMechanism.java:2759)
at org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectAllRows(ExpressionQueryMechanism.java:2712)
at org.eclipse.persistence.queries.ReadAllQuery.executeObjectLevelReadQuery(ReadAllQuery.java:584)
at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1232)
at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:911)
at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1191)
at org.eclipse.persistence.queries.ReadAllQuery.execute(ReadAllQuery.java:485)
at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:1279)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2983)
at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1898)
at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1880)
at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1845)
at org.eclipse.persistence.internal.jpa.QueryImpl.executeReadQuery(QueryImpl.java:262)
at org.eclipse.persistence.internal.jpa.QueryImpl.getResultList(QueryImpl.java:482)
... 127 common frames omitted
Caused by: java.lang.reflect.InvocationTargetException: null
at jdk.internal.reflect.GeneratedMethodAccessor233.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.foo.datasource.SchemaBasedDataSourceInvocationHandler.invoke(SchemaBasedDataSourceInvocationHandler.java:27)
... 158 common frames omitted
Caused by: java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30006ms.
at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:695)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:197)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:162)
at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:128)
at com.foo.datasource.DataSourceSelector.getConnection(DataSourceSelector.java:64)
... 162 common frames omitted
It's also worth noting that Hikari also complains:
11:50:21.0800 - DEBUG TenantId[] UserId[] TraceId[] Thread[HikariPool-1 housekeeper]
Logger[com.zaxxer.hikari.pool.HikariPool]
Msg[HikariPool-1 - Pool stats (total=8, active=8, idle=0, waiting=1)]
11:50:21.0800 - DEBUG TenantId[] UserId[] TraceId[] Thread[HikariPool-1 housekeeper]
Logger[com.zaxxer.hikari.pool.HikariPool]
Msg[HikariPool-1 - Fill pool skipped, pool is at sufficient level.]
11:50:35.0582 - DEBUG TenantId[-1] UserId[] TraceId[] Thread[pool-6-thread-1]
Logger[com.zaxxer.hikari.pool.HikariPool]
Msg[HikariPool-1 - Timeout failure stats (total=8, active=8, idle=0, waiting=0)]
It seems pretty obvious that the spawned threads are having trouble managing the connections. They either aren't getting any, or are not releasing them back to the pool properly. (Although in any case, even if I fix this, I'm not entirely certain the whole "rollback everything if an error occurs" is achievable?)
Here is an attempt at abstracting the current code's state (CF are CompletableFuture, --> are calls to other classes, <-- are the returned type, ### is a blocking operation):
#RestController
--> #Service "Service 1"
--> #Transactional #Service "Service for First Select"
--> #Repository #Async "Table A"
<-- CF
<-- CF
<-- CF#thenApply
--> #Service "Service 1"
--> #Transactional #Service "Service for Second Select"
--> #Repository #Async "Table B"
<-- CF
<-- CF
<-- CF#thenApply
--> #Transactional #Service "Service 2"
--> #Repository #Async "Delete from Table X"
<-- void
--> #Repository #Async "Delete from Table Y"
<-- void
--> #Repository #Async "Delete from Table Z"
<-- void
### Need to wait for ALL (CF and void) of the previous #Async calls to be done
--> #Repository #Async "Insert into Table X"
<-- void
--> #Repository #Async "Insert into Table Y"
<-- void
--> #Repository #Async "Insert into Table Z"
<-- void
<-- void
Other relevant threads
I've read through these to get some insight:
https://stackoverflow.com/a/52177959/9768291
https://stackoverflow.com/a/24917195/9768291
https://stackoverflow.com/a/29349819/9768291
https://stackoverflow.com/a/47352233/9768291
https://dzone.com/articles/spring-async-and-transaction
https://dzone.com/articles/spring-transaction-management-over-multiple-thread-1
Related
Tech stack:
JBeret (core, se) 1.3.0.Final
Hibernate Search (orm, jsr352-core, jsr352-jberet) 5.10.4.Final
Weld (servlet-core, se-core) 3.0.5.Final
If I trigger
BatchRuntime.getJobOperator().start(
MassIndexingJob.NAME,
MassIndexingJob.parameters().forEntity(getDomainObjectClass()).build()
);
then I had the situation that a can't access any CDI component outside of the batch job that are RequestScoped or SessionScoped, until the batch job is finished.
How I can fix this problem?
Part of the stacktrace
Caused by: org.jboss.weld.contexts.ContextNotActiveException: WELD-001303: No active contexts for scope type javax.enterprise.context.RequestScoped
at org.jboss.weld.manager.BeanManagerImpl.getContext(BeanManagerImpl.java:647) ~[weld-core-impl-3.0.5.Final.jar:3.0.5.Final]
at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.getIfExists(ContextualInstanceStrategy.java:89) ~[weld-core-impl-3.0.5.Final.jar:3.0.5.Final]
at org.jboss.weld.bean.ContextualInstanceStrategy$CachingContextualInstanceStrategy.getIfExists(ContextualInstanceStrategy.java:164) ~[weld-core-impl-3.0.5.Final.jar:3.0.5.Final]
at org.jboss.weld.bean.ContextualInstance.getIfExists(ContextualInstance.java:63) ~[weld-core-impl-3.0.5.Final.jar:3.0.5.Final]
at org.jboss.weld.bean.proxy.ContextBeanInstance.getInstance(ContextBeanInstance.java:87) ~[weld-core-impl-3.0.5.Final.jar:3.0.5.Final]
at org.jboss.weld.bean.proxy.ProxyMethodHandler.getInstance(ProxyMethodHandler.java:131) ~[weld-core-impl-3.0.5.Final.jar:3.0.5.Final]
at foo.bar.Baz$Proxy$_$$_WeldClientProxy.getFoo(Unknown Source) ~[classes/:na]
Annotated #ActivateRequestContext produce this stacktrace on startup/deployment
Caused by: org.jboss.weld.exceptions.WeldException: WELD-001524: Unable to load proxy class for bean Managed Bean [class foo.bar.Bean] with qualifiers [#Any #Default] with class class foo.bar.Bean using classloader ParallelWebappClassLoader
context: foobar
delegate: false
----------> Parent Classloader:
java.net.URLClassLoader#58a9760d
at org.jboss.weld.bean.proxy.ProxyFactory.getProxyClass(ProxyFactory.java:370)
at org.jboss.weld.injection.producer.SubclassedComponentInstantiator.createEnhancedSubclass(SubclassedComponentInstantiator.java:113)
at org.jboss.weld.injection.producer.SubclassedComponentInstantiator.initEnhancedSubclass(SubclassedComponentInstantiator.java:86)
at org.jboss.weld.injection.producer.SubclassedComponentInstantiator.<init>(SubclassedComponentInstantiator.java:79)
at org.jboss.weld.injection.producer.SubclassedComponentInstantiator.forInterceptedDecoratedBean(SubclassedComponentInstantiator.java:63)
at org.jboss.weld.injection.producer.BeanInjectionTarget.initializeAfterBeanDiscovery(BeanInjectionTarget.java:121)
at org.jboss.weld.injection.producer.InjectionTargetInitializationContext.initialize(InjectionTargetInitializationContext.java:42)
at org.jboss.weld.injection.producer.InjectionTargetService.initialize(InjectionTargetService.java:63)
at org.jboss.weld.bootstrap.WeldStartup.deployBeans(WeldStartup.java:475)
at org.jboss.weld.bootstrap.WeldBootstrap.deployBeans(WeldBootstrap.java:86)
at org.jboss.weld.environment.servlet.WeldServletLifecycle.initialize(WeldServletLifecycle.java:236)
at org.jboss.weld.environment.servlet.EnhancedListener.onStartup(EnhancedListener.java:62)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5245)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
... 42 more
Caused by: org.jboss.weld.exceptions.WeldException: Cannot load variable at 0. Local Variables: Local Variables: []
at org.jboss.weld.bean.proxy.InterceptedSubclassFactory.addMethodsFromClass(InterceptedSubclassFactory.java:262)
at org.jboss.weld.bean.proxy.InterceptedSubclassFactory.addMethods(InterceptedSubclassFactory.java:136)
at org.jboss.weld.bean.proxy.ProxyFactory.createProxyClass(ProxyFactory.java:449)
at org.jboss.weld.bean.proxy.ProxyFactory.getProxyClass(ProxyFactory.java:362)
... 55 more
Caused by: org.jboss.classfilewriter.InvalidBytecodeException: Cannot load variable at 0. Local Variables: Local Variables: []
at org.jboss.classfilewriter.code.CodeAttribute.aload(CodeAttribute.java:196)
at org.jboss.weld.bean.proxy.RunWithinInterceptionDecorationContextGenerator.startIfNotOnTop(RunWithinInterceptionDecorationContextGenerator.java:71)
at org.jboss.weld.bean.proxy.RunWithinInterceptionDecorationContextGenerator.runStartIfNotOnTop(RunWithinInterceptionDecorationContextGenerator.java:148)
at org.jboss.weld.bean.proxy.InterceptedSubclassFactory.addMethodsFromClass(InterceptedSubclassFactory.java:200)
... 58 more
I do not know what exactly JBeret does, but Weld SE out of the box does not activate request context (or session context) which in turn leads to the exception you are seeing. The reason is that in SE there are no HTTP requests (or sessions) hence Weld simply does not know when to activate it.
Although "request" can be interpreted differently and can be valuable addition even in SE - that's why there are supported ways to activate request context, for instance via interceptor. I suppose this is something JBeret does for you and that's why the beans "work" there.
Therefore in order to be able to use your request scoped beans in SE application, you will need to take extra steps. Note however that the context can be different from that of JBeret batch job (you won't see the same beans with the exact same state) as I expect JBeret to offload the work to another thread.
I have a webservice in adf that treats requests, I use methods from a jar that run through view object iterators and so on.
In treating a request I call a method from the said jar that initializes a separate thread that manipulates and compares rows from view objects in that separate thread,
I am getting random exceptions like the following:
## Detail 0 ##
java.lang.NullPointerException
at oracle.jbo.server.ViewRowSetIteratorImpl.initViewRowSetIteratorImpl(ViewRowSetIteratorImpl.java:227)
at oracle.jbo.server.ViewRowSetIteratorImpl.(ViewRowSetIteratorImpl.java:176)
at oracle.jbo.server.ViewRowSetImpl.createNewRowSetIterator(ViewRowSetImpl.java:1434)
at oracle.jbo.server.ViewRowSetImpl.createViewRowSetIterator(ViewRowSetImpl.java:1485)
at oracle.jbo.server.ViewRowSetImpl.getDefaultRowSetIterator(ViewRowSetImpl.java:1704)
at oracle.jbo.server.ViewRowSetImpl.setRangeSize(ViewRowSetImpl.java:2936)
at oracle.jbo.server.ViewObjectImpl.doCreateViewLinkAccessorRS(ViewObjectImpl.java:16080)
at oracle.jbo.server.ViewObjectImpl.createViewLinkAccessorRS(ViewObjectImpl.java:16171)
at oracle.jbo.server.AssociationDefImpl.get(AssociationDefImpl.java:472)
at oracle.jbo.server.ViewAttributeDefImpl.get(ViewAttributeDefImpl.java:949)
at oracle.jbo.server.ViewRowStorage.getViewLinkAccessorResult(ViewRowStorage.java:1528)
at oracle.jbo.server.ViewRowStorage.getAttributeInternal(ViewRowStorage.java:1827)
at oracle.jbo.server.ViewRowImpl.getAttributeValue(ViewRowImpl.java:1923)
at oracle.jbo.server.ViewRowImpl.getAttributeInternal(ViewRowImpl.java:866)
and
## Detail 0 ##
java.lang.NullPointerException
at oracle.jbo.server.ViewRowImpl.getUseViewLogicGroupDef(ViewRowImpl.java:6078)
at oracle.jbo.server.ViewRowStorage.getViewAttributeDef(ViewRowStorage.java:1432)
at oracle.jbo.server.ViewRowStorage.getAttributeInternal(ViewRowStorage.java:1778)
at oracle.jbo.server.ViewRowImpl.getAttributeValue(ViewRowImpl.java:1923)
at oracle.jbo.server.ViewRowImpl.getAttributeInternal(ViewRowImpl.java:866)
and
Caused by: java.lang.NullPointerException
at oracle.jbo.server.ViewRowSetIteratorImpl.initViewRowSetIteratorImpl(ViewRowSetIteratorImpl.java:227)
at oracle.jbo.server.ViewRowSetIteratorImpl.(ViewRowSetIteratorImpl.java:176)
at oracle.jbo.server.ViewRowSetImpl.createNewRowSetIterator(ViewRowSetImpl.java:1434)
at oracle.jbo.server.ViewRowSetImpl.createViewRowSetIterator(ViewRowSetImpl.java:1485)
at oracle.jbo.server.ViewRowSetImpl.getDefaultRowSetIterator(ViewRowSetImpl.java:1704)
at oracle.jbo.server.ViewRowSetImpl.setRangeSize(ViewRowSetImpl.java:2936)
at oracle.jbo.server.ViewObjectImpl.doCreateViewLinkAccessorRS(ViewObjectImpl.java:16080)
at oracle.jbo.server.ViewObjectImpl.createViewLinkAccessorRS(ViewObjectImpl.java:16171)
at oracle.jbo.server.AssociationDefImpl.get(AssociationDefImpl.java:472)
at oracle.jbo.server.ViewAttributeDefImpl.get(ViewAttributeDefImpl.java:949)
at oracle.jbo.server.ViewRowStorage.getViewLinkAccessorResult(ViewRowStorage.java:1528)
at oracle.jbo.server.ViewRowStorage.getAttributeInternal(ViewRowStorage.java:1827)
at oracle.jbo.server.ViewRowImpl.getAttributeValue(ViewRowImpl.java:1923)
at oracle.jbo.server.ViewRowImpl.getAttributeInternal(ViewRowImpl.java:866)
The same method when it runs without the jar ( same code like in the jar but as a part of the same project ), it executes perfectly fine.
Could you please tell me why a NullPointerException is raised?
Thanks
You can not use the Business Components(ViewObject,Entities... etc) inside a thread.
There is no context (FacesContext cannot be initialized ).So you have to pass the ApplicationModule as a paramter to your method. But it will take a lot of time for processing
So the best way to access your data is to use CallableStatments PreparedStatments with Java Data Source connection.
Sometimes when I stop my Liferay module (for instance when I put a new version of its JAR in deploy/) the module refuses to stop.
While the module should go to "Resolved" state, it stays in the "Stopping" state forever:
Usually it is due to a thread not terminated somewhere, or a network connection not properly closed, and it is often a pain to investigate.
My question: How to find out more efficiently what this Liferay module's problem is?
What I tried:
In Gogo Shell diag <module id> does not seem to provide any valuable information about why the module refuses to leave the "Stopping" state.
jstack outputs thousands of lines, the vast majority of which is outside of the Liferay module in question. If there was a way to show jstack information for only my module that would be wonderful.
First, find the PID of your webapp server:
ps aux | grep tomcat
Adapt the command if you are running another server than tomcat, or if you have several instances running.
Then, dump all threads of that server to a file:
jstack 12345 > jstack.txt
Where 12345 is the PID you found in the first step.
Then, look at the source code of your bundle, and find the service activator. It typically looks like this:
package fr.free.nrw;
[import section]
public class ServiceActivator implements BundleActivator {
private ServiceRegistration registration;
#Override
public void start(BundleContext context) throws Exception {
registration = context.registerService(
MyService.class.getName(), new MyServiceImpl(), null);
}
#Override
public void stop(BundleContext context) throws Exception {
registration.unregister();
}
}
Take note of:
the namespace,
the class name,
the stopping method name.
For instance in the example above they are fr.free.nrw, ServiceActivator and stop, and from these three get the full name fr.free.nrw.ServiceActivator.stop.
Now open jstack.txt and search for the full name. Even though the file is thousands of lines long, there will most probably only be one hit, and that is the problematic thread:
at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.unregister(ServiceRegistrationImpl.java:222)
at fr.free.nrw.ServiceActivator.stop(ServiceActivator.java:30)
at org.eclipse.osgi.internal.framework.BundleContextImpl$4.run(BundleContextImpl.java:830)
at org.eclipse.osgi.internal.framework.BundleContextImpl$4.run(BundleContextImpl.java:1)
at java.security.AccessController.doPrivileged(Native Method)
at org.eclipse.osgi.internal.framework.BundleContextImpl.stop(BundleContextImpl.java:823)
In this file, go up to the beginning of the paragraph, which will be something like this:
"fileinstall-/home/nico/p/liferay/osgi/modules" #37 daemon prio=5 os_prio=0 tid=0x00007f39480e3000 nid=0x384f waiting on condition [0x00007f395d169000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000eb8defb8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
With this information in hand, you will known what kind of threading problem is going on, and you will be able to solve it with usual Java thread debugging techniques (1 2).
The Activator you shared should never block on the stop method. So I doubt it can cause the behavior you described.
I am creating a thread in a web application in the following way. Creating a thread in web application is perhaps not the right thing to do but unfortunately that is how it has been done in my application.
The thread has to call a stored procedure making use of the same connection object passed to its runnable object. But the procedure doesn't get executed because of an error DSRA9110E: Statement is closed. Intermittently I also get "Connection is closed." Note that this happens only in IBM Websphere and there are no issues when deployed in Apache Tomcat.
Is it possible that the thread gets running i.e. thread.start() is executed before persistReconcileRecord method gets completed.
I am unable to understand what causes this statement/connection is closed issue. I appreciate any help regarding this issue. Please tell me if more information is required.
public class MyServiceImpl{
private ReconDAO reconDAO = new ReconDAO();
public String anyMethod(String infodom,ReconModel recon){
//persistReconcileRecord is a method in DAO class.
reconDAO.persistReconcileRecord(infodom, recon,"I");
Connection connection=DBManager.getConnection(infodom);
WorkerThread worker=new WorkerThread(infodom,recon.getReconciliationId(),"I",connection);
Thread thread=new Thread(worker);
thread.start();
JSONObject jsonObj=new JSONObject();
jsonObj.put("EXIST_VALIDATION", "false");
jsonObj.put("RECONCILIATION_ID", recon.getReconciliationId());
return jsonObj.toString();
}
}
public class ReconDAO{
public void persistReconcileRecord(String infodom,ReconModel reconModel) throws Exception{
try{
//This method creates a new connection and inserts records into database and then closes it.
}catch(Exception e){
}finally{
closeConnection(connection);
closePreparedStatement(pstmt);
}
}
public class WorkerThread implements Runnable{
private String infodom;
private Long reconciliationId;
private String operation;
private Connection connection;
//A parameterized constructor to initialize all instance variables
public void run(){
//Uses the connection object from this class and then closes it in finally block
//Calls a stored procedure
}
}
There are a couple of problems with what your application is attempting. First, the JDBC programming model does not support multi-threaded access to connections. Second, even if it did support this, the manner in which the application passes the connection handle to another thread and then proceeds to close the connection means that the when the thread runs, the connection is closed from underneath it. Per JDBC spec, close of a connection requires its statements to be closed. So the behavior you are seeing is by design (and you could expect to see even worse unpredictable errors like IllegalStateException/ArrayIndexOutOfBoundsException and so forth if it hits the former pattern rather than the latter)
Note that JDBC does support multi-threaded access to data sources. So the proper pattern for the application to use would be to supply the data source to the thread, and the thread can obtain its own connection and close it when finished. You should also consider a more proper approach to threading in Java EE applications. Depending on which version of WebSphere Application Server you are using, that could be Java EE Concurrency (spec standard as of Java EE 7) or Asynchronous Beans.
I don't know if this is a bug or a feature, but it's definitely unintuitive to people from a Java background to track the reason of exception.
Groovy allows variables to be referred to even if the variable is not defined.
For example consider the below class:
class B {
def infos;
public B(String param)
{
infos = param
}
public getInfo()
{
return info;
}
}
If you noticed, inside getInfo(), I am returning info which is never defined. However, Eclipse does not give a warning. So I continue on writing following:
class A
{
static main(def args)
{
B bObj = new B("Mahesh")
println "Hello groovy"
println bObj.getInfo()
println "Hello groovy"
}
}
Now this gives StackOverflowError with a huge stack trace:
Exception in thread "main" java.lang.StackOverflowError
at java.lang.Exception.<init>(Exception.java:102)
at java.lang.ReflectiveOperationException.<init>(ReflectiveOperationException.java:89)
at java.lang.reflect.InvocationTargetException.<init>(InvocationTargetException.java:72)
at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
at groovy.lang.MetaClassImpl$GetBeanMethodMetaProperty.getProperty(MetaClassImpl.java:3493)
at org.codehaus.groovy.runtime.callsite.GetEffectivePogoPropertySite.callGroovyObjectGetProperty(GetEffectivePogoPropertySite.java:67)
--> at packages.B.getInfo(ThreadDumpsExp.groovy:169) <--
at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
at groovy.lang.MetaClassImpl$GetBeanMethodMetaProperty.getProperty(MetaClassImpl.java:3493)
at org.codehaus.groovy.runtime.callsite.GetEffectivePogoPropertySite.callGroovyObjectGetProperty(GetEffectivePogoPropertySite.java:67)
:
:
The stack trace here was ok since somewhere down in the stack trace it pointed to this specific line which I highlighted in the above stack trace with arrows. I was expecting this line in the stack trace, that's why I was able to quickly track it down. The issue arose when today I came across same issue in my project. The stack trace was equally huge. I had no idea where it is actually going wrong so I can not make a guess about which line might actually be faulting. Worst it was stopping in Groovy's source during debugging. I had to repeatedly put breakpoints at different places to actually halt the execution at them. After some time I found the line where the debugger actually stopped inside my code. From there I stepped through my whole code to find the line which was causing issue. That line was a simple getter which was returning te wrong thing.
Now I know I should be more conscious while writing code and should not make such mistakes of returning a non-existent variable from a getter. But is there any way to make it not do what it did above?
Edit
Also after adding #TypeChecked, below error occurs. It was working properly earlier.
When you add a getter for a property, regardless of whether the property exists, you need to reference the property with the .# operator. This is the direct field access operator, which skips any getter and goes straight to the property. If you don't use the operator the same getter will be invoked over and over until you get a StackOverflowError.
For example:
def getInfo() {
return this.#info
}
See Section 6.2 Direct field access operator in the Groovy operator docs for a little more.
The problem lies in the signature of the getInfo-Method which implies that this is a getter. When you call a property of a class (here with "return info") than implicitly groovy uses the getter getInfo. This creates a endless loop which results in the StackOverflowError (because info calls getInfo, getInfo calls info, calls getInfo ...). If you use another not-existing property in here (i.e. return foo) you receive the expected MissingPropertyException. To avoid this behavior you should not define getter-Methods with names matching to not-existing properties.