When looking through some logs after a glassfish server was being unresponsive, I came upon this error :
{{[#|2016-06-02T08:37:51.737+0200|WARNING|oracle-
glassfish3.1.2|org.restlet.Component.ServiceDispatcher|_ThreadID=165;_ThreadName=http-thread-pool-37860(5);|Exception or error caught in status service
javax.ejb.EJBException
at com.sun.ejb.containers.BaseContainer.processSystemException(BaseContainer.java:5236)
at com.sun.ejb.containers.BaseContainer.checkExceptionNoTx(BaseContainer.java:5065)
at com.sun.ejb.containers.BaseContainer.postInvokeTx(BaseContainer.java:4900)
at com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2046)
at com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:1995)
at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:222)
at com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:89)
at com.sun.proxy.$Proxy228.getKpiValuesFlat(Unknown Source)
at com.business.renderer.TopSuppliers.getData(TopSuppliers.java:174)
at com.business.renderer.TopSuppliers.getDataAsJSONTopSuppliers.java:179)
at com.business.renderer.TopSuppliers.render(TopSuppliers.java:107)
at com.ui.renderer.engine.AbstractRenderingEngine.render(AbstractRenderingEngine.java:140)
at com.services.filter.ContainerAdapter.represent(ContainerAdapter.java:311)
at org.restlet.resource.Resource.getRepresentation(Resource.java:259)
at org.restlet.resource.Resource.handleGet(Resource.java:425)
at org.restlet.resource.Finder.handle(Finder.java:470)
at org.restlet.routing.Filter.doHandle(Filter.java:156)
at org.restlet.routing.Filter.handle(Filter.java:203)
at org.restlet.routing.Router.doHandle(Router.java:497)
at org.restlet.routing.Router.handle(Router.java:737)
at com.servlets.ServiceDispatcher$1.handle(ServiceDispatcher.java:55)
at org.restlet.routing.Filter.doHandle(Filter.java:156)
at org.restlet.routing.Filter.handle(Filter.java:203)
at org.restlet.routing.Filter.doHandle(Filter.java:156)
at org.restlet.routing.Filter.handle(Filter.java:203)
at org.restlet.routing.Filter.doHandle(Filter.java:156)
at org.restlet.engine.application.StatusFilter.doHandle(StatusFilter.java:151)
at org.restlet.routing.Filter.handle(Filter.java:203)
at org.restlet.routing.Filter.doHandle(Filter.java:156)
at org.restlet.routing.Filter.handle(Filter.java:203)
at org.restlet.engine.ChainHelper.handle(ChainHelper.java:111)
at org.restlet.engine.application.ApplicationHelper.handle(ApplicationHelper.java:72)
at org.restlet.Application.handle(Application.java:388)
at org.restlet.routing.Filter.doHandle(Filter.java:156)
at org.restlet.routing.Filter.handle(Filter.java:203)
at org.restlet.routing.Router.doHandle(Router.java:497)
at org.restlet.routing.Router.handle(Router.java:737)
at org.restlet.routing.Filter.doHandle(Filter.java:156)
at org.restlet.routing.Filter.handle(Filter.java:203)
at org.restlet.routing.Router.doHandle(Router.java:497)
at org.restlet.routing.Router.handle(Router.java:737)
at org.restlet.routing.Filter.doHandle(Filter.java:156)
at org.restlet.routing.Filter.handle(Filter.java:203)
at org.restlet.engine.ChainHelper.handle(ChainHelper.java:111)
at org.restlet.Component.handle(Component.java:387)
at org.restlet.Server.handle(Server.java:488)
at org.restlet.engine.ServerHelper.handle(ServerHelper.java:71)
at org.restlet.engine.http.HttpServerHelper.handle(HttpServerHelper.java:150)
at org.restlet.ext.servlet.ServerServlet.service(ServerServlet.java:1037)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:770)
at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1554)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:339)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at com.jamonapi.JAMonFilter.doFilter(JAMonFilter.java:59)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:278)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:655)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:595)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:161)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:655)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:595)
at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:334)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:230)
at com.sun.enterprise.v3.services.impl.ContainerMapper$AdapterCallable.call(ContainerMapper.java:311)
at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:189)
at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:850)
at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:747)
at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1032)
at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:231)
at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
at java.lang.Thread.run(Thread.java:745)
Caused by: Exception [EclipseLink-2004] (Eclipse Persistence Services - 2.3.4.v20151027-346465e): org.eclipse.persistence.exceptions.ConcurrencyException
Exception Description: A signal was attempted before wait() on ConcurrencyManager. This normally means that an attempt was made to
commit or rollback a transaction before it was started, or to rollback a transaction twice.
at org.eclipse.persistence.exceptions.ConcurrencyException.signalAttemptedBeforeWait(ConcurrencyException.java:84)
at org.eclipse.persistence.internal.helper.ConcurrencyManager.releaseReadLock(ConcurrencyManager.java:489)
at org.eclipse.persistence.internal.identitymaps.CacheKey.releaseReadLock(CacheKey.java:386)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.cloneAndRegisterObject(UnitOfWorkImpl.java:1018)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.cloneAndRegisterObject(UnitOfWorkImpl.java:929)
at org.eclipse.persistence.internal.sessions.UnitOfWorkIdentityMapAccessor.getAndCloneCacheKeyFromParent(UnitOfWorkIdentityMapAccessor.java:181)
at org.eclipse.persistence.internal.sessions.UnitOfWorkIdentityMapAccessor.getFromIdentityMap(UnitOfWorkIdentityMapAccessor.java:120)
at org.eclipse.persistence.internal.sessions.IdentityMapAccessor.getFromIdentityMap(IdentityMapAccessor.java:380)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.registerExistingObject(UnitOfWorkImpl.java:3899)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.registerExistingObject(UnitOfWorkImpl.java:3854)
at org.eclipse.persistence.mappings.CollectionMapping.buildElementUnitOfWorkClone(CollectionMapping.java:267)
at org.eclipse.persistence.mappings.CollectionMapping.buildElementClone(CollectionMapping.java:279)
at org.eclipse.persistence.internal.queries.ContainerPolicy.addNextValueFromIteratorInto(ContainerPolicy.java:213)
at org.eclipse.persistence.mappings.CollectionMapping.buildCloneForPartObject(CollectionMapping.java:205)
at org.eclipse.persistence.internal.indirection.UnitOfWorkQueryValueHolder.buildCloneFor(UnitOfWorkQueryValueHolder.java:51)
at org.eclipse.persistence.internal.indirection.UnitOfWorkValueHolder.instantiateImpl(UnitOfWorkValueHolder.java:161)
at org.eclipse.persistence.internal.indirection.UnitOfWorkValueHolder.instantiate(UnitOfWorkValueHolder.java:222)
at org.eclipse.persistence.internal.indirection.DatabaseValueHolder.getValue(DatabaseValueHolder.java:88)
at org.eclipse.persistence.indirection.IndirectList.buildDelegate(IndirectList.java:244)
at org.eclipse.persistence.indirection.IndirectList.getDelegate(IndirectList.java:414)
at org.eclipse.persistence.indirection.IndirectList$1.<init>(IndirectList.java:542)
at org.eclipse.persistence.indirection.IndirectList.listIterator(IndirectList.java:541)
at org.eclipse.persistence.indirection.IndirectList.iterator(IndirectList.java:505)
at com.business.frontend.facade.impl.FrontEndBFBean.combineAllContainerKpis(FrontEndBFBean.java:515)
at com.business.frontend.facade.impl.FrontEndBFBean.getKpiValuesFlat(FrontEndBFBean.java:405)
at sun.reflect.GeneratedMethodAccessor237.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.glassfish.ejb.security.application.EJBSecurityManager.runMethod(EJBSecurityManager.java:1052)
at org.glassfish.ejb.security.application.EJBSecurityManager.invoke(EJBSecurityManager.java:1124)
at com.sun.ejb.containers.BaseContainer.invokeBeanMethod(BaseContainer.java:5409)
at com.sun.ejb.EjbInvocation.invokeBeanMethod(EjbInvocation.java:619)
at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:800)
at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:571)
at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.doAround(SystemInterceptorProxy.java:162)
at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.aroundInvoke(SystemInterceptorProxy.java:144)
at sun.reflect.GeneratedMethodAccessor170.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:861)
at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:800)
at com.sun.ejb.containers.interceptors.InterceptorManager.intercept(InterceptorManager.java:370)
at com.sun.ejb.containers.BaseContainer.__intercept(BaseContainer.java:5381)
at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:5369)
at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:214)
... 75 more
On further inspection I notised the lines :
_ThreadID=165;_ThreadName=http-thread-pool-37860(5);
and
Exception [EclipseLink-2004] (Eclipse Persistence Services - 2.3.4.v20151027-346465e)
My initial thought were that that the HTTP thread timeouted waiting for the DB call to complete, and then the HTTP thread interrupted the EclipseLink leaving behind a ConcurrencyManager in an unexpected state (with a DeferredLock not in fact released after use). I made a small app, testing my hypothesis by tweking the server settings and making long calls to the DB and terminating the http call before the DB call was ready, but this doesn't seem to reproduce the problem. There is also the fact that the method called when the exception occurs does not use any kind of JPA(ORM), the call uses JDBC. Most of the application uses EclipseLink and JPA, but not this part. Also it is not uncommon for the application to be under heavy database usage.
So I was thinking, what can be the reason behind this error appart from the descriptive (This normally means that an attempt was made to
commit or rollback a transaction before it was started, or to rollback a transaction twice), I'm pretty sure the code is not doing such a thing.
So it must be some lazy loading conflict, but I have no idea what conditions can cause this problem.
I have no idea where to investigate from this point forward. Any ideas are well welcomed :)
In this error case, the exact reason for the concurrency exception isn't known - we can only see the one thread accessing the context, by triggering a lazy relationship on an entity. How this entity was read in and what happened in other threads that might also be using this EntityManager the entity was read from (directly or indirectly by accessing entities read from it) requires tracking down, but can usually be avoided.
EntityManagers are not meant to be accessed by multiple threads, and so do not handle concurrent access. Entities read in from EntityManagers have many hooks to the context they were read from; these hooks allow for lazy relationships, change tracking etc to occur transparent to the application. This has the consequence though that these entities too cannot be concurrently accessed - they should not be cached or passed around by the application without careful consideration. In many cases, applications might be better off letting JPA handle caching, and access entities as needed from a context.
Related
I am using Jmeter 5.2.1 along with Concurrency Thread Group with ${__tstFeedback(ThroughputShapingTimer,1,10,10)} in combination with the Throughput Shaping Timer to dynamically change the target throughput throughout the test duration.
The Test Plan consists of a Thread group having a Module Controller and the ThroughputShapingTimer underneath it. The module controller controls a Test Fragment which contains the test logic.
I am encountering an issue at the end of the test where most of the time, a few last samplers result in a "java.net.SocketException: Socket closed" exception. This happens regardless if I have a ramp-down to 0 at the end of the test or not.
By contrast, if I use the same script with the same setup but just control it from a normal Thread Group, then all of the samplers complete successfully at the end of the script without issues.
Here are some screenshots of my setup:
Full error text from the sampler:
java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at sun.security.ssl.InputRecord.readFully(Unknown Source)
at sun.security.ssl.InputRecord.read(Unknown Source)
at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)
at sun.security.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
at sun.security.ssl.AppInputStream.read(Unknown Source)
at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.executeRequest(HTTPHC4Impl.java:850)
at org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.sample(HTTPHC4Impl.java:561)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerProxy.sample(HTTPSamplerProxy.java:67)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1282)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1271)
at org.apache.jmeter.threads.JMeterThread.doSampling(JMeterThread.java:627)
at org.apache.jmeter.threads.JMeterThread.executeSamplePackage(JMeterThread.java:551)
at org.apache.jmeter.threads.JMeterThread.processSampler(JMeterThread.java:490)
at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:257)
at java.lang.Thread.run(Unknown Source)
I would like to understand what the cause could be as this is impacting the reporting.
Any help to remove or work around this issue would be very appreciated.
JMeter cannot gracefully terminate the thread after 5 tries hence it forcefully terminates the connection, in your case this termination happens during response from the server breaking SSL tunnel and causing the error.
"Normal" thread group uses different approach for stopping the threads which cannot be re-used in the Concurrency Thread Group as in your case it spawns an extra pool of threads which doesn't belong to the Thread Group unless they're explicitly used hence they're kind of beyond JMeter's control.
If you still want to use the Concurrency Thread Group and just want to ignore or filter out the error it can be done using:
JMeter Plugins Command Line Tool Plugin
Filter Results Plugin
Response Assertion and "Ignore Status" box
JSR223 Assertion and your custom code to conditionally mark the sampler(s) in scope as passed or failed
The error is thrown at the end of the test when JMeter tries to send a request but the connection is already closed.
These errors are created due to test configuration and should be ignored in the test results/reports.
You could ignore the test results during specific times with pre.setIgnore() in a JSR223 Post Processor.
The sample code is available here.
I wrote a program by using spark streaming to insert data to kerberos enabled hbase. In one batch, I met one failed task. The error is below:
java.io.IOException: Login failure for myuser#example.com from keytab ./user.keytab
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1160)
at com.framework.common.HbaseUtil$.InsertToHbase(HbaseUtil.scala:81)
at com.framework.realtime.RDDUtil$$anonfun$dwsTodwd$2.apply(RDDUtil.scala:203)
at com.framework.realtime.RDDUtil$$anonfun$dwsTodwd$2.apply(RDDUtil.scala:202)
at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$33.apply(RDD.scala:920)
at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$33.apply(RDD.scala:920)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1858)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1858)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: javax.security.auth.login.LoginException: Receive timed out
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:767)
at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:584)
at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:762)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:203)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:690)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:688)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:687)
at javax.security.auth.login.LoginContext.login(LoginContext.java:595)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1149)
... 13 more
Caused by: java.net.SocketTimeoutException: Receive timed out
at java.net.PlainDatagramSocketImpl.receive0(Native Method)
at java.net.AbstractPlainDatagramSocketImpl.receive(AbstractPlainDatagramSocketImpl.java:146)
at java.net.DatagramSocket.receive(DatagramSocket.java:816)
at sun.security.krb5.internal.UDPClient.receive(NetClient.java:207)
at sun.security.krb5.KdcComm$KdcCommunication.run(KdcComm.java:390)
at sun.security.krb5.KdcComm$KdcCommunication.run(KdcComm.java:343)
at java.security.AccessController.doPrivileged(Native Method)
at sun.security.krb5.KdcComm.send(KdcComm.java:327)
at sun.security.krb5.KdcComm.send(KdcComm.java:219)
at sun.security.krb5.KdcComm.send(KdcComm.java:191)
at sun.security.krb5.KrbAsReqBuilder.send(KrbAsReqBuilder.java:319)
at sun.security.krb5.KrbAsReqBuilder.action(KrbAsReqBuilder.java:364)
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:735)
... 25 more
But in the second attempt,the task succeed. In my opinion,the certification process is too long, so it fails, and in another attempt, the process is short. So it scceed. Am I correct? If so or not, how to solve this problem please?
My code is as below:
val ugi = UserGroupInformation.loginUserFromKeytabAndReturnUGI(princ,
keytab)
ugi.doAs(new PrivilegedAction[Unit]() {
def run(): Unit = {
// TODO Auto-generated method stub
var conn: HConnection = null
var htable: HTableInterface = null
conn = HConnectionManager.createConnection(conf)
htable = conn.getTable(tableName)
htable.setAutoFlushTo(false)
for (record <- partitionOfRecords) {
htable.put(record)
}
}
})
From Hadoop and Kerberos - the Madness beyond the Gate chapter "Error Messages to Fear"...
Receive timed out
Usually in a stack trace like
Caused by: java.net.SocketTimeoutException: Receive timed out
at java.net.PlainDatagramSocketImpl.receive0(Native Method)
...
at sun.security.krb5.internal.UDPClient.receive(NetClient.java:207)
... UDP socket ... Switch to TCP —at the very least, it will fail
faster.
And just above that:
Switching kerberos to use TCP rather than UDP
In /etc/krb5.conf:
[libdefaults]
udp_preference_limit = 1
Generally speaking, many erratic Kerberos issues seem to occur only with UDP, so it's unfortunate that it's used by default...
Note that Java also supports kdc_timeout configuration parameter, but it's a dirty mess:
not mentioned in MIT Kerberos documentation
not mentioned in Unix/Linux documentation except for BSD
mentioned only in the darkest corners of Java documentation, here for Java 9, with an interesting side note about the fact that the default value has changed from 30s-expressed-implicitly-in-milliseconds to 30s at some point
a few weeks ago, the Cloudera support team issued a recommendation about that setting -- because the 30s default timeout could create cascading failures in HDFS High Availability or something like that -- but the poor guys did not really know what they were recommending, so they suggested randomly "3" or "3s" or "3000" for the explicit timeout value
Note also that if you have multiple KDCs for high availability, and these KDCs are explicitly listed in krb5.conf (or implicitly listed via a DNS alias set with a round-robin rule, for example) then in case of "KDC timeout" Java should retry with the next KDC in line. Unless you have reached a global time-out.
I am using Hazelcast 3.2.6 as second level cache for Hibernate. The cluster has 4 servers with multiple Read/Update/Delete operations being performed on the DB. It was running fine for quite sometime suddenly I see that all the threads which are trying to perform db operation are stuck, following is an extract from thread dump, there are no exceptions being printed.
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at com.hazelcast.spi.impl.BasicInvocation$InvocationFuture.pollResponse(BasicInvocation.java:767)
- locked <0x0000000665956110> (a com.hazelcast.spi.impl.BasicInvocation$InvocationFuture)
at com.hazelcast.spi.impl.BasicInvocation$InvocationFuture.waitForResponse(BasicInvocation.java:719)
at com.hazelcast.spi.impl.BasicInvocation$InvocationFuture.get(BasicInvocation.java:697)
at com.hazelcast.spi.impl.BasicInvocation$InvocationFuture.get(BasicInvocation.java:676)
at com.hazelcast.spi.impl.BasicInvocation$InvocationFuture.getSafely(BasicInvocation.java:689)
at com.hazelcast.concurrent.lock.LockProxySupport.lock(LockProxySupport.java:80)
at com.hazelcast.concurrent.lock.LockProxySupport.lock(LockProxySupport.java:74)
at com.hazelcast.concurrent.lock.LockProxy.lock(LockProxy.java:70)
at com.xxx.database.ccsecure.persistance.impl.DataStore.get(DataStore.java:120)
Apparently the invocation doesn't get a result. This means that the invocation-future is not going to complete. The big question is: why does the operation not get a response to its request.
Do you know which operation it is?
My code of jena is broken under the multiple-thread execution environment. I use the jena sdb to save the rdf triples. However, when i start the six threads to complete the saving data action, the exception be threw out. (every thread save a graph rdf into db,every graph is different)
I need the helps are:
1. Whether jena SDB support the transaction, which is thread safe ?
2. How to implement the thread safe operation for jena sdb ?
3. How to solve my code problem? (The exception and key code is under below)
So appreciate for your help and any suggestions. Wish any replay from you. Good Luck ～～
My key code below:(Database is DB2, database pool: Websphere datasource pool)
//DBConnector is an object that get the jdbc connection from the data source
//initialize and return a sdb connection object [new SDBConnection(jdbcConnection)]
SDBConnection con = DBConnector.getSDBConnection();
store = SDBFactory.connectStore(con,storeDesc);
model.notifyEvent(GraphEvent.startRead);
model.read(in,'',"N-Triple");
model.notifyEvent(GraphEvent.finishRead);
model.close();
The exception is below:
com.hp.hpl.jena.sdb.layout2.LoaderTuplesNodes.Thread-6():
Error in thread: Problem making new tupleloader
com.hp.hpl.jena.sdb.SDBException: Problem making new tupleloader
at com.hp.hpl.jena.sdb.layout2.LoaderTuplesNodes.updateOneTuple(LoaderTuplesNodes.java:269)
at com.hp.hpl.jena.sdb.layout2.LoaderTuplesNodes.access$200(LoaderTuplesNodes.java:31)
at com.hp.hpl.jena.sdb.layout2.LoaderTuplesNodes$Commiter.run(LoaderTuplesNodes.java:334)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at com.hp.hpl.jena.sdb.layout2.LoaderTuplesNodes.updateOneTuple(LoaderTuplesNodes.java:265)
... 3 more
Caused by: com.hp.hpl.jena.sdb.SDBException: Problem initialising loader for [Quads]
at com.hp.hpl.jena.sdb.layout2.TupleLoaderBase.<init>(TupleLoaderBase.java:47)
at com.hp.hpl.jena.sdb.layout2.hash.TupleLoaderHashBase.<init>(TupleLoaderHashBase.java:17)
at com.hp.hpl.jena.sdb.layout2.hash.TupleLoaderHashDB2.<init>(TupleLoaderHashDB2.java:22)
... 8 more
My suspicion is that the store isn't being freed up after use. Could you try:
SDBConnection con = DBConnector.getSDBConnection();
store = SDBFactory.connectStore(con,storeDesc);
model.read(in,'',"N-Triple");
store.close();
(The notify isn't needed)
It should be thread safe etc.
I've just refactored a project in Eclipse to use packages instead of the default package to better organise my code. I have a test program which creates a number of Runnable objects, each of which are run sequentially to each other, but in parallel with the main program.
Before refactoring this worked fine, with each thread dutifully performing its task. However, since moving things into packages I'm getting a ClassNotFound exception as soon as one of the Runnable classes attempts to use a class from another package. Stacktrace follows:
java.lang.ClassNotFoundException: Tweet
at java.net.URLClassLoader$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Unknown Source)
at java.io.ObjectInputStream.resolveClass(Unknown Source)
at java.io.ObjectInputStream.readNonProxyDesc(Unknown Source)
at java.io.ObjectInputStream.readClassDesc(Unknown Source)
at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
at java.io.ObjectInputStream.readObject0(Unknown Source)
at java.io.ObjectInputStream.readObject(Unknown Source)
at indexing.TweetCleanup.run(TweetCleanup.java:84)
at java.lang.Thread.run(Unknown Source)
In this trace 'TweetCleanup' is the Runnable class, and 'Tweet' is the class not found. I've included it in TweetCleanup as 'common.Tweet' (where common is the package). I've used a separate test program to see if it can see the class in the main thread, which runs successfully.
All I can think of is that the Thread needs to be supplied some classpath to 'see' the Tweet class, however this wasn't the case before refactoring as packages. I believe the default behaviour of a child Thread is to use the classpath of its parent, which includes 'common.Tweet'.
I'm using Eclipse Helios as my IDE.
Any tips would be appreciated!
Cheers,
P
Not a Threading problem at all, but you're using Java Serialization somehow and you try to read an Object from a stream which has "older" objects stored within it.
I suspect you're using some kind of persistence by Java Serialization, and your TweetCleanup-Thread is reading in the old "database". Delete this file and your program will work again.
Note that renaming packages or moving classes is a binary incompatible change - you cannot deserialize such objects anymore.