Spring data redis, multi-threading issue with Jedis - multithreading

I am using redis in a heavily multi-threaded java application and getting intermittent ClassCastExceptions. Reading through various discussions seemed to point out this might be because the Jedis connection instance getting shared between multiple threads (https://github.com/xetorthio/jedis/issues/359). The solution suggested is to use JedisPool which is thread-safe.
I have configured redis through Spring redis support by using RedisTemplate. A thing to note is I am using multiple templates (to support different models for serialization and de-serialization). Here is snippet of my configuration -
<bean id="jedisConnFactory"
class="org.springframework.data.redis.connection.jedis.JedisConnectionFactory"
redis:usePool="true" redis:poolConfig-ref="jedisPoolConfig" redis:hostName="${redis.datasource.hostName}"
redis:database="${redis.database.index}" redis:port="${redis.datastore.port}"/>
<bean id="jedisPoolConfig" class="redis.clients.jedis.JedisPoolConfig">
<property name="maxIdle" value="${redis.conn.maxIdle}"/>
<property name="maxTotal" value="${redis.conn.maxTotal}"/>
<property name="minIdle" value="${redis.conn.minIdle}"/>
<property name="testOnBorrow" value="true"/>
</bean>
<bean id="redisTemplate"
class="org.springframework.data.redis.core.RedisTemplate"
redis:connectionFactory-ref="jedisConnFactory"
redis:keySerializer-ref="redisStringSerializer"
redis:valueSerializer-ref="redisStringSerializer"
redis:defaultSerializer-ref="redisStringSerializer"/>
Note the use of usePool=true which prompts spring to use JedisPool. Looking through spring code also suggests that spring is handling resource allocation and release properly.
Any help identifying the issue will be appreciated.
Edit: Stack trace -
Thread 1:
[ERROR] [03/01/2015 07:05:32.044] [events-system-akka.actor.default-dispatcher-2281] [akka://events-system/user/$YN/$b/$b/$b] java.lang.Long cannot be cast to java.util.List
java.lang.ClassCastException: java.lang.Long cannot be cast to java.util.List
at redis.clients.jedis.Connection.getRawObjectMultiBulkReply(Connection.java:230)
at redis.clients.jedis.Connection.getObjectMultiBulkReply(Connection.java:236)
at redis.clients.jedis.BinaryJedis.zscan(BinaryJedis.java:3608)
at org.springframework.data.redis.connection.jedis.JedisConnection$3.doScan(JedisConnection.java:2998)
at org.springframework.data.redis.core.KeyBoundCursor.doScan(KeyBoundCursor.java:39)
at org.springframework.data.redis.core.ScanCursor.scan(ScanCursor.java:85)
at org.springframework.data.redis.core.ScanCursor.hasNext(ScanCursor.java:168)
at org.springframework.data.redis.core.ConvertingCursor.hasNext(ConvertingCursor.java:56)
...
application specific stack trace
...
Thread 2:
[ERROR] [03/01/2015 07:03:07.295] [events-system-akka.actor.default-dispatcher-2273] [akka://events-system/user/$VN/$b/$b/$b] Unknown redis exception; nested exception is java.lang.ClassCastException: [B cannot be cast to java.lang.Long
org.springframework.data.redis.RedisSystemException: Unknown redis exception; nested exception is java.lang.ClassCastException: [B cannot be cast to java.lang.Long
at org.springframework.data.redis.FallbackExceptionTranslationStrategy.getFallback(FallbackExceptionTranslationStrategy.java:48)
at org.springframework.data.redis.FallbackExceptionTranslationStrategy.translate(FallbackExceptionTranslationStrategy.java:38)
at org.springframework.data.redis.connection.jedis.JedisConnection.convertJedisAccessException(JedisConnection.java:195)
at org.springframework.data.redis.connection.jedis.JedisConnection.zRem(JedisConnection.java:2321)
at org.springframework.data.redis.core.DefaultZSetOperations$19.doInRedis(DefaultZSetOperations.java:283)
at org.springframework.data.redis.core.DefaultZSetOperations$19.doInRedis(DefaultZSetOperations.java:280)
at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:190)
at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:152)
at org.springframework.data.redis.core.AbstractOperations.execute(AbstractOperations.java:85)
at org.springframework.data.redis.core.DefaultZSetOperations.remove(DefaultZSetOperations.java:280)
...
application specific stack trace
...
Caused by: java.lang.ClassCastException: [B cannot be cast to java.lang.Long
at redis.clients.jedis.Connection.getIntegerReply(Connection.java:210)
at redis.clients.jedis.BinaryJedis.zrem(BinaryJedis.java:1624)
at org.springframework.data.redis.connection.jedis.JedisConnection.zRem(JedisConnection.java:2319)
... 21 more

Forgot to post an update on this here.
In my case, my analysis was that the connection pool by default used ForkJoinPool which works in a work-stealing mode. This made another thread take over the results part of the operation over and gave ClassCastException when there was a mismatch in Type of the result.
This did resolve the issue I had.

Related

Provider com.sap.cloud.sdk.cloudplatform.connectivity.CertificateBasedHttpClientFactory not a subtype

In SAP Cloud SDK FAQ page, there is a QA about I'm Observing a DefaultHttpClientFactory not a subtype Exception, now I encountered a similar error Provider com.sap.cloud.sdk.cloudplatform.connectivity.CertificateBasedHttpClientFactory not a subtype, however I could not exclude connectivity-scp-cf since this is mandatory?
Any hints how to solve this error?
Caused by: com.sap.core.connectivity.jco.cf.auth.TokenFactory$GetTokenException: Could not get ClientCredentialsGrantAccessToken
at com.sap.core.connectivity.jco.cf.auth.TokenFactory.getClientCredentialsGrantAccessToken(TokenFactory.java:61)
at com.sap.core.connectivity.jco.cf.destination.ConnectivityConfigurationCF.getConfiguration(ConnectivityConfigurationCF.java:72)
... 92 more
Caused by: java.lang.NoClassDefFoundError: com/sap/cloud/security/client/HttpClientFactory : cannot initialize class because prior initialization attempt failed
at com.sap.core.connectivity.jco.cf.auth.TokenFactory.executeTokenExchange(TokenFactory.java:94)
at com.sap.core.connectivity.jco.cf.auth.TokenFactory.getClientCredentialsGrantAccessToken(TokenFactory.java:57)
... 93 more
Caused by: java.lang.ExceptionInInitializerError: java.util.ServiceConfigurationError: com.sap.cloud.security.client.HttpClientFactory: Provider com.sap.cloud.sdk.cloudplatform.connectivity.CertificateBasedHttpClientFactory not a subtype
Interface type com.sap.cloud.security.client.HttpClientFactory is part of dependency com.sap.cloud.security.xsuaa:token-client.
When using SAP Java Buildpack (e.g. for JCo), please make sure to give this dependency a provided scope. Otherwise you'll be experiencing class loading issues like in your current case. For further errors, please attach your mvn dependency:tree.

Hazelcast Supplier and Aggregation gives Concurrent Execution Exception

I am trying to get a set of the distinct values of an object's field stored in a Hazelcast map.
This line of java code:
instructions.aggregate(Supplier.all(value -> value.getWorkArea()), Aggregations.distinctValues());
has the following stacktrace :
java.util.concurrent.ExecutionException: com.hazelcast.nio.serialization.HazelcastSerializationException: java.lang.ClassNotFoundException: com.example.instruction.repository.HazelcastInstructionRepository$GeneratedEvaluationClass
com.hazelcast.nio.serialization.HazelcastSerializationException: java.lang.ClassNotFoundException: com.example.instruction.repository.HazelcastInstructionRepository$GeneratedEvaluationClass
java.lang.ClassNotFoundException: com.example.instruction.repository.HazelcastInstructionRepository$GeneratedEvaluationClass
If I were to try this line :
instructions.aggregate(Supplier.all()), Aggregations.distinctValues());
or:
instructions.aggregate(Supplier.fromPredicate(Predicates.and(Predicates.equal("type", "someType"), equal("groupId", null),
Predicates.equal("workArea", "someWorkArea"))), Aggregations.distinctValues());
It just works ... It seems to be something wrong when I am making a reference to the object's field. (I also tried it with other fields of the object and the same error gets returned)
This is running on my local environment and I am sure that the objects are being placed correctly in the Hazelcast map since the other aggregations/predicates are working.
Do you have any ideas about what am I doing wrong?
Many Thanks!
EDITED: So the problem is the closure. It's not available on all nodes. Only on the calling node.
Also. This feature is deprecated. Plz use the fast-aggregations instead.
http://docs.hazelcast.org/docs/latest/manual/html-single/#fast-aggregations

Exception in Map Reduce job

i have created a map reduce job to fetch the number of employees of a location.
I am using hazelcast 3.6.3. Each employee has name and address.
I have added my code to following git repository.
https://github.com/adasari/hazelcast-demo
Exception :
java.util.concurrent.ExecutionException: java.lang.ClassCastException: com.hazelcast.mapreduce.aggregation.impl.DistinctValuesAggregation$SimpleEntry cannot be cast to com.hazelcast.query.impl.Extractable
at com.hazelcast.mapreduce.impl.task.TrackableJobFuture.setResult(TrackableJobFuture.java:68)
at com.hazelcast.mapreduce.impl.task.JobSupervisor.notifyRemoteException(JobSupervisor.java:156)
at com.hazelcast.mapreduce.impl.operation.NotifyRemoteExceptionOperation.run(NotifyRemoteExceptionOperation.java:54)
at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:172)
at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:393)
at com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.processPacket(OperationThread.java:184)
at com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.process(OperationThread.java:137)
at com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.doRun(OperationThread.java:124)
at com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.run(OperationThread.java:99)
Caused by: java.lang.ClassCastException: com.hazelcast.mapreduce.aggregation.impl.DistinctValuesAggregation$SimpleEntry cannot be cast to com.hazelcast.query.impl.Extractable
at com.hazelcast.query.impl.predicates.AbstractPredicate.readAttributeValue(AbstractPredicate.java:129)
at com.hazelcast.query.impl.predicates.AbstractPredicate.apply(AbstractPredicate.java:55)
can you point me the issue ?
Thanks.
Not exactly sure what you're looking for or what you're trying to do (looking at the code) but your problem is here:
Caused by: java.lang.ClassCastException: com.hazelcast.mapreduce.aggregation.impl.DistinctValuesAggregation$SimpleEntry cannot be cast to com.hazelcast.query.impl.Extractable
So you have to implement the Extractable interface with your SimpleEntry class.
I haven't worked on MapReduce but below are my observation while looking/executing the code.
The place it is failing is a different SimpleEntry class, an inner class of DistinctValuesAggregation which doesn't implement Extractable.
There is already a defect in Hazelcast(#7398) but it says closed in 3.6.1 so might as well follow up with them in Git hub.
I found that the code works well when you run the Cluster with just single node. So I suspect the above defect would be impacting the aggregation over multiple nodes.
HazelcastInstance hazelcastInstance = buildCluster(1);
Following actions resolved the issue -
1. DistinctMapper implements DataSerializable
2. SimpleEntry extends QueryableEntry

Hazelcast hangs when server fails to handle query

I want to execute instance of supertype AbstractEntryProcessor (lets say SomeEntryProcessor) with IMap.executeOnKey method.
Server side ClassLoader doesn't have this class (SomeEntryProcessor).
So it expectedly fails with:
java.lang.ClassNotFoundException: com.package.SomeEntryProcessor
com.hazelcast.nio.serialization.HazelcastSerializationException: java.lang.ClassNotFoundException: com.package.SomeEntryProcessor
at com.hazelcast.nio.serialization.DefaultSerializers$ObjectSerializer.read(DefaultSerializers.java:190) ~[hazelcast-3.2.3.jar:3.2.3]
at com.hazelcast.nio.serialization.StreamSerializerAdapter.read(StreamSerializerAdapter.java:40) ~[hazelcast-3.2.3.jar:3.2.3]
at com.hazelcast.nio.serialization.SerializationServiceImpl.readObject(SerializationServiceImpl.java:279) ~[hazelcast-3.2.3.jar:3.2.3]
at com.hazelcast.nio.serialization.ByteArrayObjectDataInput.readObject(ByteArrayObjectDataInput.java:433) ~[hazelcast-3.2.3.jar:3.2.3]
at com.hazelcast.map.client.MapExecuteOnKeyRequest.read(MapExecuteOnKeyRequest.java:88) ~[hazelcast-3.2.3.jar:3.2.3]
But after this executeOnKey hangs forever.
I believe this happens due to infinite wait in method
ClientCallFuture.get(long timeout, TimeUnit unit) {
...
this.wait(waitMillis); // line 103 in hazelcast 3.2.3
}
Has anyone seen the same?
Created an issue https://github.com/hazelcast/hazelcast/issues/3842, but no response received
Sorry somehow missed the issue. Can you try a 3.3.x release? I guess the issue is fixed on those versions.

Groovy Griffon with Hibernate4 plugin class not found exception

I was looking for a fast and easy way to write a very cross platform desktop application. This leads me to thinking that the JVM is the place to be. Since Groovy (Grails) is used in my workplace I thought I would try Griffon since they claim it is essentially Grails for the desktop.
I wanted a persistence management layer and it doesn't not appear that GORM is showtime ready in this environment so I moved towards hibernate using the Hibernate4 plugin for Griffon.
Not that I've really used Hibernate in general however I believe, based on the guides, that I am doing things correctly. My gatherings indicate that this doesn't support annotations to wire up classes so I am using hbm.xml files.
The provided sample for the plugin isn't complex but I don't understand where I am deviating.
Here is a sample class file as it stands:
package gwash
import groovy.beans.Bindable
class DeliveryMethodModel {
// #Bindable String propName
}
Here is some of the stack trace:
org.hibernate.InvalidMappingException: Could not parse mapping document from res
ource gwash\DeliveryMethod.hbm.xml
at org.hibernate.cfg.Configuration$MetadataSourceQueue.processHbmXml(Con
figuration.java:3415)
at org.hibernate.cfg.Configuration$MetadataSourceQueue.processHbmXmlQueu
e(Configuration.java:3404)
at org.hibernate.cfg.Configuration$MetadataSourceQueue.processMetadata(C
onfiguration.java:3392)
at org.hibernate.cfg.Configuration.secondPassCompile(Configuration.java:
1341)
at org.hibernate.cfg.Configuration.buildSessionFactory(Configuration.jav
a:1737)
at org.hibernate.cfg.Configuration.buildSessionFactory(Configuration.jav
a:1788)
at org.hibernate.cfg.Configuration$buildSessionFactory.call(Unknown Sour
ce)
at griffon.plugins.hibernate4.Hibernate4Connector.connect(Hibernate4Conn
ector.groovy:72)
at griffon.plugins.hibernate4.Hibernate4Connector.connect(Hibernate4Conn
ector.groovy)
at griffon.plugins.hibernate4.Hibernate4Connector$connect.call(Unknown S
ource)
at Hibernate4GriffonAddon.addonInit(Hibernate4GriffonAddon.groovy:27)
at griffon.core.GriffonAddon$addonInit.call(Unknown Source)
at griffon.core.GriffonAddon$addonInit.call(Unknown Source)
at org.codehaus.griffon.runtime.util.AddonHelper.handleAddon(AddonHelper
.groovy:155)
at org.codehaus.griffon.runtime.util.AddonHelper.handleAddonsAtStartup(A
ddonHelper.groovy:105)
at org.codehaus.griffon.runtime.core.DefaultAddonManager.doInitialize(De
faultAddonManager.java:33)
at org.codehaus.griffon.runtime.core.AbstractAddonManager.initialize(Abs
tractAddonManager.java:101)
at org.codehaus.griffon.runtime.util.GriffonApplicationHelper.initialize
AddonManager(GriffonApplicationHelper.java:320)
at org.codehaus.griffon.runtime.util.GriffonApplicationHelper.prepare(Gr
iffonApplicationHelper.java:123)
at org.codehaus.griffon.runtime.core.AbstractGriffonApplication.initiali
ze(AbstractGriffonApplication.java:221)
at griffon.swing.AbstractSwingGriffonApplication.bootstrap(AbstractSwing
GriffonApplication.java:74)
at griffon.swing.AbstractSwingGriffonApplication.run(AbstractSwingGriffo
nApplication.java:131)
at griffon.swing.SwingApplication.main(SwingApplication.java:36)
Caused by: org.hibernate.PropertyNotFoundException: field [id] not found on gwas
h.DeliveryMethodModel
at org.hibernate.property.DirectPropertyAccessor.getField(DirectProperty
Accessor.java:182)
at org.hibernate.property.DirectPropertyAccessor.getField(DirectProperty
Accessor.java:189)
at org.hibernate.property.DirectPropertyAccessor.getField(DirectProperty
Accessor.java:189)
at org.hibernate.property.DirectPropertyAccessor.getField(DirectProperty
Accessor.java:189)
at org.hibernate.property.DirectPropertyAccessor.getField(DirectProperty
Accessor.java:189)
at org.hibernate.property.DirectPropertyAccessor.getField(DirectProperty
Accessor.java:174)
at org.hibernate.property.DirectPropertyAccessor.getGetter(DirectPropert
yAccessor.java:197)
at org.hibernate.internal.util.ReflectHelper.getter(ReflectHelper.java:2
53)
at org.hibernate.internal.util.ReflectHelper.reflectedPropertyClass(Refl
ectHelper.java:229)
at org.hibernate.mapping.SimpleValue.setTypeUsingReflection(SimpleValue.
java:326)
at org.hibernate.cfg.HbmBinder.bindSimpleId(HbmBinder.java:449)
at org.hibernate.cfg.HbmBinder.bindRootPersistentClassCommonValues(HbmBi
nder.java:382)
at org.hibernate.cfg.HbmBinder.bindRootClass(HbmBinder.java:322)
at org.hibernate.cfg.HbmBinder.bindRoot(HbmBinder.java:173)
at org.hibernate.cfg.Configuration$MetadataSourceQueue.processHbmXml(Con
My xml mapping file:
<!DOCTYPE hibernate-mapping PUBLIC
"-//Hibernate/Hibernate Mapping DTD 3.0//EN"
"http://www.hibernate.org/dtd/hibernate-mapping-3.0.dtd">
<hibernate-mapping package="gwash">
<class name="DeliveryMethodModel" table="[DELIVERY METHODS]">
<id name="id" column="[DELIVERY METHOD ID]">
<generator class="increment"/>
</id>
<property name="method" column="[DELIVERY METHOD]"/>
</class>
</hibernate-mapping>
EDIT: I've removed the brackets and spaces as indicated. Changed the DataSource.groovy to 'create' on the DB side. Still experiencing the same issues. The examples for hibernate integration with griffon/hsqldb/groovy are scant on details. Do I need to create all given properties for the model files for this to parse correctly? I've never used hibernate. Nor groovy. Nor griffon. I would definitely provide feedback for the community if I can get this resolved, if not I'll be rolling me own ORM since this is a rather small project. Rather not roll me own.
do you actually have the strings wrapped with [and ]?
I would suspect that for a class defined as
package gwash
import groovy.beans.Bindable
class DeliveryMethodModel {
Long id
#Bindable String method
}
the mapping file would be
<!DOCTYPE hibernate-mapping PUBLIC
"-//Hibernate/Hibernate Mapping DTD 3.0//EN"
"http://www.hibernate.org/dtd/hibernate-mapping-3.0.dtd">
<hibernate-mapping package="gwash">
<class name="DeliveryMethodModel" table="DELIVERY_METHODS">
<id name="id" column="DELIVERY_METHOD_ID">
<generator class="increment"/>
</id>
<property name="method" column="DELIVERY_METHOD"/>
</class>
</hibernate-mapping>

Resources