When I run the main class I get these errors:
**Exception in thread "main" java.lang.NoClassDefFoundError: org/slf4j/LoggerFactory
at org.apache.cassandra.cql.jdbc.CassandraDriver.<clinit>(CassandraDriver.java:52)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:188)
at oracledbtest.CqlJdbcTestBasic.main(CqlJdbcTestBasic.java:19)
Caused by: java.lang.ClassNotFoundException: org.slf4j.LoggerFactory
How can I create a new driver connection to Cassandra?
What exactly are you trying to do? The error simply means that you do not have slf4j's jar in your classpath..
Related
I was using ignite 2.8.0 and integration with cassandra was working fine. With upgrade to ignite 2.8.1, ignite startup is failing with message : Failed to validate cache configuration. Cache store factory is not serializable. Underlying exception is Null Pointer Exception.
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to start processor: GridProcessorAdapter []
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to validate cache configuration. Cache store factory is not serializable. Cache name
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to deserialize object with given class loader: sun.misc.Launcher$AppClassLoader#764c12b6
Caused by: java.lang.NullPointerException
at java.util.Collections$UnmodifiableCollection.<init>
at org.apache.ignite.cache.store.cassandra.persistence.PersistenceSettings.readObject(PersistenceSettings.java:533)
I can see that there was the Cassandra-related ticket that brought a new code and most likely caused the issue.
The new ticket is https://issues.apache.org/jira/browse/IGNITE-13431
I am not able to connect to Big Query table from Spark on GCP.
https://cloud.google.com/hadoop/bigquery-connector
I already tried steps present in above by providing project Id dataset name and table name link still no success .When I am trying to print the data using below code I am getting below error:
Exception in thread "main" java.lang.NoClassDefFoundError: com/google/cloud/hadoop/io/bigquery/BigQueryConfiguration
at Main.main(Main.scala:27)
at Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.orgapachesparkdeploySparkSubmitrunMain(SparkSubmit.scala:849)
at org.apache.spark.deploy.SparkSubmit.doRunMain1(SparkSubmit.scala:167)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmitanon2.doSubmit(SparkSubmit.scala:924)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala:933)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: com.google.cloud.hadoop.io.bigquery.BigQueryConfiguration
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 14 more
You probably miss the spark-bigquery-connector in your classpath, you can add it by adding the following parameter:
gcloud dataproc jobs submit spark --cluster "$MY_CLUSTER" --jars gs://spark-lib/bigquery/spark-bigquery-latest.jar ...
I have been trying Spark 2.4 deployment on k8s and want to establish a secured RPC communication channel between driver and executors. Was using the following configuration parameters as part of spark-submit
spark.authenticate true
spark.authenticate.secret good
spark.network.crypto.enabled true
spark.network.crypto.keyFactoryAlgorithm PBKDF2WithHmacSHA1
spark.network.crypto.saslFallback false
The driver and executors were not able to communicate on a secured channel and were throwing the following errors.
Exception in thread "main" java.lang.reflect.UndeclaredThrowableException
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1713)
at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:64)
at org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:188)
at org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:281)
at org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
Caused by: org.apache.spark.SparkException: Exception thrown in awaitResult:
at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:226)
at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75)
at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:101)
at org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:201)
at org.apache.spark.deploy.SparkHadoopUtil$$anon$2.run(SparkHadoopUtil.scala:65)
at org.apache.spark.deploy.SparkHadoopUtil$$anon$2.run(SparkHadoopUtil.scala:64)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
... 4 more
Caused by: java.lang.RuntimeException: java.lang.IllegalArgumentException: Unknown challenge message.
at org.apache.spark.network.crypto.AuthRpcHandler.receive(AuthRpcHandler.java:109)
at org.apache.spark.network.server.TransportRequestHandler.processRpcRequest(TransportRequestHandler.java:181)
at org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:103)
at org.apache.spark.network.server.TransportChannelHandler.channelRead(TransportChannelHandler.java:118)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
Can someone guide me on this?
Disclaimer: I do not have a very deep understanding of spark implementation, so, be careful when using the workaround described below.
AFAIK, spark does not have support for auth/encryption for k8s in 2.4.0 version.
There is a ticket, which is already fixed and likely will be released in a next spark version: https://issues.apache.org/jira/browse/SPARK-26239
The problem is that spark executors try to open connection to a driver, and a configuration will be sent only using this connection. Although, an executor creates the connection with default config AND system properties started with "spark.".
For reference, here is the place where executor opens the connection: https://github.com/apache/spark/blob/5fa4384/core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala#L201
Theoretically, if you would set spark.executor.extraJavaOptions=-Dspark.authenticate=true -Dspark.network.crypto.enabled=true ..., it should help, although driver checks that there are no spark parameters set in extraJavaOptions.
Although, there is a workaround (a little bit hacky): you can set spark.executorEnv.JAVA_TOOL_OPTIONS=-Dspark.authenticate=true -Dspark.network.crypto.enabled=true .... Spark does not check this parameter, but JVM uses this env variable to add this parameter to properties.
Also, instead of using JAVA_TOOL_OPTIONS to pass secret, I would recommend to use spark.executorEnv._SPARK_AUTH_SECRET=<secret>.
I am new to Cassandra. I am now working on Ubuntu and the latest Cassandra. I am using Java 1.7 on a PC. I created the installation on a PC without any cloud or server network (replication was one). I can use CQL ok but when I am trying in Java the code fails.
Here is the code:
Cluster cluster;
Session session;
cluster = Cluster.builder().addContactPoint("127.0.0.1").build();
session = cluster.connect("casslinks");
if (!tableExists(linktable, session))
{
createCassTable(linktable, session);
}
for (String url: urls) {
insertCassUrl(url, crawledUrl, session, linktable);
}
Here is the error:
Exception in thread "main"
java.lang.NoClassDefFoundError: org/slf4j/LoggerFactory
at com.datastax.driver.core.Cluster.<clinit>(Cluster.java:65)
at com.example.GetSoogrData.insertUrls(GetSoogrData.java:596)
at com.example.GetSoogrData.main(GetSoogrData.java:1112)
Caused by: java.lang.ClassNotFoundException: org.slf4j.LoggerFactory
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
I used the debugger and saw the session line failed. I assume the IP address is wrong. I am unsure how to test for the correct IP address or how to define the session variable to it will correctly connect.
I have been using mongo db and 127.0.0.1 works for that.
Has anyone any ideas?
You are probably running your code with missing dependencies.
Datastax java driver requires few addidional jars, SLF4J libraries being one of them for which exception is thrown.
Look here to see what jars you need:
http://www.datastax.com/documentation/developer/java-driver/2.0/java-driver/reference/settingUpJavaProgEnv_r.html
http://www.datastax.com/documentation/developer/java-driver/2.0/common/drivers/introduction/driverDependencies_r.html
I have a Grails 2.0.1 app and I spawn worker threads. My worker threads use GORM to read/write domain objects from MySql. Given that the code that is accessing the DB is not inside of an HTTP request, I create a Hibernate session for the save() like this:
MyClass.withTransaction { status ->
myClass.save()
}
I do that to resolve the "No Hibernate Session bound to thread" problem.
This worked fine on Grails 1.3.7. I am currently attempting to upgrade to Grails 2.0.1, and am seeing an exception I do not understand.
The following stack trace shows the exception.
Although I recognize the exception, and normally resolve it via the "withTransaction" technique shown above, the thing that baffles me about this one is that it is happening on a worker thread that I did not create.
Exception in thread "pool-12-thread-2" org.hibernate.HibernateException: No Hibernate Session bound to thread, and configuration does not allow creation of non-transactional one here
at org.springframework.orm.hibernate3.SpringSessionContext.currentSession(SpringSessionContext.java:63)
at org.hibernate.impl.SessionFactoryImpl.getCurrentSession(SessionFactoryImpl.java:687)
at org.codehaus.groovy.grails.orm.hibernate.SessionFactoryProxy.getCurrentSession(SessionFactoryProxy.java:145)
at org.codehaus.groovy.grails.orm.hibernate.validation.HibernateDomainClassValidator.validate(HibernateDomainClassValidator.java:51)
at org.codehaus.groovy.grails.validation.GrailsDomainClassValidator.validate(GrailsDomainClassValidator.java:121)
at org.codehaus.groovy.grails.orm.hibernate.metaclass.ValidatePersistentMethod.doInvokeInternal(ValidatePersistentMethod.java:119)
at org.codehaus.groovy.grails.orm.hibernate.metaclass.AbstractDynamicPersistentMethod.invoke(AbstractDynamicPersistentMethod.java:63)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.springsource.loaded.ri.ReflectiveInterceptor.jlrMethodInvoke(ReflectiveInterceptor.java:1231)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoCachedMethodSite.invoke(PojoMetaMethodSite.java:189)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:53)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:124)
at org.codehaus.groovy.grails.orm.hibernate.HibernateGormValidationApi.validate(HibernateGormEnhancer.groovy:702)
at com......MyClass.validate(MyClass.groovy)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.springsource.loaded.ri.ReflectiveInterceptor.jlrMethodInvoke(ReflectiveInterceptor.java:1231)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
at org.codehaus.groovy.grails.orm.hibernate.support.ClosureEventListener$7.call(ClosureEventListener.java:282)
at org.codehaus.groovy.grails.orm.hibernate.support.ClosureEventListener$7.call(ClosureEventListener.java:267)
at org.codehaus.groovy.grails.orm.hibernate.support.ClosureEventListener.doWithManualSession(ClosureEventListener.java:302)
at org.codehaus.groovy.grails.orm.hibernate.support.ClosureEventListener.onPreUpdate(ClosureEventListener.java:267)
at org.codehaus.groovy.grails.orm.hibernate.EventTriggeringInterceptor.onPreUpdate(EventTriggeringInterceptor.java:164)
at org.codehaus.groovy.grails.orm.hibernate.EventTriggeringInterceptor.onPersistenceEvent(EventTriggeringInterceptor.java:89)
at org.grails.datastore.mapping.engine.event.AbstractPersistenceEventListener.onApplicationEvent(AbstractPersistenceEventListener.java:46)
at org.springframework.context.event.SimpleApplicationEventMulticaster$1.run(SimpleApplicationEventMulticaster.java:92)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)
I am hoping someone can enlighten me regarding:
where this worker thread pool originates from?
why is is doing a preUpdate event (when I believe the save in my thread already did a successful validate and save
Can I somehow config my app such that these foreign worker threads are not there
If #3 can't be pulled off, how can I do the equivalent of a "withTransaction" over in the worker thread code that I do not "own"