I am learning spark. And trying to run a simple spark app that output 5 dataframe rows without installing spark. I know that one can run spark app without installing spark. The app is throwing below error and I could not resolve the issue. I have tried solution posted on snowflake to similar problem but in vain. Below is the error log for your reference.
Using below softwares:
Eclipse IDE for Enterprise Java and Web Developers (includes Incubating components)
Version: 2022-09 (4.25.0)
Build id: 20220908-1902
java -version
openjdk version "11.0.12" 2021-07-20
OpenJDK Runtime Environment Microsoft-25199 (build 11.0.12+7)
OpenJDK 64-Bit Server VM Microsoft-25199 (build 11.0.12+7, mixed mode)
Exception in thread "main" java.lang.ExceptionInInitializerError
at org.apache.spark.unsafe.array.ByteArrayMethods.<clinit>(ByteArrayMethods.java:54)
at org.apache.spark.internal.config.package$.<init>(package.scala:1095)
at org.apache.spark.internal.config.package$.<clinit>(package.scala)
at org.apache.spark.SparkConf$.<init>(SparkConf.scala:654)
at org.apache.spark.SparkConf$.<clinit>(SparkConf.scala)
at org.apache.spark.SparkConf.set(SparkConf.scala:94)
at org.apache.spark.SparkConf.set(SparkConf.scala:83)
at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$1(SparkSession.scala:916)
at scala.collection.mutable.HashMap.$anonfun$foreach$1(HashMap.scala:149)
at scala.collection.mutable.HashTable.foreachEntry(HashTable.scala:237)
at scala.collection.mutable.HashTable.foreachEntry$(HashTable.scala:230)
at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:44)
at scala.collection.mutable.HashMap.foreach(HashMap.scala:149)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:916)
at net.jgp.books.spark.ch01.lab100_csv_to_dataframe.CsvToDataframeApp.start(CsvToDataframeApp.java:32)
at net.jgp.books.spark.ch01.lab100_csv_to_dataframe.CsvToDataframeApp.main(CsvToDataframeApp.java:21)
Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make private java.nio.DirectByteBuffer(long,int) accessible: module java.base does not "opens java.nio" to unnamed module #35dab4eb
at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:354)
at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297)
at java.base/java.lang.reflect.Constructor.checkCanSetAccessible(Constructor.java:188)
at java.base/java.lang.reflect.Constructor.setAccessible(Constructor.java:181)
at org.apache.spark.unsafe.Platform.<clinit>(Platform.java:56)
... 16 more
I tried the solutions here Why am I seeing `java.lang.reflect.InaccessibleObjectException: Unable to make private java.nio.DirectByteBuffer(long,int) accessible` on a mac
I tried uninstalling and reinstalling java again. I am a newbie.
Try adding the below to your Spark session builder
SparkSession.builder()
// your setting
.config(
"spark.sql.extensions",
"org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions"
)
Related
My cassandra was working and suddenly I get the error
with 6.8 and I am using java 8
ERROR [main] 2022-03-19 21:27:23,812 DseModule.java:114 - Class not found: org/apache/lucene/uninverting/FieldCache. Exiting...
My java version
java -version java version "1.8.0_181" Java(TM) SE Runtime Environment (build 1.8.0_181-b13) Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)
I am running the command,
bin/dse cassandra
to start cassandra
I have tried removing cassandra and reinstalling. I even tried deleting the /var/lib/cassandra folder
The error on its own doesn't give us an indication of the problem. We need quite a bit more information including:
log files
cassandra.yaml and dse.yaml
OS platform and version
My suggestion is that you should log a ticket with DataStax Support so one of our engineers can assist you. Cheers!
I configured jmeter to run the test distributed using Azure Windows Virtual Machines. All machines are using windows.
The test ran successfully using the GUI mode and I can get the result saved on master
However when I try to run the test from the console jmeter -n -t plan.jmx -r the console says:
Error in rconfigure() method java.rmi.MarshalException: error marshalling arguments; nested exception is:
java.io.NotSerializableException: org.apache.jmeter.JMeter$ListenToTest
I' ve configured all the virtual machines (server and clients) like:
Java version "1.8.0_65"
Java(TM) SE Runtime Environment (build 1.8.0_65-b17)
Java HotSpot(TM) 64-Bit Server VM (build 25.65-b01, mixed mode)
Jmeter version 2.13
Jmeter Plugins 1.31
Any ideas what could be the issue?
Thanks
EDIT
I Downgrade Java version to 1.7 and now the error says
Error in rconfigure() method java.rmi.MarshalException: error marshalling arguments; nested exception is:
java.io.NotSerializableException: org.apache.jmeter.threads.RemoteThreadsListenerTestElement
Can Jmeter Plugins the cause of the issue?
EDIT - FIXED
I changed Plugin version from 1.31 to 1.21 and now works!!!.
However I think, and i'm just guessing here, the error happens when I ran the test from GUI and then on the console. Probably GUI leaves a socket open or something like that, not sure.
Anyway this combination works
Java version "1.7.0_79"
Java(TM) SE Runtime Environment (build 1.7.0_79-b15)
Java HotSpot(TM) 64-Bit Server VM (build 24.79-b02, mixed mode)
Jmeter version 2.13
Jmeter Plugins 1.21 (only Standard Version)
Check the logs in jmeter.log (inside bin folder) file and if you have somewhere an error like this:
2018-09-17 14:41:32,097 ERROR o.a.j.e.ConvertListeners: Error replacing class org.apache.jmeter.threads.RemoteThreadsListenerTestElement by wrapper: class org.apache.jmeter.threads.RemoteThreadsListenerWrapper java.rmi.RemoteException: Cannot start. yout-pc-name is a loopback address.
Then start the jmeter by giving:
./jmeter -Djava.rmi.server.hostname=your_ip_address
I was struggling with:
NotSerializableException: org.apache.jmeter.threads.RemoteThreadsListenerTestElement
The problem in my case was:
Caused by: java.net.BindException: Address already in use (Bind failed)
This error info did not appear in the standard output, just the log file.
I am a new user to Maven, as I am trying to use it to build apache spark on amazon EC2 VMs. I have mannually installed java version 1.7.0 on the VMs. However as I was running the Maven, the following error occurs:
Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.0:testCompile (scala-test-compile-first) on project spark-core_2.10: Execution scala-test-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.2.0:testCompile failed. CompileFailed
As I think the java version mismatch is the potential reason, causing the compiling problem. I opened up the pom file of the spark for maven tool, it has declared java related version in two seperate places:
<java.version>1.6</java.version>
and
<aws.java.sdk.version>1.8.3</aws.java.sdk.version>
What are the differences between these two versions?
Which one should be edited to solve the jave version mismatch?
It's two different things
<java.version>1.6</java.version>
is the java version used and
<aws.java.sdk.version>1.8.3</aws.java.sdk.version>
is the AWS SDK for Java version used.
The minumum requirement of AWS SDK 1.9 is Java 1.6+ so there is no compatibility issues.
I want to try Cassandra. When trying to run Cassandra, writes an error:
Error: Could not find or load main class org.apache.cassandra.service.CassandraDaemon
What's the problem?
Java -version
Java version "1.7.0_67"
Java (TM) SE Runtime Environment (build 1.7.0_67-b01)
Java HotSpot (TM) 64-Bit Server VM (build 24.65-b04, mixed mode)
python --version
Python 2.7.8
You'll get that error when you've downloaded a source distribution of Cassandra but haven't built it or when the CassandraDaemon.class file isn't in your classpath.
For the first problem:
You'll need the JDK 1.7 (which you already have) and ant to build C*.
Navigate to wherever you've extracted cassandra (I'll use ~/cassandra for this explanation) , Run ant and enjoy the awesome.
For the second, if your classpath is setup incorrectly something has gone wrong in the build process or the classpath has been modified. I'd verify that the classpath is what is expected by displaying it in the startup script (the cassandra executable) by adding echo $CLASSPATH near the bottom of the script (in my case it was line 212 for C* 2.1.0).
P.S. On windows you'll need to set CASSANDRA_HOME before being able to run C*.
I have a dev PC with Win7 and Java HotSpot(TM) 64-Bit Server VM (build 23.1-b03, mixed mode) installed and a test server with Ubuntu and OpenJDK Runtime Environment (IcedTea6 1.12.3) (6b27-1.12.3-0ubuntu1 ~12.04.1). I also have a Grails application with a jasper plugin to print some pdf files. While it works on the dev PC, on the test server it fails with this exception:
ERROR errors.GrailsExceptionResolver - NoSuchMethodError occurred when processing request: [GET] /test/printer/print
com.lowagie.text.pdf.PdfWriter.setRgbTransparencyBlending(Z)V. Stacktrace follows:
org.codehaus.groovy.grails.web.servlet.mvc.exceptions.ControllerExecutionException: Executing action [print] of controller [com.***.PrinterController] caused exception: Runtime error executing action
I have read so far that this may be caused by an inapropriate version of iText library, but the same itext.jar is included into the plugin on both machines. I'll appreciate any ideas that may help make this plugin work
Finally discovered that the project had a pdf 0.6 plugin that contained an itext 2.0.8 library which was used instead of the expected itext 2.1.7