I installed apache-spark and pyspark on my machine (Ubuntu), and in Pycharm, I also updated the environment variables (e.g. spark_home, pyspark_python).
I'm trying to do:
import os, sys
os.environ['SPARK_HOME'] = ".../spark-2.3.0-bin-hadoop2.7"
sys.path.append(".../spark-2.3.0-bin-hadoop2.7/bin/pyspark/")
sys.path.append(".../spark-2.3.0-bin-hadoop2.7/python/lib/py4j-0.10.6-src.zip")
from pyspark import SparkContext
from pyspark import SparkConf
sc = SparkContext('local[2]')
words = sc.parallelize(["scala", "java", "hadoop", "spark", "akka"])
print(words.count())
But, I receive some weird warnings:
py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe.
: java.lang.IllegalArgumentException
at org.apache.xbean.asm5.ClassReader.<init>(Unknown Source)
at org.apache.xbean.asm5.ClassReader.<init>(Unknown Source)
at org.apache.xbean.asm5.ClassReader.<init>(Unknown Source)
at org.apache.spark.util.ClosureCleaner$.getClassReader(ClosureCleaner.scala:46)
at org.apache.spark.util.FieldAccessFinder$$anon$3$$anonfun$visitMethodInsn$2.apply(ClosureCleaner.scala:449)
at org.apache.spark.util.FieldAccessFinder$$anon$3$$anonfun$visitMethodInsn$2.apply(ClosureCleaner.scala:432)
at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
at scala.collection.mutable.HashMap$$anon$1$$anonfun$foreach$2.apply(HashMap.scala:103)
at scala.collection.mutable.HashMap$$anon$1$$anonfun$foreach$2.apply(HashMap.scala:103)
at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
at scala.collection.mutable.HashMap$$anon$1.foreach(HashMap.scala:103)
at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
at org.apache.spark.util.FieldAccessFinder$$anon$3.visitMethodInsn(ClosureCleaner.scala:432)
at org.apache.xbean.asm5.ClassReader.a(Unknown Source)
at org.apache.xbean.asm5.ClassReader.b(Unknown Source)
at org.apache.xbean.asm5.ClassReader.accept(Unknown Source)
at org.apache.xbean.asm5.ClassReader.accept(Unknown Source)
at org.apache.spark.util.ClosureCleaner$$anonfun$org$apache$spark$util$ClosureCleaner$$clean$14.apply(ClosureCleaner.scala:262)
at org.apache.spark.util.ClosureCleaner$$anonfun$org$apache$spark$util$ClosureCleaner$$clean$14.apply(ClosureCleaner.scala:261)
at scala.collection.immutable.List.foreach(List.scala:381)
at org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$$clean(ClosureCleaner.scala:261)
at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:159)
at org.apache.spark.SparkContext.clean(SparkContext.scala:2292)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2066)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2092)
at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:939)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
at org.apache.spark.rdd.RDD.collect(RDD.scala:938)
at org.apache.spark.api.python.PythonRDD$.collectAndServe(PythonRDD.scala:153)
at org.apache.spark.api.python.PythonRDD.collectAndServe(PythonRDD.scala)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:214)
at java.base/java.lang.Thread.run(Thread.java:844)
How can I solve this problem?
Actually, I found a tricky solution. To solve the following problem:
Be sure that you installed Py4j, correctly. It's better to install it by using an official release. To do,
download the latest official release from from https://pypi.org/project/py4j/.
untar/unzip the file and navigate to the newly created directory, e.g., cd py4j-0.x.
run
sudo python(3) setup.py install
Then downgrade your Java to version 8 (previously, I used version 10.).
To do, first remove the current version of Java using:
sudo apt-get purge openjdk-\* icedtea-\* icedtea6-\*
and then Install Java 8 using:
sudo apt install openjdk-8-jre-headless
Now the code works for me properly.
I also confirm that the solution works on Ubuntu 18.04 LTS.
I had a java 10 installed and tried to run the Python examples from:
http://spark.apache.org/docs/2.3.1/, i.e. things as simple as:
./bin/spark-submit examples/src/main/python/pi.py 10
It did not work!
After applying the suggested fix:
sudo apt-get purge openjdk-\* icedtea-\* icedtea6-\*
sudo apt autoremove
sudo apt install openjdk-8-jre-headless
the example eventually worked; I mean if you consider that the right answer is:
Pi is roughly 3.142000
Thanks for the solution,
Bagvian
I had two versions of java before, java8 and java9. When I deleted Java9, the problem has been solved.
Step 1:
Downgrade or upgrade your java version to 8, if you have already installed one.
(see how to alternate among java versions)
Step 2:
Add the following to ~/.bashrc
export JAVA_HOME='/usr/lib/jvm/java-8-openjdk-amd64'
export PATH=$JAVA_HOME/bin:$PATH
export SPARK_HOME='/path/to/spark-2.x.x-bin-hadoop2.7'
export PATH=$SPARK_HOME/bin:$PATH
and run source ~/.bashrc to load it, or just start a new terminal.
An alternative approach would be to copy /path/to/spark-2.x.x-bin-hadoop2.7/conf/spark-env.sh.template to /path/to/spark-2.x.x-bin-hadoop2.7/conf/spark-env.sh. Then add the following to spark-env.sh
export JAVA_HOME='/usr/lib/jvm/java-8-openjdk-amd64'
export PYSPARK_PYTHON=python3
Then add the following to ~/.bashrc
export SPARK_HOME='/path/to/spark-2.x.x-bin-hadoop2.7'
export PATH=$SPARK_HOME/bin:$PATH
export SPARK_CONF_DIR=$SPARK_HOME/conf
and run source ~/.bashrc.
I need to maintain both OpenJDK 11 and JDK 8 for different purposes, so downgrading is not an option. For Spark Programs, I leverage by exporting (overriding) JAVA_HOME path pointing to JDK8 as below.
export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_65.jdk/Contents/Home/
direnv + adoptopenjdk8 (brew tap homebrew/cask-versions + brew cask install adoptopenjdk8) worked great for me in this situation (macOS)
# ~/.direnvrc
use_java() {
if [ "$#" -ne 1 ]; then
echo "usage: use java VERSION" >&2
return 1
fi
local v
v="$1"
if [ "$v" -le "8" ]; then
v="1.$v"
fi
export JAVA_HOME="$(/usr/libexec/java_home -v "$v")"
PATH_add $JAVA_HOME/bin
}
# .envrc in the project directory
use_java 8
If you are using anaconda, try:
conda install -c cyclus java-jdk
I had same problem. I had java-11, so I deleted Java-11 and installed java-8, the problem has been solved.
The main here of getting the error is due to the incorrect/incomplete path in the environment variable. You need to add path for java, spark, pyspark_python, hadoop(containing the bin folder).Most probably this solution can be resolved by adding right paths.
https://youtu.be/WQErwxRTiW0 ---- this video helped me in resolving my issue(video describes all the installation and correct paths)
I have the same issue.
PySpark 2.x.x supports Java 8. And PySpark 3.x.x supports Java 8 and Java 11.
So, check your PySpark and Java version.
If you are using PySpark 2.x.x then you need to install/upgrade/downgrade Java 8 and point your JAVA_HOME to java 8 jdk path.
I'm trying to start android studio in ubuntu 16.04. I'm using java 8 and $JAVA_HOME is set correctly.
but when i start android studio it throws this error
java.lang.NoClassDefFoundError: com/android/sdklib/repository/AndroidSdkHandler
at com.android.tools.idea.welcome.wizard.InstallComponentsPath.<init>(InstallComponentsPath.java:90)
at com.android.tools.idea.welcome.wizard.FirstRunWizard.init(FirstRunWizard.java:63)
at com.android.tools.idea.welcome.wizard.FirstRunWizardHost.setupWizard(FirstRunWizardHost.java:99)
at com.android.tools.idea.welcome.wizard.FirstRunWizardHost.getWelcomePanel(FirstRunWizardHost.java:91)
at com.intellij.openapi.wm.impl.welcomeScreen.FlatWelcomeFrame.<init>(FlatWelcomeFrame.java:104)
at com.intellij.openapi.wm.impl.welcomeScreen.FlatWelcomeFrameProvider.createFrame(FlatWelcomeFrameProvider.java:29)
at com.intellij.openapi.wm.impl.welcomeScreen.WelcomeFrame.showNow(WelcomeFrame.java:171)
at com.intellij.idea.IdeaApplication$IdeStarter.main(IdeaApplication.java:340)
at com.intellij.idea.IdeaApplication.run(IdeaApplication.java:200)
at com.intellij.idea.MainImpl$1$1$1.run(MainImpl.java:52)
at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:311)
at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:756)
at java.awt.EventQueue.access$500(EventQueue.java:97)
at java.awt.EventQueue$3.run(EventQueue.java:709)
at java.awt.EventQueue$3.run(EventQueue.java:703)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:76)
at java.awt.EventQueue.dispatchEvent(EventQueue.java:726)
at com.intellij.ide.IdeEventQueue.dispatchEvent(IdeEventQueue.java:366)
at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:201)
at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:116)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:105)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:93)
at java.awt.EventDispatchThread.run(EventDispatchThread.java:82)
Caused by: java.lang.ClassNotFoundException: com.android.sdklib.repository.AndroidSdkHandler PluginClassLoader[org.jetbrains.android, 10.2.2.3]
at com.intellij.ide.plugins.cl.PluginClassLoader.loadClass(PluginClassLoader.java:64)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 25 more
I can't guarantee this as a solution for you, but having installed using this method I have had no trouble.
Uninstall (by whatever method is appropriate for you)
Then reinstall with the terminal using
sudo apt-add-repository ppa:paolorotolo/android-studio
sudo apt-get update
sudo apt-get install android-studio
I must stress that this adds and installs from an unofficial repository so it is up to you to decide if you are comfortable with installing from an unofficial source
Credit to https://itsfoss.com/install-android-studio-ubuntu-linux/ where I originally found this method.
This happens when I attempt to attach the Android Studio debugger to an Android process. On occasion, this succeeds. But most of the time, it fails with an IDE internal error in the Android NDK Support plugin, displaying a RuntimeException: "Cannot find module by package name".
When this is the first time this error has occurred in the current Android Studio session, a red window pops up with the following error message:
IDE internal error occurred. Click to see details and submit a bug
report.
The IDE Fatal Errors windows has the following message:
Exception in plugin Android NDK Support.
Below is the error message and stack trace:
Cannot find module by package name
java.lang.RuntimeException: Cannot find module by package name
at com.android.tools.ndk.run.editor.NativeAndroidDebugger.attachToClient(NativeAndroidDebugger.java:152)
at com.android.tools.ndk.run.editor.AutoAndroidDebugger.attachToClient(AutoAndroidDebugger.java:98)
at org.jetbrains.android.actions.AndroidConnectDebuggerAction.closeOldSessionAndRun(AndroidConnectDebuggerAction.java:65)
at org.jetbrains.android.actions.AndroidConnectDebuggerAction.actionPerformed(AndroidConnectDebuggerAction.java:51)
at com.intellij.openapi.actionSystem.ex.ActionUtil.performActionDumbAware(ActionUtil.java:191)
at com.intellij.openapi.actionSystem.impl.ActionMenuItem$ActionTransmitter$1.run(ActionMenuItem.java:312)
at com.intellij.openapi.wm.impl.FocusManagerImpl.runOnOwnContext(FocusManagerImpl.java:958)
at com.intellij.openapi.wm.impl.IdeFocusManagerImpl.runOnOwnContext(IdeFocusManagerImpl.java:124)
at com.intellij.openapi.actionSystem.impl.ActionMenuItem$ActionTransmitter.actionPerformed(ActionMenuItem.java:282)
at javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:2022)
at com.intellij.openapi.actionSystem.impl.ActionMenuItem.fireActionPerformed(ActionMenuItem.java:110)
at com.intellij.ui.plaf.beg.BegMenuItemUI.doClick(BegMenuItemUI.java:513)
at com.intellij.ui.plaf.beg.BegMenuItemUI.access$300(BegMenuItemUI.java:45)
at com.intellij.ui.plaf.beg.BegMenuItemUI$MyMouseInputHandler.mouseReleased(BegMenuItemUI.java:533)
at java.awt.Component.processMouseEvent(Component.java:6533)
at javax.swing.JComponent.processMouseEvent(JComponent.java:3324)
at java.awt.Component.processEvent(Component.java:6298)
at java.awt.Container.processEvent(Container.java:2236)
at java.awt.Component.dispatchEventImpl(Component.java:4889)
at java.awt.Container.dispatchEventImpl(Container.java:2294)
at java.awt.Component.dispatchEvent(Component.java:4711)
at java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4888)
at java.awt.LightweightDispatcher.processMouseEvent(Container.java:4525)
at java.awt.LightweightDispatcher.dispatchEvent(Container.java:4466)
at java.awt.Container.dispatchEventImpl(Container.java:2280)
at java.awt.Window.dispatchEventImpl(Window.java:2746)
at java.awt.Component.dispatchEvent(Component.java:4711)
at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:758)
at java.awt.EventQueue.access$500(EventQueue.java:97)
at java.awt.EventQueue$3.run(EventQueue.java:709)
at java.awt.EventQueue$3.run(EventQueue.java:703)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:76)
at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:86)
at java.awt.EventQueue$4.run(EventQueue.java:731)
at java.awt.EventQueue$4.run(EventQueue.java:729)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:76)
at java.awt.EventQueue.dispatchEvent(EventQueue.java:728)
at com.intellij.ide.IdeEventQueue.defaultDispatchEvent(IdeEventQueue.java:857)
at com.intellij.ide.IdeEventQueue._dispatchEvent(IdeEventQueue.java:654)
at com.intellij.ide.IdeEventQueue.dispatchEvent(IdeEventQueue.java:386)
at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:201)
at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:116)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:105)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:93)
at java.awt.EventDispatchThread.run(EventDispatchThread.java:82)
I am using Android Studio v. 2.2.2, with Android NDK Support v. 1.0, on Windows 10 Pro (64-bit).
1.when debug dialog pop, pick Debugger Java instead of auto.
this works for me .
2.and the second choice,to delete Android Studio's config directory in .AndroidStudioXXX
Above way 1 works for me. Maybe Android studio try with dual mode by default, when you project also has C++ module.
I am trying to install android studio in ubuntu 16.04 LTS.
I downloaded the latest release and installed jdk using
sudo apt install openjdk-9-jre-headless
After the above step I am trying to run ./studio.sh after extracting Android studio zip file but It is giving me the following error. Please help.
com.intellij.ide.plugins.PluginManager$StartupAbortedException: java.lang.reflect.InvocationTargetException
at com.intellij.ide.plugins.PluginManager$2.run(PluginManager.java:96)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.intellij.ide.plugins.PluginManager$2.run(PluginManager.java:93)
... 1 more
Caused by: java.lang.ExceptionInInitializerError
at com.intellij.openapi.application.ApplicationNamesInfo.getInstance(ApplicationNamesInfo.java:49)
at com.intellij.ui.AppUIUtil.getFrameClass(AppUIUtil.java:127)
at com.intellij.ui.AppUIUtil.updateFrameClass(AppUIUtil.java:120)
at com.intellij.idea.StartupUtil.prepareAndStart(StartupUtil.java:114)
at com.intellij.idea.MainImpl.start(MainImpl.java:34)
... 6 more
Caused by: java.lang.RuntimeException: Cannot load resource: /idea/AndroidStudioApplicationInfo.xml
at com.intellij.openapi.application.ApplicationNamesInfo.<init>(ApplicationNamesInfo.java:59)
at com.intellij.openapi.application.ApplicationNamesInfo.<init>(ApplicationNamesInfo.java:30)
at com.intellij.openapi.application.ApplicationNamesInfo$ApplicationNamesInfoHolder.<clinit>(ApplicationNamesInfo.java:43)
... 11 more
Caused by: java.io.FileNotFoundException: /idea/AndroidStudioApplicationInfo.xml
at com.intellij.openapi.util.JDOMUtil.loadDocument(JDOMUtil.java:351)
at com.intellij.openapi.application.ApplicationNamesInfo.<init>(ApplicationNamesInfo.java:55)
... 13 more
I saw similar problem on StackOverflow. Please, do not install Java 9 before previous would be officially deprecated. This version is still in sdevelopment process and might be highly unstable
Unistall openJdk-9 and install Oracle-jdk-8 following this post:
http://www.webupd8.org/2012/09/install-oracle-java-8-in-ubuntu-via-ppa.html
I installed Light table in 0.7.2 for linux x64(Ubuntu 14.04).
I installed plugin Groovy 0.0.7
I stored in file test.groovy
println (1..10)
When I try to evaluate it(ctrl+enter) i got
We couldn't connect. Looks like there was an issue trying to connect to the project. Here's what we got:
null
null
find: `/usr/bin/../embeddable': No such file or directory
Exception in thread "main" java.lang.NoClassDefFoundError: groovy/lang/GroovyObject
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:482)
Caused by: java.lang.ClassNotFoundException: groovy.lang.GroovyObject
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 13 more
I am new to Lighttable, how could I fix it ?
Reinstall the plugin and check where it is being installed would be the quickest route. The other alternative is to embed the plugin into the project which would take you a bit of extra time.
Here would be the two quickest suggestions:
Pull the plugin out. Then do git clean -fdx and be sure the plugin is gone. Then reinstall from scratch.
Find where the plugin is located. By default, clojure and groovy typically are located at LightTable/deploy/plugins. If it is, then this is easy, do a git pull on the directory because the .jar may not be in the same location as the plugin itself.
Try number 1 first. It is simpler and faster to accomplish.