I want to install Spark 2.1.1 on windows 10, I used a step by step guide mentioned in http://www.eaiesb.com/blogs/?p=334
I did all the steps, but when I come to the last part where I should run the spark-shell and I get the following:
C:\>”spark\spark-2.1.1-bin-hadoop2.7\bin\spark-shell”
I keep getting this
The system cannot find the path specified.
I am running this on Windows 10 machine on a virtual box.
I didn’t have another partition (D is used in the site) so I set it to C:\spark where everything is there (i.e. Hadoop, Spark, and the tmp folders).
UPDATE:/// I reinstalled Java and selected another folder with no spaces within its name, The message that am getting now is (The system cannot find the path specified)
The environment variables are :
JAVA_HOME ——> C:\Java8\jdk1.8.0_131
HADOOP_HOME —–> C:\spark\Hadoop
SPARK_HOME —-> C:\spark\spark-2.1.1-bin-hadoop2.7
You Have TO Set Jdk > BinDir Path Like This :
JAVA_HOME ——> C:\Program Files\Java\jdk1.8.0_131\bin
NOTE: If You Set Another Path of File ENV Then You Have TO separate It using Semicolon ;
Example:
JAVA_HOME ——> C:\Program Files\Java\jdk1.8.0_131\bin;C:\Program Files\nodejs
Well, before anything, thanks for #RïshïKêsh Kümar on your comments I really appreciate you tried to help me out.
The problem was with Java path, the "location_of_java_folder" didn't work for me, So I just named a folder java8 and reinstalled java in that location,
oh, and don' forget to restart your machine for changes to take effect.
Related
I am trying to run spark-submit command from drive/folder where my python script and dataset is H:\spark_material. It just won't work !
But if I copy my python script into this folder C:\spark\bin then it works.
I believe it has something to do with environment variables.
Here is my Path = %JAVA_HOME%\bin; %SPARK_HOME%\bin
Here are my variables:
HADOOP_HOME = C:\winutils
JAVA_HOME = C:\jdk
SPARK_HOME = C:\spark
Java is properly installed as I have tried typing "java -version" anywhere in CMD and it works!!
Open your cmd and type path and check is apache spark path specify till bin folder
If not please fix your path
It was/is mystery - I re-installed everything one by one on my machine except operating system and it was an issue with Python distribution in my opinion. When I reinstalled Canopy(enthought), spark-submit command started to work. I still don't know why it happened as even in my previous version of Canopy (Python) was working fine properly.
Thank you everyone for your response and contribution. Learnt a lot from you guys.
I tried installing Apache Spark on my 64 bit Windwos 7 machine.
I used the guides -
Installing Spark on Windows 10
How to run Apache Spark on Windows 7
Installing Apache Spark on Windows 7 environment
This is what I did -
Install Scala
Set environment variable SCALA_HOME and add %SCALA_HOME%\bin to Path
Result: scala command works on command prompt
Unpack pre-built Spark
Set environment variable SPARK_HOME and add %SPARK_HOME%\bin to Path
Download winutils.exe
Place winutils.exe under C:/hadoop/bin
Set environment variable HADOOP_HOME and add %HADOOP_HOME%\bin to Path
I already have JDK 8 installed.
Now, the problem is, when I run spark-shell from C:/spark-2.1.1-bin-hadoop2.7/bin, I get this -
"C:\Program Files\Java\jdk1.8.0_131\bin\java" -cp "C:\spark-2.1.1-bin-hadoop2.7\conf\;C:\spark-2.1.1-bin-hadoop2.7\jars\*" "-Dscala.usejavacp=true" -Xmx1g org spark.repl.Main --name "Spark shell" spark-shell
Is it an error? Am I doing something wrong?
Thanks!
I have the same issue when trying to install Spark local with Windows 7. Please make sure the below paths is correct and I am sure I will work with you.
Create JAVA_HOME variable: C:\Program Files\Java\jdk1.8.0_181\bin
Add the following part to your path: ;%JAVA_HOME%\bin
Create SPARK_HOME variable: C:\spark-2.3.0-bin-hadoop2.7\bin
Add the following part to your path: ;%SPARK_HOME%\bin
The most important part Hadoop path should include bin file before winutils.ee as the following: C:\Hadoop\bin Sure you will locate winutils.exe inside this path.
Create HADOOP_HOME Variable: C:\Hadoop
Add the following part to your path: ;%HADOOP_HOME%\bin
Now you can run the cmd and write spark-shell it will work.
I have successfully installed and run apache spark in the past on my machine. Today I returned to it and tried to run it using : bin/spark-shell in the spark directory (bin file exists in this dir) but I am getting:
bin is not recognized as an internal or external command,
operable program or batch file.
It s running on windows 10 cmd shell, in case this is helpful. What could cause this?
I belive we need more info, to be able to answr your question.
Using './' specifies a path, starting in the root of your working directory. (Bash or powershell)
Are you running this in the cmd shell/powershell/bash shell?
What directory are you working in, when trying to execute your command?
Is there a bin folder in your current directory? (LS command or dir command)
JAVA_HOME was outdated... I had updated java without updating the path! That was the problem.
Check version of java installed and location where environment variable JAVA_HOME is pointing to.
In my case JAVA_HOME = C:\Program Files\Java\jdk1.7.0_79 (this is old version)
The cause of this issue was that I installed a new version of JDK and removed the previous installation but JAVA_HOME was pointing to the old environment which was missing.
I am installing Apache Spark on linux. I already have Java, Scala and Spark downloaded and they are all in the Downloads folder inside the Home folder with the path /home/alex/Downloads/X where X=scala, java, spark, literally that's what the folders are called.
I got scala to work but when I try to run spark by typing ./bin/spark-shell it says:
/home/alex/Downloads/spark/bin/saprk-class: line 100: /usr/bin/java/bin/java: Not a directory
I have already included the file path by editing the bashrc with sudo gedit ~/.bashrc:
# JAVA
export JAVA_HOME=/home/alex/Downloads/java
export PATH=$PATH:$JAVA_HOME/bin
# scala
export SCALA_HOME=/home/alex/Downloads/scala
export PATH=$PATH:$SCALA_HOME/bin
# spark
export SPARK_HOME=/home/alex/Downloads/spark
export PATH=$PATH:$SPARK_HOME/bin
When I try to type sbt/sbt package in the spark folder it say no such file or directory is found also. What should I do from here?
It seems you have a few issues, namely your JAVA_HOME is not pointed to a directory with java, when you are running sbt in spark you should run ./sbt/sbt (or in new versions ./build/sbt). While you can download Java & Scala by hand, you may find that your system packages are sufficient (make sure to get jdk 7 or later).
Furthermore, after using system packages as Holden points out, in Linux you may use the command whereis to make sure of the right paths.
Finally, the following link may prove useful:
http://www.tutorialspoint.com/apache_spark/apache_spark_installation.htm
Hope this helps.
Note: It looks like there may be a configuration issue, misspelling, of the directory name
/home/alex/Downloads/spark/bin/saprk-class: line 100: /usr/bin/java/bin/java: Not a directory
saprk-class
That could be a configuration issue only, but it's worth a look if it is called /spark-class elsewhere to see if it's causing related issues.
I'm running IntelliJ Idea under linux. I have created a project and a module inside it, and in that module I have a class (MyClass.class) and when I'm trying to run it from IDE, I get
ERROR: MyClass.class (No Such file or directory)
Can somebody explain me why IntelliJ Idea doesn't recognize the classes inside my module? I know it should be a problem regarding module settings but I can't figure it out. I'm using Ubuntu 11.10
OK I place here the paths and everything for all to see :)
type : echo $PATH
Result:
/usr/lib/lightdm/lightdm:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/lib/jvm/java-7-openjdk-i386/bin:/usr/lib/jvm/java-7-openjdk-i386/bin:/usr/lib/jvm/java-7-openjdk-i386/bin
type: echo $JAVA_HOME
Result:
/usr/lib/jvm/java-7-openjdk-i386
type: ./idea.sh
Result:
NOTE: If you have both Sun JDK and OpenJDK installed
please validate either IDEA_JDK or JDK_HOME environment variable points to valid Sun JDK installation
Arkde, I have a possible explanation why Jaroslav's solution with JDK7 didn't work for you.
Maybe you had mixed Java versions in various alternatives items, possibly conflicting with the version that environment variables like JAVA_HOME and JDK_HOME point to?
Maybe something points to the /usr/lib/jvm/default-java symlink as the JDK home, and that symlink points to a different version of JDK than intended?
Did you try resetting alternatives for all Java tools to version 7? Like this:
update-java-alternatives --list
# ...see what JDK's are available, choose the one that corresponds to Java 7
# and set it to be the default in alternatives:
sudo update-java-alternatives --set java-1.7.0-openjdk-amd64
# or interactively:
sudo update-alternatives --config java
What do the following commands output on your system?
echo $JAVA_HOME
echo $JDK_HOME
ls -l /usr/lib/jvm/default-java
update-java-alternatives --list
update-alternatives --list java
I had exactly the same problem.
I've performed strace on the Idea process and in the log I saw it trying to open several .class files without the path to them specified - like open("SomeClass.class", O_RDONLY) = -1 ENOENT (No such file or directory) - no path to the project output directory and to appropriate package.
So I've apt-get installed JDK 7 along JDK 6:
apt-get install openjdk-7-doc openjdk-7-jdk openjdk-7-jre openjdk-7-jre-headless openjdk-7-jre-lib openjdk-7-source
In Ubuntu 11.10 Oneiric, OpenJDK 6 isn't removable if you want OpenJDK 7. JDK 7 is dependent on JDK 6...
So I've:
updated alternatives configuration as specified above,
changed the /usr/lib/jvm/default-java symlink to point to java-7-openjdk-amd64 ,
double checked all the environment variables (my JAVA_HOME and JDK_HOME both point to /usr/lib/jvm/default-java),
reconfigured my project's SDK appropriately (and for all the modules in the project),
and voila - problem solved!
Solved it. Remove all jdk/jre you have, install openjdk7.
Add this line to .bashrc
export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-i386/
export PATH=$PATH:$JAVA_HOME/bin
Run Idea. Profit. :)
I experienced the same problem. What I found is that the underlying system-wide Java version doesn't matter, so there is no need to set JAVA_HOME or update_alternatives. All I had to do was change some settings in IDEA:
Add a Java SDK, either OpenJDK 7, or Oracle JDK 6 or 7 (File -> Project structure -> SDKs)
Select it as Project SDK (File -> Project structure -> Project)
Check that Make checkbox is enabled, otherwise IDEA will not compile your project, also verify that class file is available in the output directory and you are running with the classpath of the correct module.
If the problem remains, send a sample project to support.
I've run into the same problem - I moved my projects (and Idea settings) from a laptop with Ubuntu 10.04 and sun-jdk-6 to a PC with Ubunty 11.10 and openjdk-6. Upon project rebuild I got MyClass.class (No Such file or directory) errors for ALL classes.
Thanks to Jaroslav, his (almost) solution did helped - I can't explain why, perhaps it would work with sun-jdk-6 too... So, I installed openjdk-7, without removing openjdk-6, and set 7th as a project's JDK in Idea. (I did not change anything in environment variables.) With jdk7 it compiled.
PS I should've written it as a comment to Jaroslav's post, not a separate answer, but I don't yet have enough reputation to do this...
Try to run IDEA using
sh -c "export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-i386 && ./idea.sh"
When it starts press Ctrl+Alt+Shift+S to open Project Settings dialog. In the left panel choose Project and verify that Project SDK is configured correctly (at least it is not red).