I am trying to install Spark in Win7 and am getting an error.
These are my environment settings:
SPARK_HOME : C:\spark (this is where I have unzipped the spark files)
JAVA_HOME : C:\Program Files\Java\jdk1.7.0_71;
SCALA_HOME: C:\Program Files (x86)\scala
PATH : C:\Program Files\Java\jdk1.7.0_71\bin;C:\app\Sampad\product\11.2.0\dbhome_1\bin;C:\Python27;C:\Program Files\Python27\;C:\Program Files\Python27\Scripts;C:\Program Files (x86)\Intel\iCLS Client\;C:\Program Files\Intel\iCLS Client\;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;%SystemRoot%\system32;%SystemRoot%;%SystemRoot%\System32\Wbem;%SYSTEMROOT%\System32\WindowsPowerShell\v1.0\;C:\Program Files\Intel\Intel(R) Management Engine Components\DAL;C:\Program Files\Intel\Intel(R) Management Engine Components\IPT;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\DAL;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\IPT;C:\Program Files\Java\jdk1.6.0_45\bin;C:\Program Files\nodejs\;C:\Program Files\Python27;C:\Anaconda;C:\Anaconda\Scripts;C:\HashiCorp\Vagrant\bin;C:\Program Files (x86)\scala\bin;C:\spark\bin;
HADOOP_HOME : C:\winutils; (set this after reading this blog
Please let me know what mistake I have made. Thanks in advance.
I installed java and spark in folder with out spaces and I am using windows 10.
In my case I added
JAVA_HOME=C:\Java\jdk1.8.0_191\bin
so it was trying to search for executables in "C:\Program Files\Java\jdk1.8.0_191\bin\bin"
make sure to add variable as
JAVA_HOME = C:\Java\jdk1.8.0_191
and In path environmental variable add
%JAVA_HOME%\bin
""same for SPARK_HOME""
It is working for me now !!!
Probably, it happens, because different java versions:
JAVA_HOME : C:\Program Files\Java\jdk1.7.0_71;
C:\Program Files\Java\jdk1.6.0_45\bin
Instead of "C:\Program Files\Java\jdk1.6.0_45\bin" use "%JAVA_HOME%\bin"
Spark: Trying to run spark-shell, but get 'cmd' is not recognized as an internal or
Try modifying the spark-shell2.cmd file first line from
#echo off
to
rem #echo off
For me it showed me that it was trying to load a file from c:\spark\bin\bin on the following line
"%SPARK_HOME%\bin\spark-submit2.cmd" --class org.apache.spark.repl.Main --name "Spark shell" %*
In my env %SPARK_HOME% was set to c:\spark\bin. My installation is at c:\spark\bin
So I set the %SPARK_HOME% to c:\Spark and added %SPARK_HOME%\bin to my PATH.
you need to setup JAVA_HOME and path
the first thing is you have to go
edit system Environment variable -> Environment variable -> under the User varibales for <user>
add New
1. JAVA_HOME = C:\Program Files\Java\jdk1.8.0_191
In path
add %JAVA_HOME%\bin
2. SPARK_HOME = C:\spark //*spark is my folder where i have installed spark*
In path
add %SPARK_HOME%\bin
This will fix your problem
For me everything that is specified above was correct still it was not working.
The reason I found is all the environment variables where added in "system variables" section so when I added them in "user variables" section it started working.
I added JAVA_HOME,SPARK_HOME, HADOOP_HOME and PATH in user variables.
I had the same problem, but I solved it by correctly setting up JAVA_HOME environment variable. Basically, you need JAVA version 8 to run Spark. So you need to install JAVA 8 and properly set up the environment and the path just like the way you set up SPARK_HOME. For example variable name: JAVA_HOME, variable value: C:\JAVA. Then you go to the path and set up the path for java as: %JAVA_HOME%
I had similar issue, I reinstalled java(new version) and corrected JAVA_HOME. This resolved the issue for me
I had same issue on windows 10, make sure that only JAVA_HOME value is absolute path, rest everything is relative to %JAVA_HOME%
One of the reasons is either JAVA_HOME or SPARK_HOME has a space in the path. In this case :
"SCALA_HOME: C:\Program Files (x86)\scala"
Here after "Program Files (x86)" has space in its path. Try to move files to location where there won't be space involved in the full path. In my case I had space in JAVA_HOME path.
In my case I noticed my JAVA_HOME env variable had semicolon(;) at the end of the path resulting in above mentioned error (as this was leading to incorrect path resolution).
I could find this issue by commenting #echo off.
Related
To avoid installing SDKs on my C: drive which is already low on space, I search a little bit and understand that I have to change SDK installation location through environment variables.
So I find them from here and:
Create ANDROID_HOME=D:\Android\Sdk\ as a User variable
Create ANDROID_USER_HOME=D:\Android\Sdk\ as a User variable
Add ANDROID_HOME and ANDROID_USER_HOME to Path
Run set command in the terminal to make sure everything is set properly
Download Android Studio and start the installation
Then this is what I see:
Android SDK Location still points to my C: drive.
So I quit the installer and this time set both ANDROID_HOME and ANDROID_USER_HOME as SYSTEM variables. Same result.
C:\Users\omids>set
ALLUSERSPROFILE=C:\ProgramData
ANDROID_HOME=D:\Android\Sdk\
ANDROID_USER_HOME=D:\Android\Sdk\
APPDATA=C:\Users\omids\AppData\Roaming
CommonProgramFiles=C:\Program Files\Common Files
CommonProgramFiles(x86)=C:\Program Files (x86)\Common Files
CommonProgramW6432=C:\Program Files\Common Files
COMPUTERNAME=DESKTOP-P8G5R28
ComSpec=C:\WINDOWS\system32\cmd.exe
DriverData=C:\Windows\System32\Drivers\DriverData
HOMEDRIVE=C:
HOMEPATH=\Users\omids
LOCALAPPDATA=C:\Users\omids\AppData\Local
LOGONSERVER=\\DESKTOP-P8G5R28
NUMBER_OF_PROCESSORS=4
OneDrive=C:\Users\omids\OneDrive
OneDriveConsumer=C:\Users\omids\OneDrive
OS=Windows_NT
Path=C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\Program Files\dotnet\;C:\Program Files\Git\cmd;C:\Program Files\Docker\Docker\resources\bin;C:\ProgramData\DockerDesktop\version-bin;C:\Users\omids\AppData\Local\Programs\Python\Python310\Scripts\;C:\Users\omids\AppData\Local\Programs\Python\Python310\;C:\Users\omids\AppData\Local\Microsoft\WindowsApps;C:\Users\omids\AppData\Local\Programs\Microsoft VS Code Insiders\bin;D:\Android\Sdk\;D:\Android\Sdk\;
PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC
PROCESSOR_ARCHITECTURE=AMD64
PROCESSOR_IDENTIFIER=Intel64 Family 6 Model 94 Stepping 3, GenuineIntel
PROCESSOR_LEVEL=6
PROCESSOR_REVISION=5e03
ProgramData=C:\ProgramData
ProgramFiles=C:\Program Files
ProgramFiles(x86)=C:\Program Files (x86)
ProgramW6432=C:\Program Files
PROMPT=$P$G
PSModulePath=C:\Program Files\WindowsPowerShell\Modules;C:\WINDOWS\system32\WindowsPowerShell\v1.0\Modules
PUBLIC=C:\Users\Public
SESSIONNAME=Console
SystemDrive=C:
SystemRoot=C:\WINDOWS
TEMP=C:\Users\omids\AppData\Local\Temp
TMP=C:\Users\omids\AppData\Local\Temp
USERDOMAIN=DESKTOP-P8G5R28
USERDOMAIN_ROAMINGPROFILE=DESKTOP-P8G5R28
USERNAME=omids
USERPROFILE=C:\Users\omids
windir=C:\WINDOWS
ZES_ENABLE_SYSMAN=1
I know I can change that path manually and proceed with installation, but if Android Studio cannot read those variables or ignores them I'm sure I'll have issues down the road.
I have successfully installed aws cli, but I'm getting
"'aws' is not recognized as an internal or external command,
operable program or batch file."
I think it is problem with the path settings. How do I know the path of AWSCLI for windows?
Ideally, it should be configured with the installation.
Please try the following.
On my system for version
The path entry is
C:\Program Files\Amazon\AWSCLIV2\
Check that this path "C:\Program Files\Amazon\AWSCLIV2" exists on your system. If it's nonexistent then search for aws.exe in C:\Program Files. The folder in which aws.exe is present will be your base path. If you don't find the aws.exe do a re-install.
If the path "C:\Program Files\Amazon\AWSCLIV2" exists on your system. Or there is some other folder in which aws.exe is present then that path needs to be configured manually.
Set the path manually. by following steps.
Steps to add this entry in a path environment variable.
Click on windows and search edit environment.
In the System variable section click on the path.
Add the path "C:\Program Files\Amazon\AWSCLIV2"
Save. Open a new command line and type aws.
I use windows10.
My node.js's path is C:\Program Files\nodejs\node.exe and I can use node command.
But I haven't set my environment variable path.
It does not mean just that I have not set it myself. I checked the user environment variables and the system environment variables but could not find them. (The path of npm was in the user environment variable.)
Why can I use node command without setting path?
For the node command to work in Windows from a command shell, one of the following must be true:
Your current directory in the command shell is C:\Program Files\nodejs and thus node.exe or node.bat can be found in that current directory.
C:\Program Files\nodejs is in the search path which can be either a system wide path setting or a local user path setting (what you see in the environment is a combination of those two).
There is a node.bat file somewhere in your system path or in the current directory that launches node.exe for you by directly referencing its path.
On Windows, you can type "where node" in the command shell and it will tell you where it's finding the file to run. If what it is finding is not in the current directory, then you must have its directory in your path somewhere.
I'm completely new to hadoop framework and for the past few months I've been using linux . After I installing hadoop to /usr/local directory. I tried to run hadoop command in CLI and it responds as hadoop command not found, then I figured out environment variables aren't set, so I set the environment variables by the following commands
export HADOOP_HOME=/usr/local/hadoop/
export PATH=$PATH:$HADOOP_HOME/bin/
It worked. Also I know what is an environment variable but my doubt is how does the Shell refers hadoop command by using the HADOOP_HOME variable
HADOOP_HOME does nothing for when you type the hadoop command. (Or anything else in $HADOOP_HOME/bin, for that matter).
The $PATH where all the commands you type into the terminal are looked for.
Just echo $PATH and you'll see all the folders.
HADOOP_HOME is looked for by the hadoop components themselves, and isn't Linux specific.
HADOOP_HOME variable is used by shell files like yarn-config.sh, mapred-config.sh that is why it is required for setting HADOOP_HOME variable so that when config files access it they can reach to main hadoop folder.
If you do not want to define HADOOP_HOME then you need to edit config script files by replacing HAOOP_HOME with the required directory address
During my installation of Nodist, the node version manager for Windows, I encountered an alert message that stated:
PATH not updated, original length 1030 > 1024
The result was that even Nodist was installed successfully(per installation feedback), entering 'nodist' at git bash at directory levels other than its resident directory (C:\Program Files (x86)\Nodist\bin) yielded:
'nodist' is not recognized as an internal or external command,operable program or batch file.
How can one set up PATH manually?
Since nodist can be accessed from the directory of C:\Program Files (x86)\Nodist\bin and not anywhere else, placing the path
C:\Program Files (x86)\Nodist\bin
in the System Variables (accessible from Window's Advanced System Settings/Environmental Variables) corrects the problem. Many thanks to Marcel Klehr for the pointer.