I am using Ubuntu 16.04 LTS and installed hadoop 2.7.2. The Output of
hadoop version
is
Hadoop 2.7.2
Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r b165c4fe8a74265c792ce23f546c64604acf0e41
Compiled by jenkins on 2016-01-26T00:08Z
Compiled with protoc 2.5.0
From source with checksum d0fda26633fa762bff87ec759ebe689c
This command was run using /usr/local/hadoop-2.7.2/share/hadoop/common/hadoop-common-2.7.2.jar
and when i run
whereis hadoop
it gives output as
hadoop: /usr/local/hadoop /usr/local/hadoop-2.7.2/bin/hadoop.cmd /usr/local/hadoop-2.7.2/bin/hadoop
But when i run command
start-all.sh
it says command not found.
also when i run
start-dfs.sh
it gives output as command not found.
I am able to run these command when i navigate to hadoop directory but i want to run these command without navigating into hadoop directory.
Your problem is that bash doesn't know where to look for ./start-all.sh.
You can fix this by opening $HOME/.bashrc and adding a line that looks like this:
PATH=$PATH:/usr/local/hadoop/sbin
This tells bash that it should look in '/usr/local/hadoop/sbin' for start-all.sh.
Note:
Changes to $HOME/.bashrc will not take affect in any terminals that are currently open.
If you need the changes to take affect in a terminal that is currently open, run
source $HOME/.bashrc
I had to search for it with find.
find / -iname start-all.sh 2> /dev/null
It found:
/usr/local/sbin/start-all.sh
/usr/local/Cellar/hadoop/3.3.4/libexec/sbin/start-all.sh
/usr/local/Cellar/hadoop/3.3.4/sbin/start-all.sh
So in addition to the previous answer, the variable in $HOME/.bashrc looks like this:
PATH=$PATH:/usr/local/sbin/
Note: I'm not sure which one should be set as PATH
Related
I'm new to spark, I downloaded precompiled spark.
When I try to run spark-shell from bin folder on command line, it returns
:cd /users/denver/spark-1.6/bin
:spark-shell
command not found
But if I run it like
:cd /users/denver/spark-1.6
:./bin/spark-shell
it launches spark ..
can you please let me know why it is throwing error in the 1st case
The reason why you can not run the spark-shell command in first case is because of environment variable
The terminal searches for executables in $PATH. This is a Unix environment variable that lists directories containing system binaries (such as ls, echo, or gcc). If you call an executable that's not in a $PATH directory (such as spark-shell), you need to indicate its absolute path in the file system.
In the terminal . is a synonym for the current working directory, thus ./bin/spark-shell can work properly. You could equally well call ./some/path/bin/spark-shell.
Hope it helps.
In linux:
Add to your ~/.bashrc
export SPARK_HOME=/home/das/spark-1.6.2-bin-hadoop2.6
export PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin
Source ~/.bashrc.
Relaunch terminal
then spark-shell works from anywhere
If it doesn't work add to ~/.profile
This happened to me too. Using ./bin/spark-shell after changing to the spark directory should solve the issue.
I have zero experience in hadoop and trying to set up hadoop in ec2 environment. After formatted the filesystem, I tried to start hadoop and it keeps saying command not found.
I think I have tried every advice I found on stackoverflow previous questions/answers.
Here is the line I am having trouble with:
[root#ip-172-31-22-92 ~]# start-hadoop.sh
-bash: start-hadoop.sh: command not found
I have tried all the following commands (which I found on previous answers)
[root#ip-172-31-22-92 ~]# hadoop-daemon.sh start namenode
-bash: hadoop-daemon.sh: command not found
[root#ip-172-31-22-92 ~]# ./start-all.sh
-bash: ./start-all.sh: No such file or directory
[root#ip-172-31-22-92 ~]# cd /usr/local/hadoop/
-bash: cd: /usr/local/hadoop/: No such file or directory
Honestly, I don't know what I am doing wrong. Plus, I am doing this as root...is this right? it seems like I should be in user...?! (discard this question if i just sounded dumber)
I am not sure whether you have downloaded/installed the hadoop package or not, so let me walk you through the process of it briefly:
Download the latest package using wget:
wget http://apache.cs.utah.edu/hadoop/common/hadoop-2.7.1/hadoop-2.7.1.tar.gz
Extract the package relative to where you have downloaded it:
tar xzf hadoop-2.7.1.tar.gz
change the dir into the extracted directory
cd hadoop-2.7.1
Now you would be able to find or start the hadoop daemons using:
sbin/start-all.sh
You can find the script's you are trying to use in the extracted dir's (hadoop-2.7.1) sbin folder.
Make sure you follow the proper documentation to get it completed properly, because I haven't really covered installing Java or configuring hadoop which are extensively covered in the following documentation link:
http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html
The scripts in this repository could help you to understand the steps to install hadoop. https://github.com/lalosam/EasyHadoop (hadoop.sh). You could try to download it and execute it. The script should download the hadoop library and confgure it as pseudo cluster. start-hadoop and stop-hadoop scripts start and stop all the services required for hadoop.
First You may have to add your HADOOP_HOME variable in .bashrc file .
Ex:
export HADOOP_HOME=/usr/local/bigdata/hadoop/hadoop-1.2.1
export CLASSPATH=$JAVA_HOME:/usr/local/bigdata/hadoop/hadoop-1.2.1/hadoop-core-1.2.1.jar
export PATH=$PATH:$HADOOP_HOME/bin
Then open a new session and execute ./start-all.sh
I am trying to check my installation of hadoop. I did create the environment variables and when I call printenv, I do see my HADOOP_HOME and PATH variables printed and correct (home/hadoop and HADOOP_HOME/bin respectively).
If I go to home/hadoop in the terminal and call ls, I see the hadoop file there. If I try to run it by calling hadoop, it still tells me command not found.
First day on Linux, so there may be a stupid answer to this problem.
Your current working directory is probably not part of your path.
That is default on linux systems.
If you are in the same directory, where your hadoop file is, run that command with an relative path, like: ./hadoop
HOME DIRECTORY:
/home/hadoop is a home directory created by linux similar to Document and settings in windows.
Open your terminal and type:
ls -l /home/hadoop
Post your result for this command: ls -l /home/hadoop
SETTING GLOBAL PATH:
Go to /home/hadoop and open .bashrc in text editor.
Add these lines at the end:
export HADOOP_HOME=/path/to/your/hadoop/installation/folder
export PATH=$PATH:$HADOOP_HOME/bin
Save and exit. Now type, this in your teminal:
echo $PATH
echo $HADOOP_HOME
If these commands shows correct directories, try hadoop command. It should work.
Post your result for these command: echo $PATH and echo $HADOOP_HOME
Go to Hadoop-x.x.x/bin folder
check for hadoop folder there
run ./hadoop version
You must run “hadoop version” command.
If the hadoop setup is fine, then you should see the following result:
Hadoop 2.4.1
Subversion https://svn.apache.org/repos/asf/hadoop/common -r 1529768
Compiled by hortonmu on 2013-10-07T06:28Z
Compiled with protoc 2.5.0
From source with checksum 79e53ce7994d1628b240f09af91e1af4
For installation related guide you can refer here:
Hadoop Environment Setup
Link to my quora answer https://qr.ae/TWngHN
Hope this helps.
Thanks
Enter which hadoop in your terminal. If you see a path as an output, hadoop is set in PATH of your system. If you get something similar to this,
usr/bin/which: no hadoop in (/usr/local/hadoop.... you might not have setup everything properly. Modify the /etc/bash.bashrc with
export HADOOP_HOME = /path/to/hadoop/folder and add it to PATH using export PATH=$PATH:HADOOP_HOME/bin
You may be editing the wrong ~/.bashrc file.
Open terminal and run sudo gedit ~/.bashrc and edit these command
export HADOOP_HOME=/usr/local/hadoop
export PATH=$PATH:$HADOOP_HOME/bin
export PATH=$PATH:$HADOOP_HOME/sbin
Note: You must not use sudo gedit ~/.bashrc.sh these both work differently on newer OS
I am new to hadoop so i am following this tuturial to install a single node on my computer. My OS is Windows 7 so I installed cygwin. In the tuturial I've been asked to:
Try the following command:
$ bin/hadoop
but when I try it I get this response:
-bash: binhadoop: command not found
The file "hadoop" is in the directory and have not changed it. As I mentioned I am new to Hadoop and Linux\Cygwin so maybe its something trivial that I do wrong.
Thanks
The command would be $cd /bin/hadoop to change directory. Use Linux for making things easier I would suggest.
I installed Scala on Ubuntu using the following
sudo apt-get install scala
~$ which scala
/usr/bin/scala
~$ whereis scala
scala: /usr/bin/scala /usr/bin/X11/scala /usr/share/man/man1/scala.1.gz
~$ scala -version
Scala code runner version 2.9.1 -- Copyright 2002-2011, LAMP/EPFL
My question is what should I put in the variable SCALA_HOME? /usr/bin ?
Today I installed scala using "apt-get install scala" and confirmed the scala jar files are located in /usr/share/java
You should be able to set your SCALA_HOME to /usr/share/java and have it all work. I assume you want to use NetBeans so you will need to set SCALA_HOME in your .profile (or .bash_profile) rather than in your .bashrc because NetBeans won't see any variables set in your .bashrc unless you start it from the command line
$ find / -maxdepth 6 -iname \*scala\*jar 2> /dev/null
/usr/share/java/scala-dbc.jar
/usr/share/java/scala-partest.jar
/usr/share/java/scala-partest-2.9.1.jar
/usr/share/java/scala-dbc-2.9.1.jar
/usr/share/java/scalacheck.jar
/usr/share/java/scalap.jar
/usr/share/java/scala-library-2.9.1.jar
/usr/share/java/scala-compiler-2.9.1.jar
/usr/share/java/scala-library.jar
/usr/share/java/scalacheck-2.9.1.jar
/usr/share/java/scala-compiler.jar
/usr/share/java/scala-swing-2.9.1.jar
/usr/share/java/scalap-2.9.1.jar
/usr/share/java/scala-swing.jar
For me its: /usr/share/java/scala
I determined this by doing
dpkg -L scala
This assumes you install scala using APT.
As of today I couldn't find an easy (and reliable) way of setting this.
As per Alex (in the comment above) installing from tarball (downloaded from scala-lang.org) into /location/of/scala/untar
Then I set export SCALA_HOME=/location/of/scala/untar in my .bashrc
Everything works for now!
I had the same issue and I did some digging
This takes into account that you are using sudo dpkg -i scala-2.11.4.deb; where the debian package has been downloaded
The SCALA_HOME should be /usr/share/scala; This is based on the following
/usr/bin/scala is a symbolic link to /usr/share/scala/bin/scala
/usr/bin/X11/scala is also a symbolic link to
/usr/share/scala/bin/scala
The way I see the scala package is installed in /usr/share/scala which should be your SCALA_HOME
I installed the untarred scala into /usr/local/share as it is on the scala download site.
In my .bashrc, I placed the following line:
export PATH="/usr/local/share/scala-2.11.8/bin:$PATH"
works great from terminal regardless of what directory I'm in.
If you have installed Scala using
$apt-get install scala
then, after a successful install to see where it installed, run
which scala
If this command shows you the path to scala binaries.
Now run
pwd
Now export SCALA_HOME path into either of these environment files
~/.bashrc
or
/etc/profile
export SCALA_HOME=<output of pwd>
The SCALA_HOME should be the directory where you install scala from.
For example, the name of this directory may be scala-2.9.2.