namenode port is getting blocked - linux

Question

I have installed 7 VM instances of Ubuntu 14.04 LTS servers. First instance runs the namenode service and all other 6 nodes run datanode service.I think my NameNode is getting crashed or blocked due to some issue.
After rebooting if I check JPS command output my namenode is running. In core-site.xml the fs.defaultfs property is set to hdfs://instance-1:8020.
but in the netstat -tulpn output 8020 port is not there.
this is the JPS output right after rebooting.
root#instance-1:~# jps
3017 VersionInfo
2613 NameNode
3371 VersionInfo
3313 ResourceManager
3015 Main
2524 QuorumPeerMain
2877 HeadlampServer
1556 Main
3480 Jps
2517 SecondaryNameNode
3171 JobHistoryServer
2790 EventCatcherService
2842 AlertPublisher
2600 Bootstrap
2909 Main
this is the netstat output that I checked after jps.
root#instance-1:~# netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 600/rpcbind
tcp 0 0 0.0.0.0:9010 0.0.0.0:* LISTEN 2524/java
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1164/sshd
tcp 0 0 127.0.0.1:5432 0.0.0.0:* LISTEN 1158/postgres
tcp 0 0 127.0.0.1:19001 0.0.0.0:* LISTEN 1496/python
tcp 0 0 0.0.0.0:42043 0.0.0.0:* LISTEN 2524/java
tcp 0 0 10.240.71.132:9000 0.0.0.0:* LISTEN 1419/python
tcp 0 0 0.0.0.0:7432 0.0.0.0:* LISTEN 1405/postgres
tcp6 0 0 :::111 :::* LISTEN 600/rpcbind
tcp6 0 0 :::22 :::* LISTEN 1164/sshd
tcp6 0 0 :::7432 :::* LISTEN 1405/postgres
udp 0 0 0.0.0.0:68 0.0.0.0:* 684/dhclient
udp 0 0 0.0.0.0:111 0.0.0.0:* 600/rpcbind
udp 0 0 10.240.71.132:123 0.0.0.0:* 3323/ntpd
udp 0 0 127.0.0.1:123 0.0.0.0:* 3323/ntpd
udp 0 0 0.0.0.0:123 0.0.0.0:* 3323/ntpd
udp 0 0 0.0.0.0:721 0.0.0.0:* 600/rpcbind
udp 0 0 0.0.0.0:29611 0.0.0.0:* 684/dhclient
udp6 0 0 :::111 :::* 600/rpcbind
udp6 0 0 :::123 :::* 3323/ntpd
udp6 0 0 :::721 :::* 600/rpcbind
udp6 0 0 :::22577 :::* 684/dhclient
As I said I don't see 8020 port. After one minute I checked JPS output and the namenode is gone.
this is the jps output one minute after rebooting.
root#instance-1:~# jps
3794 Main
3313 ResourceManager
3907 EventCatcherService
4325 Jps
2530 RunJar
3082 RunJar
2524 QuorumPeerMain
2656 Bootstrap
2877 HeadlampServer
1556 Main
2517 SecondaryNameNode
3171 JobHistoryServer
2842 AlertPublisher
2600 Bootstrap
As I said namenode is not there. I repeated the above process couple of times and everytime I get the same results port 8020 not there and namenode getting crashed. I think it is a firewall issue , what do you think?
Thanks in advance.

Answer 1

Looks like your namenode is indeed getting crashed. try stopping all the hadoop daemons then delete all the datanode data and format your namenode.
for stopping hadoop daemons use
stop-all.sh
now delete all the data from datanodes manually for terminal using rm -r command
for formatting your namenode use this
hadoop namenode -format
then start all the daemons again by using this
start-all.sh
hope it helps.

Answer 2

I don't have a full answer, but I know that you can go to the Hadoop folder in the machine where the namenode is running, and go to logs folder, and open the file that contains the log for the namenode, it should have a name like hadoop-username-namenode-machineName.log
where username is the username for your computer and machine name is the name of the host of this machine.
Go untill the end of that file and you will probably see the exact error causing the problem
Best of luck