Automate SSH Configuration for new Hadoop cluster - linux

Question

Guessing this has been done many times, but I could not find a clean answer, so I am appealing to your expertise for a better solution:
Objective: I am setting up a moderate sized RHEL Hadoop cluster and want to automate the configuration of SSH connectivity between all nodes. On the first node I have a list of all of the IPs in the file 'remote_ips', and I have the hduser password in the file 'hduser_pw' (chmod 600).
Step 1) Create "hduser_pw" password file on each host
for x in $(cat remote_ips); do sshpass -p$(cat hduser_pw) ssh -o StrictHostKeyChecking=no hduser#$x "echo $(cat hduser_pw) > hduser_pw; chmod 600 hduser_pw"; done
Step 2) Generate RSA keys for each node in cluster:
for x in $(cat remote_ips); do sshpass -p$(cat hduser_pw) ssh -o StrictHostKeyChecking=no hduser#$x "echo -e 'y'| ssh-keygen -t rsa -N \"\"
"; done
Step 3) Copy the file 'remote_ips' to each node in the cluster:
for x in $(cat remote_ips); do sshpass -p$(cat hduser_pw) scp -o StrictHostKeyChecking=no remote_ips hduser#$x:~; done
Step 4) For each node, copy the RSA public key to "authorized_keys" in every other node:
for x in $(cat remote_ips); do sshpass -p$(cat hduser_pw) ssh -o StrictHostKeyChecking=no hduser#$x 'for y in $(cat remote_ips); do cat /home/hduser/.ssh/id_rsa.pub | sshpass -p$(cat hduser_pw) ssh -o StrictHostKeyChecking=no hduser#$y '\''cat >> .ssh/authorized_keys'\'' ; done '; done
Is there a better way to do this? Really appreciate your help.
EDIT: Here are my revisions:
I incorporated the feedback from #janos and #likewhoa --> I handled the UUOC with redirection, looped across each IP, removed any pwds from the shell history with variables, etc. Thanks so much!
hduser_pw=$(< hduser_pw)
remote_ips=$(< remote_ips)
for x in $remote_ips; do
echo "Create hduser_pw password file on node: ${x}"
sshpass -p$hduser_pw scp -o StrictHostKeyChecking=no hduser_pw hduser#$x:~
echo "chmod 600 hduser_pw on node: ${x}"
sshpass -p$hduser_pw ssh -o StrictHostKeyChecking=no hduser#$x "chmod 600 hduser_pw"
echo "Generate RSA keys for: ${x}"
sshpass -p$hduser_pw ssh -o StrictHostKeyChecking=no hduser#$x "echo y | ssh-keygen -f ~/.ssh/id_rsa -t rsa -N \"\""
echo "SCP the file remote_ips to node: ${x}"
sshpass -p$hduser_pw scp -o StrictHostKeyChecking=no remote_ips hduser#$x:~
done
for x in $remote_ips; do
for y in $remote_ips; do
echo "ssh-copy-id from node ${x} to node ${y}:"
sshpass -p$hduser_pw ssh -o StrictHostKeyChecking=no hduser#$x "sshpass -p${hduser_pw} ssh-copy-id -i ~/.ssh/id_rsa.pub hduser#${y} -o StrictHostKeyChecking=no";
done
done

Answer 1

Instead of running $(cat hduser_pw) and $(cat remote_ips) multiple times,
it would be better to run them only once and, save in variables and reuse.
For example:
hduser_pw=$(cat hduser_pw)
remote_ips=$(cat remote_ips)
# Step 1) Create "hduser_pw" password file on each host
for x in $remote_ips; do
sshpass -p$hduser_pw ssh -o StrictHostKeyChecking=no hduser#$x "echo $hduser_pw > hduser_pw; chmod 600 hduser_pw"
done
Can you spot the security problem? The echo $hduser_pw will be saved in the shell history, and may also be visible in ps printings. It's better to avoid it. Instead of this:
ssh server "echo $hduser_pw > hduser_pw; chmod 600 hduser_pw"
You can do like this:
ssh server "cat > hduser_pw; chmod 600 hduser_pw" < hduser_pw
That's a bit hacky and may be confusing.
A simpler option is to scp and then ssh (for the chmod):
scp hduser_pw server:
ssh server "chmod 600 hduser_pw"
Other simplifications:
Instead of echo -e 'y' | ... you can simplify to echo y | ...
Instead of the messy Step 4, take a look into ssh-copy-id if it's available (usually it is in Linux systems)

Answer 2

First off UUOC, you really should use a while loop and re-directions instead of CAT.
Use this instead for the following steps which merges all steps into one big while loop.
while read ips; do sshpass -p$(<hduser_pw) ssh -o StrictHostKeyChecking=no hduser#$ips "echo $(<hduser_pw) > hduser_pw; chmod 600 hduser_pw"; sshpass -p$(<hduser_pw) ssh -o StrictHostKeyChecking=no hduser#$ips ssh-keygen -f ~/.ssh/id_rsa -t rsa -N \"\"";sshpass -p$(<hduser_pw) scp -o StrictHostKeyChecking=no remote_ips hduser#$ips:;done <remote_ips
I'll let you fix Step 4 on your own. GL!
Few pointers.
Use while loops when reading through lines
ssh-keygen -f ~/.ssh/somekey to avoid 'echo y | ssh-keygen ...'
Use redirect '
Good luck! Perhaps this is best if you use some kind of Configuration Management tools to automate this and more.