This question already has answers here:
While loop stops reading after the first line in Bash
(5 answers)
Closed last year.
#!/bin/bash
input="/home/ptx/script/input.file"
while IFS= read -r line;
do
sshpass -f password_file ssh -tt admin#$line 'echo password | sudo -S -s ls -ltra &> collect.txt'
done < "$input"
password_file content:
password
input.file contents:
mescmb49
mescmb46
mescmb44
mescmb33
Question
need to collect remote servers home directory contents to remote server file.
want to automate this task.
all the servers are listed in input.file
issue is, once script works only for the first server that is "mescmb42"
sshpass is reading from its stdin, which it inherited from the while loop. So sshpass is consuming all of the data that you intended to go to read. The easiest fix is probably to close sshpass's input stream (or redirect it:
while IFS= read -r line; do
</dev/null sshpass ...
done < "$input"
while IFS= read -r line; do
<&- sshpass ...
done < "$input"
Another option is to have read read from an alternate fd:
while IFS= read -r line <&3; do
sshpass ...
done 3< "$input"
Related
I have the following code but it doesnt work when i execute the code, the file th2.csv its empty.
The function of the sed is replace two words. I dont know how to make the script work correctly.
It must be done with the while.
bash th1.csv > th2.csv
Script bash
#!/bin/bash
while read -r line; do
echo "$line" | sed -E "s/,True,/,ll,/g;s/,False,/,th,/" th1.csv
done < th1.csv
Given the requirements that you must loop and apply regex, line by line, then consider:
#!/bin/bash
while read -r line; do
echo "$line" | sed -E "s/,True,/,ll,/g;s/,False,/,th,/" >> th2.csv
done < th1.csv
This reads, line by line, via a while loop. Each line is passed as stdin to sed. Note we remove the th1.csv at the end of your original sed attempt, as that will override sed reading from stdin (causing it to ignore it and instead process the file over and over again, every iteration). Lastly we append >> to your th2.csv file each iteration.
Guessing a step ahead, that you may want to pass the two files in as parameters to the script (just based on your first code snippet) then you can change this to:
#!/bin/bash
while read -r line; do
echo "$line" | sed -E "s/,True,/,ll,/g;s/,False,/,th,/" >> "$2"
done < "$1"
And, assuming this script is called myscript.sh you can call it like:
/bin/bash myscript.sh 'th1.csv' 'th2.csv'
Or, if you make it executable with chmod +x myscript.sh then:
./myscript.sh 'th1.csv' 'th2.csv'.
I have a config file that has details like
#pem_file username ip destination
./test.pem ec2-user 00.00.00.11 /Desktop/new/
./test1.pem ec2-user 00.00.00.22 /Desktop/new/
Now I need to know how can I fix the below script to get all the details using scp
while read "$(cat $conf | awk '{split($0,array,"\n")} END{print array[]}')"; do
scp -i array[1] array[2]#array[3]:/home/ubuntu/documents/xyz.xml array[4]
done
please help me.
Build your while read like this:
#!/bin/bash
while read -r file user ip destination
do
echo $file
echo $user
echo $ip
echo $destination
echo ""
done < <(grep -Ev "^#" "$conffile")
Use these variables to build your scp command.
The grep is to remove commented out lines.
If you prefer using an array, you can do this:
#!/bin/bash
while read -a line
do
echo ${line[0]}
echo ${line[1]}
echo ${line[2]}
echo ${line[3]}
echo ""
done < <(grep -Ev "^#" "$conffile")
See https://mywiki.wooledge.org/BashFAQ/001 for looping on files and commands output using while.
This question already has answers here:
ssh breaks out of while-loop in bash [duplicate]
(2 answers)
Closed 4 years ago.
bash script to read a file and process commands remotely.
It currently only processes the first line (server1)
Need to remotely process 3 commands on server1 , then server2 ......
#!/bin/bash
while read line; do
sshpass -f password ssh -o StrictHostKeyChecking=no user#$line zgrep "^A30=" /var/tmp/logs1/messages.* | >> Output.txt
sshpass -f password ssh -o StrictHostKeyChecking=no user#$line zgrep "^A30=" /var/tmp/logs2/messages.* | >> Output.txt
sshpass -f password ssh -o StrictHostKeyChecking=no user#$line zgrep "^A30=" /var/tmp/logs3/messages.* | >> Output.txt
done < file1
file1:
server1
server2
server3
sshpass is reading from the same file descriptor as the while loop, and exhausting that input before read is able to read it. Best thing is to close its stdin explicitly sshpass <&- ... or redirect from /dev/null sshpass < /dev/null.
Another option is to let sshpass inherit stdin from the script and read from a different file descriptor:
while read line <&3; do
...
done 3< file1
I have a working bash script to create backups and upload them as a tar archive to a remote sftp server.
After the upload, the script should remove all but the latest 20 backup files. I can't use any, pipe, grep, whatever on the sftp. Also I don't get the file-listing result handled in my bash-script.
export SSHPASS=$(cat /etc/backup/pw)
SFTPCONNECTION=$(cat /etc/backup/sftp-connection)
sshpass -e sftp $SFTPCONNECTION - << SOMEDELIMITER
ls -lt backup-*.tar
quit
SOMEDELIMITER
There is this nice oneliner, but I did not figure out how the use it in my case (sftp).
This script deletes all tar files in the given directory except the last 20 ones. The -t flag sorts by time & date. The <<< redirect expands $RESULT feed's it into the stdin of the while loop. I'm not entirely pleased with it as it has to create multiple connections, but with sftp I don't believe there is another way.
RESULT=`echo "ls -t path/to/old_backups/" | sftp -i ~/.ssh/your_ssh_key user#server.com | grep tar`
i=0
max=20
while read -r line; do
(( i++ ))
if (( i > max )); then
echo "DELETE $i...$line"
echo "rm $line" | sftp -i ~/.ssh/your_ssh_key user#server.com
fi
done <<< "$RESULT"
Thanks to codelitt I went with this solution:
export SSHPASS=$(cat /etc/backup/pw)
SFTPCONNECTION="username#host"
RESULT=`echo "ls -tl backup*.tar" | sshpass -e sftp $SFTPCONNECTION | grep -oP "backup.*\.tar" `
i=0
max=24
while read -r line; do
# echo "$line "
(( i++ ))
if (( i > max )); then
echo "DELETE $i...$line"
echo "rm $line" | sshpass -e sftp $SFTPCONNECTION
fi
done <<< "$RESULT"
It's a slight modification of his version:
it counts/removes only files named backup*.tar
it uses ls -l (for line based listings)
I had to use sshpass instead of a certificate-based authentication. The sftp password is inside /etc/backup/pw
There are other threads with this same topic but my issue is unique. I am running a bash script that has a function that sshes to a remote server and runs a sudo command on the remote server. I'm using the ssh -t option to avoid the requiretty issue. The offending line of code works fine as long as it's NOT being called from within the while loop. The while loop basically reads from a csv file on the local server and calls the checkAuthType function:
while read inputline
do
ARRAY=(`echo $inputline | tr ',' ' '`)
HOSTNAME=${ARRAY[0]}
OS_TYPE=${ARRAY[1]}
checkAuthType $HOSTNAME $OS_TYPE
<more irrelevant code>
done < configfile.csv
This is the function that sits at the top of the script (outside of any while loops):
function checkAuthType()
{
if [ $2 == linux ]; then
LINE=`ssh -t $1 'sudo grep "PasswordAuthentication" /etc/ssh/sshd_config | grep -v "yes\|Yes\|#"'`
fi
if [ $2 == unix ]; then
LINE=`ssh -n $1 'grep "PasswordAuthentication" /usr/local/etc/sshd_config | grep -v "yes\|Yes\|#"'`
fi
<more irrelevant code>
}
So, the offending line is the line that has the sudo command within the function. I can change the command to something simple like "sudo ls -l" and I will still get the "stdin is not a terminal" error. I've also tried "ssh -t -t" but to no avail. But if I call the checkAuthType function from outside of the while loop, it works fine. What is it about the while loop that changes the terminal and how do I fix it? Thank you one thousand times in advance.
Another option to try to get around the problem would be to redirect the file to a different file descriptor and force read to read from it instead.
while read inputline <&3
do
ARRAY=(`echo $inputline | tr ',' ' '`)
HOSTNAME=${ARRAY[0]}
OS_TYPE=${ARRAY[1]}
checkAuthType $HOSTNAME $OS_TYPE
<more irrelevant code>
done 3< configfile.csv
I am guessing you are testing with linux. You should try add the -n flag to your (linux) ssh command to avoid having ssh read from stdin - as it normally reads from stdin the while loop is feeding it your csv.
UPDATE
You should (usually) use the -n flag when scripting with SSH, and the flag is typically needed for 'expected behavior' when using a while read-loop. It does not seem to be the main issue here, though.
There are probably other solutions to this, but you could try adding another -t flag to force pseudo-tty allocation when stdin is not a terminal:
ssh -n -t -t
BroSlow's approach with a different file descriptor seems to work! Since the read command reads from fd 3 and not stdin,
ssh and hence sudo still have or get a tty/pty as stdin.
# simple test case
while read line <&3; do
sudo -k
echo "$line"
ssh -t localhost 'sudo ls -ld /'
done 3<&- 3< <(echo 1; sleep 3; echo 2; sleep 3)