I'm writing a script to perform an offsite rsync backup, and whenever the rsyncline recieves some output it goes into a single variable. I then want to split that variable into an array upon the ^M token, so that I can send them to two different logger-sessions (so I get them on seperate lines in the log).
My current line to perform the rsync
result=`rsync --del -az -e "ssh -i $cert" $source $destination 2>&1`
Result in the log, when the server is unavailable
ssh: connect to host offsite port 22: Connection timed out^M rsync: connection unexpectedly closed (0 bytes received so far) [sender] rsync error: unexplained error (code 255) at io.c(601) [sender=3.0.7]
IFS=$'\n' read -a foo <<< $'12\n345\n67'
echo "${foo[#]}"
^M is $'\r' so you might try that as your delimiter:
IFS=$'\r' read -ar result <<< $(rsync --del -az -e "ssh -i $cert" $source $destination 2>&1)
echo ${result[0]} # first part
echo ${result[1]} # second part
IFS=$'\r'
set -- $result
echo $1
echo $2
Thanks guys, trough a mix of your answers, I came up with a working solution. Here it is.
result=`rsync --del -az -e "ssh -i $cert" $source $destination 2>&1`
IFS=$'\n'
for variable in $result; do
logger $variable
done
unset IFS
Now the entire script works!
Related
I have a config file that has details like
#pem_file username ip destination
./test.pem ec2-user 00.00.00.11 /Desktop/new/
./test1.pem ec2-user 00.00.00.22 /Desktop/new/
Now I need to know how can I fix the below script to get all the details using scp
while read "$(cat $conf | awk '{split($0,array,"\n")} END{print array[]}')"; do
scp -i array[1] array[2]#array[3]:/home/ubuntu/documents/xyz.xml array[4]
done
please help me.
Build your while read like this:
#!/bin/bash
while read -r file user ip destination
do
echo $file
echo $user
echo $ip
echo $destination
echo ""
done < <(grep -Ev "^#" "$conffile")
Use these variables to build your scp command.
The grep is to remove commented out lines.
If you prefer using an array, you can do this:
#!/bin/bash
while read -a line
do
echo ${line[0]}
echo ${line[1]}
echo ${line[2]}
echo ${line[3]}
echo ""
done < <(grep -Ev "^#" "$conffile")
See https://mywiki.wooledge.org/BashFAQ/001 for looping on files and commands output using while.
I write this for add SSH connection from the list automatically but when I run this script have error! I think this problem relate to read ip from $line variable in script.
My Script:
#!/bin/bash
filename='iplist.txt'
n=1
USER=root
SSHPASS=123456
while read line; do
echo "No. $n : IP = $line"
echo "yes \n" | sshpass -p "$SSHPASS" \
ssh-copy-id -o StrictHostKeyChecking=no $USER#$line \
&& echo "Add successfully!" || echo "FAILED"
echo "########################################"
n=$((n+1))
sleep 2
done < $filename
iplist.txt is a file that's contain my IPs:
172.25.25.1
172.25.25.2
This is the result of my script:
No. 1 : IP = 172.25.25.1
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
: Name or service not known: ssh: Could not resolve hostname 172.25.25.1
FAILED
########################################
No. 2 : IP = 172.25.25.2
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
: Name or service not known: ssh: Could not resolve hostname 172.25.25.2
FAILED
########################################
check the file endings, if they are CRLF for windows, CR for mac, or LF for linux.
while read -r line; do COMMAND; done
The -r option passed to read command prevents backslash escapes from being interpreted.
Add IFS= option before read command to prevent leading/trailing whitespace from being trimmed -
while IFS= read -r line; do COMMAND_on $line; done
the code above is an example, you may want to use the -r parameter. For IFS you probably do not want to use this, because if there was any whitespace, then IFS would keep this and not remove them.
I'm trying to SSH into another machine and caputre it's ip address and hostname into a variable.
However, the varaible seems to be empty when i echo it.
I have tried out answers from other posts, but it didnt solve my problem. I'm not able to figure out as what the problem is.
#!/bin/bash
FILE=/home/admin/Vishal/output.txt
input=host.txt
while IFS= read -r line
do
echo "$line"
if [ $line = $HOSTNAME ]
then
ip=`hostname -i`
domain=`hostname -A`
host=`hostname`
sudo echo $ip $domain $line localhost >> $FILE
else
output=$(ssh -i -t admin#$line << "ENDSSH"
ip2=`hostname -i`
domain2=`hostname -A`
host2=`hostname`
ENDSSH
)
echo $output
fi
done <"$input"
The input file contains a list of hostnames
The variable FILE contains the path of the file where the results are to be stored.
The varaible output is the one in which i want to store the results.
Note: The script works for first part of the if where ssh isnt required.
Ony this command is relevant for your quesion:
output=$(ssh -i -t admin#$line << "ENDSSH"
ip2=`hostname -i`
domain2=`hostname -A`
host2=`hostname`
ENDSSH
)
The command sets some variables, but doesn't produce any output, so it's expected that output does't contain your values.
If you really want three lines with hostname related values, then something simple like this should work
output=$(ssh admin#$line "hostname -i; hostname -A; hostname")
I'm using scp command to copy file from one Linux host to another.
I run scp commend on host1 and copy file from host1 to host2. File is quite big and it takes for some time to copy it.
On host2 file appears immediately as soon as copying was started. I can do everything with this file even if copying is still in progress.
Is there any reliable way to find out if copying was finished or not on host2?
Off the top of my head, you could do something like:
touch tinyfile
scp bigfile tinyfile user#host:
Then when tinyfile appears you know that the transfer of bigfile is complete.
As pointed out in the comments, this assumes that scp will copy the files one by one, in the order specified. If you don't trust it, you could do them one by one explicitly:
scp bigfile user#host:
scp tinyfile user#host:
The disadvantage of this approach is that you would potentially have to authenticate twice. If this were an issue you could use something like ssh-agent.
On sending side (host1) use script like this:
#!/bin/bash
echo 'starting transfer'
scp FILE USER#DST_SERVER:DST_PATH
OUT=$?
if [ $OUT = 0 ]; then
echo 'transfer successful'
touch successful
scp successful USER#DST_SERVER:DST_PATH
else
echo 'transfer faild'
fi
On receiving side (host2) make script like this:
#!/bin/bash
SLEEP_TIME=30
MAX_CNT=10
CNT=0
while [[ ! -e successful && $CNT < $MAX_CNT ]]; do
((CNT++))
sleep($SLEEP_TIME);
done;
if [[ -e successful ]]; then
echo 'successful'
rm successful
# do somethning with FILE
fi
With CNT and MAX_CNT you disable endless loop (in case file successful isn't transferred).
Product MAX_CNT and SLEEP_TIME should be equal or greater expected transfer time. In my example expected transfer time is less than 300 seconds.
A checksum (md5sum, sha256sum ,sha512sum) of the local and remote files would tell you if they're identical.
For the situation where you don't have SSH access to the remote system - like an FTP server - you can download the file after it's uploaded and compare the checksums. I do this for files I send from production scripts at work. Below is a snippet from the script in which I do this.
MD5SRC=$(md5sum $LOCALFILE | cut -c 1-32)
MD5TESTFILE=$(mktemp -p /ramdisk)
curl \
-o $MD5TESTFILE \
-sS \
-u $FTPUSER:$FTPPASS \
ftp://$FTPHOST/$REMOTEFILE
MD5DST=$(md5sum $MD5TESTFILE | cut -c 1-32)
if [ "$MD5SRC" == "$MD5DST" ]
then
echo "+Local and Remote files match!"
else
echo "-Local and Remote files don't match"
fi
if you use inotify-tools,
then the solution will looks like this:
while ! inotifywait -e close $(dirname ${bigfile_fullname}) 2>/dev/null | \
grep -Eo "CLOSE $(basename ${bigfile_fullname})$">/dev/null
do true
done
echo "File ${bigfile_fullname} closed"
After some investigation, and discussion of the problem on other forums I have found one more solution. Maybe it can help somebody.
There is a command "lsof". It lists open files. During copying the file will be opened, so the command
lsof | grep filename
will return non empty result.
So you might want to make a while loop to wait until lsof returns nothing and proceed with your task.
Example:
# provide your file name here
f=<nameOfYourFile>
lsofresult=`lsof | grep $f | wc -l`
while [ $lsofresult != 0 ]; do
echo still copying file $f...
sleep 5
lsofresult=`lsof | grep $f | wc -l`
done; echo copying file $f is finished: `ls $f`
For the duplicate question, How to check if file has been scp 100% to the remote location , which was for an expect script, to know if a file is transferred completely, we can add expect 100% .. .. i.e something like this ...
expect -c "
set timeout 1
spawn scp user#$REMOTE_IP:/tmp/my.file user#$HOST_IP:/home/.
expect yes/no { send yes\r ; exp_continue }
expect password: { send $SCP_PASSWORD\r }
expect 100%
sleep 1
exit
"
if [ -f "/home/my.file" ]; then
echo "Success"
fi
If avoiding a second SSH handshake is important, you can use something like the following:
ssh host cat \> bigfile \&\& touch complete < bigfile
Then wait for the "complete" file to get created on the remote end.
There are other threads with this same topic but my issue is unique. I am running a bash script that has a function that sshes to a remote server and runs a sudo command on the remote server. I'm using the ssh -t option to avoid the requiretty issue. The offending line of code works fine as long as it's NOT being called from within the while loop. The while loop basically reads from a csv file on the local server and calls the checkAuthType function:
while read inputline
do
ARRAY=(`echo $inputline | tr ',' ' '`)
HOSTNAME=${ARRAY[0]}
OS_TYPE=${ARRAY[1]}
checkAuthType $HOSTNAME $OS_TYPE
<more irrelevant code>
done < configfile.csv
This is the function that sits at the top of the script (outside of any while loops):
function checkAuthType()
{
if [ $2 == linux ]; then
LINE=`ssh -t $1 'sudo grep "PasswordAuthentication" /etc/ssh/sshd_config | grep -v "yes\|Yes\|#"'`
fi
if [ $2 == unix ]; then
LINE=`ssh -n $1 'grep "PasswordAuthentication" /usr/local/etc/sshd_config | grep -v "yes\|Yes\|#"'`
fi
<more irrelevant code>
}
So, the offending line is the line that has the sudo command within the function. I can change the command to something simple like "sudo ls -l" and I will still get the "stdin is not a terminal" error. I've also tried "ssh -t -t" but to no avail. But if I call the checkAuthType function from outside of the while loop, it works fine. What is it about the while loop that changes the terminal and how do I fix it? Thank you one thousand times in advance.
Another option to try to get around the problem would be to redirect the file to a different file descriptor and force read to read from it instead.
while read inputline <&3
do
ARRAY=(`echo $inputline | tr ',' ' '`)
HOSTNAME=${ARRAY[0]}
OS_TYPE=${ARRAY[1]}
checkAuthType $HOSTNAME $OS_TYPE
<more irrelevant code>
done 3< configfile.csv
I am guessing you are testing with linux. You should try add the -n flag to your (linux) ssh command to avoid having ssh read from stdin - as it normally reads from stdin the while loop is feeding it your csv.
UPDATE
You should (usually) use the -n flag when scripting with SSH, and the flag is typically needed for 'expected behavior' when using a while read-loop. It does not seem to be the main issue here, though.
There are probably other solutions to this, but you could try adding another -t flag to force pseudo-tty allocation when stdin is not a terminal:
ssh -n -t -t
BroSlow's approach with a different file descriptor seems to work! Since the read command reads from fd 3 and not stdin,
ssh and hence sudo still have or get a tty/pty as stdin.
# simple test case
while read line <&3; do
sudo -k
echo "$line"
ssh -t localhost 'sudo ls -ld /'
done 3<&- 3< <(echo 1; sleep 3; echo 2; sleep 3)