Bash reading from file and parsing each line [duplicate] - linux

This question already has an answer here:
Loop only iterates once with `ssh` in the body [duplicate]
(1 answer)
Closed 8 years ago.
I need to manage updates of a particular software on several customers' linux servers. The updates are provided by sending .tar.gz files to each server and inflating it overwriting the old software files.
I've put the customers' server list in a txt file like this:
user1#customer-server-1-address:22:/path/to/software/server1:NULL:Customer1name
user2#customer-server-2-address:2222:/path/to/software/server2:/etc/vpnc/client1.conf:Customer2name
user3#customer-server-3-address:22:/path/to/software/server3:NULL:Customer3name
Char ":" acts as field separator; field 1 is the server address, 2 is port, 3 is path to software, 4 is used to tell if a VPN tunnel is to be enabled first, 5 is customer name.
I've managed to create the following script:
#!/bin/bash
[...]
while IFS=':' read -ra array; do
if [ "${array[3]}" != "NULL" ]; then
echo
echo "### USING VPN CONF '${array[3]}' FOR SERVER '${array[0]}'... ###"
sleep 1
#vpnc ${array[3]}
TUNNEL1=$(ifconfig | grep -o eth0)
TUNNEL2='eth0'
if [ "$TUNNEL1" = "$TUNNEL2" ]; then
echo "### TUNNEL IS UP ###"
scp -P${array[1]} $FILENAME.gz ${array[0]}:${array[2]}
scp -P${array[1]} $FILELIST ${array[0]}:${array[2]}
ssh -p${array[1]} ${array[0]} 'cd '${array[2]}' && cat '${array[2]}'/'$FILELIST' | cpio --quiet -H ustar -o -0 -L -F '${array[2]}'/'$BACKUP_FILENAME
ssh -p${array[1]} ${array[0]} 'gzip '${array[2]}'/'$BACKUP_FILENAME
ssh -p${array[1]} ${array[0]} 'tar -C '${array[2]}' -xf '${array[2]}'/'$FILENAME.gz
ssh -p${array[1]} ${array[0]} 'cd '${array[2]}' && mkdir -p '$REMOTE_BACKUP_DIRECTORY' && rm -f '$FILELIST' && mv -f '$BACKUP_FILENAME.gz $REMOTE_BACKUP_DIRECTORY
echo "### SERVER '${array[0]}' UPDATED ###"
#vpnc-disconnect
sleep 1
else
echo "### VPN ERROR IN SERVER '${array[4]}' ###"
fi
else
echo
scp -P${array[1]} $FILENAME.gz ${array[0]}:${array[2]}
scp -P${array[1]} $FILELIST ${array[0]}:${array[2]}
ssh -p${array[1]} ${array[0]} 'cd '${array[2]}' && cat '${array[2]}'/'$FILELIST' | cpio --quiet -H ustar -o -0 -L -F '${array[2]}'/'$BACKUP_FILENAME-${array[4]}
ssh -p${array[1]} ${array[0]} 'gzip '${array[2]}'/'$BACKUP_FILENAME-${array[4]}
ssh -p${array[1]} ${array[0]} 'tar -C '${array[2]}' -xf '${array[2]}'/'$FILENAME.gz
ssh -p${array[1]} ${array[0]} 'cd '${array[2]}' && mkdir -p '$REMOTE_BACKUP_DIRECTORY' && rm -f '$FILELIST' && mv -f '$BACKUP_FILENAME-${array[4]}.gz $REMOTE_BACKUP_DIRECTORY
echo "### SERVER '${array[4]}' UPDATED ###"
echo
fi;
done < "$CUSTOMER_SERVERS_FILE"
$CUSTOMER_SERVERS_FILE is the container file as described earlier, $FILENAME.gz is the update package,
$FILELIST is the list of the files to update. Other variables set for other purposes.
Problem is, this whole thing is effectively running only for the first customer in my $CUSTOMER_SERVERS_FILE. The lines above the first are being skipped, and I can't tell why. Any hints? Thank you.
EDIT: As an answer to the duplicate objection, being the script rather large I didn't think of any link between the while and ssh objects. I was rather thinking more of a syntax error. However, I can still delete the question if asked to do so.

This happens because your while read loop and ssh both read from stdin.
You can fix it by using /dev/null as ssh's stdin:
ssh host cmd < /dev/null

Related

Linux/sh: How to list files one by one, compress each (by p7zip without save file on disk) and upload to ftp server (by curl/ncftp)?

Linux/sh: How to list all files one by one in specific folder,
compress each (by p7zip without save file on disk) and
upload to ftp server (by curl/ncftp) with same folder structure?
This script below work perfect but
I don't want to save 7z file on a disk each time. Because I always need to delete them all after uploaded.
I prefer stio from 7zip to curl, how to do that?
#!/bin/sh
FOLDER="/volume3/backup_3/kopia_nas/tmp"
BACKUP_DIR="/volume3/backup_3/kopia_nas/tmp2"
FTP_HOST=""
FTP_USER=""
FTP_PASS=""
FTP_PORT="21"
PASSWORD="abc123"
FTP_FOLDER="/backup2"
#####################################################################
echo "[$(date +'%d-%m-%Y %H:%M:%S')] starting..."
echo ""
/usr/bin/find "${FOLDER}" -type f | while read line; do
# echo "$line" #path+file
# echo "${line##*/}" #file
# echo "${line%/*}" #path
#
/usr/bin/p7zip/7za a "${BACKUP_DIR}${line}.7z" "${line}" -t7z -ms=off -m0=Copy -mhe -mmt -mx0 -p"${PASSWORD}"
curl -s --disable-epsv -v -T "${BACKUP_DIR}${line}.7z" -u "${FTP_USER}:${FTP_PASS}" "ftp://${FTP_HOST}/${FTP_FOLDER}${line%/*}/" --ftp-create-dirs;
#-S -show errors
#-s -silent mode
#-an - no file name
#v- verbose
#/usr/bin/ncftp/ncftpput -m -u -c "${FTP_USER}" -p "${FTP_PASS}" -P "${FTP_PORT}" "${FTP_HOST}" "${FTP_FOLDER}${line%/*}/" "${line##*/}.7z"
# if [ $? -ne 0 ]; then echo "[$(date +'%d-%m-%Y %H:%M:%S')] Upload failed"; fi
done
#rm -rf "${BACKUP_DIR}/" #delete temporary folder
echo ""
echo "[$(date +'%d-%m-%Y %H:%M:%S')] completed..."
exit 0
I try this but it doesn't work for me...
/usr/bin/p7zip/7za a -an -t7z -ms=off -m0=Copy -mhe -mmt -mx0 -so -p"${PASSWORD}" | curl -S --disable-epsv -v -T - -u "${FTP_USER}:${FTP_PASS}" "ftp://${FTP_HOST}/${FTP_FOLDER}${line}/" --ftp-create-dirs;

shell script ssh command not working

I have a small list of servers, and I am trying to add a user on each of these servers. I can ssh individually to each server and run the command.
sudo /usr/sbin/useradd -c "Arun" -d /home/amurug -e 2014-12-12 -g users -u 1470 amurug
I wrote a script to loop through the list and run this command but I get some errors.
#!/bin/bash
read -p "Enter server list: " file
if [[ $file == *linux* ]]; then
for i in `cat $file`
do
echo "creating amurug on" $i
ssh $i sudo /usr/sbin/useradd -c "Arun" -d /home/amurug -e 2014-12-12 -g users -u 1470 amurug
echo "==============================================="
sleep 5
done
fi
When I run the script it does not execute the command.
creating amurug on svr102
Usage: useradd [options] LOGIN
Options:
What is wrong with my ssh crommand in my script?
Try this script:
#!/bin/bash
read -p "Enter server list: " file
if [[ "$file" == *linux* ]]; then
while read -r server
do
echo "creating amurug on" "$server"
ssh -t -t "$server" "sudo /usr/sbin/useradd -c Arun -d /home/amurug \
-e 2014-12-12 -g users -u 1470 amurug"
echo "==============================================="
sleep 5
done < "$file"
fi
As per man bash:
-t
Force pseudo-tty allocation. This can be used to execute arbitrary screen-based programs on a remote which can be very useful, e.g. when implementing menu services. Multiple -t options force tty allocation, even if ssh has no local tty.

Detect host operating system distro in chef-solo deploy bash script

When deploying a chef-solo setup you need to switch between using sudo or not eg:
bash install.sh
and
sudo bash install.sh
Depending on the distro on the host server. How can this be automated?
ohai already populates these attributes and are readily available in your recipe
for example,
"platform": "centos",
"platform_version": "6.4",
"platform_family": "rhel",
you can reference to these as
if node[:platform_family].include?("rhel")
...
end
To see what other attributes ohai sets, just type
ohai
on the command line.
You can detect the distro on the remote host and deploy accordingly. in deploy.sh:
DISTRO=`ssh -o 'StrictHostKeyChecking no' ${host} 'bash -s' < bootstrap.sh`
The DISTRO variable is populated by whatever is echoed by the bootstrap.sh script, which is run on the host machine. So we can now use bootstrap.sh to detect the distro or any other server settings we need to and echo, which will be bubbled to the local script and you can respond accordingly.
example deploy.sh:
#!/bin/bash
# Usage: ./deploy.sh [host]
host="${1}"
if [ -z "$host" ]; then
echo "Please provide a host - eg: ./deploy root#my-server.com"
exit 1
fi
echo "deploying to ${host}"
# The host key might change when we instantiate a new VM, so
# we remove (-R) the old host key from known_hosts
ssh-keygen -R "${host#*#}" 2> /dev/null
# rough test for what distro the server is on
DISTRO=`ssh -o 'StrictHostKeyChecking no' ${host} 'bash -s' < bootstrap.sh`
if [ "$DISTRO" == "FED" ]; then
echo "Detected a Fedora, RHEL, CentOS distro on host"
tar cjh . | ssh -o 'StrictHostKeyChecking no' "$host" '
rm -rf /tmp/chef &&
mkdir /tmp/chef &&
cd /tmp/chef &&
tar xj &&
bash install.sh'
elif [ "$DISTRO" == "DEB" ]; then
echo "Detected a Debian, Ubuntu distro on host"
tar cj . | ssh -o 'StrictHostKeyChecking no' "$host" '
sudo rm -rf ~/chef &&
mkdir ~/chef &&
cd ~/chef &&
tar xj &&
sudo bash install.sh'
fi
example bootstrap.sh:
#!/bin/bash
# Fedora/RHEL/CentOS distro
if [ -f /etc/redhat-release ]; then
echo "FED"
# Debian/Ubuntu
elif [ -r /lib/lsb/init-functions ]; then
echo "DEB"
fi
This will allow you to detect the platform very early in the deploy process.

Update bash script, file check, how?

#!/bin/sh
LOCAL=/var/local
TMP=/var/tmp
URL=http://um10.eset.com/eset_upd
USER=""
PASSWD=""
WGET="wget --user=$USER --password=$PASSWD -t 15 -T 15 -N -nH -nd -q"
UPDATEFILE="update.ver"
cd $LOCAL
CMD="$WGET $URL/$UPDATEFILE"
eval "$CMD" || exit 1;
if [ -n "`file $UPDATEFILE|grep -i rar`" ]; then
(
cd $TMP
rm -f $TMP/$UPDATEFILE
unrar x $LOCAL/$UPDATEFILE ./
)
UPDATEFILE=$TMP/$UPDATEFILE
URL=`echo $URL|sed -e s:/eset_upd::`
fi
TMPFILE=$TMP/nod32tmpfile
grep file=/ $UPDATEFILE|tr -d \\r > $TMPFILE
FILELIST=`cut -c 6- $TMPFILE`
rm -f $TMPFILE
echo "Downloading updates..."
for FILE in $FILELIST; do
CMD="$WGET \"$URL$FILE\""
eval "$CMD"
done
cp $UPDATEFILE $LOCAL/update.ver
perl -i -pe 's/\/download\/\S+\/(\S+\.nup)/\1/g' $LOCAL/update.ver
echo "Done."
So I have this code to download definitions for my antivirus. The only problem is that, it downloads all files everytime i run script. Is it possible to implement some sort file checking ?, let's say for example,
"if that file is present and have same filesize skip it"
Bash Linux
The -nc argument to wget will not re-fetch files that already exist. It is, however, not compatible with the -N switch. So you'll have to change your WGET line to:
WGET="wget --user=$USER --password=$PASSWD -t 15 -T 15 -nH -nd -q -nc"

Consuming bandwidth

I know how to write a basic bash script which uses wget to download a file, but how do I run this in an endless loop to download the specified file, delete it when the download is complete, then download it again.
you're looking for
while :
do
wget -O - -q "http://some.url/" > /dev/null
done
this will not save the file, not output useless info, and dump the contents over and over again in /dev/null
edit to just consume bandwidth, use ping -f or ping -f -s 65507
If your goal is to max out your bandwidth, especially for the purposes of benchmarking, use iperf. You run iperf on your server and client, and it will test your bandwidth using the protocol and parameters you specify. It can test one-way or two-way throughput and can optionally try to achieve a "target" bandwidth utilization (i.e. 3Mbps).
Everything is possible with programming. :)
If you want to try and max out your internet bandwidth, you could start many many processes of wget and let them download some big disk image files at the same time, while at the same time sending some huge files back to some server.
The details are left for the implementation, but this is one method to max out your bandwidth.
In case you want to consume network bandwidth, you'll need another computer. Then from computer A, IP 192.168.0.1, listen on a port (e.g. 12345).
$ netcat -l -p 12345
Then, from the other computer, send data to it.
$ netcat 192.168.0.1 12345 < /dev/zero
I perfer to use curl to wget. it is more editable. here is an excrpt from a bash script i wrote which checks the SVN version, and then gives the user a choice to download stable or latest. It then parses out the file, separating the "user settings" from the rest of the script.
svnrev=`curl -s -m10 mythicallibrarian.googlecode.com/svn/trunk/| grep -m1 Revision | sed s/"<html><head><title>mythicallibrarian - "/""/g| sed s/": \/trunk<\/title><\/head>"/""/g`
if ! which librarian-notify-send>/dev/null && test "$LinuxDep" = "1"; then
dialog --title "librarian-notify-send" --yesno "install librarian-notify-send script for Desktop notifications?" 8 25
test $? = 0 && DownloadLNS=1 || DownloadLNS=0
if [ "$DownloadLNS" = "1" ]; then
curl "http://mythicallibrarian.googlecode.com/files/librarian-notify-send">"/usr/local/bin/librarian-notify-send"
sudo chmod +x /usr/local/bin/librarian-notify-send
fi
fi
if [ ! -f "./librarian" ]; then
DownloadML=Stable
echo "Stable `date`">./lastupdated
else
lastupdated="`cat ./lastupdated`"
DownloadML=$(dialog --title "Version and Build options" --menu "Download an update first then Build mythicalLibrarian" 10 70 15 "Latest" "Download and switch to SVN $svnrev" "Stable" "Download and switch to last stable version" "Build" "using: $lastupdated" 2>&1 >/dev/tty)
if [ "$?" = "1" ]; then
clear
echo "mythicalLibrarian was not updated."
echo "Please re-run mythicalSetup."
echo "Done."
exit 1
fi
fi
clear
if [ "$DownloadML" = "Stable" ]; then
echo "Stable "`date`>"./lastupdated"
test -f ./mythicalLibrarian.sh && rm -f mythicalLibrarian.sh
curl "http://mythicallibrarian.googlecode.com/files/mythicalLibrarian">"./mythicalLibrarian.sh"
cat "./mythicalLibrarian.sh"| sed s/' '/'\\t'/g |sed s/'\\'/'\\\\'/g >"./mythicalLibrarian1" #sed s/"\\"/"\\\\"/g |
rm ./mythicalLibrarian.sh
mv ./mythicalLibrarian1 ./mythicalLibrarian.sh
parsing="Stand-by Parsing mythicalLibrarian"
startwrite=0
test -f ./librarian && rm -f ./librarian
echo -e 'mythicalVersion="'"`cat ./lastupdated`"'"'>>./librarian
while read line
do
test "$line" = "########################## USER JOBS############################" && let startwrite=$startwrite+1
if [ $startwrite = 2 ]; then
clear
parsing="$parsing""."
test "$parsing" = "Stand-by Parsing mythicalLibrarian......." && parsing="Stand-by Parsing mythicalLibrarian"
echo $parsing
echo -e "$line" >> ./librarian
fi
done <./mythicalLibrarian.sh
clear
echo "Parsing mythicalLibrarian completed!"
echo "Removing old and downloading new version of mythicalSetup..."
test -f ./mythicalSetup.sh && rm -f ./mythicalSetup.sh
curl "http://mythicallibrarian.googlecode.com/files/mythicalSetup.sh">"./mythicalSetup.sh"
chmod +x "./mythicalSetup.sh"
./mythicalSetup.sh
exit 0
fi
if [ "$DownloadML" = "Latest" ]; then
svnrev=`curl -s mythicallibrarian.googlecode.com/svn/trunk/| grep -m1 Revision | sed s/"<html><head><title>mythicallibrarian - "/""/g| sed s/": \/trunk<\/title><\/head>"/""/g`
echo "$svnrev "`date`>"./lastupdated"
test -f ./mythicalLibrarian.sh && rm -f mythicalLibrarian.sh
curl "http://mythicallibrarian.googlecode.com/svn/trunk/mythicalLibrarian">"./mythicalLibrarian.sh"
cat "./mythicalLibrarian.sh"| sed s/' '/'\\t'/g |sed s/'\\'/'\\\\'/g >"./mythicalLibrarian1" #sed s/"\\"/"\\\\"/g |
rm ./mythicalLibrarian.sh
mv ./mythicalLibrarian1 ./mythicalLibrarian.sh
parsing="Stand-by Parsing mythicalLibrarian"
startwrite=0
test -f ./librarian && rm -f ./librarian
echo -e 'mythicalVersion="'"`cat ./lastupdated`"'"'>>./librarian
while read line
do
test "$line" = "########################## USER JOBS############################" && let startwrite=$startwrite+1
if [ $startwrite = 2 ]; then
clear
parsing="$parsing""."
test "$parsing" = "Stand-by Parsing mythicalLibrarian......." && parsing="Stand-by Parsing mythicalLibrarian"
echo $parsing
echo -e "$line" >> ./librarian
fi
done <./mythicalLibrarian.sh
clear
echo "Parsing mythicalLibrarian completed!"
echo "Removing old and downloading new version of mythicalSetup..."
test -f ./mythicalSetup.sh && rm -f ./mythicalSetup.sh
curl "http://mythicallibrarian.googlecode.com/svn/trunk/mythicalSetup.sh">"./mythicalSetup.sh"
chmod +x "./mythicalSetup.sh"
./mythicalSetup.sh
exit 0
fi
EDIT: NEVERMIND I THOUGHT YOU WERE SAYING IT WAS DOWNLOADING IN AN ENDLESS LOOP

Resources