rsync bash script duplicating dir structure? - linux

I have a script attaching an Amazon S3 bucket as a mount point on my CentOS 6.5 machine
I am attempting to utilize rsync to copy files from 2 locations to the bucket
Code:
#!/bin/bash
# SET THE BUCKET NAME HERE
S3_BUCKET="my-bucketname";
# SET THE MOUNT POINT HERE
MNT_POINT="/mnt/my-mountpoint";
# Create the mount point if it does not already exist, and set permissions on it
if [[ ! -e $MNT_POINT ]]; then
mkdir $MNT_POINT;
chmod -R 0777 $MNT_POINT;
fi;
# Mount the bucket
riofs -c ~/.config/riofs/riofs.conf.xml -o rw,allow_other,umask=2777,uid=1000,gid=1000 --cache-dir=/tmp/cache $S3_BUCKET $MNT_POINT;
mkdir $MNT_POINT/home;
mkdir $MNT_POINT/mysqlbak;
# Copy all "User" directories, except those owned by root
for filename in /home/* ; do
# Get the owner of $filename.
ACCT=$(stat -c '%U' "$filename");
# If the file is a directory NOT owned by root, run backup.
if [ -d "$filename" -a "$ACCT" != "root" ]; then
# Rsync to the mount
rsync -a /home/$filename $MNT_POINT/home;
fi;
done;
# Copy all mysql backups
for mysqlbak in /mysqlbak/* ; do
# Rsync to the mount
rsync -a /mysqlbak/$mysqlbak $MNT_POINT/mysqlbak;
done;
# No need to keep it mounted
umount $MNT_POINT;
as you can see I am attempting to keep a backup of /mysqlbackup folder's contents, and the /home's contents (minus anything attached to the root account)
The issue is, when I run this script on my server I am getting the following errors:
rsync: change_dir "/home//home" failed: No such file or directory (2)
rsync error: some files/attrs were not transfered (see previous errors) (code 23) at main.c(1039) [sender=3.0.6]
and
rsync: change_dir "/mysqlbak//mysqlbak" failed: No such file or directory (2)
rsync error: some files/attrs were not transfered (see previous errors) (code 23) at main.c(1039) [sender=3.0.6]
I can assure you that /home and /mysqlbak both exist.
How can I fix this so it properly syncs up to this mount?
/home and /mysqlbak are not created in the bucket

Replace
rsync -a /home/$filename $MNT_POINT/home;
by
rsync -a $filename $MNT_POINT/home;
and replace
rsync -a /mysqlbak/$mysqlbak $MNT_POINT/mysqlbak;
by
rsync -a $mysqlbak $MNT_POINT/mysqlbak;

Related

Automate SCP copy files from multiple directories (in brackets) to appropraite directories

I have a bash script used for copy some files from different directories in remote host. All of them have the same parent. So i put them into list:
LIST=\{ADIR, BDIR, CDIR\}
and i use the scp command
sshpass -p $2 scp -o LogLevel=debug -r $1#192.168.121.1$/PATH/$LIST/*.txt /home/test/test
that command makes me able to copy all of .txt files from ADIR, BDIR, CDIR to my test directory. Is there any option which can put all of .txt files in appropriate directory like /home/test/test/ADIR or /home/test/test/BDIR ... ?
Have you considered using rsync?
You could try something along these lines:
# Rsync Options
# -a, --archive archive mode; equals -rlptgoD (no -H,-A,-X)
# -D same as --devices --specials
# -g, --group preserve group
# -l, --links copy symlinks as symlinks
# -o, --owner preserve owner (super-user only)
# -O, --omit-dir-times omit directories from --times
# -p, --perms preserve permissions
# -r, --recursive recurse into directories
# -t, --times preserve modification times
# -u, --update skip files that are newer on the receiver
# -v, --verbose increase verbosity
# -z, --compress compress file data during the transfer
for DIR in 'ADIR' 'BDIR' 'CDIR'
do
rsync -zavu --rsh="ssh -l {username}" 192.168.121.1:/$PATH/$DIR /home/test/test/
done
Finally my working code:
SOURCE='/usr/.../'
DEST='/home/test/test'
DIRS_EXCLUDED='test/ADIR test/BDIR'
EXTENSIONS_EXCLUDED='*.NTX *.EXE'
EXCLUDED_STRING=''
for DIR in $DIRS_EXCLUDED
do
EXCLUDED_STRING=$EXCLUDED_STRING'--exclude '"$DIR"' '
done
for EXTENSION in $EXTENSIONS_EXCLUDED
do
EXCLUDED_STRING=$EXCLUDED_STRING'--exclude '"$EXTENSION"' '
done
rsync -zavu $EXCLUDED_STRING --rsh="sshpass -p $2 ssh -l $1" 192.168.xxx.xxx:$SOURCE $DEST

Copying syslog file to a new directory in linux

I'm currently having an assignment to write a bash script that can perform backup log (syslog, dmesg and message) files to a new directory. I wrote my script like this:
cd /var/log
sudo cp syslog Assignment
The file "Assignment" is in my home directory. When I used the "ls" command in my Assignment folder, I don't find a copy of syslog in there. Can someone tell me where did I go wrong? Thanks in advance.
I think you mean Assignment folder, not Assignment file. Anyways if you cd to /var/log, then when you do a cp in /var/log it will think Assignment is local to /var/log. If you do an ls in /var/log now you will see a copy of syslog called Assignment in /var/log. To get syslog copied to the assignment folder in your home directory you need to specify the absolute path not the relative path. Use the tilde, ~, to specify the home directory. So your script should say
cd /var/log
sudo cp syslog ~/Assignment/
You can try this:
#!/bin/sh
if ! [ $1 ] ; then
echo "Usage:";
echo $0 "<directory_where_to_save_logs>";
return;
fi
if [ ! -d "$1" ]; then
echo "Creating directory $1";
mkdir $1;
fi
cp /var/log/syslog* $1
cp /var/log/dmesg* $1
Thanks

script to copy, install and execute on multiple hosts

I am trying to copy few files into multiple hosts, install/configure those on each with running specific commands depending on OS type. The IP addresses for each host are read from host.txt file.
It appears when I run the script, it does not execute on the remote hosts. Can someone help identify the issues with this script? Sorry for this basic one, I am quite new into shell scripting.
#!/bin/bash
export AGENT=agent-x86-64-linux-5.8.1.tar.gz
export AGENT_PROPERTIES_NONDMZ=agent.properties.nondmz
export agent_INIT=agent.sh
echo "####Installing hqagent####"
while read host; do
scp $AGENT $AGENT_PROPERTIES_NONDMZ $agent_INIT root#$host:/opt
if ssh -n root#$host '[ "$(awk "/CentOS/{print}" /etc/*release)" ] '
then
cd /opt
tar -xvzf $AGENT
mv -f /opt/agent.properties.nondmz /opt/agent-5.8.1/conf/agent.properties
mkdir /opt/hqagent/
ln -s /opt/agent-5.8.1/ /opt/hqagent/agent-current
useradd hqagent
groupadd hqagent
chown -R hqagent:hqagent /opt/hqagent /opt/agent-5.8.1/
cd /etc/init.d
chmod 755 hqagent.sh
chkconfig --add hqagent.sh
su - hqagent
/opt/agent-5.8.1/bin/hq-agent.sh start
else
cd /opt
tar -xvzf $AGENT
mv -f /opt/agent.properties.nondmz /opt/agent-5.8.1/conf/agent.properties
rm -rf /opt/hqagent.sh
mkdir /opt/hqagent/
ln -s /opt/agent-5.8.1/ /opt/hqagent/agent-current
useradd hqagent
groupadd hqagent
chown -R hqagent:hqagent /opt/hqagent /opt/agent-5.8.1
cd /etc/init.d
ln -s /opt/hqagent/agent-current/bin/hq-agent.sh hqagent.sh
cd /etc/init.d/rc3.d/
ln -s /etc/init.d/hqagent.sh S99hqagent
ln -s /etc/init.d/hqagent.sh K01hqagent
cd ../rc5.d
ln -s /etc/init.d/hqagent.sh S99hqagent
ln -s /etc/init.d/hqagent.sh K01hqagent
chkconfig --add hqagent.sh
su - hqagent
/opt/agent-5.8.1/bin/hq-agent.sh start
fi
done < hosts.txt
error:
tar (child): agent-x86-64-linux-5.8.1.tar.gz: Cannot open: No such file or directory
tar (child): Error is not recoverable: exiting now
tar: Child returned status 2
tar: Error is not recoverable: exiting now
mv: cannot stat `/opt/agent.properties.nondmz': No such file or directory
mkdir: cannot create directory `/opt/hqagent/': File exists
ln: creating symbolic link `/opt/hqagent/agent-current': File exists
useradd: user 'hqagent' already exists
groupadd: group 'hqagent' already exists
chown: cannot access `/opt/agent-5.8.1/': No such file or directory
chmod: cannot access `hqagent.sh': No such file or directory
error reading information on service hqagent.sh: No such file or directory
-bash: line 1: 10.145.34.6: command not found
-bash: line 2: 10.145.6.10: command not found
./hq-install.sh: line 29: /opt/agent-5.8.1/bin/hq-agent.sh: No such file or directory
It appears that the problem is that you run this script on the "master" server, but somehow expect the branches of your if-statement to be run on the remote hosts. You need to factor those branches out into their own files, copy them to the remote hosts along with the other files, and in your if-statement, each branch should just be a ssh command to the remote host, triggering the script you copied over.
So your master script would look something like:
#!/bin/bash
export AGENT=agent-x86-64-linux-5.8.1.tar.gz
export AGENT_PROPERTIES_NONDMZ=agent.properties.nondmz
export agent_INIT=agent.sh
# Scripts containing the stuff you want done on the remote hosts
centos_setup=centos_setup.sh
other_setup=other_setup.sh
echo "####Installing hqagent####"
while read host; do
echo " ++ Copying files to $host"
scp $AGENT $AGENT_PROPERTIES_NONDMZ $agent_INIT root#$host:/opt
echo -n " ++ Running remote part on $host "
if ssh -n root#$host '[ "$(awk "/CentOS/{print}" /etc/*release)" ] '
then
echo "(centos)"
scp $centos_setup root#$host:/opt
ssh root#host "/opt/$centos_setup"
else
echo "(generic)"
scp $other_setup root#$host:/opt
ssh root#host "/opt/$other_setup"
fi
done < hosts.txt
The contents of the two auxiliary scrips would be the current contents of the if-branches in your original.

FTP File upload - script STUCK

I have a basic bash script that I'm using to copy a file and upload it to my FTP:
cp -i /var/mobile/file.db /var
cd /var
HOST=MYFTPHOST
USER=USERNAME
PASS=PASSWORD
ftp -inv $HOST << EOF
user $USER $PASS
cd websitefolder
put sms.db
bye
EOF
rm -f file.db
When I run the script, it saves the file to my FTP perfectly. But I'm running the script from different computer's so somehow, I'd like the script to upload the file.db to my FTP like this everytime it uploads it:
file1.db
file2.db
file3.db
file4.db
Your question is a little unclear, but if I understand correctly, you're trying to name the database files in sequential order without overwriting any old files. You'll have to get the list of files from the FTP server in order to find out what files have already been uploaded.
This code will get the list of files from the server that begin with "file" and end with ".db", count them, then change the name of your "file.db" to "fileXX.db", where "XX" is the next number in the naming sequence (i.e. file1.db, file2.db, file3.db, etc).
I'm not sure where "sms.db" came from. I've changed it to "file.db" in the script.
cp -i /var/mobile/file.db /var
cd /var
HOST=MYFTPHOST
USER=USERNAME
PASS=PASSWORD
ftp -inv $HOST << EOF
user $USER $PASS
cd websitefolder
LIST=$(ls | grep file*.db)
bye
EOF
FILECOUNT=0
for FILE in $LIST
do
if [ -f $FILE ];
then
FILECOUNT+=1
done
FILECOUNT+=1
NEXTDB="file$FILECOUNT.db"
mv file.db $NEXTDB
ftp -inv $HOST << EOF
put $NEXTDB
bye
EOF

Bash: move file/directory and create a link of it

I am trying to make a bash script that moves a file or directory from source directory to destination directory and puts a symlink to it into source directory.
So, <source_path> can be a file or directory, <destination_dir_path> is the directory where I want the original moved to.
Sample usage:
$ mvln /source_dir/file.txt /destination_dir/
OR
$ mvln /source_dir/dir_I_want_to_move/ /destination_dir/
This is what I have managed to put together, but it does not work properly.
It works only if source is a directory, otherwise mv returns an error:
mv: unable to rename `/source_dir/some_file.txt': Not a directory
And the directory is not moved into destination_directory but only its contents are moved.
#!/bin/bash
SCRIPT_NAME='mvln'
USAGE_STRING='usage: '$SCRIPT_NAME' <source_path> <destination_dir_path>'
# Show usage and exit with status
show_usage_and_exit () {
echo $USAGE_STRING
exit 1
}
# ERROR file does not exist
no_file () {
echo $SCRIPT_NAME': '$1': No such file or directory'
exit 2
}
# Check syntax
if [ $# -ne 2 ]; then
show_usage_and_exit
fi
# Check file existence
if [ ! -e "$1" ]; then
no_file $1
fi
# Get paths
source_path=$1
destination_path=$2
# Check that destination ends with a slash
[[ $destination_path != */ ]] && destination_path="$destination_path"/
# Move source
mv "$source_path" "$destination_path"
# Get original path
original_path=$destination_path$(basename $source_path)
# Create symlink in source dir
ln -s "$original_path" "${source_path%/}"
Can some one please help?
The problem is that $destination_path refers to a directory that doesn't exist. Something like this:
mv /path/to/file.txt /path/to/non/existent/directory/
returns an error, and
mv /path/to/directory/ /path/to/non/existent/directory/
will rename /path/to/directory/ to /path/to/non/existent/directory/ (provided that /path/to/non/existent/ is an existent directory, just without a subfolder named directory).
If you are expecting that $destination_path doesn't already exist, then you can add a mkdir command:
mkdir "$destination_path"
mv "$source_path" "$destination_path"
if you're expecting that it might not exist, then you can add it conditionally:
[[ -d "$destination_path" ]] || mkdir "$destination_path"
mv "$source_path" "$destination_path"
and if you're expecting that it does exist, then you have some debugging to do!
(By the way, depending on your exact situation, you might find mkdir -p to be helpful. It recursively creates a directory and all necessary parent directories, and it doesn't mind if the directory already exists.)

Resources