I am attempting to write a script in rsync to save daily backups in new directories named after the date they are created, before they are deleted 30 days after being created. The code below works, but it will quickly fill up my memory because the -u option will not see that several files in the directory structure already exist in a previous backup. Is there a better way to do this to preserve memory/bandwidth? I have had the --delete and --backup-dir options mentioned to me, but I have no idea how they would apply to this specific scenario.
#!/bin/bash
#User who's files are being backed up
BNAME=username
#directory to back up
BDIR=/home/username/BackThisUp
#directory to backup to
BackupDir=/var/home/username_local/BackupTo
#user
RUSER=$USER
#SSH Key
KEY=/var/home/username_local/.ssh
#Backupname
RBackup=`date +%F`
#Backup Server
BServ=backup.server
#Path
LPATH='Data for backup'
#date
DATE=`date +%F`
#make parent directory for backup
mkdir $BackupDir/$BNAME > /dev/null 2>&1
#Transfer new backups
rsync -avpHrz -e "ssh -i $KEY" $BNAME#$BServ:$BDIR $BackupDir/$BNAME/$DATE
find $BackupDir/$BNAME -type d -ctime +30 -exec rm -rf {} \;
I might do somethign simpler. Create a hash that only has the date's
day in it. For example, 8/11/2015 would hash to 11
Then do something like
# this number changes based on date.
hash=`date +%d`
rm -rf backup_folder/$hash
# then recreate backup_folder/$hash
You'll have around 30 days of backups. You may want to zip/compress these folders, assuming you have 30 times the size of the folder available on the disk.
Related
How I can compress files in this scenario:
I have folder structure like this:
User1:
/home/user1/websites/website1
/home/user1/websites/website2
User2:
/home/user2/websites/website1
/home/user2/websites/website2
/home/user2/websites/website3
And I try now (need) to do backup like this:
Folders for backups per user:
/backup/date/websites/user1/
/backup/date/websites/user1/
And I need backup tar in user directory separately per website like this:
/backup/date/websites/user1/website1.tar.gz
/backup/date/websites/user1/website2.tar.gz
/backup/date/websites/user2/website1.tar.gz
/backup/date/websites/user2/website2.tar.gz
/backup/date/websites/user2/website3.tar.gz
I have script like this one to do half of this work:
#VARIABLES
BKP_DATE=`date +"%F"`
BKP_DATETIME=`date +"%H-%M"`
#BACKUPS FOLDERS
BKP_DEST=/backup/users
BKP_DEST_DATE=$BKP_DEST/$BKP_DATE
BKP_DEST_TIME=$BKP_DEST_DATE/$BKP_DATETIME
BACKUP_DIR=$BKP_DEST_TIME
#NUMBER OF DAYS TO KEEP ARCHIVES IN BACKUP DIRECTORY
KEEP_DAYS=7
#Create folders
mkdir -p $BKP_DEST_DATE
mkdir -p $BKP_DEST_TIME
mkdir -p $BACKUP_DIR
#DELETE FILES OLDER THAN {*} DAYS IN BACKUP SERVER DIRECTORY
#echo 'Deleting backup folder older than '${KEEP_DAYS}' days'
find $BKP_DEST/* -type d -ctime +${KEEP_DAYS} -exec rm -rf {} \;
#Do backups
#List only names available users data directories
usersdirectories=`cd /home && find * -maxdepth 0 -type d | grep -Ev "(tmp|root)"`
#Creating directories per user name
for i in $usersdirectories; do
mkdir -p $BACKUP_DIR/$i/websites
done
But if u see, i haven't how to do tar this for separately archives. In my half script I have done:
Create folder structure for backup by datetime (/backup/users/day/hour-minutes)
Create folder structure for backup by users names (/backup/users/day/hour-minutes/user1)
Thanks for all users who try to help me!
I will try to complete your script, but I can't debug it because your environment is hard to reproduce. It is better in the future that you provide a minimal reproducible example.
#VARIABLES
BKP_DATE=$(date +"%F")
BKP_DATETIME=$(date +"%H-%M")
#BACKUPS FOLDERS
BKP_DEST=/backup/users
BKP_DEST_DATE=$BKP_DEST/$BKP_DATE
BKP_DEST_TIME=$BKP_DEST_DATE/$BKP_DATETIME
BACKUP_DIR=$BKP_DEST_TIME
#NUMBER OF DAYS TO KEEP ARCHIVES IN BACKUP DIRECTORY
KEEP_DAYS=7
#Create folders
mkdir -p $BKP_DEST_DATE
mkdir -p $BKP_DEST_TIME
mkdir -p $BACKUP_DIR
#DELETE FILES OLDER THAN {*} DAYS IN BACKUP SERVER DIRECTORY
#echo 'Deleting backup folder older than '${KEEP_DAYS}' days'
find $BKP_DEST/* -type d -ctime +${KEEP_DAYS} -exec rm -rf {} \;
#Do backups
#List only names available users data directories
usersdirectories=$(cd /home && find * -maxdepth 0 -type d | grep -Ev "(tmp|root)")
#Creating directories per user name
for i in $usersdirectories; do
for w in $(/home/$i/websites/*); do
ws=$(basename $w)
mkdir -p $BACKUP_DIR/$i/websites/$ws
tar -czvf $BACKUP_DIR/$i/websites/$ws.tar.gz /home/$i/websites/$ws
done
done
I suppose there are no blanks inside the directory names website1, etc...
I also replaced the deprecated backticks operators of your code by $(...).
I wish I could comment to ask for more clarification but I will attempt to help you.
#!/bin/bash
# example for user1
archive_path=/backup/data/websites/user1
file_path=/home/user1/websites/*
for i in $file_path
do
tar czf $archive_path/$(basename $i).tar.gz $file_path
done
Okay, after small fix. Version from Pierre working now. If u want, u can adjust it for yours need.
#VARIABLES
BKP_DATE=$(date +"%F")
BKP_DATETIME=$(date +"%H-%M")
#BACKUPS FOLDERS
BKP_DEST=/backup
BKP_DEST_DATE=$BKP_DEST/$BKP_DATE
BKP_DEST_TIME=$BKP_DEST_DATE/$BKP_DATETIME
BACKUP_DIR=$BKP_DEST_TIME
#NUMBER OF DAYS TO KEEP ARCHIVES IN BACKUP DIRECTORY
KEEP_DAYS=7
#Create folders
mkdir -p $BKP_DEST_DATE
mkdir -p $BKP_DEST_TIME
mkdir -p $BACKUP_DIR
#DELETE FILES OLDER THAN {*} DAYS IN BACKUP SERVER DIRECTORY
#echo 'Deleting backup folder older than '${KEEP_DAYS}' days'
find $BKP_DEST/* -type d -ctime +${KEEP_DAYS} -exec rm -rf {} \;
#Do backups
#List only names available users data directories
usersdirectories=$(cd /home/user && find * -maxdepth 0 -type d | grep -Ev "(tmp|root)")
#Creating directories per user name
for i in $usersdirectories; do
for w in /home/user/$i/*; do
#echo ok
ws=$(basename $w)
echo mkdir -p $BACKUP_DIR/$i
echo tar -czvf $BACKUP_DIR/$i/$ws.tar.gz /home/user/$i/$ws
done
done
I am having the following directory with multiple Backup from $(date +"%d.%m.%Y at %H:%M:%S").zip files.
/opt/
/opt/files/
/opt/files/private/*
/opt/files/backup.sh
/opt/files/backup.txt
/opt/files/Backup from $(date +"%d.%m.%Y at %H:%M:%S").zip
With a daily cronjob 0 0 * * * cd /opt/files/ && ./backup.sh > /opt/files/backup.txt I am currently managing my backups.
As you can imagine, this directory gets bigger and bigger over time. I now would like to create another script (or cronjob if it works with one command) to delete the oldest /opt/files/Backup from $(date +"%d.%m.%Y at %H:%M:%S").zip after 14 days (so that I have 14 recent backups all the time).
It would be great if you could explain your answer.
find /opt/files/Backup -name \*.zip -a -mtime +14 -ls
If you are satisfied the files being matched are the ones to delete, replace -ls with "-exec rm {} \;"
I run some kind of server on my linux machine and I use simple bash script to delete files every 3 and some files every 7 days. I use find command for doing that.But my files are saved periodically, meaning that the last modification day is the current day. So files never get deleted. Only worked for me the first time, because it met the conditions. I can't find a way to delete those files using a creation date, not modification date.
Here's my simple script:
#!/bin/sh
while true
do
java -server file.jar nogui
echo ">$(tput setaf 3)STARTING REBOOT$(tput sgr0) $(tput setaf 7)(Ctrl+C To Stop!)$(tput sgr0)"
find /folder/* -mtime +7 -exec rm -rf {} \;
find /folder/* -mtime +3 -exec rm -rf {} \;
find /logs/* -mtime +1 -exec rm -rf {} \;
echo ">Rebooting in:"
for i in 5 4 3 2 1
do
echo ">$i..."
sleep 1
done
done
If someone could help me with this, I would be really thankful!
Just an idea-don't shoot... :-)
If the files are not system files automatically generated by some process but is lets say server log files, you could possibly echo inside the file the creation date (i.e at the end or beginning) and grep that value later to decide if must be removed or kept.
No, it is not possible. Standard Linux filesystems do not track creation time at all. (ctime, sometimes mistaken for creation time, is metadata change time -- as compared to mtime, which is data modification time).
That said, there are certainly ugly hacks available. For instance, if you have the following script invoked by incron (or, less efficiently, cron) to record file creation dates:
#!/bin/bash
mkdir -p .ctimes
for f in *; do
if [[ -f $f ]] && [[ ! -e .ctimes/$f ]]; then
touch ".ctimes/$f"
fi
done
...then you can look for files in the .ctimes directory that are older than three days, and delete both the markers and the files they stand for:
#!/bin/bash
find .ctimes -mtime +3 -type f -print0 | while IFS= read -r -d '' filename; do
realname=${filename#.ctimes/}
rm -f -- "$filename" "$realname"
done
If you are on ext4 Filesystem there is some hope. You can retrieve it using stat and debugfs utilities. ext4 stores creation time with inode table entry i_crtime which is 'File creation time, in seconds since the epoch' per docs. Reference Link.
I am using following simple script script to take backup of all of my websites via tar
TIME=`date +%b-%d-%y`
FILENAME=backup-$TIME.tar.gz
#Parent backup directory
backup_parent_dir="/backup/httpdocs"
#Create backup directory and set permissions
backup_date=`date +%Y_%m_%d_%H_%M`
backup_dir="${backup_parent_dir}/${backup_date}"
echo "Backup directory: ${backup_dir}"
mkdir -p "${backup_dir}"
chmod 755 "${backup_dir}"
SRCDIR=/var/www/sites #Location of Important Data Directory (Source of backup).
tar -cpzf $backup_dir/$FILENAME $SRCDIR
Now, this is wokring fine but I need 2 things if I can do via same script
Can I exclude some folder within /var/www/sites directory, like if I don't want /var/www/sites/abc.com/logs folder to be backup. Can I define it and some other sub-directories within this script?
This script takes all sites in tar format in specified folder /backup/httpdocs through cronjob which runs daily during night and for old tarballs(older than 7 days) I have to delete them manually, so it there any possibility through same script so when it runs, it checks if there is any backup exists older than 7 days and it deletes it automatically?
EDIT:
Thanks everyone, this is what I am using now which takes backup excluding log files and delete anything older than 7 days
#!/bin/bash
#START
TIME=`date +%b-%d-%y` # This Command will add date in Backup File Name.
FILENAME=backup-$TIME.tar.gz # Here i define Backup file name format.
# Parent backup directory
backup_parent_dir="/backup/httpdocs"
# Create backup directory and set permissions
backup_date=`date +%Y_%m_%d_%H_%M`
backup_dir="${backup_parent_dir}/${backup_date}"
echo "Backup directory: ${backup_dir}"
mkdir -p "${backup_dir}"
chmod 755 "${backup_dir}"
SRCDIR=/var/www/vhosts # Location of Important Data Directory (Source of backup).
tar -cpzf $backup_dir/$FILENAME $SRCDIR --exclude=$SRCDIR/*/logs
find ${backup_parent_dir} -name '*' -type d -mtime +2 -exec rm -rfv "{}" \;
#END
Exclude from tar:
tar -cpzf $backup_dir/$FILENAME --exclude=/var/www/sites/abc.com/logs $SRCDIR
Find and delete old backups:
find ${backup_parent_dir} -type f -name 'backup-*.tar.gz' -mtime +7 -delete
The filter of find is conservative: selecting names that match backup-*.tar.gz probably renders the -type f (files only) option useless. I added it just in case you also have directories with such names. The -mtime +7 option is to be checked by you because older than 7 days is not accurate enough. Depending on what you have in mind it may be +6, +7 or +8. Please have a look at the find man page and decide by yourself. Note that the selection of backups to delete is not based on their names but on their date of last modification. If you modify them after they are created it may not be what you want. Let us know.
Use the --exclude option
tar -cpzf $backup_dir/$FILENAME $SRCDIR --exclude=$SRCDIR/*/logs
Name your backup files with an identifier that is derived from the day of the week. This will ensure the new file for each day will overwrite any existing file.
Day:
a Day of the week - abbreviated name (Mon)
A Day of the week - full name (Monday)
u Day of the week - number (Monday = 1)
d Day of the month - 2 digits (05)
e Day of the month - digit preceded by a space ( 5)
j Day of the year - (1-366)
w Same as 'u'
From: http://ss64.com/bash/date.html
For example:
TARGET_BASENAME= `date +%u`
option 1 when not many files/directories to exclude
tar -cpzf $backup_dir/$FILENAME --exclude=$SRCDIR/dir_ignore --exclude=$SRCDIR/*.log $SRCDIR
or if you have many entries to exclude much better done it in file
tar -cpzf $backup_dir/$FILENAME -X /path/to/exclude.txt $SRCDIR
where /path/to/exclude.txt file looks like
/var/www/dir_to_ignore
/var/www/*.log
you cannot use variables,but can use wildcards
second question answered very good by both guys before,i personalty love
find ${backup_parent_dir} -type f -name 'backup-*.tar.gz' -mtime +7 -delete
I am using a combination of find and copy command in my backup script.
it is used on a fairly huge amount of data,
first, out of 25 files it needs to find all the files older than 60 mins
then copy these files to a temp directory - each of these files are 1.52GB to 2GB
one of these 25 files will have data being appended continuously.
I have learnt from googling that Tarring operation will fail if there is an update going on to the file being attempted to tar, is it the same thing with find and copy also??
I am trying something like this,
/usr/bin/find $logPath -mmin +60 -type f -exec /bin/cp {} $logPath/$bkpDirectoryName \;
after this I have a step where I tar the files copied to the temp directory as mentioned above(&bkpDirectoryName), here I use as mentioned below,
/bin/tar -czf $bkpDir/$bkpDirectoryName.tgz $logPath/$bkpDirectoryName
and this also fails.
the same backup script was running from past many days and suddenly it has started failing and causing me headache! can someone please help me on this??
can you try these steps please
instead of copying files older than 60 min, move them.
run the tar on the moved files
If you do the above, the file which is continuously appended will not be moved.
In case any of your other 24 files might be updated after 60 min, you can do the following
Once you move a file, touch a file with the same name in case there are async updates which are not continuous.
When tarring the file, give a timestamp name to the tar file.This way you have a rolling tar of your logs
If nothing works due to some custom requirement on your side, try doing a rsync and then do the same operations on the rsynced files (i.e find and tar or just tar)
try this
output=`find $logPath -mmin 60 -type f`
if [ "temp$output" != "temp" ];then
cp -rf $output $other_than_logPath/$bkpDirectoryName/
else
echo sorry
fi
I think, you are using +60 instead of 60.
I also want to know, at what interval your script gets called.
#!/bin/bash
for find in `find / -name "*" -mmin 60`
do
cp $find / ## Choose directory
done
That's basically what you need, just change the directory I guess//