I have a remote server that copies 30-some backup files to a local server every day and I want to remove the old backups if and only if a newer backup successfully copied.
With different codes I tried, I managed to erase older files, but I got the problem that if it found one new backup, it deleted ALL older ones.
I have something like (picture this with 20 virtual machines):
vm001-2019-08-01.bck
vm001-2019-07-28.bck
vm002-2019-08-01.bck
vm003-2019-07-29.bck
vm004-2019-08-01.bck
vm004-2019-07-31.bck
vm004-2019-07-30.bck
vm004-2019-07-29.bck
...
And I'd want to erase all but keep only the most recent ones.
i.e.: erase:
vm001-2019-07-28.bck
vm002-2019-07-29.bck
vm004-2019-07-31.bck
vm004-2019-07-30.bck
vm004-2019-07-29.bck
and keep only:
vm001-2019-08-01.bck
vm002-2019-08-01.bck
vm003-2019-07-29.bck
vm004-2019-08-01.bck
the problem I had is that if I have any recent backup of any machine, files like vm-003-2019-07-29 get deleted, because they are older, even if they are of different machines.
I know there are several variants of this question in the site, but I can't quite get this to work.
I've been trying variants of this code:
#!/bin/bash
for i in ./*.bck
do
echo "found" "$i"
if [[ -n $(find "$i" -type f -mmin -1440) ]]
then
echo "$i"
find "$i" -type f -mmin +1440 -exec rm -f "$i" {} +
fi
done
(The echos are for debugging purposes only)
At this time, this code finds the newer and the older files, but doesn't delete anything. If I put find "$i" -type f -mmin +1440 -exec echo "$i" {} +, it never prints anything, as if find $i is not finding anything, but when I run it as a solo command in the terminal, it does (minus the -exec part).
I've tested this script generating files with different timestamps using touch -d, but I had no success.
Unless you add the -name test before the filename find is going to consider "$i" to be the name of a directory to search in. So your find command should be:
find -name "$i" -type f -mmin -1440
which will search in the current directory. Or
find /path/to/dir -name "$i" -type f -mmin -1440
which will search in a directory named "/path/to/dir".
But, based on BashFAQ/099, I would do this to delete all but the newest file for each VM (untested):
#!/bin/bash
declare -A newest # associative array to store name of newest file for each VM
for f in *
do
vm=${f%%-*} # extracts vm name from filename (i.e. vmm001 from vm001-2019-08-01.bck)
if [[ -f $f && $f -nt ${newest["$vm"]} ]]
then
newest["$vm"]=$f
fi
done
for f in *
do
vm=${f%%-*}
if [[ -f $f && $f != ${newest["$vm"]} ]]
then
rm "$f"
fi
done
This is set up to run against files in the current directory. It assumes that the files are named as shown in the question (the VM name is separated from the rest of the file name by a hyphen). In order to use an associative array, Bash 4 or higher is required.
Related
I have multiple folders with multiple files. I need to rename those files with the same name like the folder where the file stored with "_partN" prefix.
As example,
I have a folder named as "new_folder_for_upload" which have 2 files. I need to convert the name of these 2 files like,
new_folder_for_upload_part1
new_folder_for_upload_part2
I have so many folders like above which have multiple files. I need to convert all the file names as I describe above.
Can anybody help me to find out for a single linux command or script to do this work automatically?
Assuming bash shell, and assuming you want the file numbering to restart for each subdirectory, and doing the moving of all files to the top directory (leaving empty subdirectories). Formatted as script for easier reading:
find . -type f -print0 | while IFS= read -r -d '' file
do
myfile=$(echo $file | sed "s#./##")
mydir=$(dirname "$myfile")
if [[ $mydir != $lastdir ]]
then
NR=1
fi
lastdir=${mydir}
mv "$myfile" "$(dirname "$myfile")_part${NR}"
((NR++))
done
Or as one-line command:
find . -type f -print0 | while IFS= read -r -d '' file; do myfile=$(echo $file | sed "s#./##"); mydir=$(dirname "$myfile"); if [[ $mydir != $lastdir ]]; then NR=1; fi; lastdir=${mydir}; mv "$myfile" "$(dirname "$myfile")_part${NR}"; ((NR++)); done
Beware. This is armed, and will do a bulk renaming / moving of every file in or below your current work directory. Use at your own risk.
To delete the empty subdirs:
find . -depth -empty -type d -delete
I run some kind of server on my linux machine and I use simple bash script to delete files every 3 and some files every 7 days. I use find command for doing that.But my files are saved periodically, meaning that the last modification day is the current day. So files never get deleted. Only worked for me the first time, because it met the conditions. I can't find a way to delete those files using a creation date, not modification date.
Here's my simple script:
#!/bin/sh
while true
do
java -server file.jar nogui
echo ">$(tput setaf 3)STARTING REBOOT$(tput sgr0) $(tput setaf 7)(Ctrl+C To Stop!)$(tput sgr0)"
find /folder/* -mtime +7 -exec rm -rf {} \;
find /folder/* -mtime +3 -exec rm -rf {} \;
find /logs/* -mtime +1 -exec rm -rf {} \;
echo ">Rebooting in:"
for i in 5 4 3 2 1
do
echo ">$i..."
sleep 1
done
done
If someone could help me with this, I would be really thankful!
Just an idea-don't shoot... :-)
If the files are not system files automatically generated by some process but is lets say server log files, you could possibly echo inside the file the creation date (i.e at the end or beginning) and grep that value later to decide if must be removed or kept.
No, it is not possible. Standard Linux filesystems do not track creation time at all. (ctime, sometimes mistaken for creation time, is metadata change time -- as compared to mtime, which is data modification time).
That said, there are certainly ugly hacks available. For instance, if you have the following script invoked by incron (or, less efficiently, cron) to record file creation dates:
#!/bin/bash
mkdir -p .ctimes
for f in *; do
if [[ -f $f ]] && [[ ! -e .ctimes/$f ]]; then
touch ".ctimes/$f"
fi
done
...then you can look for files in the .ctimes directory that are older than three days, and delete both the markers and the files they stand for:
#!/bin/bash
find .ctimes -mtime +3 -type f -print0 | while IFS= read -r -d '' filename; do
realname=${filename#.ctimes/}
rm -f -- "$filename" "$realname"
done
If you are on ext4 Filesystem there is some hope. You can retrieve it using stat and debugfs utilities. ext4 stores creation time with inode table entry i_crtime which is 'File creation time, in seconds since the epoch' per docs. Reference Link.
I'm using freeradius with daloradius appliance.
Now there are lots of routers connected to this radius server and the database and log files are growing too much.
The log files from freeradius are saved on:
/radacct/xxx.xxx.xxx.xxx/detail-20150404
xxx.xxx.xxx.xxx are differents IP clients, so there are lots of folders and lots of files inside these folders.
I can add this directory to rotate log because the file detail-TODAY can't be modified during the day and can be accessible all the 24h.
So I'm asking for:
A script to move /radacct/xxx.xxx.xxx.xxx/detail-yyyymmdd to a new folder /radacct_old/xxx.xxx.xxx.xxx/detail-yyyymmdd.
We have to move all files except where yyyymmdd is the current date (date when the script is executed).
After this I can rotate log radacct_old or just add to zip radacct_old_yyyymmdd.
I'm planning to do this job every week or so.
What’s the best way do you suggest?
Try something like this:
function move {
today=$(date +%Y%m%d)
file="${1#/radacct/}"
ip="${file%%/*}"
name="${file##*/}"
if [[ ! $name =~ detail-$today ]]; then
dir="/radacct_old/$ip"
[ -d "${dir}" ] || mkdir "${dir}"
mv "${1}" "${dir}/${name}"
fi
}
export -f move
find /radacct -type d -mindepth 2 -maxdepth 2 -name '*detail*' -exec bash -c 'move "$0"' {} \;
Beware this is untested, you will certainly be able to fill the gaps. I will test it out and debug later if you can't seem to make it work. Post when you have further questions.
Explanation: generally the script looks for all directories of the required format and moves them (last two lines) by calling a function (beginning).
Move function
today=$(date +%Y%m%d) constructs the date in the required format.
file="${1#/radacct/}" remove leading directory name from the directory we found using find.
ip="${file%%/*}" extract the ip address.
name="${file##*/}" extract the dir name.
if [[ ! $name =~ detail-$today ]]; then if dir name is from today.
dir="/radacct_old/$ip" construct target directory.
[ -d "${dir}" ] || mkdir "${dir}" create it if it doesn't exist.
mv "${1}" "${dir}/${name}" move the dir to the new location.
export -f move export the function so it can be called in subshell
Find function
find /radacct look in /radacct dir
-type d -mindepth 2 -maxdepth 2 look for dirs in dirs.
-name '*detail*' which contain the word detail.
-exec bash -c 'move "$0"' {} \; and execute the move function, supplying the name of the dir as argument.
Note that I will add more details and test it later today.
To perform this weekly, use a cron job.
I need a Bash Script to Execute a program for all directories that do not have a specific file and create the output file on the same directory.This program needs an input file which exist in every directory with the name *.DNA.fasta.Suppose I have the following directories that may contain sub directories also
dir1/a.protein.fasta
dir2/b.protein.fasta
dir3/anyfile
dir4/x.orf.fasta
I have started by finding the directories that don't have that specific file whic name is *.protein.fasta
in this case I want the dir3 and dir4 to be listed (since they do not contain *.protein.fasta)
I have tried this code:
find . -maxdepth 1 -type d \! -exec test -e '{}/*protein.fasta' \; -print
but it seems I missed some thing it does not work.
also I do not know how to proceed for the whole story.
This is a tricky one.
I can't think of a good solution. But here's a solution, nevertheless. Note that this is guaranteed not to work if your directory or file names contain newlines, and it's not guaranteed to work if they contain other special characters. (I've only tested with the samples in your question.)
Also, I haven't included a -maxdepth because you said you need to search subdirectories too.
#!/bin/bash
# Create an associative array
declare -A excludes
# Build an associative array of directories containing the file
while read line; do
excludes[$(dirname "$line")]=1
echo "excluded: $(dirname "$line")" >&2
done <<EOT
$(find . -name "*protein.fasta" -print)
EOT
# Walk through all directories, print only those not in array
find . -type d \
| while read line ; do
if [[ ! ${excludes[$line]} ]]; then
echo "$line"
fi
done
For me, this returns:
.
./dir3
./dir4
All of which are directories that do not contain a file matching *.protein.fasta. Of course, you can replace the last echo "$line" with whatever you need to do with these directories.
Alternately:
If what you're really looking for is just the list of top-level directories that do not contain the matching file in any subdirectory, the following bash one-liner may be sufficient:
for i in *; do test -d "$i" && ( find "$i" -name '*protein.fasta' | grep -q . || echo "$i" ); done
#!/bin/bash
for dir in *; do
test -d "$dir" && ( find "$dir" -name '*protein.fasta' | grep -q . || Programfoo"$dir/$dir.DNA.fasta");
done
I'm trying to move media and other files which are in a specified directory to another directory and create another one if it does not exits (where the files will go), and create a directory the remaining files with different extensions will go. My first problem is that my script is not making a new directory and it is not moving the files to other directories and what code can I use to move files with different extensions to one directory?
This is what i have had so far, correct me where I'm wrong and help modify my script:
#!/bin/bash
From=/home/katy/doc
To=/home/katy/mo #directory where the media files will go
WA=/home/katy/do # directory where the other files will go
if [ ! -d "$To" ]; then
mkdir -p "$To"
fi
cd $From
find path -type f -name"*.mp4" -exec mv {} $To \;
I'd solve it somewhat like this:
#!/bin/bash
From=/home/katy/doc
To=/home/katy/mo # directory where the media files will go
WA=/home/katy/do # directory where the other files will go
cd "$From"
find . -type f \
| while read file; do
dir="$(dirname "$file")"
base="$(basename "$file")"
if [[ "$file" =~ \.mp4$ ]]; then
target="$To"
else
target="$WA"
fi
mkdir -p "$target/$dir"
mv -i "$file" "$target/$dir/$base"
done
Notes:
mkdir -p will not complain if the directory already exists, so there's no need to check for that.
Put double quotes around all filenames in case they contain spaces.
By piping the output of find into a while loop, you also avoid getting bitten by spaces, because read will read until a newline.
You can modify the regex according to taste, e.g. \.(mp3|mp4|wma|ogg)$.
In case you didn't know, $(...) will run the given command and stick its output back in the place of the $(...) (called command substitution). It is almost the same as `...` but slightly better (details).
In order to test it, put echo in front of mv. (Note that quotes will disappear in the output.)
cd $From
find . -type f -name "*.mp4" -exec mv {} $To \;
^^^
or
find $From -type f -name "*.mp4" -exec mv {} $To \;
^^^^^
cd $From
mv *.mp4 $To;
mv * $WA;