How to write a Linux shell script that removes files older than X days, but leaves the first file of the day by modification time? - linux

Question

As the title says, how could that shell script be implemented. I know its easy to find files and delete files older than v.gr. 29 days using:
find /some_folder/ -name "file_prefix*" -mtime +30 -exec rm {} \;
But how to add exception, that first file of each day by modification time is not removed?

Answer 1

Not the most elegant - but is a combination of a few answers - something like this will work:
d=2020-01-01
end_date=2020-02-03
while [ "$d" != $end_date ]; do
d2=$(date -I -d "$d - 1 day")
d=$(date -I -d "$d + 1 day")
echo $d2
echo $d
find -type f -newerct "${d2}" ! -newerct "${d}" -printf "%T# %Tc %p\n" | sort -n | tail -n +2 | awk '{print $9}' | xargs rm
done
I'd suggest adding paths and hashing out the xargs rm bit (just to print and double check what you're removing).
There's probably a more elegant way to do this other than the print stuff but it works.

Answer 2

For common filenames
find /some_folder/ -name "file_prefix*" -mtime +30 -printf '%TD %TT %p\n' |
sort |
awk '{if ($1==prevdate) print $3; prevdate=$1}' |
xargs rm
The find command will print %TD %TT %p, i.e. the last modification date followed by the last modification time and then the filepath (folder and filename).
The list is sorted by sort. Because of the date/time/filepath structure, this will sort by date then time so that the oldest files are printed first, which is important afterward.
awk parses each line and calls {if ($1==prevdate) print $3; prevdate=$1}. Because of the date/time/filepath structure, the date is $1, the time is $2 and the filepath is $3. This prints the filepath whenever the date if similar to the previously parsed date. So, the first file of the day is not printed out because its date differs from the date of the previous day, and all subsequent files of the same day are printed. Please note that prevdate is initially unassigned which is roughly equivalent to the null string. You can call this if you find it more readable:
awk 'BEGIN{prevdate=""} {if ($1==prevdate) print $3; prevdate=$1}'
Finally, xargs rm will call rm for each line from the standard input which, at this moment, contains the list of files printed by awk.
Handling spaces
If your filenames contain space characters, the solution can be slightly adjusted:
find /some_folder/ -name "file_prefix*" -mtime +30 -printf '%TD %TT %p\n' |
sort |
awk '{if ($1==prevdate) print; prevdate=$1}' |
cut -d ' ' -f3- |
xargs rm
awk prints the whole line instead of printing the filepath only, then the filename is extracted with cut -d ' ' -f3- before calling xargs rm.
Handling weird filenames
The above solutions do not work with filenames containing newlines and possibly won’t work with backslashes either.
I assume you won’t run into these issues because if you want to clean up the directory, chances are you already know what’s inside this directory, and these are probably log files or other type of files created automatically.
However, should you need to handle all types of filenames, the command below will do the trick:
unset prevdate currentdate filepath
find /some_folder/ -name "file_prefix*" -mtime +30 -printf '%TD %TT %p\0' |
sort -z |
while IFS= read -r -d '' line
do
currentdate=${line%% *}
if [ "$currentdate" = "$prevdate" ]
then
filepath=$(cut -d ' ' -f3- <<< $line)
rm -- "$filepath"
fi
prevdate=$currentdate
done
It behaves essentially like the initial solution but strings are separated by the null character (which is a forbidden character in a filename) instead of the traditional newline separation.
find outputs results with %TD %TT %p\0 instead of %TD %TT %p\n which means results are null-separated, then sort -z makes use of this null-separated result, and finally the while loop is a rewrite of the awk script, but makes use of null-separated strings (which is almost impossible to do with awk). There is no call to xargs rm because rm is called directly inside the while loop.
While the ability to handle all types of filenames is tempting, please note that this solution is significantly less efficient than the other solutions. The code that I wrote is non-optimal for educational purposes, but it would still be slower even if I optimized it.
Same date and time
If several "first file of the day" occur at the exact same time within a same day, it will only skip the first file with the "lowest" file path, i.e. sorted by its alphanumeric characters. If you want to keep all first files of the day with the exact same time, this is slightly more complicated but this is doable.