Delete all files older than 30 days, based on file name as date - linux

I'm new to bash, I have a task to delete all files older than 30 days, I can figure this out based on the files name Y_M_D.ext 2019_04_30.txt.
I know I can list all files with ls in a the folder containing the files. I know I can get todays date with $ date and can configure that to match the file format $ date "+%Y_%m_%d"
I know I can delete files using rm.
How do I tie all this together into a bash script that deletes files older than 30 days from today?
In pseudo-python code I guess it would look like:
for file in folder:
if file.name to date > 30 day from now:
delete file

I am by no means a systems administrator, but you could consider a simple shell script along the lines of:
# Generate the date in the proper format
discriminant=$(date -d "30 days ago" "+%Y_%m_%d")
# Find files based on the filename pattern and test against the date.
find . -type f -maxdepth 1 -name "*_*_*.txt" -printf "%P\n" |
while IFS= read -r FILE; do
if [ "${discriminant}" ">" "${FILE%.*}" ]; then
echo "${FILE}";
fi
done
Note that this is will probably be considered a "layman" solution by a professional. Maybe this is handled better by awk, which I am unfortunately not accustomed to using.

Here is another solution to delete log files older than 30 days:
#!/bin/sh
# A table that contains the path of directories to clean
rep_log=("/etc/var/log" "/test/nginx/log")
echo "Cleaning logs - $(date)."
#loop for each path provided by rep_log
for element in "${rep_log[#]}"
do
#display the directory
echo "$element";
nb_log=$(find "$element" -type f -mtime +30 -name "*.log*"| wc -l)
if [[ $nb_log != 0 ]]
then
find "$element" -type f -mtime +30 -delete
echo "Successfull!"
else
echo "No log to clean !"
fi
done
allows to include multiple directory where to delete files
rep_log=("/etc/var/log" "/test/nginx/log")
we fill the var: we'r doing a search (in the directory provided) for files which are older than 30 days and whose name contains at least .log. Then counts the number of files.
nb_log=$(find "$element" -type f -mtime +30 -name "*.log*"| wc -l)
we then check if there is a result other than 0 (posisitive), if yes we delete
find "$element" -type f -mtime +30 -delete

For delete file older than X days you can use this command and schedule it in /etc/crontab
find /PATH/TO/LOG/* -mtime +10 | xargs -d '\n' rm
or
find /PATH/TO/LOG/* -type f -mtime +10 -exec rm -f {} \

Related

Find all files containing the filename of specific date range on Terminal/Linux

I have a surveillance camera which is capturing image base on my given condition. The images are saved on my Linux. Image naming convention are given below-
CAPTURE04.YYYYMMDDHHMMSS.jpg
The directory contains the following files -
CAPTURE04.20171020080501.jpg
CAPTURE04.20171021101309.jpg
CAPTURE04.20171021101913.jpg
CAPTURE04.20171021102517.jpg
CAPTURE04.20171021103422.jpg
CAPTURE04.20171022103909.jpg
CAPTURE04.20171022104512.jpg
CAPTURE04.20171022105604.jpg
CAPTURE04.20171022110101.jpg
CAPTURE04.20171022112513.jpg ... and so on.
However, Actually, now I'm trying to find a way to get all files between a specific date time (filename) range by using the terminal command.
Note: Need to follow the filename (YYYYMMDDHHMMSS), not the file created/modified time.
Such as I need to get all files whose file name is between 2017-10-20 08:30:00 and 2017-10-22 09:30:00
I'm trying and searching google around and got the following command -
find -type f -newermt "2017-10-20 08:30:00" \! -newermt "2017-10-22 09:30:00" -name '*.jpg'
It returns the files which are created/modified on that given date range. But I need to find files base on the given filenames range. So I think it does not work on my condition.
Also trying with the following command-
find . -maxdepth 1 -size +1c -type f \( -name 'CAPTURE04.20171020083000*.jpg' -o -name 'CAPTURE04.2017102209300*.jpg' \) | sort -n
This is not working.. :(
Please help me to write the actual command. Thanks, in advance.
Complete find + bash solution:
find . -type f -regextype posix-egrep -regex ".*CAPTURE04\.[0-9]{14}\.jpg" -exec bash -c \
'fn=${0##*/}; d=${fn:10:-4};
[[ $d -ge 20171020083000 && $d -le 20171022093000 ]] && echo "$0"' {} \;
fn=${0##*/} - obtaining file basename
d=${fn:10:-4} - extracting datetime section from the file's basename
[[ $d -ge 20171020083000 && $d -le 20171022093000 ]] && echo "$0" - print the filepath only if its datetime "section" is in specified range
One way(bash), not an elegant one:
ls CAPTURE04.2017102008{30..59}*.jpg CAPTURE04.2017102009{00..30}*.jpg 2>/dev/null
as maxdepth option is used means all files are in current directory so can be done in a loop with globs
for file in CAPTURE04.201710{20..22}*.jpg; do
if [[ $file > CAPTURE04.20171020083000 && $file < CAPTURE04.2017102209300 ]]; then
... do something with "$file"
fi
done

How to get echo to print only deleted file paths?

I'm trying to write a script to create mysqldumps daily in a directory as well as check all the backups in that directory and remove any older than 7 days that is going to run on cron.
So my functions work correctly, it's just my last echo command that is not doing what I want it to. This is what I have so far:
DBNAME=database
DATE=`date +\%Y-\%m-\%d_\%H\%M`
SQLFILE=$DBNAME-${DATE}.sql
curr_dir=$1
#MAIN
mysqldump -u root -ppassword --databases $DBNAME > $SQLFILE
echo "$SQLFILE has been successfully created."
#Remove files older than 7 days
for filepath in "$curr_dir"*
do
find "$filepath" -mtime +7 -type f -delete
echo "$filepath has been deleted."
done
exit
So the backup creations and removal of old files both work. But, my problem is that echo "$filepath has been deleted." is printing all files in the directory instead of just the files older than 7 days that were deleted. Where am I going wrong here?
EDIT (Full solution):
This is the full solution that wound up working for me using everyone's advice from the answers and comments. This works for cron jobs. I had to specify the main function's output filepath because the files were being created in the root directory instead of the path specified in Argument $1.
Thank you everyone for the help! The if statement also checks whether or not $1 is the specified directory I want files to be deleted in.
#Variables
DBNAME=database
DATE=`date +\%Y-\%m-\%d_\%H\%M`
SQLFILE=$DBNAME-${DATE}.sql
curr_dir=$1
#MAIN
mysqldump -u root -ppassword --databases $DBNAME > /path/to/db-backups/directory/$SQLFILE
echo "$SQLFILE has been successfully created."
#Remove files older than 7 days
for filepath in "$curr_dir"*
do
if [[ $1 = "/path/to/db-backups/directory" ]]; then
find "$filepath" -mtime +7 -type f -delete -exec sh -c 'printf "%s has been deleted.\n" "$#"' _ {} +
fi
done
exit
You can merge the echo into the find:
find "$filepath" -mtime +7 -type f -delete -exec echo '{}' "has been deleted." \;
The -delete option is just a shortcut for -exec rm '{}' \; and all the -exec commands are run in the sequence you specify them in.

Shell script to loop and delete

Could someone help me on this.I have below folder structure as shown below .I want to loop through every folder inside the backuptest and delete all the folders except today date folder.i want it run as a cron job
Use find for this:
today="$(date +%Y-%m-%d)"
find /path/to/backuptest/Server* -mindepth 1 -maxdepth 1 -type d -not -name "$today" -exec rm -R {} \;
Edit
To not delete directories other than those containing a date structure, use something like
find /path/to/backuptest/Server* -mindepth 1 -maxdepth 1 -type d -regex ".*2016-[0-1]*[0-9]-[0-3][0-9]$" -not -name "$today"
You can get today's date in whatever format you require via the date command. For example,
TODAY=$(date +%Y-%m-%d)
You can loop over the subfolders you want with a simple wildcard match:
for d in /path/to/backuptest/*/*; do
# ...
done
You can strip the directory portion from a file name with the basename command:
name=$(basename path/to/file)
You can glue that together something like this:
#!/bin/bash
TODAY=$(date +%Y-%m-%d)
for d in /path/to/backuptest/*/*; do
test "$(basename "$d")" = "$TODAY" || rm -rf "$d"
done
Update:
If you don't actually want to purge all subfolders except today's, but rather only those matching some particular name pattern, then one way to accomplish that would be to insert that pattern into the glob in the for command. For example, here
for d in /path/to/backuptest/*/+([0-9])-+([0-9])-+([0-9]); do
test "$(basename "$d")" = "$TODAY" || rm -rf "$d"
done
the only files / directories considered for deletion are those whose names consist of three nonempty, hyphen-separated strings of decimal digits. One could write patterns that more precisely match date string format if one preferred, but it does get messier the more discriminating you want the pattern to be.
You can do it with find:
set date=`date +%Y-%m-%d`
find backuptest -type d -not -name $date -not -name "backuptest" -not -name "Server*" -exec rm -rf {} \;
This:
find backuptest -type d -not -name $date -not -name "backuptest" -not -name "Server*"
will look for directories name different than:
backuptest
Server*
$date -> current date
and remove them with:
rm -rf

Linux command to check new files in file system

We have linux machine we would like to check what new files have been added between a certain date range.
I only have SSH access to this box and it's openSUSE 11.1
Is there some sort of command that can give me a list of files that have been added to the filesystem between say 04/05/2011 and 05/05/2011
Thanks
Regards
Gabriel
There are bunch of ways for doing that.
First one:
start_date=201105040000
end_date=201105042359
touch -t ${start_date} start
touch -t ${end_date} end
find /you/path -type f -name '*you*pattern*' -newer start ! -newer end -exec ls -s {} \;
Second one:
find files modified between 20 and 21 days ago:
find -ctime +20 -ctime -21
finds files modified between 2500 and 2800 minutes ago:
find -cmin +2500 -cmin -2800
And read this topic too.
Well, you could use find to get a list of all the files that were last-modified in a certain time window, but that isn't quite what you want. I don't think you can tell just from a file's metadata when it came into existence.
Edit: To list the files along with their modification dates, you can pipe the output of find through xargs to run ls -l on all the files, which will show the modification time.
find /somepath -type f ... -print0 | xargs -0 -- ls -l
I misunderstood your question. Depending on what filesystem you are using, it may or may not store creation time.
My understanding is that ext2/3/4 do not store creation time, but modified, changed (status, which is slightly different), and access times are.
Fat32 on the other hand does contain creation timestamps IIRC.
If you are using an ext filesystem, you have two options it seems:
1.Settle for finding all of the files that were modified between two dates (which will include created files, but also files that were just edited). You could do this using find.
2.Create a script/cronjob that will document the contents of your filesystem at some interval, e.g.
find / > filesystem.$(date +%s).log
and then run diffs to see what has been added. This, of course, would prevent you from looking backwards to time before you started making these logs.
You can try one of these:
find -newerct "1 Aug 2013" ! -newerct "1 Sep 2013" -ls
find . -mtime $(date +%s -d"Jan 1, 2013 23:59:59") -mtime $(date +%s -d"Jan 2, 2016 23:59:59")
find /media/WD/backup/osool/olddata/ -newermt 20120101T1200 -not -newermt 20130101T1400
find . -mtime +1 -mtime -3
find . -mtime +1 -mtime -3 > files_from_yesterday.txt 2>&1
find . -mtime +1 -mtime -3 -ls > files_from_yesterday.txt 2>&1
touch -t 200506011200 first
touch -t 200507121200 last
find / -newer first ! -newer last
#!/bin/bash
for i in `find Your_Mail_Dir/ -newermt "2011-01-01" ! -newermt "2011-12-31"`; do
mv $i /moved_emails_dir/
Hope this helps.

Find files older than X days excluding some other files

i'm trying to write a shell script, for linux and solaris, that finds some specific files older than X days and then deletes them. the trick is that during this process there are a couple of files that must not be deleted.
for example from the following list of files i need to delete *.zip and keep *.log and *.something.*
1.zip
2.zip
3.log
prefix.something.suffix
finding the files and feeding them to rm was easy, but i'm having difficulties in excluding the files from the deletion list.
experimenting around i discovered one can benefit from multiple complex expressions grouped with logical operators like this:
find -L path -type f \( -name '*.log' \) -a ! \( -name '*.zip' -o -name '*something*' \) -mtime +3
cheers,
G
or you could do this:
find /appl/ftp -type f -mtime +30 |grep -vf [exclude_file] | xargs rm -rf;
I needed to find a way to provide a hard coded list of exclude files to not remove, but remove everything else that was older than 30 days. Here is a little script to perform a remove of all files older that 30 days, except files that are listed in the [exclude_file].
EXCL_FILES=`/bin/cat [exclude_file]`;
RM_FILE=`/usr/bin/find [path] -type f -mtime +30`;
for I in $RM_FILES;
do
for J in $EXCL_FILES;
do
grep $J $I;
if [[ $? == 0 ]]; then
/bin/rm $I;
if [[ $? != 0 ]]; then echo "PROBLEM: Could not remove $I"; exit 1; fi;
fi;
done;
done;

Resources