I want to create a script that will delete any files older than 7 days on a specified list of directories, but wondering what would be the best way to go about it...
I want to perform the following command on all directories specified:
find DIRECTORY_PATH -type f -mtime +7 -exec rm {} \;
Maybe an array holding a list of directories, and loop through each element in the array performing the find command on that?
Any help/advice would be appreciated.
You can directly store all the directories in a file, say dirs.txt and loop through it:
while read dir
do
find "$dir" -type f -mtime +7 -exec rm {} \;
done < dirs.txt
Related
I have some automated downloads in a proprietary linux distro.
They go to a temp scratch disk. I want to move them when they're finished to the main RAID array. The best way I can see to do this is to check the folders on the disk to see if the contents have changed in the last minute. If not then its probably finished downloading and then move it.
Assuming there could be hundreds of folders or just one in this location and its all going to the same place. Whats the best way to write this?
I can get a list of folder sizes with
du -h directory/name
The folders can contain multiple files anywhere from 1.5mb to 10GB
Temp Loc: /volume2/4TBScratch/Processing
Dest Loc when complete: /volume1/S/00 Landing
EDIT:
Using this:
find /volume2/4TBScratch/Processing -mindepth 1 -type d -not -mmin +10 -exec mv "{}" "/volume1/S/00 Landing" \;
find: `/volume2/4TBScratch/Processing/test': No such file or directory
4.3#
yet it DOES copy the relevant folders and all files. But the error worries me that something might go wrong in the future.... is it because there is multiple files and it's running the same move command for EACH file or folder in the root folder? But since it moves it all on the first iteration it cant find it on the next ones?
EDIT2:
Using Rsync
4.3# find /volume2/4TBScratch/Processing -mindepth 1 -type d -not -mmin +10 -exec rsync --remove-source-files "{}" "/volume1/S/00 Landing" \;
skipping directory newtest
skipping directory erw
RESOLVED: EDIT3
Resolved with the help in the comments below. Final script looks like this:
find /volume2/4TBScratch/Processing -mindepth 1 -type d -not -mmin +10 -exec rsync -a --remove-source-files "{}" "/volume1/S/00 Landing" \;
find /volume2/4TBScratch/Processing -depth -type d -empty -delete
rsync to move folders and files but leaves empty root dir
the next command finds empty folders and removes them.
Thanks all!
You can use GNU find with options -size for detecting files/folders of certain size and use mv with the -exec option to move to destination directory. The syntax is
find /volume2/4TBScratch/Processing -type d -maxdepth 1 -size -10G -exec mv "{}" "/volume1/S/00 Landing" \;
Using rsync
find /volume2/4TBScratch/Processing -type d -maxdepth 1 -size -10G -exec rsync --remove-source-files "{}" "/volume1/S/00 Landing" \;
The size with a - sign to indicate less than the mentioned size which in this case is 10GB. A note on each of the flags used
-type d -> For identifying only the folders from the source path.
-maxdepth 1 -> To look only on the current source directory and not
being recursive.
-exec -> Execute command following it.
Alternatively, if you want to find files that are last modified over a certain time(minutes), find has an option for -mmin which can be set to a value. E.g. -mmin -5 would return files modified five minutes ago.
So suggest adding it to your requirement, for x as you need and see if the directories are listed, then you can add the -exec option for moving the directories
find /volume2/4TBScratch/Processing -type d -maxdepth 1 -mmin -2 -size -10G
Refer to the GNU documentation for finding files according to size on how this works.
Note:- The double quotes("") are added to avoid Bash from splitting the names containing spaces.
What I want:
In a bash script: Find all files in current directory that contain a certain string "teststring" and cop them into a subfolder "./testfolder"
Found this to find the filenames which im looking for
find . -type f -print0 | xargs -0 grep -l "teststring"
..and this to copy found files to another folder (here selecting by strings in filename):
find . -type f -iname "stringinfilename" -exec cp {} ./testfolder/ \;
Whats the best way to combine both commands to achieve what I described at the top?
Just let find do both:
find . -name subdir -prune -o -type f -exec \
grep -q teststring "{}" \; -exec cp "{}" subdir \;
Note that things like this are much easier if you don't try to add to the directory you're working in. In other words, write to a sibling dir instead of writing to a subdirectory. If you want to wind up with the data in a subdir, mv it when you're done. That way, you don't have to worry about the prune (ie, you don't have to worry about find descending into the subdir and attempting to duplicate the work).
I need to remove any file in the directory that is older than 2 years old. It is very important that I keep the newest files and delete the old files.
I have searched and found this.
find /path/to/files* -mtime +365 -exec rm {} \;
Can I just multiply the number?
find /path/to/files* -mtime +1095 -exec rm {} \;
Is there a way to add a switch that will print the file name to the screen as it removes it? To make sure it is doing what I am expecting?
I have also found this:
find /rec -mtime +365 -print0 | xargs -0 rm -f
Is there a major difference between the two? Is one better than the other? What I have read says that xargs is faster. Would I be able to multiply the mtime number out to a 2nd or 3rd year?
And finally would would I be able to place the code as it is into a cron job that can run daily?
Thank you!
Can I just multiply the number?
find /path/to/files -mtime +1095 -exec rm {} \;
Yes. And to "echo" before you remove
find /path/to/files -mtime +1095 -print
Then the version with -exec rm {} \; to remove the files (when you are ready).
find /path/to/files* -mtime +1095 -exec rm {} \;
That should work fine, you can run a dry a run of this by simply listing the files that are found by the command:
find /path/to/files* -mtime +1095 -exec ls {} \;
To be safe though I would also add in a -type to ensure that other things dont get deleted:
find /path/to/files* -type f -mtime +1095 -exec rm {} \;
To answer the second part of your question.
Yes there is a major difference in using -exec or xargs.
-exec starts a new process of rm for every file found. This creates a lot of overhead and can seriously slow down Systems if you delete a lot of files.
xargs creates only as much rm processes as needed, as it creates a command line containing as much files as possible. So only a few rm processes are created.
But both are better than -delete, because delete is unsave
I want to copy the recently updated multiple file into another directory.
I am having 1.xml,2.xml,3.xml.... in this directory recently someone updated file or added new file into the directory,So i want to copy those files into the destination directory ..Its like synchronization of 2 directories.
For that I have tried below commend
find home/deployment/server/services/ -type f -mtime 1 | xargs cp /home/application/
and below one also
find home/deployment/server/services/ -type f -mtime 1 -exec cp /home/application/
I am not getting any file into destination after updating 1.xml file,So I have added new file 4.xml even that also not updating in destination directory.
How to process recently updated or newly added multiple files.
Thanks in advance.
Short answer:
use xargs to mv the "find" directory into another directory
Long answer: As I recall (not tested) for exec syntax is
find . -type f --mtime 1 -exec cp {} /destination/path/ +
"{}" is an argument which came from command "find"
For xargs
find . -type f --mtime 1 | xargs -0 -I {} cp {} /destination/path/
I do this often but use \; instead of + and usually -cnewer rather than -mtime.
\; executes the cp command on files individually instead of as a group.
+ executes as a group with as many paths as xterm will take. It may do this multiple time if there are a lot of files.
the \ in front of the ; option is required or bash will think it is the end of the command.
find ./ -mtime -1 -exec cp {} /path/ \; -print
Use the -print at the end to get a list of the files that were copied.
I've got a log file directory that has 82000 files and directories in it (about half and half).
I need to delete all the file and directories which are older than 3 days.
In a directory that has 37000 files in it, I was able to do this with:
find * -mtime +3 -exec rm {} \;
But with 82000 files/directories, I get the error:
/usr/bin/find: Argument list too long
How can I get around this error so that I can delete all files/directories that are older than 3 days?
To delete all files and directories within the current directory:
find . -mtime +3 | xargs rm -Rf
Or alternatively, more in line with the OP's original command:
find . -mtime +3 -exec rm -Rf -- {} \;
Can also use:
find . -mindepth 1 -mtime +3 -delete
To not delete target directory
Another solution for the original question, esp. useful if you want to remove only SOME of the older files in a folder, would be smth like this:
find . -name "*.sess" -mtime +100
and so on.. Quotes block shell wildcards, thus allowing you to "find" millions of files :)