I've been set a mini project to run a bash script to archive and remove files that are older than 'x' number of days. The file will be archived in the /nfs/archive directory and they need to be compressed (TAR) or removed... e.g. '/test.sh 15' would remove files older than 15 days. Moreover, I also need to input some validation checking before removing files...
My code so far:
> #!/bin/bash
>
> #ProjectEssentials:
>
> # TAR: allows you to back up files
> # cronjob: schedule taks
> # command: find . -mtime +('x') -exec rm {} \; this will remove files older than 'x' number of days
>
> find /Users/alimohamed/downloads/nfs/CAMERA -type f -name '*.mov'
> -mtime +10 -exec mv {} /Users/limohamed/downloads/nfs/archive/ \;
>
> # TAR: This will allow for the compression
>
> tar -cvzf doc.tar.gz /Users/alimohamed/downloads/nfs/archive/
>
> # Backup before removing files 'cp filename{,.bak}'? find /Users/alimohamed/downloads/nfs/CAMERA -type f name '*.mov' -mtime +30
> -exec rm {} \; ~
Any help would much appreciated!!
Modified script to fix few typos. Note backup file will have a YYYY-MM-DD, to allow for multiple backups (limited to one backup per day).Using TOP to make script generic - work on any account.
X=15 # Number of days
# Move old files (>=X days) to archive, via work folder
TOP=~/downloads/nfs
mkdir -p "$TOP/work"
find $TOP/CAMERA -type f -name '*.mov' -mtime +"$X" -exec mv {} "$WORK/work" \;
# Create daily backup (note YYYY-MM-DD in file name from work folder
tar -cvzf $TOP/archive/doc.$(date +%Y-%m-%d).tar.gz -C "$TOP/work" .
# Remove all files that were backed-up, If needed
find "$TOP/work" -type f -name '*.mov' -exec rm {} \; ~
Related
Can i find any directories with a condition like older than 3 days
and zip them then delete the directories?
I have 2 solutions.
zip all directories in 1 zip under working directory
I tried
zip -rm ${WORKDIR}/date +%Y%m%d -d "${DAY_TO_ZIP} days ago".zipfind ${WORKDIR} -daystart -mtime +${DAY_TO_ZIP} -type d ! -name "*.zip"``
this command will zip all files include non-directory file.
1 directory 1 zip same path with a directory
thank you very much
Execute bellow command to find all directory older than 3 days and zip all file
# find / -mtime +3 -type d -exec zip -r zipfile.zip {} +
-mtime +3 means you are looking for a file modified 3 days ago.
-mtime -3 means less than 3 days.
-mtime 3 If you skip + or – it means exactly 3 days.
Finally If you delete all directory then execute bellow command
# find / -mtime +3 -type d -exec rm -f {} \;
find ./ -mtime +x -print -exec gzip {} ;
I have the following directories:
P922_101
P922_102
.
.
Each directory, for instance P922_101 has following subdirectories:
140311_AH8MHGADXX 140401_AH8CU4ADXX
Each subdirectory, for instance 140311_AH8MHGADXX has the following files:
1_140311_AH8MH_P922_101_1.fastq.gz 1_140311_AH8MH_P922_101_2.fastq.gz
2_140311_AH8MH_P922_101_1.fastq.gz 2_140311_AH8MH_P922_101_2.fastq.gz
And files in 140401_AH8CU4ADXX are:
1_140401_AH8CU_P922_101_1.fastq.gz 1_140401_AH8CU_P922_4001_2.fastq.gz
2_140401_AH8CU_P922_101_1.fastq.gz 2_140401_AH8CU_P922_4001_2.fastq.gz
I want to do 'cat' for the files in the subdirectories in the following way:
cat 1_140311_AH8MH_P922_101_1.fastq.gz 2_140311_AH8MH_P922_101_1.fastq.gz
1_140401_AH8CU_P922_101_1.fastq.gz 2_140401_AH8CU_P922_101_1.fastq.gz > P922_101_1.fastq.gz
which means that files ending with _1.fastq.gz should be concatenated into a single file and files ending with _2.fatsq.gz into another file.
It should be run for all files in subdirectories in all directories. Could someone give a linux solution to do this?
Since they're compressed, you should probably use gzip -dc (decompress and write to stdout) -
find /somePath -type f -name "*.fastq.gz" -exec gzip -dc {} \; | \
tee -a /someOutFolder/out.txt
You can use find for this:
find /top/path -mindepth 2 -type f -name "*_1.fastq.gz" -exec cat {} \; > one_file
find /top/path -mindepth 2 -type f -name "*_2.fastq.gz" -exec cat {} \; > another_file
This will look for all the files starting from /top/path and having a name matching the pattern _1.fastq.gz / _2.fastq.gz and cat them into the desired file. -mindepth 2 makes find look for files that are at least under the current directory; this way, files in /top/path won't be matched.
Note that you will probably need zcat instead of cat, for gz files.
As you keep adding details in comments, let's see what else we can do:
Say you have the list of directories in a file directories_list, each line containing one:
while read directory
do
find $directory -mindepth 2 -type f -name "*_1.fastq.gz" -exec cat {} \; > $directory/output
done < directories_list
I want to move all files and directories are located on /etc/ that are older than 90 days to /old-etc directory but with the same structure in the source directory.
Thanks
Try doing this :
find /etc -mtime +90 -type f -exec bash -c 'install -D "$1" "/old-etc/$1" && rm -f "$1"' -- {} \;
I need to move files from ORIGIN and place them to DESTINATION accordingly to the information contained in text file "toto.txt"
I do NOT know how to code the part which says:
place these files accordingly with the information contained in toto.txt which states
the sub-folder structure on DESTINATION folder"
toto.txt conatins the folder structure of ORIGIN and the files must be moved accordingly to DESTINATION but with the original folder structure location.
# My working Paths
MY_DIR1="/media/nss/MBVOL1/TEST/ORIGIN"
MY_DIR2="/media/nss/MBVOL1/TEST/DESTINATION"
# Flag files older than 1 day and list their name\full path to “TOTO” text file
echo "REPORT Created"
cd $MY_DIR1 && find . -mindepth 0 -maxdepth 40 -mtime +1 -type f > toto.txt
cp $MY_DIR1/toto.txt /$MY_DIR2
# Flag files older than 1 day then MOVE file to “DESTINATION” Folder
echo "FILES Moved"
find $MY_DIR1 -mindepth 0 -maxdepth 400 -type f -mtime +14 -exec mv '{}' $MY_DIR2 \;
Try this:
cd "$MY_DIR1"
# Duplicate directory structure
find . -type d -exec mkdir -p "$MY_DIR2"/{} \;
# move files older than 1 day
find . -type f -mtime +1 -exec mv {} "$MY_DIR2"/{} \;
You can combine them into one command:
find . -type d -exec mkdir -p "$MY_DIR2"/{} \; -o -type f -mtime +1 -exec mv {} "$MY_DIR2"/{} \;
Use something like this...
cat ${MY_DIR2}/toto.txt | while read FILE ; do
mv -v "${MY_DIR1}/${FILE}" "${MY_DIR2}"
done
I am trying to put together a script which will run through all the files on my server (under various subdirectories) , look for .jpeg files and run them through a translator which converts them to non progressive jpgs.
I have:
find /home/disk2/ -type f -iname "*.jpg"
Which finds all the files.
Then if it finds for example 1.jpg, I need to run:
/usr/bin/jpegtrans /file location/1.jpg > /file location/1.jpg.temp
The jpegtrans app converts the file to a temp file which needs to replace the original file.
So then I need to delete the original and rename 1.jpg.temp to 1.jpg
rm /file location/1.jpg
mv /file location/1.jpg.temp /file location/1.jpg
I can easily do this for single files but i need to do it for 100's on my server.
Use find with -exec:
find /home/disk2/ -type f -iname "*.jpg" -exec sh -c "/usr/bin/jpegtrans {} > {}.temp; mv -f {}.temp {}" \;
EDIT: For handling spaces in filenames, say:
find /home/disk2/ -type f -iname "*.jpg" -exec sh -c "/usr/bin/jpegtrans '{}' > '{}.temp'; mv -f '{}.temp' '{}'" \;