I have a surveillance camera which is capturing image base on my given condition. The images are saved on my Linux. Image naming convention are given below-
CAPTURE04.YYYYMMDDHHMMSS.jpg
The directory contains the following files -
CAPTURE04.20171020080501.jpg
CAPTURE04.20171021101309.jpg
CAPTURE04.20171021101913.jpg
CAPTURE04.20171021102517.jpg
CAPTURE04.20171021103422.jpg
CAPTURE04.20171022103909.jpg
CAPTURE04.20171022104512.jpg
CAPTURE04.20171022105604.jpg
CAPTURE04.20171022110101.jpg
CAPTURE04.20171022112513.jpg ... and so on.
However, Actually, now I'm trying to find a way to get all files between a specific date time (filename) range by using the terminal command.
Note: Need to follow the filename (YYYYMMDDHHMMSS), not the file created/modified time.
Such as I need to get all files whose file name is between 2017-10-20 08:30:00 and 2017-10-22 09:30:00
I'm trying and searching google around and got the following command -
find -type f -newermt "2017-10-20 08:30:00" \! -newermt "2017-10-22 09:30:00" -name '*.jpg'
It returns the files which are created/modified on that given date range. But I need to find files base on the given filenames range. So I think it does not work on my condition.
Also trying with the following command-
find . -maxdepth 1 -size +1c -type f \( -name 'CAPTURE04.20171020083000*.jpg' -o -name 'CAPTURE04.2017102209300*.jpg' \) | sort -n
This is not working.. :(
Please help me to write the actual command. Thanks, in advance.
Complete find + bash solution:
find . -type f -regextype posix-egrep -regex ".*CAPTURE04\.[0-9]{14}\.jpg" -exec bash -c \
'fn=${0##*/}; d=${fn:10:-4};
[[ $d -ge 20171020083000 && $d -le 20171022093000 ]] && echo "$0"' {} \;
fn=${0##*/} - obtaining file basename
d=${fn:10:-4} - extracting datetime section from the file's basename
[[ $d -ge 20171020083000 && $d -le 20171022093000 ]] && echo "$0" - print the filepath only if its datetime "section" is in specified range
One way(bash), not an elegant one:
ls CAPTURE04.2017102008{30..59}*.jpg CAPTURE04.2017102009{00..30}*.jpg 2>/dev/null
as maxdepth option is used means all files are in current directory so can be done in a loop with globs
for file in CAPTURE04.201710{20..22}*.jpg; do
if [[ $file > CAPTURE04.20171020083000 && $file < CAPTURE04.2017102209300 ]]; then
... do something with "$file"
fi
done
Related
I'm new to bash, I have a task to delete all files older than 30 days, I can figure this out based on the files name Y_M_D.ext 2019_04_30.txt.
I know I can list all files with ls in a the folder containing the files. I know I can get todays date with $ date and can configure that to match the file format $ date "+%Y_%m_%d"
I know I can delete files using rm.
How do I tie all this together into a bash script that deletes files older than 30 days from today?
In pseudo-python code I guess it would look like:
for file in folder:
if file.name to date > 30 day from now:
delete file
I am by no means a systems administrator, but you could consider a simple shell script along the lines of:
# Generate the date in the proper format
discriminant=$(date -d "30 days ago" "+%Y_%m_%d")
# Find files based on the filename pattern and test against the date.
find . -type f -maxdepth 1 -name "*_*_*.txt" -printf "%P\n" |
while IFS= read -r FILE; do
if [ "${discriminant}" ">" "${FILE%.*}" ]; then
echo "${FILE}";
fi
done
Note that this is will probably be considered a "layman" solution by a professional. Maybe this is handled better by awk, which I am unfortunately not accustomed to using.
Here is another solution to delete log files older than 30 days:
#!/bin/sh
# A table that contains the path of directories to clean
rep_log=("/etc/var/log" "/test/nginx/log")
echo "Cleaning logs - $(date)."
#loop for each path provided by rep_log
for element in "${rep_log[#]}"
do
#display the directory
echo "$element";
nb_log=$(find "$element" -type f -mtime +30 -name "*.log*"| wc -l)
if [[ $nb_log != 0 ]]
then
find "$element" -type f -mtime +30 -delete
echo "Successfull!"
else
echo "No log to clean !"
fi
done
allows to include multiple directory where to delete files
rep_log=("/etc/var/log" "/test/nginx/log")
we fill the var: we'r doing a search (in the directory provided) for files which are older than 30 days and whose name contains at least .log. Then counts the number of files.
nb_log=$(find "$element" -type f -mtime +30 -name "*.log*"| wc -l)
we then check if there is a result other than 0 (posisitive), if yes we delete
find "$element" -type f -mtime +30 -delete
For delete file older than X days you can use this command and schedule it in /etc/crontab
find /PATH/TO/LOG/* -mtime +10 | xargs -d '\n' rm
or
find /PATH/TO/LOG/* -type f -mtime +10 -exec rm -f {} \
Can you please suggest efficient way to move files from one location to their sub directory in a loop.
Ex:
/MY_PATH/User1/1234/Daily/abc.txt to /MY_PATH/User1/1234/Daily/Archive/abc.txt
/MY_PATH/User2/3456/Daily/def.txt to /MY_PATH/User2/3456/Daily/Archive/def.txt
/MY_PATH/User1/1111/Daily/hij.txt to /MY_PATH/User1/1111/Daily/Archive/hij.txt
/MY_PATH/User2/2222/Daily/def.txt to /MY_PATH/User2/2222/Daily/Archive/def.txt
I started in this way, but need your suggestions and best way to write it:
#!/bin/bash
dir1="/MyPath/"
subs= `ls $dir1`
for i in $subs; do
mv $dir1/$i/*/Daily $dir1/$i/*/Daily/Archive
done
My one line bash
for dir in $(
find MY_PATH -mindepth 3 -maxdepth 3 -type d -name Daily
);do
mkdir -p $dir/Archives
find $dir -maxdepth 1 -mindepth 1 ! -name Archives \
-exec mv -t $dir/Archives {} +
done
To quickly test:
mkdir -p MY_PATH/User{1,2,3,4}/{1234,2346,3333,2323}/Daily
touch MY_PATH/User{1,2,3,4}/{1234,2346,3333,2323}/Daily/{abc,bcd,def,feg,fds}.txt
for dir in $( find MY_PATH -mindepth 3 -maxdepth 3 -type d -name Daily );do
mkdir -p $dir/Archives; find $dir -maxdepth 1 -mindepth 1 ! -name Archives \
-exec mv -t $dir/Archives {} + ; done
ls -lR MY_PATH
This seem match OP's request
For more robust solution
There is a solution wich work with spaces somewhere in path...
Edited to include #mklement0's well pointed suggestion.
while IFS= read dir;do
mkdir -p "$dir"/Archives
find "$dir" -maxdepth 1 -mindepth 1 ! -name Archives \
-exec mv -t "$dir/Archives" {} +
done < <(
find MY_PATH -mindepth 3 -maxdepth 3 -type d -name Daily
)
Same demo;
mkdir -p MY_PATH/User{1,2,3,"4 3"}/{1234,"23 6",3333,2323}/Daily
touch MY_PATH/User{1,2,3,"4 3"}/{1234,"23 6",3333,2323}/Daily/{abc,"b c",def,hgz0}.txt
while read dir;do mkdir -p "$dir"/Archives;find "$dir" -maxdepth 1 -mindepth 1 \
! -name Archives -exec mv -t "$dir/Archives" {} +; done < <(
find MY_PATH -mindepth 3 -maxdepth 3 -type d -name Daily )
ls -lR MY_PATH
Assuming the directory structure is as you have shown in your examples, i.e.
MY_PATH/
subdir-level-1/
subdir-level-2/
Daily/
files
Archive/
Here's what you can do:
shopt -s nullglob # defend against globbing failure -- inspired by mklement0's answer
root="/MyPath"
for dir in "${root}"/*/*/Daily/; do
mkdir -p "${dir}/Archive" # if Archive might not exist; to be pedantic you should look at David C. Rankin's answer for error handling, but usually we know what we're doing so that's not necessary
find "${dir}" -maxdepth 1 -type f -print0 | xargs -0 mv -t "${dir}/Archive"
done
The reason I use find and xargs is to save a few processes; you can as well move files in each ${dir} one by one.
Update: #mklement0 suggested that find "${dir}" -maxdepth 1 -type f -print0 | xargs -0 mv -t "${dir}/Archive" can be further improved to
find "${dir}" -maxdepth 1 -type f -exec mv -t "${dir}/Archive" +
which is a very good point.
Try the following:
dir1="/MyPath"
for d in "$dir1"/*/*/Daily/; do
[[ -d $d ]] || break # break, if no subdirectories match
for f in "$d"/*; do # loop over files in */*/Daily/
[[ -f "$f" ]] || continue # skip non-files or if nothing matches
mv "$f" "$d"/Archive/
done
done
"$dir1"*/*/Daily/ matches all grandchild subdirectories of $dir1; thanks to the terminating /, only directories match; note that, as a result, $d ends in /.
Note that $d therefore ends in /, and, strictly speaking, needs no / later on when synthesizing paths with it (e.g., "$d"/*), but doing so does no harm and helps readability, as #4ae1e1 points out in a comment.
[[ -d $d ]] || break ensures that the loop is exited if no grandchild directories match (by default, a glob (pattern) that has no matches is passed as is to the loop).
for f in "$d"* loops over all entries (files and/or subdirs.) in $d:
[[ -f "$f" ]] || continue ensures that only files are processed or, in the event that nothing matches, the loop is exited.
mv "$f" "$d"/Archive/ then moves each file to subdir. Archive.
You need to check for, and if not present, create the destination directory before moving the file to Archive. If you cannot create the directory (due to permissions or otherwise), you skip the move. The following does not assume any limitation on depth, but will omit any directory containing Archive as an intermediate subdirectory:
oldifs="$IFS"
IFS=$'\n'
for i in $(find /MY_PATH -type f); do
[[ "$i" =~ Archive ]] && continue
[ -d "${i%/*}/Archive" ] || mkdir -p "${i%/*}/Archive"
[ -d "${i%/*}/Archive" ] || {
printf "error: unable to create '%s'\n" "${i%/*}/Archive"
continue
}
mv -fv "$i" "${i/Daily/Daily\/Archive}"
done
IFS="$oldifs"
Output when run
$ bash archive_daily.sh
mv -fv /MY_PATH/User1/1111/Daily/hij.txt /MY_PATH/User1/1111/Daily/Archive/hij.txt
mv -fv /MY_PATH/User1/1234/Daily/abc.txt /MY_PATH/User1/1234/Daily/Archive/abc.txt
mv -fv /MY_PATH/User2/3456/Daily/def.txt /MY_PATH/User2/3456/Daily/Archive/def.txt
mv -fv /MY_PATH/User2/2222/Daily/def.txt /MY_PATH/User2/2222/Daily/Archive/def.txt
Note: you can limit/tighten the file selection by adjusting the call to find populating the for loop (e.g. -name or -iname). This simply checks/moves every file to its Archive folder. To limit to only files with the .txt extension, you can specify find /MY_PATH -type f -name "*.txt". To limit to only files in the /MY_PATH/User1 and /MY_PATH/User2directories with a .txt extension, use find /MY_PATH/User[12] -type f -name "*.txt".
Note2: when looping on filenames, the paths & filenames should not contain non-standard characters for the current locale. Certainly you should not have the '\n' as a character in your filename. Setting IFS is required to protect against word splitting on spaces in either the path or filename.
Since you said efficient, anything with a subshell will fail in funny ways with lots of entries. You're better off using xargs:
#!/bin/bash
dir1="/MyPath/"
find $dir1 -name Daily -type d -depth 3 | while read i
do
pushd .
cd $i
mkdir Archive
find . -type f -depth 1 | xargs -J {} mv {} Archive
popd
done
The outer find will look for you Daily directories. It's very specific in that they have to be at a certain depth and directories, not regular files. The results gets piped into read, where each directory is entered, Archive is created, and files batch-copied with xargs ... mv. Complete file lists and directory lists are never stored in memory, so it scales very well.
I am trying to create a small bash script that essentially looks through a directory that includes hundreds of sub directories. in SOME of these subdirectories include a textfile.txt and a htmlfile.html where the names textfile and htmlfile are variable.
I only really care about sub directories that have both the .txt and the .html, all other subdirecories can be ignored.
I then want to list all the .html files and .txt files that are in the same sub directory
this seems like a pretty simple issue to solve but I am at a loss. all I can really get working is a line of code that outputs sub directories that have either a .html file or .txt with no association with the actual sub directory they are in, and I am pretty new at bash scripting so I can't go any further
#!/bin/bash
files="$(find ~/file/ -type f -name '*.txt' -or -name '*.html')"
for file in $files
do
echo $file
done
The following find command looks checks every subdirectory and, if it has both html and txt files, it lists all of them:
find . -type d -exec env d={} bash -c 'ls "$d"/*.html &>/dev/null && ls "$d"/*.txt &>/dev/null && ls "$d/"*.{html,txt}' \;
Explanation:
find . -type d
This looks for all subdirectories of the current directory.
-exec env d={} bash -c '...' \;
This sets the environment variable d to the value of the found subdirectory and then executes the bash command that is contained within the single quotes (see below).
ls "$d"/*.html &>/dev/null && ls "$d"/*.txt &>/dev/null && ls "$d/"*.{html,txt}
This is the bash command that is executed. It consists of three statements and-ed together. The first checks to see if directory d has any html files. If so, the second statement runs and it checks to see if there are any txt files. If so, the last statement is executed and it lists all html and txt files in the directory d.
This command is safe for all file and directory names containing spaces, tabs, or other difficult characters.
You could do it by searching recursively with the globstar option:
shopt -s globstar
for file in **; do
if [[ -d $file ]]; then
for sub_file in "$file"/*; do
case "$sub_file" in
*.html)
html=1;;
*.txt)
txt=1;;
esac
done
[[ $html && $txt ]] && echo "$file"
html=""
txt=""
fi
done
You can make use of -o
#!/bin/bash
files=$(find ~/file/ -type f -name '*.txt' -o -name '*.html')
for file in $files
do
echo $file
done
#!/bin/bash
#A quick peek into a dir to see if there's at least one file that matches pattern
dir_has_file() { dir="$1"; pattern="$2";
[ -n "$(find "$dir" -maxdepth 1 -type f -name "$pattern" -print -quit)" ]
}
#Assumes there are no newline characters in the filenames, but will behave correctly with subdirectories that match *.html or *.txt
find "$1" -type d|\
while read d
do
dir_has_file "$d" '*.txt' &&
dir_has_file "$d" '*.html' &&
#Now print all the matching files
find "$d" -maxdepth 1 -type f -name '*.txt' -o -name '*.html'
done
This script takes the root directory to look into as the first argument ($1).
The test command is what you need to check for the existence of each file in each of the subdirs:
find . -type d -exec sh -c "if test -f {}/$file1 -a -f {}/$file2 ; then ls {}/*.{txt,html} ; fi" \;
where $file1 and $file2 are the two .txt and .html files you are looking for.
i have the following bash script on my server:
today=$(date +"%Y-%m-%d")
find /backups/www -type f -mtime -1|xargs tar uf /daily/backup-$today.tar
as you can see it creates backups of files modified/created in the last 24h. However if no files are found, it creates corrupted tar file. I would like to wrap it in if..fi statement so id doesn't create empty/corrupted tar files.
Can someone help me modify this script?
Thanks
You can check if result is ok then check if result is empty :
today=$(date +"%Y-%m-%d")
results=`find /backups/www -type f -mtime -1`
if [[ 0 == $? ]] ; then
if [[ -z $results ]] ; then
echo "No files found"
else
tar uf /daily/backup-$today.tar $results
fi
else
echo "Search failed"
fi
find /backups/www -type f -mtime -1 -exec tar uf /daily/backup-$today.tar {} +
Using -exec is preferable to xargs. There's no pipeline needed and it will handle file names with spaces, newlines, and other unusual characters without extra work. The {} at the end is a placeholder for the file names, and + marks the end of the -exec command (in case there were more arguments to find).
As a bonus it won't execute the command if no files are found.
One relatively simple trick would be this:
today=$(date +"%Y-%m-%d")
touch /backups/www/.timestamp
find /backups/www -type f -mtime -1|xargs tar uf /daily/backup-$today.tar
That way you're guaranteed to always find at least one file (and it's minimal in size).
xargs -r does nothing if there is no input.
i'm trying to write a shell script, for linux and solaris, that finds some specific files older than X days and then deletes them. the trick is that during this process there are a couple of files that must not be deleted.
for example from the following list of files i need to delete *.zip and keep *.log and *.something.*
1.zip
2.zip
3.log
prefix.something.suffix
finding the files and feeding them to rm was easy, but i'm having difficulties in excluding the files from the deletion list.
experimenting around i discovered one can benefit from multiple complex expressions grouped with logical operators like this:
find -L path -type f \( -name '*.log' \) -a ! \( -name '*.zip' -o -name '*something*' \) -mtime +3
cheers,
G
or you could do this:
find /appl/ftp -type f -mtime +30 |grep -vf [exclude_file] | xargs rm -rf;
I needed to find a way to provide a hard coded list of exclude files to not remove, but remove everything else that was older than 30 days. Here is a little script to perform a remove of all files older that 30 days, except files that are listed in the [exclude_file].
EXCL_FILES=`/bin/cat [exclude_file]`;
RM_FILE=`/usr/bin/find [path] -type f -mtime +30`;
for I in $RM_FILES;
do
for J in $EXCL_FILES;
do
grep $J $I;
if [[ $? == 0 ]]; then
/bin/rm $I;
if [[ $? != 0 ]]; then echo "PROBLEM: Could not remove $I"; exit 1; fi;
fi;
done;
done;