Copy files with date/time range in filename - linux

Question

I have a bash script, which contains the following lines:
for ((iTime=starttime;iTime<=endtime;iTime++))
do
find . -name "*${iTime}*" -exec cp --parents \{\} ${dst} \;
done
I have a structure with a few folders including subfolders and many files at the bottom of the tree. These files are labeled with date and time info in the filename, like "filename_2021063015300000_suffix". The time is in format yyyymmddhhmmss and two digits for 1/10 and 1/100 seconds. I have a lot of files, which means, that my approach is very slow. The files have a time distance of a few minutes, so only a couple of files (e.g. 10 per subfolder out of >10000) should be copied.
How can i find all the files in the time range and get them all in one find and copy command? Maybe get a list of all the files to copy with one find command and then copy the list of filepathes? But how can i do this?

Answer 1

If your time span is reasonably limited, just inline the acceptable file names into the single find command.
find . \( -false $(for ((iTime=starttime;iTime<=endtime;iTime++)); do printf ' %s' -o -name "*$iTime*"; done) \) -exec cp --parents \{\} ${dst} \;
The initial -false predicate inside the parentheses is just to simplify the following predicates so that they can all start with -o -name.
This could end up with an "argument list too long" error if your list of times is long, though. Perhaps a more robust solution is to pass the time resolution into the command.
find . -type f -exec bash -c '
for f; do
for ((iTime=starttime;iTime<=endtime;iTime++)); do
if [[ $f == *"$iTime"* ]]; then
cp --parents "$f" "$0"
break
fi
done' "$dst" {} +
The script inside -exec could probably be more elegant; if your file names have reasonably regular format, maybe just extract the timestamp and compare it numerically to check whether it's in range. Perhaps also notice how we abuse the $0 parameter after bash -c '...' to pass in the value of $dst.

Answer 2

Lose the find. I created -
filename_2020063015300000_suffix
filename_2021053015300000_suffix
filename_2021063015300000_suffix
filename_2022063015300000_suffix
foo/filename_2021053015312345_suffix
bar/baz/filename_2021053015310101_suffix
So if I execute
starttime=2021000000000000
endtime=2022000000000000
shopt -s globstar
for f in **/*_[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]_*; do # for all these
ts=${f//[^0-9]/} # trim to date
(( ts >= starttime )) || continue # skip too old
(( ts <= endtime )) || continue # skip too new
echo "$f" # list matches
done | xargs -I{} echo cp {} /new/dir/ # pass to xargs
I get
cp bar/baz/filename_2021053015310101_suffix /new/dir/
cp filename_2021053015300000_suffix /new/dir/
cp filename_2021063015300000_suffix /new/dir/
cp foo/filename_2021053015312345_suffix /new/dir/
There are ways to simplify that glob. If you use extglob you can make it shorter, and check more carefully with a regex - for example,
shopt -s globstar extglob
for f in **/*_+([0-9])_*; do
[[ "$f" =~ _[0-9]{16}_ ]] || continue;
It starts looking complicated and hard to maintain for the next guy, though.

Answer 3

Try these, replace the dst, starttime, endtime in your case, both work for me on Ubuntu16.04.
find . -type f -regextype sed -regex "[^_]*_[0-9]\{16\}_[^_]*" -exec bash -c 'dt=$(echo "$0" | grep -oP "\d{16}"); [ "$dt" -gt "$2" ] && [ "$dt" -lt "$3" ] && cp -p "$0" "$1"' {} 'dst/' 'starttime' 'endtime' \;
$0 is filename which contain the datetime, $1 is dst directory path, $2 is starttime, $3 is endtime
Or
find . -type f -regextype sed -regex "[^_]*_[0-9]\{16\}_[^_]*" | awk -v dst='/tmp/test_find/' '{if (0 == system("[ $(echo \"" $0 "\"" " | grep -oP \"" "(?<=_)\\d+(?=_)\") -gt starttime ] && [ $(echo \"" $0 "\"" " | grep -oP \"" "(?<=_)\\d+(?=_)\") -lt endtime ]")) {system("cp -p " $0 " " dst)}}'
Both of them, first, use find to find the file name which has the pattern like _2021063015300000_ (maybe this has 16 digital but you say this pattern format yyyymmddhhmmss only has 14 digital) with sed regex.
Then use -exec bash -c "get datetime in filename, compare them with times, and exec cp action"
Or use awk to get the datetime and compare with start or end time by system command, and will execute cp to dst directory at last also by system command.
PS. this pattern are dependent the filename which only has the datetime between two _.