FInd and copy multiple files that contain a pattern in Linux - linux

When I need to copy multiple files in the same dir I can just:
cp file{20..30} newLocation
But when I combine that with find, it doesn't work.
find . -name 'file{20..30}' -exec cp '{}' newLocation ';'
What am I doing wrong?

The {20..30} range syntax is a special feature of Bash's command-line parser. It is not part of standard POSIX globbing, such as find's -name test performs, and it's not even recognized by Bash in some contexts where you might like it to be.
You already know the simpler, more direct alternative that I would otherwise recommend for your example case. You could also do something like
find . '(' -name 'file2[0-9]' -o -name file30 ')' -exec cp '{}' newLocation ';'
, though that doesn't work very well if the endpoints of the range are determined dynamically.
If the point of using find is to avoid problems arising from some of the files not existing, then you might consider addressing it like this:
for f in file{20..30}; do
[[ -e "$f" ]] && cp "$f" newLocation
done

As mentioned from the other post/answer, enclosing the names with a ( and closing ) plus the -o and -name in between.
Something like this should be able to add those strings from your input file/string.
#!/usr/bin/env bash
format() {
local f
declare -ag files
for f; do
files+=( -o -name "$f" )
done
}
format file{10..20}
Now check the value of "${files[#]}"
declare -p files
Output
declare -a files=([0]="-o" [1]="-name" [2]="file10" [3]="-o" [4]="-name" [5]="file11" [6]="-o" [7]="-name" [8]="file12" [9]="-o" [10]="-name" [11]="file13" [12]="-o" [13]="-name" [14]="file14" [15]="-o" [16]="-name" [17]="file15" [18]="-o" [19]="-name" [20]="file16" [21]="-o" [22]="-name" [23]="file17" [24]="-o" [25]="-name" [26]="file18" [27]="-o" [28]="-name" [29]="file19" [30]="-o" [31]="-name" [32]="file20")
To see what would it look like when used as an input to find
printf '%s\n' "\( ${files[*]} \)"
Output
\( -o -name file10 -o -name file11 -o -name file12 -o -name file13 -o -name file14 -o -name file15 -o -name file16 -o -name file17 -o -name file18 -o -name file19 -o -name file20 \)
To use that array files as an input to find
find . -type f \( "${files[#]:1}" \) -exec bash -c 'echo cp -v -- "$#" /destination' _ {} +
The "${files[#]:1}" removes the leading -o
Remove the echo if you're satisfied with the output so files copying should occur.
Or just use globstar with nullglob'
#!/usr/bin/env bash
shopt -s globstar nullglob
cp -v ./**/file{10..20} /destination
The leading ./ means the current directory, It could be
/path/to/source/**/file{10..20}
where /path/to/source/ is the directory where the files in question are.

Related

Circumvent Argument list too long in script (for loop)

I've seen a few answers regarding this, but as a newbie, I don't really understand how to implement that in my script.
it should be pretty easy (for those who can stuff like this)
I'm using a simple
for f in "/drive1/"images*.{jpg,png}; do
but this is simply overloading and giving me
Argument list too long
How is this easiest solved?
Argument list too long workaroud
Argument list length is something limited by your config.
getconf ARG_MAX
2097152
But after discuss around differences between bash specifics and system (os) limitations (see comments from that other guy), this question seem wrong:
Regarding discuss on comments, OP tried something like:
ls "/simple path"/image*.{jpg,png} | wc -l
bash: /bin/ls: Argument list too long
This happen because of OS limitation, not bash!!
But tested with OP code, this work finely
for file in ./"simple path"/image*.{jpg,png} ;do echo -n a;done | wc -c
70980
Like:
printf "%c" ./"simple path"/image*.{jpg,png} | wc -c
Reduce line length by reducing fixed part:
First step: you could reduce argument length by:
cd "/drive1/"
ls images*.{jpg,png} | wc -l
But when number of file will grow, you'll be buggy again...
More general workaround:
find "/drive1/" -type f \( -name '*.jpg' -o -name '*.png' \) -exec myscript {} +
If you want this to NOT be recursive, you may add -maxdepth as 1st option:
find "/drive1/" -maxdepth 1 -type f \( -name '*.jpg' -o -name '*.png' \) \
-exec myscript {} +
There, myscript will by run with filenames as arguments. The command line for myscript is built up until it reaches a system-defined limit.
myscript /drive1/file1.jpg '/drive1/File Name2.png' /drive1/...
From man find:
-exec command {} +
This variant of the -exec action runs the specified command on
the selected files, but the command line is built by appending
each selected file name at the end; the total number of invoca‐
tions of the command will be much less than the number of
matched files. The command line is built in much the same way
that xargs builds its command lines. Only one instance of `{}'
Inscript sample
You could create your script like
#!/bin/bash
target=( "/drive1" "/Drive 2/Pictures" )
[ "$1" = "--run" ] && exec find "${target[#]}" -type f \( -name '*.jpg' -o \
-name '*.png' \) -exec $0 {} +
for file ;do
echo Process "$file"
done
Then you have to run this with --run as argument.
work with any number of files! (Recursively! See maxdepth option)
permit many target
permit spaces and special characters in file and directrories names
you could run same script directly on files, without --run:
./myscript hello world 'hello world'
Process hello
Process world
Process hello world
Using pure bash
Using arrays, you could do things like:
allfiles=( "/drive 1"/images*.{jpg,png} )
[ -f "$allfiles" ] || { echo No file found.; exit ;}
echo Number of files: ${#allfiles[#]}
for file in "${allfiles[#]}";do
echo Process "$file"
done
There's also a while read loop:
find "/drive1/" -maxdepth 1 -mindepth 1 -type f \( -name '*.jpg' -o -name '*.png' \) |
while IFS= read -r file; do
or with zero terminated files:
find "/drive1/" -maxdepth 1 -mindepth 1 -type f \( -name '*.jpg' -o -name '*.png' \) -print0 |
while IFS= read -r -d '' file; do

find command: delete everything but one folder

I have this command:
find ~/Desktop/testrm -mindepth 1 -path ~/Desktop/testrm/.snapshot -o -mtime +2 -prune -exec rm -rf {} +
I want it to work as is, but it must avoid to remove a specific directory ($ROOT_DIR/$DATA_DIR).
it must remove the files inside the directory but not the directory itself
the flag "r" in rm is needed because it has to delete other directories
-prune is not suitable since it will discard the content and also sub directories
You can exclude individual paths using the short circuiting behavior of -o (like you already did with ~/Desktop/testrm/.snapshot).
However, for each excluded path you also have to exclude all of its parent directories. Otherwise you would delete a/b/c by deleting a/b/ or a/ with rm -rf.
In the following script, the function orParents generates a part of the find command. Example:
find $(orParents a/b/c) ... would run
find -path a/b/c -o -path a/b -o -path a -o ....
#! /usr/bin/env bash
orParents() {
p="$1"
while
printf -- '-path %q -o' "$p"
p=$(dirname "$p")
[ "$p" != . ]
do :; done
}
find ~/Desktop/testrm -mindepth 1 \
$(orParents "$ROOT_DIR/$DATA_DIR") -path ~/Desktop/testrm/.snapshot -o \
-mtime +2 -prune -exec rm -rf {} +
Warning: You have to make sure that $ROOT_DIR/$DATA_DIR does not end with a / and does not contain glob characters like *, ?, and [].
Spaces are ok as printf %q escapes them correctly. However, find -path interprets its argument as a glob pattern independently. We could do a double quoting mechanism. Maybe something like printf %q "$(sed 's/[][*?\]/\\&/' <<< "$p")", but I'm not so sure about how exactly find -path interprets its argument.
Alternatively, you could write a script isParentOf and do ...
find ... -exec isParentOf "$ROOT_DIR/$DATA_DIR" {} \; -o ...
... to exclude $ROOT_DIR/$DATA_DIR and all of its parents. This is probably safer and more portable, but slower and a hassle to set up (find -exec bash -c ... and so on) if you don't want to add a script file to your path.

BASH: Filter list of files by return value of another command

I have series of directories with (mostly) video files in them, say
test1
1.mpg
2.avi
3.mpeg
junk.sh
test2
123.avi
432.avi
432.srt
test3
asdf.mpg
qwerty.mpeg
I create a variable (video_dir) with the directory names (based on other parameters) and use that with find to generate the basic list. I then filter based on another variable (video_type) for file types (because there is sometimes non-video files in the dirs) piping it through egrep. Then I shuffle the list around and save it out to a file. That file is later used by mplayer to slideshow through the list.
I currently use the following command to accomplish that. I'm sure it's a horrible way to do it, but it works for me and it's quite fast even on big directories.
video_dir="/test1 /test2"
video_types=".mpg$|.avi$|.mpeg$"
find ${video_dir} -type f |
egrep -i "${video_types}" |
shuf > "$TEMP_OUT"
I now would like to add the ability to filter out files based on the resolution height of the video file. I can get that from.
mediainfo --Output='Video;%Height%' filename
Which just returns a number. I have tried using the -exec functionality of find to run that command on each file.
find ${video_dir} -type f -exec mediainfo --Output='Video;%Height%' {} \;
but that just returns the list of heights, not the filenames and I can't figure out how to reject ones based on a comparison, like <480.
I could do a for next loop but that seems like a bad (slow) idea.
Using info from #mark-setchell I modified it to,
video_dir="test1"
find ${video_dir} -type f \
-exec bash -c 'h=$(mediainfo --Output="Video;%Height%" "$1"); [[ $h -gt 480 ]]' _ {} \; -print
Which works.
You can replace your egrep with the following so you are still inside the find command (-iname is case insensitive and -o represents a logical OR):
find test1 test2 -type f \
\( -iname "*.mpg" -o -iname "*.avi" -o -iname "*.mpeg" \) \
NEXT_BIT
The NEXT_BIT can then -exec bash and exit with status 0 or 1 depending on whether you want the current file included or excluded. So it will look like this:
-exec bash -c 'H=$(mediainfo -output ... "$1"); [ $H -lt 480 ] && exit 1; exit 0' _ {} \;
So, taking note of #tripleee advice in comments about superfluous exit statements, I get this:
find test1 test2 -type f \
\( -iname "*.mpg" -o -iname "*.avi" -o -iname "*.mpeg" \) \
-exec bash -c 'h=$(mediainfo ...options... "$1"); [ $h -lt 480 ]' _ {} \; -print
This Q&A was focused on one particular case, so the accepted answer is not as general as it could be.
find
If the list of files comes from find, one can use its filtering facilities, e.g. -exec:
find ${video_dir} -type f \
-exec COMMAND \; \
-print
Here
COMMAND is not enclosed in quotes -- find reads everything after -exec and up to a \;
find will expand {} to the current file name (including path -- you might find -execdir helpful, which will cd to the file's directory and replace {} with the leaf file name)
The exit code of COMMAND is treated as follows:
0 -> true
non-0 -> false
Note that you can build more complex expressions (e.g. -not -exec ...), which will be evaluated "from left to right, according to the rules of precedence ... -and is assumed where the operator is omitted." (per man find)
xargs
If the list of files comes from elsewhere (and is available on stdin), you can use xargs as follows (from
If xargs is map, what is filter? )
ls | xargs -I{} bash -c "COMMAND '{}' && echo '{}'"
Here is my solution.
#!/bin/bash
shopt -s nullglob
video_dir=(/test1 /test2)
while IFS= read -rd '' file; do
if [[ $file = *.#(mpg|avi|mpeg|mp4) ]]; then
h=$(mediainfo --Output="Video;%Height%" "$file")
(( h >= 480 )) && echo "$file"
fi
done < <(find "${video_dir[#]}" -type f -print0)
This solution you can process everything inside the while read loop.

How to rename multiple files at once

I have lots of files, directories and sub-directories at my file system.
For example:
/path/to/file/test-poster.jpg
/anotherpath/my-poster.jpg
/tuxisthebest/ohyes/path/exm/bold-poster.jpg
I want to switch all file names from *-poster.jpg to folder.jpg
I have tried with sed and awk with no success.
little help?
You can do it with find:
find -name "*poster.jpg" -exec sh -c 'mv "$0" "${0%/*}/folder.jpg"' '{}' \;
Explanation
Here, for each filename matched, executes:
sh -c 'mv "$0" "${0%/*}/folder.jpg"' '{}'
Where '{}' is the filename passed as an argument to the command_string:
mv "$0" "${0%/*}/folder.jpg"
So, at the end, $0 will have the filename.
Finally, ${0%/*}/folder.jpg expands to the path of the old filename and adds /folder.jpg.
Example
Notice I'm replacing mv with echo
$ find -name "*poster.jpg" -exec sh -c 'echo "$0" "${0%/*}/folder.jpg"' '{}' \;
./anotherpath/my-poster.jpg ./anotherpath/folder.jpg
./path/to/file/test-poster.jpg ./path/to/file/folder.jpg
./tuxisthebest/ohyes/path/exm/bold-poster.jpg ./tuxisthebest/ohyes/path/exm/folder.jpg
Try this script, it should rename all the files as required.
for i in $(find . -name "*-poster.jpg") ; do folder=`echo $i | awk -F"-poster.jpg" {'print $1'}`; mv -iv $i $folder.folder.jpg; done
You can replace . to the directory where these files are placed in the command find . -name "*-poster.jpg" in the script. Let me know if it is working fine for you.
you can try it like
find -name '*poster*' -type f -exec sh -c 'mv "{}" "$(dirname "{}")"/folder.jpg' \;
find all files containing poster == find -name '*poster*' -type f
copy the directory path of the file and store it in a temporary variable and afterwards affix "folder.jpg" to directory path == -exec sh -c 'mv "{}" "$(dirname "{}")"/folder.jpg' \;

Run command from variables in shell script

I wrote this piece of code to scan a directory for files newer than a reference file while excluding specific subdirectories.
#!/bin/bash
dateMarker="date.marker"
fileDate=$(date +%Y%m%d)
excludedDirs=('./foo/bar' './foo/baz' './bar/baz')
excludedDirsNum=${#excludedDirs[#]}
for (( i=0; i < $excludedDirsNum; i++)); do
myExcludes=${myExcludes}" ! -wholename '"${excludedDirs[${i}]}"*'"
done
find ./*/ -type f -newer $dateMarker $myExcludes > ${fileDate}.changed.files
However the excludes are just being ignored. When I "echo $myExcludes" it looks just fine and furthermore the script behaves just as intended if I replace "$myExcludes" in the last line with the output of the echo command. I guess it's some kind of quoting/escaping error, but I haven't been able to eliminate it.
Seems to be a quoting problem, try using arrays:
#!/bin/bash
dateMarker=date.marker
fileDate=$(date +%Y%m%d)
excludedDirs=('./foo/bar' './foo/baz' './bar/baz')
args=(find ./*/ -type f -newer "$dateMarker")
for dir in "${excludedDirs[#]}"
do
args+=('!' -wholename "$dir")
done
"${args[#]}" > "$fileDate.changed.files"
Maybe you also need -prune:
args=(find ./*/)
for dir in "${excludedDirs[#]}"
do
args+=('(' -wholename "$dir" -prune ')' -o)
done
args+=('(' -type f -newer "$dateMarker" -print ')')
you need the myExcludes to evaluate to something like this:
\( -name foo/bar -o -name foo/baz -o -name bar/baz \)

Resources