I wrote a bash script to retrieve the last few days file info from the file system, and the file under some sub-folders will be excluded. Here is the script(test.sh):
#!/bin/bash
date_range=$1
base_dir=$2
excluded_dir=$3
# Command initialization
cmd="find $base_dir"
for item in ${excluded_dir[#]}
do
cmd="$cmd -not \( -path '$base_dir/$item' -prune \)"
done
cmd="$cmd -type f -mtime -$date_range -ls"
echo $cmd
$cmd
I tried an example as below:
./test.sh 3 /root "excluded_folder1 excluded_folder2"
The command has been initialized as:
find /root -not \( -path '/root/excluded_folder1' -prune \) -not \( -path '/root/excluded_folder2' -prune \) -type f -mtime -3 -ls
If I run this find command in my terminal, it works fine, I can get the results that I want. While if it's executed in the bash script. I always get such an error:
find: paths must precede expression: \(
Usage: find [-H] [-L] [-P] [-Olevel] [-D help|tree|search|stat|rates|opt|exec] [path...] [expression]
Does anybody knows what is the problem and how to fix this?
Thanks for all the answers and suggestions I received here. But none of that solved my problem. The problem is finally solved by using 'eval' to execute the command. The final working bash script is as below:
#!/bin/bash
date_range=$1
base_dir=$2
excluded_dir=$3
# Command initialization
cmd="find $base_dir"
for item in ${excluded_dir[#]}
do
cmd="$cmd -not \( -path '$base_dir/$item' -prune \)"
done
cmd="$cmd -type f -mtime -$date_range -ls"
eval $cmd
While there're some posts saying using eval in bash script is a bad and insecure choice, I still don't know how can I solve this problem with some other approaches. If someone got a better idea, please post it here.
Reference:
What is the eval command in bash?
Why and when should eval use be avoided in shell scripts?
Based on my guess above, I suggest the following code:
#!/bin/bash
date_range=$1
base_dir=$2
excluded_dir=$3
# Command initialization
cmd="find $base_dir"
for item in ${excluded_dir[#]}
do
cmd="$cmd -not ( -path '$base_dir/$item' -prune )"
done
cmd="$cmd -type f -mtime -$date_range -ls"
echo $cmd
$cmd
Related
This question already has answers here:
Expanding a bash array only gives the first element
(1 answer)
Why does shell ignore quoting characters in arguments passed to it through variables? [duplicate]
(3 answers)
Closed 11 months ago.
Expected Function: write a bash script showing all the regular files
with filename ended with ".xml" or ".yml" and with path not begined
with "./target/", i.e., exclude the "target" subdirectory.
Example File List: ./a/a1.xml; ./a/a2.txt; ./b/b1.yml; ./target/t.xml;
Example Outout: ./a/a1.xml; ./b/b1.yml
I construct find options in a bash shell script like
#!/bin/bash
find_opts=( -type f -a ( -not -path "./target/*" ) -a ( -false -o -name "*.xml" -o -name "*.yml" ) )
find . $find_opts
But it does not output the expected result. Howver, when I tpye the full command string in bash terminal as follows:
[root#localhost]#find . \( -type f -a \( -not -path "./target/*" \) -a \( -false -o -name "*.xml" -o -name "*.yml" \) \)
it works. What is the problem about the above bash script ?
==============================================================
Someone gives this reference link : Expanding a bash array only gives the first element
It is about "bash array". But it seems that my problem is not about "bash array". Would anyone give any reasons?
Please see How to exclude a directory in find . command to known why I use parentheses whiches look like an array. Anyway, I try two others attempts:
#the first one
find_opts=( -type f -a ( -not -path "./target/*" ) -a ( -false -o -name "*.xml" -o -name "*.yml" ) )
#the second one
find_opts=\\( -type f -a \\( -not -path "./target/*" \\) -a \\( -false -o -name "*.xml" -o -name "*.yml" \\) \\)
#the third one
find_opts=\( -type f -a \( -not -path "./target/*" \) -a \( -false -o -name "*.xml" -o -name "*.yml" \) \)
The first one and the third one give the same output which is unexpeced result. The second one occures an error.
syntax error near unexpected token `('
The problem is still here.
==============================================================
Someone give one more reference:Why does shell ignore quoting characters in arguments passed to it through variables?
It is about how to use bash array to pass arguments and the problem is solved by the following code:
#!/bin/bash
find_opts=(\( -type f -a \( -not -path './target/*' \) -a \( -false -o -name '*.xml' -o -name '*.yml' \) \))
find . "${find_opts[#]}"
I've seen a few answers regarding this, but as a newbie, I don't really understand how to implement that in my script.
it should be pretty easy (for those who can stuff like this)
I'm using a simple
for f in "/drive1/"images*.{jpg,png}; do
but this is simply overloading and giving me
Argument list too long
How is this easiest solved?
Argument list too long workaroud
Argument list length is something limited by your config.
getconf ARG_MAX
2097152
But after discuss around differences between bash specifics and system (os) limitations (see comments from that other guy), this question seem wrong:
Regarding discuss on comments, OP tried something like:
ls "/simple path"/image*.{jpg,png} | wc -l
bash: /bin/ls: Argument list too long
This happen because of OS limitation, not bash!!
But tested with OP code, this work finely
for file in ./"simple path"/image*.{jpg,png} ;do echo -n a;done | wc -c
70980
Like:
printf "%c" ./"simple path"/image*.{jpg,png} | wc -c
Reduce line length by reducing fixed part:
First step: you could reduce argument length by:
cd "/drive1/"
ls images*.{jpg,png} | wc -l
But when number of file will grow, you'll be buggy again...
More general workaround:
find "/drive1/" -type f \( -name '*.jpg' -o -name '*.png' \) -exec myscript {} +
If you want this to NOT be recursive, you may add -maxdepth as 1st option:
find "/drive1/" -maxdepth 1 -type f \( -name '*.jpg' -o -name '*.png' \) \
-exec myscript {} +
There, myscript will by run with filenames as arguments. The command line for myscript is built up until it reaches a system-defined limit.
myscript /drive1/file1.jpg '/drive1/File Name2.png' /drive1/...
From man find:
-exec command {} +
This variant of the -exec action runs the specified command on
the selected files, but the command line is built by appending
each selected file name at the end; the total number of invoca‐
tions of the command will be much less than the number of
matched files. The command line is built in much the same way
that xargs builds its command lines. Only one instance of `{}'
Inscript sample
You could create your script like
#!/bin/bash
target=( "/drive1" "/Drive 2/Pictures" )
[ "$1" = "--run" ] && exec find "${target[#]}" -type f \( -name '*.jpg' -o \
-name '*.png' \) -exec $0 {} +
for file ;do
echo Process "$file"
done
Then you have to run this with --run as argument.
work with any number of files! (Recursively! See maxdepth option)
permit many target
permit spaces and special characters in file and directrories names
you could run same script directly on files, without --run:
./myscript hello world 'hello world'
Process hello
Process world
Process hello world
Using pure bash
Using arrays, you could do things like:
allfiles=( "/drive 1"/images*.{jpg,png} )
[ -f "$allfiles" ] || { echo No file found.; exit ;}
echo Number of files: ${#allfiles[#]}
for file in "${allfiles[#]}";do
echo Process "$file"
done
There's also a while read loop:
find "/drive1/" -maxdepth 1 -mindepth 1 -type f \( -name '*.jpg' -o -name '*.png' \) |
while IFS= read -r file; do
or with zero terminated files:
find "/drive1/" -maxdepth 1 -mindepth 1 -type f \( -name '*.jpg' -o -name '*.png' \) -print0 |
while IFS= read -r -d '' file; do
find . -maxdepth 1 ! -path . -type f ! -name "*.gz" ${FILE_PATTERN} -mtime +${DAYS_AFTER_ARCHIVE}
I am trying to execute above command inside the script where ${FILE_PATTERN} and ${DAYS_AFTER_ARCHIVE} are the variable provided while executing the script. The variable value for ${FILE_PATTERN} would be ! -name "*warnings*".
I am looking for executing above command as like below command in script
find . -maxdepth 1 ! -path . -type d ! -name "*warnings*" -mtime +7
I am providing file pattern argument as "! -name "warnings""
but receiving following error message
find: paths must precede expression
Usage: find [-H] [-L] [-P] [path...] [expression]
suggest on above.
First of all
-name "*.gz" ${FILE_PATTERN}
has too many option values (this is what usually causes the message shown)
If you use bash or similar, escape the exclamation marks
find . -maxdepth 1 \! -path . -type f \! -name "*.gz" ${FILE_PATTERN} -mtime +${DAYS_AFTER_ARCHIVE}
I am providing file pattern argument as "! -name "warnings"" but receiving following error message
You can't combine flags and their values like that. Also, you can't nest " like that. So, it could be like
! -name "$FILE_PATTERN"
If you're using BASH you can make use of BASH arrays:
# find options
FILE_PATTERN=(! -name "warnings*")
# build the find command
cmd=(find . -maxdepth 1 ! -path . -type f \( ! -name "*.gz" "${FILE_PATTERN[#}}" \) -mtime +${DAYS_AFTER_ARCHIVE})
# execute the command
"${cmd[#]}"
If not using BASH then you will have to use eval with caution:
FILE_PATTERN='! -name "warnings*"'
eval find . -maxdepth 1 ! -path . -type f ! -name "*.gz" "${FILE_PATTERN}" -mtime +${DAYS_AFTER_ARCHIVE}
If I have a list of filenames in a text file that I want to exclude when I run find, how can I do that? For example, I want to do something like:
find /dir -name "*.gz" -exclude_from skip_files
and get all the .gz files in /dir except for the files listed in skip_files. But find has no -exclude_from flag. How can I skip all the files in skip_files?
I don't think find has an option like this, you could build a command using printf and your exclude list:
find /dir -name "*.gz" $(printf "! -name %s " $(cat skip_files))
Which is the same as doing:
find /dir -name "*.gz" ! -name first_skip ! -name second_skip .... etc
Alternatively you can pipe from find into grep:
find /dir -name "*.gz" | grep -vFf skip_files
This is what i usually do to remove some files from the result (In this case i looked for all text files but wasn't interested in a bunch of valgrind memcheck reports we have here and there):
find . -type f -name '*.txt' ! -name '*mem*.txt'
It seems to be working.
I think you can try like
find /dir \( -name "*.gz" ! -name skip_file1 ! -name skip_file2 ...so on \)
find /var/www/test/ -type f \( -iname "*.*" ! -iname "*.php" ! -iname "*.jpg" ! -iname "*.png" \)
The above command gives list of all files excluding files with .php, .jpg ang .png extension. This command works for me in putty.
Josh Jolly's grep solution works, but has O(N**2) complexity, making it too slow for long lists. If the lists are sorted first (O(N*log(N)) complexity), you can use comm, which has O(N) complexity:
find /dir -name '*.gz' |sort >everything_sorted
sort skip_files >skip_files_sorted
comm -23 everything_sorted skip_files_sorted | xargs . . . etc
man your computer's comm for details.
This solution will go through all files (not exactly excluding from the find command), but will produce an output skipping files from a list of exclusions.
I found that useful while running a time-consuming command (file /dir -exec md5sum {} \;).
You can create a shell script to handle the skipping logic and run commands on the files found (make it executable with chmod, replace echo with other commands):
$ cat skip_file.sh
#!/bin/bash
found=$(grep "^$1$" files_to_skip.txt)
if [ -z "$found" ]; then
# run your command
echo $1
fi
Create a file with the list of files to skip named files_to_skip.txt (on the dir you are running from).
Then use find using it:
find /dir -name "*.gz" -exec ./skip_file.sh {} \;
This should work:
find * -name "*.gz" $(printf "! -path %s " $(<skip_files.txt))
Working out
Assuming skip_files has a filename on each line, you can get the list of filenames via $(<skip_files.txt). E.g. echo $(<skip_files.txt) should print them all out.
For each filename you want to have a ! -path filename expression. To build this, use $(printf "! -path %s " $(<skip_files.txt))
Then, put it together with a filter on -name "*.gz"
I wrote this piece of code to scan a directory for files newer than a reference file while excluding specific subdirectories.
#!/bin/bash
dateMarker="date.marker"
fileDate=$(date +%Y%m%d)
excludedDirs=('./foo/bar' './foo/baz' './bar/baz')
excludedDirsNum=${#excludedDirs[#]}
for (( i=0; i < $excludedDirsNum; i++)); do
myExcludes=${myExcludes}" ! -wholename '"${excludedDirs[${i}]}"*'"
done
find ./*/ -type f -newer $dateMarker $myExcludes > ${fileDate}.changed.files
However the excludes are just being ignored. When I "echo $myExcludes" it looks just fine and furthermore the script behaves just as intended if I replace "$myExcludes" in the last line with the output of the echo command. I guess it's some kind of quoting/escaping error, but I haven't been able to eliminate it.
Seems to be a quoting problem, try using arrays:
#!/bin/bash
dateMarker=date.marker
fileDate=$(date +%Y%m%d)
excludedDirs=('./foo/bar' './foo/baz' './bar/baz')
args=(find ./*/ -type f -newer "$dateMarker")
for dir in "${excludedDirs[#]}"
do
args+=('!' -wholename "$dir")
done
"${args[#]}" > "$fileDate.changed.files"
Maybe you also need -prune:
args=(find ./*/)
for dir in "${excludedDirs[#]}"
do
args+=('(' -wholename "$dir" -prune ')' -o)
done
args+=('(' -type f -newer "$dateMarker" -print ')')
you need the myExcludes to evaluate to something like this:
\( -name foo/bar -o -name foo/baz -o -name bar/baz \)