I have this block and the for loop doesn't repeat even if the path has more than 2 files.. It executes only once and that's all.. What's the problem? How can I make it run for all files in the list?
list=$(find $path -type f \( -name "*.c" -or -name "*.cpp" -or -name "*.cxx" -or -name "*.cc" \))
for file in "$list";do
#commands
done
You can avoid the use of find entirely here (Assuming the only files with those extensions are regular files; no directories etc.), via bash's extended globbing:
shopt -s extglob globstar
for file in "$path"/**/*.#(c|cpp|cxx|cc); do
# commands
done
Putting $list in quotes makes it just one word, so it doesn't loop.
But if you take out the quotes, it won't work properly if any of the filenames contain whitespace, since they'll be split into multiple words.
Instead of assigning to a variable, pipe the output to a while read loop.
find $path -type f \( -name "*.c" -or -name "*.cpp" -or -name "*.cxx" -or -name "*.cc" \) | while read -r file
do
# commands
done
Related
I'm trying to create a cronjob that will delete everything with a pattern *.jar, except for master.jar and anything in a directory pattern */jarkeeper/*/staging/*
I'm close but not luck in finding the correct command. Here's what i have so far:
find /var/lib/jenkins/workspace/ ! -path "*/jarkeeper/*/staging/*" -or -type f ! -name master.jar -name \*.jar
and
find /var/lib/jenkins/workspace/ \( ! -path "*/jarkeeper/*/staging/*" \) -or \( -type f ! -name master.jar \) -name \*.jar
What should the correct format be?
The issue looks like you are using -or as opposed to -or. I would also suggest using -path as opposed to -name throughout to keep everything consistent and so:
find /var/lib/jenkins/workspace/ -type f ! -path "*master.jar" -or ! -path "*/jarkeeper/*/staging/*" -or -path "*.jar"
As an idea, I've always felt more comfortable combining more primitive tools than to use find's complex syntax, like:
find $somewhere -name \*.jar | grep -v master.jar | \
grep -vE "jarkeeper/.*/staging/" | xargs rm -rf
This also comes at the advantage that you can test/check/debug your scripts part by part.
This question already has answers here:
How to loop through file names returned by find?
(17 answers)
Closed 2 years ago.
How can I select files one by one from a list and work on them? This is my code:
list=$( find $path \( -name "*.c" -or -name "*.cpp" -or -name "*.cxx" -or -name "*.cc" \) )
for file in "$list"
do
done
You can pass to find as many commands as you like:
find $path \( -name "*.c" -or -name "*.cpp" -or -name "*.cxx" -or -name "*.cc" \) \
-exec bash -c "echo cmd1 '{}'; echo cmd2 '{}'; echo etc." \;
an alternative is to write a function:
function safe_copy() {
echo "$1"
}
export -f safe_copy
find $path \( -name "*.c" -or -name "*.cpp" -or -name "*.cxx" -or -name "*.cc" \) \
-exec bash -c "safe_copy '{}'" \;
You can also write a script and call it with exec instead of a function.
You can use -print0 as find argument, to print a null delimited stream of filenames.
Then loop read the null delimited stream.
find "$path" \( -name '*.c' -or -name '*.cpp' -or -name '*.cxx' -or -name '*.cc' \) -print0 |
while IFS= read -r file
do
done
Trying to append output of find command to a variable in a Bash script
Can append output of find command to a log file ok, but can't append it to a variable i.e.
This works ok:
find $DIR -type d -name "*" >> $DIRS_REMOVED_LOG
But this won't:
FILES_TO_EVAL=find $DIR -type f \( -name '*.sh' -or -name '*.txt' -or -name '*.xml' -or -name '*.log' \)
ENV=`basename $PS_CFG_HOME | tr "[:lower:]" "[:upper:]"`
FILE_TYPES=(*.log *.xml *.txt *.sh)
DIRS_TO_CLEAR="$PS_CFG_HOME/data/files $PS_CFG_HOME/appserv/prcs/$ENV/files $PS_CFG_HOME/appserv/prcs/$ENV/files/CQ"
FILES_REMOVED_LOG=$PS_CFG_HOME/files_removed.log
DIRS_REMOVED_LOG=$PS_CFG_HOME/dirs_removed.log
##Cycle through directories
##Below for files_removed_log works ok but can't get the find into a variable.
for DIR in `echo $DIRS_TO_CLEAR`
do
echo "Searching $DIR for files:"
FILES_TO_EVAL=find $DIR -type f \( -name '*.sh' -or -name '*.txt' -or -name '*.xml' -or -name '*.log' \)
find $DIR -type d -name "*" >> $DIRS_REMOVED_LOG
done
Expected FILES_TO_EVAL to be populated with results of find command but it is empty.
Run your scripts through ShellCheck. It finds lots of common mistakes, much like a compiler would.
FILES_TO_EVAL=find $DIR -type f \( -name '*.sh' -or -name '*.txt' -or -name '*.xml' -or -name '*.log' \)
SC2209: Use var=$(command) to assign output (or quote to assign string).
In addition to the problems that shellcheck.net will point out, there are a number of subtler problems.
For one thing, you're using all-caps variable names. This is dangerous, because there are a large number of all-caps variables that have special meanings to the shell and/or other tools, and if you accidentally use one of those, it can have weird effects. Lower- or mixed-case variables are much safer (except when you specifically want the special meaning).
Also, you should almost always put double-quotes around variable references (e.g. find "$dir" ... instead of find $dir ...). Without them, the variables will be subject to word splitting and wildcard expansion, which can have a variety of unintended consequences. In some cases, you need word splitting and/or wildcard expansion on a variable's value, but usually not quite the way the shell does it; in these cases, you should look for a better way to do the job.
In the line that's failing,
FILES_TO_EVAL=find $DIR -type f \( -name '*.sh' -or -name '*.txt' -or -name '*.xml' -or -name '*.log' \)
the immediate problem is that you need to use $(find ...) to capture the output from the find command. But this is still dangerous, because it's just storing a newline-delimited list of file paths, and the standard way to expand this (just using an unquoted variable reference) has all the problems I mentioned above. In this case, it will lead to trouble if any filenames contain spaces or wildcards (which are perfectly legal in filenames). In you're in a controlled environment where you can guarantee this won't happen, you'll get away with it... but it's really not the best idea.
Correctly handling a list of filepaths from find is a little complicated, but there are a number of ways to do it. There's a lot of good info in BashFAQ #20: "How can I find and safely handle file names containing newlines, spaces or both?" I'll summarize some common options below:
If you don't need to store the list, just run commands on individual files, you can use find -exec:
find "$dir" -type f \( -name '*.sh' -or -name '*.txt' -or -name '*.xml' -or -name '*.log' \) -exec somecommand {} \;
If you need to run something more complex, you can use find -print0 to output the list in an unambiguous form, and then use read -d '' to read them. There are a bunch of potential pitfalls here, so here's the version I use to avoid all the trouble spots:
while IFS= read -r -d '' filepath <&3; do
dosomethingwith "$filepath"
done 3< <(find "$dir" -type f \( -name '*.sh' -or -name '*.txt' -or -name '*.xml' -or -name '*.log' \) -print0)
Note that the <(command) syntax (known as process substitution) is a bash-only feature, so use an explicit bash shebang (#!/bin/bash or #!/usr/bin/env bash) on your script, and don't override it by running the script with sh.
If you really do need to store the list of paths for later, store it as an array:
files_to_eval=()
while IFS= read -r -d '' filepath; do
files_to_eval+=("$filepath")
done < <(find "$dir" -type f \( -name '*.sh' -or -name '*.txt' -or -name '*.xml' -or -name '*.log' \) -print0)
..or, if you have bash v4.4 or later, it's easier to use readarray (aka mapfile):
readarray -td '' files_to_eval < <(find "$dir" -type f \( -name '*.sh' -or -name '*.txt' -or -name '*.xml' -or -name '*.log' \) -print0)
In either case, you should then expand the array with "${files_to_eval[#]}" to get all the elements without subjecting them to word splitting and wildcard expansion.
On to some other problems. In this line:
FILE_TYPES=(*.log *.xml *.txt *.sh)
In this context, the wildcards will be expanded immediately to a list of matches in the current director. You should quote them to prevent this:
file_types=("*.log" "*.xml" "*.txt" "*.sh")
In these lines:
DIRS_TO_CLEAR="$PS_CFG_HOME/data/files $PS_CFG_HOME/appserv/prcs/$ENV/files $PS_CFG_HOME/appserv/prcs/$ENV/files/CQ"
...
for DIR in `echo $DIRS_TO_CLEAR`
You're storing a list as a single string with entries separated by spaces, which has all the word-split and wildcard problems I've been harping on. Also, the echo here is a complication that doesn't do anything useful, and actually makes the wildcard problem worse. Use an array, and avoid all the mess:
dirs_to_clear=("$ps_cfg_home/data/files" "$ps_cfg_home/appserv/prcs/$env/files" "$ps_cfg_home/appserv/prcs/$env/files/CQ")
...
for dir in "${dirs_to_clear[#]}"
I'm looking for a proper way to list
all filenames (without extension)
matching a specific extension list
recursively in a specific folder
with some exclusions patterns
and then export that to a file.
Currently i'm doing the following which is working properly:
ls -R --ignore={"Sample","Sample.*","sample.*","*_sample.*","*.sample.*","*-sample.*","*.sample-*","*-sample-*","*trailer]*"} "$filesSource" | grep -E '\.mkv$|\.mp4$|\.avi$' | sed -e 's/\(.*\)/\L\1/' | sort >> "$listFile"
Thanks to ShellChecker, I have a feedback on this line and I don't know how to do that properly!
Thanks for your help!
Why don't you try find command?
something like
find YOUR_PATH -type f -name "*.FIRST_EXTENSION" -o -name "*.SECOND_EXTENSION"| grep -v SOME_EXCLUSION | awk -F. '{print $(NF-1)}' | sort > SOME_FILE
note: this will work only if the filenames contain only 1 "." character for the extension, otherwise you need to modify a little bit the awk part.
If you are searching just on filenames, then you can use:
I split the command line in multiple lines:
$ find /path/to/folder -type f \( \( -name '*.ext1' -or -name '*.ext2' -or -name '*.ext3' \) -and -not \( -name '*excl1*' -or -name 'excl2*' \) \) -print
This will do:
/path/to/folder: the folder you are searching
-type f : you are searching for files in the above folder which satisfy
\(: open the conditional test
\( -name '*.ext1' -or -name '*.ext2' -or -name '*.ext3' \): who have one of the three listed extensions (with a conditional or)
-and -not \( -name '*excl1*' -or -name 'excl2*' \): if the above condition mathches it will check (-and) if one of the patterns *excl1* or excl2* do -not match.
\) close the main conditional test
-print perform the action to print the found paths.
I'm trying to find all files whose name matches certain C++ file extensions but exclude certain directories matching a pattern with this:
find /home/palchan/code -name "*.[CcHh]" -o -name "*.cpp" -o -name "*.hpp" -a ! -name "*pattern*"
and this still gives me as output certain files like:
/home/palchan/code/libFox/pattern/hdr/fox/RedFox.H
which has the pattern in it?
Here is an example:
> ls -R .
.:
libFox
./libFox:
RedFox.C RedFox.H pattern
./libFox/pattern:
RedFox.C RedFox.H
and then I run:
> find . \( -name "*.[HC]" -a ! -name "*pattern*" \)
./libFox/pattern/RedFox.C
./libFox/pattern/RedFox.H
./libFox/RedFox.C
./libFox/RedFox.H
The following should work:
find /home/palchan/code \( -name "*pattern*" \) -prune -o -type f \( -name "*.[CcHh]" -o -name "*.cpp" -o -name "*.hpp" \) -print
From man find:
-name pattern
Base of file name (the path with the leading directories removed) matches shell pattern pattern. The metacharacters (`*', `?', and `[]') match
a `.' at the start of the base name (this is a change in findutils-4.2.2; see section STANDARDS CONFORMANCE below). To ignore a directory and
the files under it, use -prune; see an example in the description of -path. Braces are not recognised as being special, despite the fact that
some shells including Bash imbue braces with a special meaning in shell patterns. The filename matching is performed with the use of the
fnmatch(3) library function. Don't forget to enclose the pattern in quotes in order to protect it from expansion by the shell.
So, basically, you should use -prune to exclude directories instead of ! -name something
Try doing this :
find /home/palchan/code \( -name "*.[CcHh]" -o -name "*.cpp" -o -name "*.hpp" -a ! -name "*pattern*" \)