I'm studying shell in Linux these days. and I've had one question.
Please, look at below command:
$ find . -name '*.c' -or -name '*.cpp'
Exact command of above command is processed like below command?
$ find . -name '*.c' -and -print -or -name '*.cpp' -and -print
You are combining different search expressions with the logical operator or.
Basically your command will find all files in the current directory ending with .c or .cppand will print them to STDOUT.
For further info check the man page of find command.
Also note that this question would be more suitable to ask here.
Related
I'm trying to create a cronjob that will delete everything with a pattern *.jar, except for master.jar and anything in a directory pattern */jarkeeper/*/staging/*
I'm close but not luck in finding the correct command. Here's what i have so far:
find /var/lib/jenkins/workspace/ ! -path "*/jarkeeper/*/staging/*" -or -type f ! -name master.jar -name \*.jar
and
find /var/lib/jenkins/workspace/ \( ! -path "*/jarkeeper/*/staging/*" \) -or \( -type f ! -name master.jar \) -name \*.jar
What should the correct format be?
The issue looks like you are using -or as opposed to -or. I would also suggest using -path as opposed to -name throughout to keep everything consistent and so:
find /var/lib/jenkins/workspace/ -type f ! -path "*master.jar" -or ! -path "*/jarkeeper/*/staging/*" -or -path "*.jar"
As an idea, I've always felt more comfortable combining more primitive tools than to use find's complex syntax, like:
find $somewhere -name \*.jar | grep -v master.jar | \
grep -vE "jarkeeper/.*/staging/" | xargs rm -rf
This also comes at the advantage that you can test/check/debug your scripts part by part.
Trying to append output of find command to a variable in a Bash script
Can append output of find command to a log file ok, but can't append it to a variable i.e.
This works ok:
find $DIR -type d -name "*" >> $DIRS_REMOVED_LOG
But this won't:
FILES_TO_EVAL=find $DIR -type f \( -name '*.sh' -or -name '*.txt' -or -name '*.xml' -or -name '*.log' \)
ENV=`basename $PS_CFG_HOME | tr "[:lower:]" "[:upper:]"`
FILE_TYPES=(*.log *.xml *.txt *.sh)
DIRS_TO_CLEAR="$PS_CFG_HOME/data/files $PS_CFG_HOME/appserv/prcs/$ENV/files $PS_CFG_HOME/appserv/prcs/$ENV/files/CQ"
FILES_REMOVED_LOG=$PS_CFG_HOME/files_removed.log
DIRS_REMOVED_LOG=$PS_CFG_HOME/dirs_removed.log
##Cycle through directories
##Below for files_removed_log works ok but can't get the find into a variable.
for DIR in `echo $DIRS_TO_CLEAR`
do
echo "Searching $DIR for files:"
FILES_TO_EVAL=find $DIR -type f \( -name '*.sh' -or -name '*.txt' -or -name '*.xml' -or -name '*.log' \)
find $DIR -type d -name "*" >> $DIRS_REMOVED_LOG
done
Expected FILES_TO_EVAL to be populated with results of find command but it is empty.
Run your scripts through ShellCheck. It finds lots of common mistakes, much like a compiler would.
FILES_TO_EVAL=find $DIR -type f \( -name '*.sh' -or -name '*.txt' -or -name '*.xml' -or -name '*.log' \)
SC2209: Use var=$(command) to assign output (or quote to assign string).
In addition to the problems that shellcheck.net will point out, there are a number of subtler problems.
For one thing, you're using all-caps variable names. This is dangerous, because there are a large number of all-caps variables that have special meanings to the shell and/or other tools, and if you accidentally use one of those, it can have weird effects. Lower- or mixed-case variables are much safer (except when you specifically want the special meaning).
Also, you should almost always put double-quotes around variable references (e.g. find "$dir" ... instead of find $dir ...). Without them, the variables will be subject to word splitting and wildcard expansion, which can have a variety of unintended consequences. In some cases, you need word splitting and/or wildcard expansion on a variable's value, but usually not quite the way the shell does it; in these cases, you should look for a better way to do the job.
In the line that's failing,
FILES_TO_EVAL=find $DIR -type f \( -name '*.sh' -or -name '*.txt' -or -name '*.xml' -or -name '*.log' \)
the immediate problem is that you need to use $(find ...) to capture the output from the find command. But this is still dangerous, because it's just storing a newline-delimited list of file paths, and the standard way to expand this (just using an unquoted variable reference) has all the problems I mentioned above. In this case, it will lead to trouble if any filenames contain spaces or wildcards (which are perfectly legal in filenames). In you're in a controlled environment where you can guarantee this won't happen, you'll get away with it... but it's really not the best idea.
Correctly handling a list of filepaths from find is a little complicated, but there are a number of ways to do it. There's a lot of good info in BashFAQ #20: "How can I find and safely handle file names containing newlines, spaces or both?" I'll summarize some common options below:
If you don't need to store the list, just run commands on individual files, you can use find -exec:
find "$dir" -type f \( -name '*.sh' -or -name '*.txt' -or -name '*.xml' -or -name '*.log' \) -exec somecommand {} \;
If you need to run something more complex, you can use find -print0 to output the list in an unambiguous form, and then use read -d '' to read them. There are a bunch of potential pitfalls here, so here's the version I use to avoid all the trouble spots:
while IFS= read -r -d '' filepath <&3; do
dosomethingwith "$filepath"
done 3< <(find "$dir" -type f \( -name '*.sh' -or -name '*.txt' -or -name '*.xml' -or -name '*.log' \) -print0)
Note that the <(command) syntax (known as process substitution) is a bash-only feature, so use an explicit bash shebang (#!/bin/bash or #!/usr/bin/env bash) on your script, and don't override it by running the script with sh.
If you really do need to store the list of paths for later, store it as an array:
files_to_eval=()
while IFS= read -r -d '' filepath; do
files_to_eval+=("$filepath")
done < <(find "$dir" -type f \( -name '*.sh' -or -name '*.txt' -or -name '*.xml' -or -name '*.log' \) -print0)
..or, if you have bash v4.4 or later, it's easier to use readarray (aka mapfile):
readarray -td '' files_to_eval < <(find "$dir" -type f \( -name '*.sh' -or -name '*.txt' -or -name '*.xml' -or -name '*.log' \) -print0)
In either case, you should then expand the array with "${files_to_eval[#]}" to get all the elements without subjecting them to word splitting and wildcard expansion.
On to some other problems. In this line:
FILE_TYPES=(*.log *.xml *.txt *.sh)
In this context, the wildcards will be expanded immediately to a list of matches in the current director. You should quote them to prevent this:
file_types=("*.log" "*.xml" "*.txt" "*.sh")
In these lines:
DIRS_TO_CLEAR="$PS_CFG_HOME/data/files $PS_CFG_HOME/appserv/prcs/$ENV/files $PS_CFG_HOME/appserv/prcs/$ENV/files/CQ"
...
for DIR in `echo $DIRS_TO_CLEAR`
You're storing a list as a single string with entries separated by spaces, which has all the word-split and wildcard problems I've been harping on. Also, the echo here is a complication that doesn't do anything useful, and actually makes the wildcard problem worse. Use an array, and avoid all the mess:
dirs_to_clear=("$ps_cfg_home/data/files" "$ps_cfg_home/appserv/prcs/$env/files" "$ps_cfg_home/appserv/prcs/$env/files/CQ")
...
for dir in "${dirs_to_clear[#]}"
I'm looking for a proper way to list
all filenames (without extension)
matching a specific extension list
recursively in a specific folder
with some exclusions patterns
and then export that to a file.
Currently i'm doing the following which is working properly:
ls -R --ignore={"Sample","Sample.*","sample.*","*_sample.*","*.sample.*","*-sample.*","*.sample-*","*-sample-*","*trailer]*"} "$filesSource" | grep -E '\.mkv$|\.mp4$|\.avi$' | sed -e 's/\(.*\)/\L\1/' | sort >> "$listFile"
Thanks to ShellChecker, I have a feedback on this line and I don't know how to do that properly!
Thanks for your help!
Why don't you try find command?
something like
find YOUR_PATH -type f -name "*.FIRST_EXTENSION" -o -name "*.SECOND_EXTENSION"| grep -v SOME_EXCLUSION | awk -F. '{print $(NF-1)}' | sort > SOME_FILE
note: this will work only if the filenames contain only 1 "." character for the extension, otherwise you need to modify a little bit the awk part.
If you are searching just on filenames, then you can use:
I split the command line in multiple lines:
$ find /path/to/folder -type f \( \( -name '*.ext1' -or -name '*.ext2' -or -name '*.ext3' \) -and -not \( -name '*excl1*' -or -name 'excl2*' \) \) -print
This will do:
/path/to/folder: the folder you are searching
-type f : you are searching for files in the above folder which satisfy
\(: open the conditional test
\( -name '*.ext1' -or -name '*.ext2' -or -name '*.ext3' \): who have one of the three listed extensions (with a conditional or)
-and -not \( -name '*excl1*' -or -name 'excl2*' \): if the above condition mathches it will check (-and) if one of the patterns *excl1* or excl2* do -not match.
\) close the main conditional test
-print perform the action to print the found paths.
I want to write a cleanup routine for my make file that removes every thing except the necessary source files in my folder. For example, my folder contains files with the following extensions: .f .f90 .F90 .F03 .o .h .out .dat .txt .hdf .gif.
I know I can accomplish this with:
find . -name \( '*.o' '*.out' '*.dat' '*.txt' '*.hdf' '*.gif' \) -delete
Using negation, I can do this:
find . -not -name '*.f*' -not -name '*.F*' -not -name '*.h' -delete
But, when I try to do this:
find . -not -name \( '*.f*' '*.F*' '*.h' \)
I get an error:
find: paths must exceed expression: [first expression in the above list]
(In this case, I would get:
find: paths must exceed expression: *.f*
)
Can you explain why this happens, and how to do what I am trying to do? I just hate writing -not -name every time I want to add a file extension to the list. Also, I want to find out why this is giving me an error so that I can learn Linux better.
Thanks!
find . -not -name \( '*.f' '*.F' '*.h' \)
is interpreted as
find
. # path to search
-not # negate next expression
-name \( # expression for files named "("
'*.f' '*.F' .'*.h' \) # more paths to search?
leading to the error.
Since these are single-letter extensions, you can collapse them to a single glob:
find . -not -name '*.[fFh]'
but if they are longer, you have to write out the globs
find . -not -name '*.f' -not -name '*.F' -not -name '*.h'
or
find . -not \( -name '*.f' -o -name '*.F' -o -name '*.h' \)
or switch to using regular expressions.
find . -not -regex '.*\.(f|F|h)$'
Note that regular expressions in find is not part of the POSIX standard and might not be available in all implementations.
I'm trying to find all files whose name matches certain C++ file extensions but exclude certain directories matching a pattern with this:
find /home/palchan/code -name "*.[CcHh]" -o -name "*.cpp" -o -name "*.hpp" -a ! -name "*pattern*"
and this still gives me as output certain files like:
/home/palchan/code/libFox/pattern/hdr/fox/RedFox.H
which has the pattern in it?
Here is an example:
> ls -R .
.:
libFox
./libFox:
RedFox.C RedFox.H pattern
./libFox/pattern:
RedFox.C RedFox.H
and then I run:
> find . \( -name "*.[HC]" -a ! -name "*pattern*" \)
./libFox/pattern/RedFox.C
./libFox/pattern/RedFox.H
./libFox/RedFox.C
./libFox/RedFox.H
The following should work:
find /home/palchan/code \( -name "*pattern*" \) -prune -o -type f \( -name "*.[CcHh]" -o -name "*.cpp" -o -name "*.hpp" \) -print
From man find:
-name pattern
Base of file name (the path with the leading directories removed) matches shell pattern pattern. The metacharacters (`*', `?', and `[]') match
a `.' at the start of the base name (this is a change in findutils-4.2.2; see section STANDARDS CONFORMANCE below). To ignore a directory and
the files under it, use -prune; see an example in the description of -path. Braces are not recognised as being special, despite the fact that
some shells including Bash imbue braces with a special meaning in shell patterns. The filename matching is performed with the use of the
fnmatch(3) library function. Don't forget to enclose the pattern in quotes in order to protect it from expansion by the shell.
So, basically, you should use -prune to exclude directories instead of ! -name something
Try doing this :
find /home/palchan/code \( -name "*.[CcHh]" -o -name "*.cpp" -o -name "*.hpp" -a ! -name "*pattern*" \)