I have one scenario where I need to select all files having aliencoders.numeric-digits
like alienoders.1206
and find command should search all subdirectories too. If there is no such file it should not do anything.
I wrote :
find /home/jassi/ -name "aliencoders.[0-9]+" | xargs ls -lrt | awk print '$9'
Bit it says no such file or directory if there no such file starts with aliencoders.xx...
How I can by pass this error. I have to run it for several such directories and it should give output for those directories only in which such file pattern exists else no warning and doesn't do xargs etc stuffs.
Currently, if no such file is there then it is taking current directory t find instead of /home/jassi
If you don't want xargs to execute if the input is empty you can use -r or --no-run-if-empty which are GNU extensions as pointed out in the man page. So if you have that support, you can try
find /home/jassi/ -name "aliencoders.[0-9]+" | xargs -r ls -lrt | awk '{print $9}'
Alternatively you can make use of exec option with find to achieve this something on these lines
find /home/jassi/ -name "aliencoders.[0-9]+" -exec ls -lrt {} + | awk '{print $9}'
Hope this helps!
bash:
Try this command (under the bash shell since most people use it, and no shell was specified):
find /home/jassi/ -name "aliencoders.[0-9]+" 2>&1 | xargs ls -lrt | awk '{print $9}'
With 2>&1 you are redirecting the error messages from stderr to stdout. Once you have this single output stream, you can process it with your pipes etc.
Without this redirection, your error messages to stderr would have continued to go to the console, creating clutter in the output and only stdout was getting processed by the pipes.
All about redirection will give you more details and control about redirection.
UPDATE:
tcsh:
Based on use of the tcsh (sometimes I think I'm the only one using it), it is possible to redirect stderr to stdout with the following construct:
command |& ...
so
find /home/jassi/ -name "aliencoders.[0-9]+" |& xargs ls -lrt | awk '{print $9}'
should help. Before your error messages were bypassing your pipes and going directly to the console, only stdout was getting processed by your pipes. With this all of your output goes through the pipes and you can filter out what you want.
Also, note the use of the awk command at the end of the command:
awk '{print $9}'
is different from what you posted originally.
Related
I want to pass each output from a command as multiple argument to a second command, e.g.:
grep "pattern" input
returns:
file1
file2
file3
and I want to copy these outputs, e.g:
cp file1 file1.bac
cp file2 file2.bac
cp file3 file3.bac
How can I do that in one go? Something like:
grep "pattern" input | cp $1 $1.bac
You can use xargs:
grep 'pattern' input | xargs -I% cp "%" "%.bac"
You can use $() to interpolate the output of a command. So, you could use kill -9 $(grep -hP '^\d+$' $(ls -lad /dir/*/pid | grep -P '/dir/\d+/pid' | awk '{ print $9 }')) if you wanted to.
In addition to Chris Jester-Young good answer, I would say that xargs is also a good solution for these situations:
grep ... `ls -lad ... | awk '{ print $9 }'` | xargs kill -9
will make it. All together:
grep -hP '^\d+$' `ls -lad /dir/*/pid | grep -P '/dir/\d+/pid' | awk '{ print $9 }'` | xargs kill -9
For completeness, I'll also mention command substitution and explain why this is not recommended:
cp $(grep -l "pattern" input) directory/
(The backtick syntax cp `grep -l "pattern" input` directory/ is roughly equivalent, but it is obsolete and unwieldy; don't use that.)
This will fail if the output from grep produces a file name which contains whitespace or a shell metacharacter.
Of course, it's fine to use this if you know exactly which file names the grep can produce, and have verified that none of them are problematic. But for a production script, don't use this.
Anyway, for the OP's scenario, where you need to refer to each match individually and add an extension to it, the xargs or while read alternatives are superior anyway.
In the worst case (meaning problematic or unspecified file names), pass the matches to a subshell via xargs:
grep -l "pattern" input |
xargs -r sh -c 'for f; do cp "$f" "$f.bac"; done' _
... where obviously the script inside the for loop could be arbitrarily complex.
In the ideal case, the command you want to run is simple (or versatile) enough that you can simply pass it an arbitrarily long list of file names. For example, GNU cp has a -t option to facilitate this use of xargs (the -t option allows you to put the destination directory first on the command line, so you can put as many files as you like at the end of the command):
grep -l "pattern" input | xargs cp -t destdir
which will expand into
cp -t destdir file1 file2 file3 file4 ...
for as many matches as xargs can fit onto the command line of cp, repeated as many times as it takes to pass all the files to cp. (Unfortunately, this doesn't match the OP's scenario; if you need to rename every file while copying, you need to pass in just two arguments per cp invocation: the source file name and the destination file name to copy it to.)
So in other words, if you use the command substitution syntax and grep produces a really long list of matches, you risk bumping into ARG_MAX and "Argument list too long" errors; but xargs will specifically avoid this by instead copying only as many arguments as it can safely pass to cp at a time, and running cp multiple times if necessary instead.
The above will still work incorrectly if you have file names which contain newlines. Perhaps see also https://mywiki.wooledge.org/BashFAQ/020
#!/bin/bash
for f in files; do
if grep -q PATTERN "$f"; then
echo cp -v "$f" "${f}.bac"
fi
done
files can be *.txt or *.text which basically means files ending in *.txt or *text or replace with something that you want/need, of course replace PATTERN with yours. Remove echo if you're satisfied with the output. For a recursive solution take a look at the bash shell option globstar
I'm trying to get rid of a hacker issue on some of my wordpress installs.
This guy puts 9 lines of code in the head of multiple files on my server... I'm trying to use grep and sed to solve this.
Im trying:
grep -r -l "//360cdn.win/c.css" | xargs -0 sed -e '1,9d' < {}
But nothing is happening, if I remove -0 fromxargs, the result of the files found are clean, but they are not overwriting the origin file with thesed` result, can anyone help me with that?
Many thanks!
You should use --null option in grep command to output a NUL byte or \0 after each filename in the grep output. Also use -i.bak in sed for inline editing of each file:
grep -lR --null '//360cdn.win/c\.css' . | xargs -0 sed -i.bak '1,9d'
What's wrong with iterating over the files directly¹?
And you might want to add the -i flat to sed so that files are edited in-place
grep -r -l "//360cdn.win/c.css" | while read f
do
sed -e '1,9d' -i "${f}"
done
¹ well, you might get problems if your files contain newlines and the like.
but then...if your website contains files with newlines, you probably have other problems anyhow...
I would like to do a find in some dir, and do a awk on the files in this direcory, and then replace the original files by each result.
find dir | xargs cat | awk ... | mv ... > filename
So I need the filename (of each of the files found by find) in the last command. How can I do that?
I would use a loop, like:
for filename in `find . -name "*test_file*" -print0 | xargs -0`
do
# some processing, then
echo "what you like" > "$filename"
done
EDIT: as noted in the comments, the benefits of -print0 | xargs -0 are lost because of the for loop. And filenames containing a white space are still not handled correctly.
The following while loop would not handle unusual filenames neither (good to know it, though it was not in the question), but filenames with a standard white space at least, so it works better, indeed:
find . -name "*test*file*" -print > files_list
while IFS= read -r filename
do
# some process
echo "what you like" > "$filename"
done < files_list
You could do something like this (but I wouldn't recommend it at all).
find dir -print0 |
xargs -0 -n 2 awk -v OFS='\0' '<process the input and write to temporary file>
END {print "temporaryfile", FILENAME}' |
xargs -0 -n 2 mv
This passes the files to awk directly two at a time (which avoids the problem with your original where cat will get hundreds (perhaps more) files as arguments all at once and spit all their content at awk via standard input at once and thus lose their individual contents and filenames entirely).
It then has awk write the processed output to a temporary file and then outputs the temporary filename and the original filename where xargs picks them up (again two at a time) and runs mv on the pairs of temporary file/original file names.
As I said at the beginning however this is a terrible way to do this.
If you have a new enough version of GNU awk (version 4.1.0 or newer) then you could just use the -i (in-place) argument to awk and use (I believe):
find dir | xargs awk -i '......'
Without that I would use a while loop of the form in Bash FAQ 001 to read the find output line-by-line and operate on it in the loop.
I have got the following Problem.
I´m doing a grep like:
$command = grep -r -i --include=*.cfg 'host{' /omd/sites/mesh/etc/icinga/conf.d/objects
I got the following output:
/omd/sites/mesh/etc/icinga/conf.d/objects/testsystem/test1.cfg:define host{
/omd/sites/mesh/etc/icinga/conf.d/objects/testsystem/test2.cfg:define host{
/omd/sites/mesh/etc/icinga/conf.d/objects/testsystem/test3.cfg:define host{
...
for all *.cfg files.
With exec($command,$array)
I passed the result in an array.
Is it possible to get only the filenames as result of the grep-command.
I have tried the following:
$Command= grep -l -H -r -i --include=*.cfg 'host{' /omd/sites/mesh/etc/icinga/conf.d/objects
but I got the same result.
I know that on the forum a similar topic exists.(How can I use grep to show just filenames (no in-line matches) on linux?), but the solution doesn´t work.
With "exec($Command,$result_array)" I try to get an array with the results.
The mentioned solutions works all, but I can´t get an resultarray with exec().
Can anyone help me?
Yet another simpler solution:
grep -l whatever-you-want | xargs -L 1 basename
or you can avoid xargs and use a subshell instead, if you are not using an ancient version of the GNU coreutils:
basename -a $(grep -l whatever-you-want)
basename is the bash straightforward solution to get a file name without path. You may also be interested in dirname to get the path only.
GNU Coreutils basename documentation
Is it possible to get only the filenames as result of the grep command.
With grep you need the -l option to display only file names.
Using find ... -execdir grep ... \{} + you might prevent displaying the full path of the file (is this what you need?)
find /omd/sites/mesh/etc/icinga/conf.d/objects -name '*.cfg' \
-execdir grep -r -i -l 'host{' \{} +
In addition, concerning the second part of your question, to read the result of a command into an array, you have to use the syntax: IFS=$'\n' MYVAR=( $(cmd ...) )
In that particular case (I formatted as multiline statement in order to clearly show the structure of that expression -- of course you could write as a "one-liner"):
IFS=$'\n' MYVAR=(
$(
find objects -name '*.cfg' \
-execdir grep -r -i -l 'host{' \{} +
)
)
You have then access to the result in the array MYVAR as usual. While I while I was testing (3 matches in that particular case):
sh$ echo ${#MYVAR[#]}
3
sh$ echo ${MYVAR[0]}
./x y.cfg
sh$ echo ${MYVAR[1]}
./d.cfg
sh$ echo ${MYVAR[2]}
./e.cfg
# ...
This should work:
grep -r -i --include=*.cfg 'host{' /omd/sites/mesh/etc/icinga/conf.d/objects | \
awk '{print $1}' | sed -e 's|[^/]*/||g' -e 's|:define$||'
The awk portion finds the first field in it and the sed command trims off the path and the :define.
I'm having some rather unusual problems using grep in a bash script. Below is an example of the bash script code that I'm using that exhibits the behaviour:
UNIQ_SCAN_INIT_POINT=1
cat "$FILE_BASENAME_LIST" | uniq -d >> $UNIQ_LIST
sed '/^$/d' $UNIQ_LIST >> $UNIQ_LIST_FINAL
UNIQ_LINE_COUNT=`wc -l $UNIQ_LIST_FINAL | cut -d \ -f 1`
while [ -n "`cat $UNIQ_LIST_FINAL | sed "$UNIQ_SCAN_INIT_POINT"'q;d'`" ]; do
CURRENT_LINE=`cat $UNIQ_LIST_FINAL | sed "$UNIQ_SCAN_INIT_POINT"'q;d'`
CURRENT_DUPECHK_FILE=$FILE_DUPEMATCH-$CURRENT_LINE
grep $CURRENT_LINE $FILE_LOCTN_LIST >> $CURRENT_DUPECHK_FILE
MATCH=`grep -c $CURRENT_LINE $FILE_BASENAME_LIST`
CMD_ECHO="$CURRENT_LINE matched $MATCH times," cmd_line_echo
echo "$CURRENT_DUPECHK_FILE" >> $FILE_DUPEMATCH_FILELIST
let UNIQ_SCAN_INIT_POINT=UNIQ_SCAN_INIT_POINT+1
done
On numerous occasions, when grepping for the current line in the file location list, it has put no output to the current dupechk file even though there have definitely been matches to the current line in the file location list (I ran the command in terminal with no issues).
I've rummaged around the internet to see if anyone else has had similar behaviour, and thus far all I have found is that it is something to do with buffered and unbuffered outputs from other commands operating before the grep command in the Bash script....
However no one seems to have found a solution, so basically I'm asking you guys if you have ever come across this, and any idea/tips/solutions to this problem...
Regards
Paul
The `problem' is the standard I/O library. When it is writing to a terminal
it is unbuffered, but if it is writing to a pipe then it sets up buffering.
try changing
CURRENT_LINE=`cat $UNIQ_LIST_FINAL | sed "$UNIQ_SCAN_INIT_POINT"'q;d'`
to
CURRENT LINE=`sed "$UNIQ_SCAN_INIT_POINT"'q;d' $UNIQ_LIST_FINAL`
Are there any directories with spaces in their names in $FILE_LOCTN_LIST? Because if they are, those spaces will need escaped somehow. Some combination of find and xargs can usually deal with that for you, especially xargs -0
A small bash script using md5sum and sort that detects duplicate files in the current directory:
CURRENT="" md5sum * |
sort |
while read md5sum filename;
do
[[ $CURRENT == $md5sum ]] && echo $filename is duplicate;
CURRENT=$md5sum;
done
you tagged linux, some i assume you have tools like GNU find,md5sum,uniq, sort etc. here's a simple example to find duplicate files
$ echo "hello world">file
$ md5sum file
6f5902ac237024bdd0c176cb93063dc4 file
$ cp file file1
$ md5sum file1
6f5902ac237024bdd0c176cb93063dc4 file1
$ echo "blah" > file2
$ md5sum file2
0d599f0ec05c3bda8c3b8a68c32a1b47 file2
$ find . -type f -exec md5sum "{}" \; |sort -n | uniq -w32 -D
6f5902ac237024bdd0c176cb93063dc4 ./file
6f5902ac237024bdd0c176cb93063dc4 ./file1