I have the following files :
#ls
test test2 test3 test4
now according to xargs -0 the input is trimmed based on null instead of whitespace, hence as there is no null in my input the 4 files should be printed in one echo command . The result is one echo command However there is a new line between each file :
#ls | xargs -0
test
test2
test3
test4
To be sure there is one command only :
ls | xargs -0 -p
echo test
test2
test3
test4
?...n
why there is new line between each file ?
When ls prints to a TTY it formats the file names in columns, but when it's writing to a file, pipe, or other non-TTY it behaves like ls -1 and prints one file name per line. You can check this by running ls | cat in place of ls.
With ls | xargs -0, the newlines are coming from ls, not from xargs. xargs is indeed reading a single string as you expect it to, but that string has newlines.
See also:
BashFAQ - Why you shouldn't parse the output of ls(1)
You are not supplying a command to xargs which is unusual. The default command xargs uses when none is supplied is /bin/echo. xargs reads the output from ls and splits it into words; each word becomes a parameter for the command xargs executes. The -0 option makes the null character the word delimiter used for the splitting. Because null does not appear in the output of ls, the entire output is considered one word and passed to /bin/echo as a single parameter, effectively resulting in the command line
/bin/echo 'test
test2
test3
test4'
That the output from ls contains newlines, on the other hand, is a property of ls when the putput is not a terminal, as John correctly explained.
When xarging over files use find instead of ls!
$ find /my/path -print0 | xargs -0 -p
With -print0 you can handle also files where space (0x20) is used in filename.
Related
I just found difference between two commands:
echo sum.txt| xargs cat
This will output content of sum.txt
echo sum.txt| xargs -0 cat
This shows error:
cat: sum.txt
: No such file or directory
I know -0 will treat null bytes as delimiter. And i think the new line starts with : is because echo command produce new lines. And doesn't produce output like:
cat: sum.txt: No such file or directory
But if echo produce new lines why the first command can succeed? since xargs use white spaces as delimiter by default.
I think what is happening will become more clear if you replace cat with echo -n as an experiment.
echo sum.txt| xargs echo -n
echo sum.txt | xargs -0 echo -n
In the first example, xargs breaks on the newline and discards the newline, leaving just sum.txt.
In the second example, xargs breaks on the end of the EOF at the end of the output of the first echo, resulting in the string 'sum.txt\n'.
I have seen the Inone option for linux command xargs ,but I googled and did not find out what the option means.
seq 2 | xargs -Inone cat file
Using seq with xargs:
This is a clever use of xargs and seq to avoid writing a loop. It is basically the equivalent of:
for i in {1..2}; do
cat file
done
That is, it will run cat file once for each line of output from the seq command. The -Inone simply prevents xargs from appending the value read from seq to the command; see the xargs man page for details on the -I option:
-I replace-str
Replace occurrences of replace-str in the initial-ar‐
guments with names read from standard input. Also,
unquoted blanks do not terminate input items; instead
the separator is the newline character. Implies -x
and -L 1.
I'm currently making a shell program and I want to display the total amount of bytes in a specific file using the pipe command. I know that the pipe command takes whatever is on the left side and gives it to the right as input. (Assuming you are in the directory the file is in)
I know that the command (wc -c) displays the number of bytes in a file but I'm not sure how to pipe it. What I've tried was:
ls fileName.sh | wc -c
wc takes the filename as argument, not as input. Try this:
wc -c fileName.sh
The wc program takes multiple arguments. You can do this to apply it to all entries in the current working directory:
wc -c $(ls)
Another approach is to use xargs to convert input to arguments:
ls | xargs wc -c
You may need to use a more complex line if you have spaces in your filenames. ls can output a single file per line, and xargs can be told to split only on \n:
ls -1 | xargs -d '\n' wc -c
If you prefer to use find instead of ls (a more powerful tool), the -print0 option for find plays along with the -0 option to xargs.
I am having a problem where my perl script will fail upon having an input piped, but works fine when I just list all the file names individually.
For reference, input of the perl script is read with while(<>).
Example:
script.pl file1.tag file2.tag file3.tag
runs fine.
But the following all fail.
find ./*.tag | chomp | script.pl
ls -l *.tag | perl -pe 's/\n/ /g' | script.pl
find ./*.tag | perl -pe 's/\n/ /g' | script.pl
I also tested dumping it into a text file and catting that into the perl:
cat files.text | script.pl
All of them fail the same way. It is like the script is passed no input arguments and the program just finishes.
From perldoc perlop:
The null filehandle <> is special [...] Input from <> comes either from standard input, or from each file listed on the command line. Here's how it works: the first time <> is evaluated, the #ARGV array is checked, and if it is empty, $ARGV[0] is set to -, which when opened gives you standard input. The #ARGV array is then processed as a list of filenames.
You're not passing any command line arguments to your Perl scripts, so everything you pipe into them is read into STDIN instead of being treated as filenames:
$ echo foo > foo.txt
$ echo bar > bar.txt
$ ls | perl -e 'print "<$_>\n" while <>'
<bar.txt
>
<foo.txt
>
Notice that the files foo.txt and bar.txt are not actually read; all we get is the file names. If you want the files to be opened and read, you have to pass them as command line arguments or explicitly set #ARGV:
$ perl -e 'print "<$_>\n" while <>' *
<bar
>
<foo
>
If you have a large number of files, like you're likely to get from find, you should use xargs as Dyno Hongjun Fu suggested.
However, you don't need find, ls, cat, or your Perl one-liner to run your script on all the .tag files in the current directory. Simply do:
script.pl *.tag
you need xargs, e.g.
find ./ -type f -name "*.tag" | xargs -i script.pl {}
what is chomp?
How do I pipe the output of grep as the search pattern for another grep?
As an example:
grep <Search_term> <file1> | xargs grep <file2>
I want the output of the first grep as the search term for the second grep. The above command is treating the output of the first grep as the file name for the second grep. I tried using the -e option for the second grep, but it does not work either.
You need to use xargs's -i switch:
grep ... | xargs -ifoo grep foo file_in_which_to_search
This takes the option after -i (foo in this case) and replaces every occurrence of it in the command with the output of the first grep.
This is the same as:
grep `grep ...` file_in_which_to_search
Try
grep ... | fgrep -f - file1 file2 ...
If using Bash then you can use backticks:
> grep -e "`grep ... ...`" files
the -e flag and the double quotes are there to ensure that any output from the initial grep that starts with a hyphen isn't then interpreted as an option to the second grep.
Note that the double quoting trick (which also ensures that the output from grep is treated as a single parameter) only works with Bash. It doesn't appear to work with (t)csh.
Note also that backticks are the standard way to get the output from one program into the parameter list of another. Not all programs have a convenient way to read parameters from stdin the way that (f)grep does.
I wanted to search for text in files (using grep) that had a certain pattern in their file names (found using find) in the current directory. I used the following command:
grep -i "pattern1" $(find . -name "pattern2")
Here pattern2 is the pattern in the file names and pattern1 is the pattern searched for
within files matching pattern2.
edit: Not strictly piping but still related and quite useful...
This is what I use to search for a file from a listing:
ls -la | grep 'file-in-which-to-search'
Okay breaking the rules as this isn't an answer, just a note that I can't get any of these solutions to work.
% fgrep -f test file
works fine.
% cat test | fgrep -f - file
fgrep: -: No such file or directory
fails.
% cat test | xargs -ifoo grep foo file
xargs: illegal option -- i
usage: xargs [-0opt] [-E eofstr] [-I replstr [-R replacements]] [-J replstr]
[-L number] [-n number [-x]] [-P maxprocs] [-s size]
[utility [argument ...]]
fails. Note that a capital I is necessary. If i use that all is good.
% grep "`cat test`" file
kinda works in that it returns a line for the terms that match but it also returns a line grep: line 3 in test: No such file or directory for each file that doesn't find a match.
Am I missing something or is this just differences in my Darwin distribution or bash shell?
I tried this way , and it works great.
[opuser#vjmachine abc]$ cat a
not problem
all
problem
first
not to get
read problem
read not problem
[opuser#vjmachine abc]$ cat b
not problem xxy
problem abcd
read problem werwer
read not problem 98989
123 not problem 345
345 problem tyu
[opuser#vjmachine abc]$ grep -e "`grep problem a`" b --col
not problem xxy
problem abcd
read problem werwer
read not problem 98989
123 not problem 345
345 problem tyu
[opuser#vjmachine abc]$
You should grep in such a way, to extract filenames only, see the parameter -l (the lowercase L):
grep -l someSearch * | xargs grep otherSearch
Because on the simple grep, the output is much more info than file names only. For instance when you do
grep someSearch *
You will pipe to xargs info like this
filename1: blablabla someSearch blablabla something else
filename2: bla someSearch bla otherSearch
...
Piping any of above line makes nonsense to pass to xargs.
But when you do grep -l someSearch *, your output will look like this:
filename1
filename2
Such an output can be passed now to xargs
I have found the following command to work using $() with my first command inside the parenthesis to have the shell execute it first.
grep $(dig +short) file
I use this to look through files for an IP address when I am given a host name.