I have the following problem.
Got a file which includes certain paths/files of a FS.
These for some reason do include the whole range of special characters, like space, single/double quotes, even sometimes the Copyright ASCII.
I need to run each line of the file and pass it to another command.
What I tried so far is:
<input_file xargs -I % command %
Which was working until I got this message from xargs
xargs: unmatched single quote; by default quotes are special to xargs unless you use the -0 option
But usinf this option did not work at all for me
xargs: argument line too long
Does anybody have a solution which does work ok with special characters.
Doesn't have to be with xargs, but I need to pass the line as it is to the command.
Many thanks in advance.
You should separate the filenames with the \0 NULL character for processing.
This can be done with
find . <args> -print0 | xargs -0
or if you must process the file with filenames, change the '\n` to '\0', e.g.
tr '\n' '\0' < filename | xargs -0 -n1 -I% echo "==%=="
the -n 1 says,
-n max-args
Use at most max-args arguments per command line.
and you should to use "%" quotes to enclosing %
The xargs -0 -n1 -I% echo "==%==" solution didn't work for me on my Mac OS X, so I found another one.
<input_with_single_quotes cat | sed "s/'/\\\'/" | xargs -I {} echo {}
This replaces the ' character with \' that works well as an input to the commands in xargs.
Related
If I execute a find command, with grep and sort etc. in the local command line, I get returned lines like so:
# find ~/logs/ -iname 'status' | xargs grep 'last seen' | sort --field-separator=: -k 4 -g
0:0:line:1
0:0:line:2
0:0:line:3
If I execute the same command over ssh, the returned text prints without newlines, like so:
# VARcmdChk="$(ssh ${VARuser}#${VARserver} "find ~/logs/ -iname 'status' | xargs grep 'last seen' | sort --field-separator=: -k 4 -g")"
# echo ${VARcmdChk}
0:0:line:1 0:0:line:2 0:0:line:3
I'm trying to understand why ssh is sanitising the returned text, so that newlines are converted to spaces. I have not yet tried output'ing to file, and then using scp to pull that back. Seems a waste, since I just want to view the remote results locally.
When you echo the variable VARcmdChk, you should enclose it with ".
$ VARcmdChk=$(ssh ${VARuser}#${VARserver} "find tmp/ -iname status -exec grep 'last seen' {} \; | sort --field-separator=: -k 4 -g")
$ echo "${VARcmdChk}"
last seen:11:22:33:44:55:66:77:88:99:00
last seen:00:99:88:77:66:55:44:33:22:11
Note that I've replaced your xargs for -exec.
Ok, the question is a duplicate of this one, Why does shell Command Substitution gobble up a trailing newline char?, so partly answered.
However, I say partly, as the answers tell you the reasons for this happening as such, but the only clue to a solution is a small answer right at the end.
The solution is to quote the echo argument, as the solution suggests:
# VARcmdChk="$(ssh ${VARuser}#${VARserver} "find ~/logs/ -iname 'status' | xargs grep 'last seen' | sort --field-separator=: -k 4 -g")"
# echo "${VARcmdChk}"
0:0:line:1
0:0:line:2
0:0:line:3
but there is no explanation as to why this works as such, since assumption is that the variable is a string, so should print as expected. However, reading Expansion of variable inside single quotes in a command in Bash provides the clue regarding preserving newlines etc. in a string. Placing the variable to be printed by echo into quotes preserves the contents absolutely, and you get the expected output.
The echo of the variable is why its putting it all into one line. Running the following command will output the results as expected:
ssh ${VARuser}#${VARserver} "find ~/logs/ -iname 'status' | xargs grep 'last seen' | sort --field-separator=: -k 4 -g"
To get the command output to have each result on a new line, like it does when you run the command locally you can use awk to split the results onto a new line.
awk '{print $1"\n"$2}'
This method can be appended to your command like this:
echo ${VARcmdChk} | awk '{print $1"\n"$2"\n"$3"\n"$4}'
Alternatively, you can put quotes around the variable as per your answer:
echo "${VARcmdChk}"
I have seen the Inone option for linux command xargs ,but I googled and did not find out what the option means.
seq 2 | xargs -Inone cat file
Using seq with xargs:
This is a clever use of xargs and seq to avoid writing a loop. It is basically the equivalent of:
for i in {1..2}; do
cat file
done
That is, it will run cat file once for each line of output from the seq command. The -Inone simply prevents xargs from appending the value read from seq to the command; see the xargs man page for details on the -I option:
-I replace-str
Replace occurrences of replace-str in the initial-ar‐
guments with names read from standard input. Also,
unquoted blanks do not terminate input items; instead
the separator is the newline character. Implies -x
and -L 1.
I have a file that has a list of malicious file names. There are many file names contains blank spaces. I need to find them and change their permissions. I have tried the following:
grep -E ". " suspicious.txt | xargs -0 chmod 000
But I am getting an error:
:File name too long
An ideas?
OK, you have one filename per line in your file, and the problem is that xargs without -0 will treat spaces and tabs as well as the newlines as file separators, while xargs with -0 expects the filenames to be separated by NUL characters and won't care about the newlines at all.
So turn the newlines into NULs before feeding the result into the xargs -0 command:
grep -E ". " suspicious.txt | tr '\n' '\0' | xargs -0 chmod 000
Update:
See Mark Reeds correct answer. This was wrong because nulls were needed for the filenames from the file, not the filenames generated by grep.
Original:
You need something more like this:
grep -Z -E ". " suspicious.txt | xargs -0 chmod 000
From xargs man page:
Because Unix filenames can contain blanks and newlines, this default behaviour is often problematic; filenames containing blanks and/or newlines are incorrectly processed by xargs. In these situations it is better to use the -0 option, which prevents such problems. When using this option you will need to ensure that the program which produces the input for xargs also uses a null character as a separator.
From grep man page:
-Z, --null
Output a zero byte (the ASCII NUL character) instead of the character that normally follows a file name. For example, grep -lZ outputs a zero byte after each file name instead of the usual newline. This option makes the output unambiguous, even in the presence of file names containing unusual characters like newlines. This option can be used with commands like find -print0, perl -0, sort -z, and xargs -0 to process arbitrary file names, even those that contain newline characters.
I'm trying to get rid of a hacker issue on some of my wordpress installs.
This guy puts 9 lines of code in the head of multiple files on my server... I'm trying to use grep and sed to solve this.
Im trying:
grep -r -l "//360cdn.win/c.css" | xargs -0 sed -e '1,9d' < {}
But nothing is happening, if I remove -0 fromxargs, the result of the files found are clean, but they are not overwriting the origin file with thesed` result, can anyone help me with that?
Many thanks!
You should use --null option in grep command to output a NUL byte or \0 after each filename in the grep output. Also use -i.bak in sed for inline editing of each file:
grep -lR --null '//360cdn.win/c\.css' . | xargs -0 sed -i.bak '1,9d'
What's wrong with iterating over the files directly¹?
And you might want to add the -i flat to sed so that files are edited in-place
grep -r -l "//360cdn.win/c.css" | while read f
do
sed -e '1,9d' -i "${f}"
done
¹ well, you might get problems if your files contain newlines and the like.
but then...if your website contains files with newlines, you probably have other problems anyhow...
$ find ~/AppData/Local/atom/ -name atom.sh -type f | grep -FzZ 'cli/atom.sh' | sed '/^\s*$/d' | cat -n
1 /c/Users/devuser/AppData/Local/atom/app-1.12.1/resources/cli/atom.sh
2 /c/Users/devuser/AppData/Local/atom/app-1.12.2/resources/cli/atom.sh
3
I tried a number of sed/awk-based options to get rid of the blank line. (#3 in the output). But I can't quite get it right...
I need to get the last line into a variable...
The below actual command I am working with fails to give an output...
find ~/AppData/Local/atom/ -name atom.sh -type f | grep -FzZ 'cli/atom.sh' | sed '/^$/d' | tail -n 1
Below sed command would remove all the empty lines.
sed '/^$/d' file
or
Below sed command would remove all the empty lines and also the lines having only spaces.
sed '/^[[:blank:]]*$/d' file
Add -i parameter to do an in-place edit.
sed -i '/^[[:blank:]]*$/d' file
The problem is the -Z flag of grep. It causes grep to terminate matched lines with a null character instead of a newline. This won't work well with sed, because sed processes input line by line, and it expects each line to be terminated by a newline. In your example grep doesn't emit any newline characters, so as far as sed is concerned, it receives a block of text without a terminating newline, so it processes it as a single line, and so the pattern /^\s*$/ doesn't match anything.
Furthermore, the -z flag would only make sense if the filenames in the output of find were terminated by null characters, that is with the -print0 flag. But you're not using that, so the -z flag in grep is pointless. And the -Z flag is pointless too, because that should be used when the next command in the pipeline expects null-terminated records, which is not your case.
Do like this:
find ~/AppData/Local/atom/ -name atom.sh -type f -print0 | grep -zF 'cli/atom.sh' | tr '\0' '\n' | tail -n 1
Not all sed's support the token \s for any whitespace character.
You can use the character class [[:blank:]] (for space or tab), or [[:space:]] (for any whitespace):
sed '/^[[:blank:]]*$/d'
sed '/^[[:space:]]*$/d'