I'm trying the whole day to find a good way for parsing some strings with a shell script. the strings are used as calling parameter for some applications.
they looks like:
parsingParams -c "id=uid5 prog=/opt/bin/example arg=\"-D -t5 >/dev/null 1>&2\" info='fdhff fd'" start
I'm only allowed to use shell-script. I tried to use some sed and cut commands but nothing works fine.
My tries are like:
prog=$(echo $# | cut -d= -f3 | sed 's|\s.*$||')
that return the correct value of prog but for the value of arg I couldn't find a good way to get it.
the info parameter is optional also it may be left.
may any one have a good idea that can solve this problem?
many thanks in advance
Looks like you could use eval to let the shell parse your input string, but if you don't control the input (if it comes from an unreliable source), that will introduce a major vulnerability (imagine an attacker somehow passes -c "rm -rf /" to your program).
A safer way would be to explicitly specify allowed forms of user input.
The problem you have with splitting on space (with cut) if the space is quoted, can be avoided if you specify valid fields (content, not separator), for example in GNU awk, you can use FPAT:
$ params="id=uid5 prog=/opt/bin/example arg=\"-D -t5 >/dev/null 1>&2\" info='fdhff fd'"
$ awk -v FPAT="[^=]+=(\"[^\"]*\"|'[^']*'|[^ ]*) *" '{for (i=1; i<=NF; i++) print $i}' <<<"$params"
id=uid5
prog=/opt/bin/example
arg="-D -t5 >/dev/null 1>&2"
info='fdhff fd'
Valid fields will be in one of the following forms:
var="val with spaces"
var='val with spaces'
var=val_no_spaces
Now with assignments split (one per line, assuming newline is not allowed in params), you can process them further, even with cut:
$ awk ... | cut -d $'\n' -f3
arg="-D -t5 >/dev/null 1>&2"
eval
$ eval "id=uid5 prog=/opt/bin/example arg=\"-D -t5 >/dev/null 1>&2\" info='fdhff fd'"
$ echo $id
uid5
$ echo $prog
/opt/bin/example
$ echo $arg
-D -t5 >/dev/null 1>&2
$ echo $info
fdhff fd
Related
I'm learning about grep commands.
I want to make a program that when a user enters more than one word, outputs a line containing the word in the data file.
So I connected the words that the user typed with '|' and put them in the grep command to create the program I intended.
But this is OR operation. I want to make AND operation.
So I learned how to use AND operation with grep commands as follows.
cat <file> | grep 'pattern1' | grep 'pattern2' | grep 'pattern3'
But I don't know how to put the user input in the 'pattern1', 'pattern2', 'pattern3' position. Because the number of words the user inputs is not determined.
As user input increases, grep must be executed using more and more pipes, but I don't know how to build this part.
The user input is as follows:
$ [the name of my program] 'pattern1' 'pattern2' 'pattern3' ...
I'd really appreciate your help.
With grep -f you can grep multiple items, when each of them is on a line in a file.
With <(command) you can let Bash think that the result of command is a file.
With printf "%s\n" and a list of arguments, each argument is printed on a new line.
Together:
grep -f <(printf "%s\n" "$#") datafile
suggesting to use awk pattern logic:
awk '/RegExp-pattern-1/ && /RegExp-pattern-2/ && /RegExp-pattern-3/ 1' input.txt
The advantages: you can play with logic operators && || on RegExp patterns. And your are scanning the whole file once.
The disadvantages: must provide files list (can't traverse sub directories), and limited RegExp syntax compared to grep -E or grep -P
In principle, what you are asking could be done with a loop with output to a temporary file.
file=inputfile
temp=$(mktemp -d -t multigrep.XXXXXXXXX) || exit
trap 'rm -rf "$temp"' ERR EXIT
for regex in "$#"; do
grep "$regex" "$file" >"$temp"/output
mv "$temp"/output "$temp"/input
file="$temp"/input
done
cat "$temp"/input
However, a better solution is probably to arrange for Awk to check for all the patterns in one go, and avoid reading the same lines over and over again.
Passing the arguments to Awk with quoting intact is not entirely trivial. Here, we simply pass them as command-line arguments and process those into an array within the Awk script itself.
awk 'BEGIN { for(i=1; i<ARGC; ++i) a[i]=ARGV[i];
ARGV[1]="-"; ARGC=1 }
{ for(n=1; n<=i; ++n) if ($0 !~ a[n]) next; }1' "$#" <file
In brief, in the BEGIN block, we copy the command-line arguments from ARGV to a, then replace ARGV and ARGC to pass Awk a new array of (apparent) command-line arguments which consists of just - which means to read standard input. Then, we simply iterate over a and skip to the next line if the current input line from standard input does not match. Any remaining lines have matched all the patterns we passed in, and are thus printed.
I have a file, the content of file has a string like this:
'/ad/e','#'.base64_decode("ZXZhbA==").'($zad)', 'add'
I want to check the file has this string. But when I use grep to check, It always return false. I try some ways:
grep "'/ad/e','#'.base64_decode("ZXZhbA==").'($zad)', 'add'" foo.txt
grep "'/ad/e','#'\.base64_decode\("ZXZhbA\=\="\)\.'\(\$zad\)', 'add'" foo.txt
str="'/ad/e','#'\.base64_decode\("ZXZhbA\=\="\)\.'\(\$zad\)', 'add'"
grep "$str" foo.txt
Can you help me? Maybe, another command line.
This is my case:
while read str; do
if [ ! -z "$str" ]; then
if grep -Fxq "$str" "$file_path"; then
do somthing
fi
fi
done < <(cat /usr/local/caotoc/db.dat)
Thank you so much!
First, you need to make sure the string is quoted properly. This is a bit of an art form, since your string contains both single and double quotes.
One thought would be to use read and a here-document to avoid having to escape anything.
Second, you need to use -F to perform exact string matching instead of more general regular-expression matching.
IFS= read -r str <<'EOF'
'/ad/e','#'.base64_decode("ZXZhbA==").'($zad)', 'add'
EOF
grep -F "$str" foo.txt
Based on the update, you can use a simple loop to read them one at a time.
while IFS= read -r str; do
grep -F "$str" foo.txt
done < /usr/local/caotoc/db.dat
You may be able to simply use the -f option to grep, which will cause grep to output lines from foo.txt that match any line from db.dat.
grep -f /usr/local/caotoc/db.dat -F foo.txt
Instead of trying to workaround regexes, the simplest way is to turn off regular expressions using -F (or --fixed-strings) option, which makes grep act like a simple string search
-F, --fixed-strings PATTERN is a set of newline-separated strings
like this:
grep -F "'/ad/e','#'.base64_decode(\"ZXZhbA==\").'(\$zad)', 'add'" test
Note: because of the shell, you still need to escape:
double quotes
dollar sign or else $zad is evaluated as an environment variable
I am trying to just echo a command within my bash script code.
OVERRUN_ERRORS="$ifconfig | egrep -i "RX errors" | awk '{print $7}'"
echo ${OVERRUN_ERRORS}
however it gives me an error and the $7 does not show up in the command. I have to store it in a variable, because I will process the output (OVERRUN_ERRORS) at a later point in time. What's the right syntax for doing this? Thanks.
On Bash Syntax
foo="bar | baz"
...is assigning the string "bar | baz" to the variable named foo; it doesn't run bar | baz as a pipeline. To do that, you want to use command substitution, in either its modern $() syntax or antiquated backtick-based form:
foo="$(bar | baz)"
On Storing Code For Later Execution
Since your intent isn't clear in the question --
The correct way to store code is with a function, whereas the correct way to store output is in a string:
# store code in a function; this also works with pipelines
get_rx_errors() { cat /sys/class/net/"$1"/statistics/rx_errors; }
# store result of calling that function in a string
eth0_errors="$(get_rx_errors eth0)"
sleep 1 # wait a second for demonstration purposes, then...
# compare: echoing the stored value, vs calculating a new value
echo "One second ago, the number of rx errors was ${eth0_errors}"
etho "Right now, it is $(get_rx_errors eth0)"
See BashFAQ #50 for an extended discussion of the pitfalls of storing code in a string, and alternatives to same. Also relevant is BashFAQ #48, which describes in detail the security risks associated with a eval, which is often suggested as a workaround.
On Collecting Interface Error Counts
Don't use ifconfig, or grep, or awk for this at all -- just ask your kernel for the number you want:
#!/bin/bash
for device in /sys/class/net/*; do
[[ -e $device/statistics/rx_errors ]] || continue
rx_errors=$(<"${device}/statistics/rx_errors")
echo "Number of rx_errors for ${device##*/} is $rx_errors"
done
Use $(...) to capture the output of a command, not double quotes.
overrun_errors=$(ifconfig | egrep -i "RX errors" | awk '{print $7}')
Your double quotes around RX errors are a problem. Try;
OVERRUN_ERRORS="$ifconfig | egrep -i 'RX errors' | awk '{print $7}'"
To see the commands as they are executing, you can use
set -v
or
set -x
For example;
set -x
OVERRUN_ERRORS="$ifconfig | egrep -i 'RX errors' | awk '{print $7}'"
set +x
I have an array of numbers - sipPeers.
I want to iterate trough that array and pass each value into the command
asterisk -rx "sip show peer 1234" - where 1234 is obviously an extension.
The output of sip show peer is piped and manipulated twice to output one value which I want to store in a second array sipUserAgent.
temp="asterisk -rx \"sip show peer "${sipPeers[q]}"\" | grep Useragent | awk \'{print$3}\'" #(1)
echo $temp #(2)
sipUserAgent[q]=$($temp) #(3)
The output of one iteration yields the constructed command as a string (2) and the tries to execute it (3):
asterisk -rx "sip show peer 564" | grep Useragent | awk '{print }'
No such command '"sip' (type 'core show help sip' for other possible commands)
If I copy and paste the echo'd command it runs, but when the script executes I get that error. Somewhere the " character changes meaning when the scipt executes it?
Don't try to store the command in a variable before executing it. There are all sorts of problems with doing this, and you're running into several of them (see BashFAQ #50: I'm trying to put a command in a variable, but the complex cases always fail!). Just put the pipeline directly inside $( ... ) where it belongs:
sipUserAgent[q]=$(asterisk -rx "sip show peer ${sipPeers[q]}" | grep Useragent | awk '{print $3}')
1st problem
In the step (1) you write: awk \'{print$3}\'
In the output of (2) you receive: awk '{print }'
The problem is an missed escape of $3
Try is again with awk \'{print\$3}\' in (1)
2nd problem
sipUserAgent[q]=$($temp) tries to invoke a command asterisk_-rx...
This is surely not what you want.
The solution is to use eval - see help eval for more informations.
example
cmd="echo TEST"
$(${cmd})
gives:
TEST: command not found
cmd="echo TEST"
eval ${cmd}
gives:
TEST
Note about code style
If possible it is better to avoid complex command creation and execution via eval.
Better is to write more simple code, if applicable.
So you should use (as also mentioned by Gordon Davisson in another answer):
sipUserAgent[q]=$(asterisk -rx "sip show peer ${sipPeers[q]}" \
| grep Useragent \
| awk '{print $3}' \
)
I am not aware of a way to escape quotes during command substitution, but a workaround is to use eval. See the following example:
$ cat test.sh
echo "$1"
$ temp="./test.sh \"2 34 434\" "
$ echo "$temp"
./test.sh "2 34 434"
$ result=$(eval $temp)
$ echo "$result"
2 34 434
$ result=$($temp)
$ echo "$result"
"2
Note that using eval is generally considered bad practice in most situations, if you can avoid it.
I want to check inside a file if it matches a binary pattern.
For that, I'm using clamAV signature database
Trojan.Bancos-166:1:*:3d415d736715ab5ee347238cacac61c7123fe35427224d25253c7b035558baf19e54e8d1a82742d6a7b37afc6d91015f751de1102d0a31e66ec33b74034b1ab471cc1381884dfdf0bb3e4233bd075fef235f342302ffd72ecabfa5aedf1b3dc99b3348346db4d9001026aef44c592fee61493f7262ad2bd1bce8a7ce60d81022533f6473ae184935f25cf6cc07c3aebfdf70a5a09139
I code this to retrieve the hex string representation signature
signature=$(echo "$line" |awk -F':' '{ print $4 }')
Moreover I change hex string to binary
printf -v variable $(sed 's/\(..\)/\\x\1/g;' <<< "$signature")
Until here It works perfectly.
Finally I would like to check if my file ( *$raw_file_path* ) matches my binary pattern (now in $variable)
I try this
test_var=$(grep -qU "$variable" "$raw_file_path")
or
test_var=$(grep -qU --regexp="$variable" "$raw_file_path")
I don't know why it doesn't work, Grep doesn't match anything
.
And sometimes some errors:
grep: Trailing backslash
grep: Invalid regular expression
I know it deals with pattern matching problems.
In my test I don't want use regular expression.
If you have any idea, or other bash tool.
Thanks.
You are currently using the --quiet option for grep by specifying q in -qU. This prevents grep from printing anything to stdout, therefore nothing will be saved to test_var.
Change your code to:
test_var=$(grep -UE "$variable" "$raw_file_path")
First the extra sub-shell can be avoided:
#!/bin/bash
signature="Trojan.Bancos-166:1:*:3d415d736715ab5ee347238cacac61c7123fe35427224d25253c7b035558baf19e54e8d1a82742d6a7b37afc6d91015f751de1102d0a31e66ec33b74034b1ab471cc1381884dfdf0bb3e4233bd075fef235f342302ffd72ecabfa5aedf1b3dc99b3348346db4d9001026aef44c592fee61493f7262ad2bd1bce8a7ce60d81022533f6473ae184935f25cf6cc07c3aebfdf70a5a09139"
variable=$(echo "${signature//*:/}" | sed 's/\(..\)/\\x\1/g;')
Require only confirmation of a match:
if grep -qU "$variable" "$raw_file_path"; then
# matches
fi
Or require the result for further processing:
test_var=$(grep -U "$variable" "$raw_file_path")
# contents of match in test_var
When returning to a variable, greps -q opt suppresses stdout
Edit
Tested working example
> signature="Trojan.Bancos-166:1:All_text before-the last : should be trimed:3d415d736715ab5ee347238cacac61c7123fe35427224d25253c7b035558baf19e54e8d1a82742d6a7b37afc6d91015f751de1102d0a31e66ec33b74034b1ab471cc1381884dfdf0bb3e4233bd075fef235f342302ffd72ecabfa5aedf1b3dc99b3348346db4d9001026aef44c592fee61493f7262ad2bd1bce8a7ce60d81022533f6473ae184935f25cf6cc07c3aebfdf70a5a09139" \
> hex_string=$( echo "${signature//*:/}" | sed 's/\(..\)/\\x\1/g;' ) \
> echo "$hex_string"
\x3d\x41\x5d\x73\x67\x15\xab\x5e\xe3\x47\x23\x8c\xac\xac\x61\xc7\x12\x3f\xe3\x54\x27\x22\x4d\x25\x25\x3c\x7b\x03\x55\x58\xba\xf1\x9e\x54\xe8\xd1\xa8\x27\x42\xd6\xa7\xb3\x7a\xfc\x6d\x91\x01\x5f\x75\x1d\xe1\x10\x2d\x0a\x31\xe6\x6e\xc3\x3b\x74\x03\x4b\x1a\xb4\x71\xcc\x13\x81\x88\x4d\xfd\xf0\xbb\x3e\x42\x33\xbd\x07\x5f\xef\x23\x5f\x34\x23\x02\xff\xd7\x2e\xca\xbf\xa5\xae\xdf\x1b\x3d\xc9\x9b\x33\x48\x34\x6d\xb4\xd9\x00\x10\x26\xae\xf4\x4c\x59\x2f\xee\x61\x49\x3f\x72\x62\xad\x2b\xd1\xbc\xe8\xa7\xce\x60\xd8\x10\x22\x53\x3f\x64\x73\xae\x18\x49\x35\xf2\x5c\xf6\xcc\x07\xc3\xae\xbf\xdf\x70\xa5\xa0\x91\x39