I have this text and more stored in a $hello:
"id":"1234"
How can I find id, and then get just the number?
There will be a lot of other text in the string.
Whole string is:
After making says:
{"self":"http://xxxxx:8081/rest/api/latest/version/1234","id":"1234","description":"desc","name":"vi1teast","archived":false,"released":false,"releaseDate":"2016-02-06","overdue":true,"userReleaseDate":"06/Feb/16","projectId":xxxxx}
I cannot use any JSON parsers, must be with string manipulation!
If you want to parse a JSON, use jq!
$ jq '.id' file
"1234"
And if you don't want to get the quotes, use -r:
$ jq -r '.id' file
1234
From the manual:
--raw-output / -r:
With this option, if the filter’s result is a string then it will be
written directly to standard output rather than being formatted as a
JSON string with quotes. This can be useful for making jq filters talk
to non-JSON-based systems.
Note I had to tweak your JSON and write "projectId":"xxxxx" so that it is a valid one.
If you just want to get the id and can make sure that it will only contain digits, you can use a pure bash solution like this. It will use bash regex and BASH_REMATCH for referencing the capturing groups.
#!/bin/bash
str='{"self":"http://xxxxx:8081/rest/api/latest/version/1234","id":"1234","description":"desc","name":"vi1teast","archived":false,"released":false,"releaseDate":"2016-02-06","overdue":true,"userReleaseDate":"06/Feb/16","projectId":xxxxx}'
if [[ "$str" =~ ^.*\"id\":\"([0-9]*)\".*$ ]];
then
echo ${BASH_REMATCH[1]} ;
else
echo "Not propper format";
fi
This will output the desired 1234.
You can also shorten it up:
#short version
[[ "$str" =~ ^.*\"id\":\"([0-9]*)\".*$ ]] && id=${BASH_REMATCH[1]}
echo $id
If you cannot use a parser, try this sed:
$ sed 's/.*"id":"\([0-9]\+\)".*/\1/' <<< "$hello"
1234
Related
Let's say I have the following text file
test.txt
ABC_01:Testing-ABCDEFG
If I want to retrieve the string after colon, I will be using
awk -F ":" '/ABC_01/{print $NF}' test.txt
which will return Testing-ABCDEFG
But what should I do if I only want to retrieve the string after the colon and before the hyphen?
You are so close. That is where split() comes in, e.g.
awk -F: '/ABC_01/{ split($NF,arr,"-"); print arr[1] }'
Which will output
Testing
The GNU Awk User's Guide - String Manipulation Functions provides the details on split(). Give it a try and let me know if you have any further questions.
Using Bash's built'in Extended Regex Engine
#!/usr/bin/env bash
while read -r; do
[[ $REPLY =~ :(.*)- ]] || :
echo "${BASH_REMATCH[1]}"
done
Using standard POSIX shell IFS field separators:
#!/usr/bin/env sh
while IFS=':-' read -r _ m _; do
echo "$m"
done
Using (GNU) grep and look-around:
$ grep -oP '(?<=:)[^-]*(?=-)' file
Testing
Explained:
grep GNU grep supports PCRE and look-around
`-o Print only the matched (non-empty) parts of a matching line
-P Interpret PATTERNS as Perl-compatible regular expressions
(?<=:) positive look-behind, ie. preceeded by a colon
[^-]* anything but a hyphen
(?=-) positive look-ahead, ie. followed by a hyphen
My shell script is following:
#!/bin/bash
account0=0xf2de2e86b9b634f655e441a4e8353c9bf59352d7
passwd=123456
data={"jsonrpc":"2.0","method":"personal_unlockAccount","id":1,"params":[$account0,$passwd]}
echo $data
My expected is (NOTICE "):
{"jsonrpc":"2.0","method":"personal_unlockAccount","id":1,"params":[0xf2de2e86b9b634f655e441a4e8353c9bf59352d7,123456]}
NOT
{jsonrpc:2.0,method:personal_unlockAccount,id:1,params:[0xf2de2e86b9b634f655e441a4e8353c9bf59352d7,123456]}
:
And I do not want to use escape character, how ?
like content insert into xml tag "<![CDATA[..."..]]>"
The echo isn't your problem. (It's a problem, but not your immediate problem).
Your problem is that your quotes aren't being assigned to the variable at all. Quotes are syntax to bash; it reads them as part of the instructions on how to parse a string. Consequently, they're consumed by bash itself, and not assigned as a value unless they are themselves quoted or escaped.
To make the whole thing literal, you can put the entire line in single quotes:
data='{"jsonrpc":"2.0","method":"personal_unlockAccount","id":1,"params":[$account0,$passwd]}'
echo "$data"
...or you can generate it with a heredoc, at some cost to efficiency:
{ IFS= read -r -d '' data || [[ $data ]]; } <<'EOF'
{"jsonrpc":"2.0","method":"personal_unlockAccount","id":1,"params":[$account0,$passwd]}
EOF
echo "$data"
Assuming you want to perform expansions, replacing account0 with the name of the like-named shell variable, the wrong way to do it is to switch from a single-quoted context to a double-quoted context before your variables are referenced:
# BAD: Does not guarantee result is valid JSON
account0=exampleName; passwd=examplePassword
data='{"jsonrpc":"2.0","method":"personal_unlockAccount","id":1,"params":["'"$account0"'","'"$passwd"'"]}'
echo "$data"
...or to switch to an unquoted heredoc (using <<EOF, not <<'EOF'):
# BAD: Does not guarantee result is valid JSON
account0=exampleName; passwd=examplePassword
{ IFS= read -r -d '' data || [[ $data ]]; } <<EOF
{"jsonrpc":"2.0","method":"personal_unlockAccount","id":1,"params":["$account0","$passwd"]}
EOF
echo "$data"
And the right way to do it is to use jq to generate safely-escaped JSON containing your literal values:
# GOOD: Result will always be syntactically valid JSON.
account0=exampleName; passwd=examplePassword
data=$(jq -cn --arg account0 "$account0" --arg passwd "$passwd" '
{"jsonrpc":"2.0","method":"personal_unlockAccount","id":1,"params":[$account0,$passwd]}
')
echo "$data"
And remember where I said echo is a problem, even if it's not your immediate problem? See the APPLICATION USAGE section of its POSIX specification to understand why it's innately unreliable when handling arbitrary data, keeping in mind that bash can be configured at runtime to behave according to any of the variants described in that spec. Use printf '%s\n' "$foo" instead of echo "$foo" to get consistent and reliable behavior.
I have a file, the content of file has a string like this:
'/ad/e','#'.base64_decode("ZXZhbA==").'($zad)', 'add'
I want to check the file has this string. But when I use grep to check, It always return false. I try some ways:
grep "'/ad/e','#'.base64_decode("ZXZhbA==").'($zad)', 'add'" foo.txt
grep "'/ad/e','#'\.base64_decode\("ZXZhbA\=\="\)\.'\(\$zad\)', 'add'" foo.txt
str="'/ad/e','#'\.base64_decode\("ZXZhbA\=\="\)\.'\(\$zad\)', 'add'"
grep "$str" foo.txt
Can you help me? Maybe, another command line.
This is my case:
while read str; do
if [ ! -z "$str" ]; then
if grep -Fxq "$str" "$file_path"; then
do somthing
fi
fi
done < <(cat /usr/local/caotoc/db.dat)
Thank you so much!
First, you need to make sure the string is quoted properly. This is a bit of an art form, since your string contains both single and double quotes.
One thought would be to use read and a here-document to avoid having to escape anything.
Second, you need to use -F to perform exact string matching instead of more general regular-expression matching.
IFS= read -r str <<'EOF'
'/ad/e','#'.base64_decode("ZXZhbA==").'($zad)', 'add'
EOF
grep -F "$str" foo.txt
Based on the update, you can use a simple loop to read them one at a time.
while IFS= read -r str; do
grep -F "$str" foo.txt
done < /usr/local/caotoc/db.dat
You may be able to simply use the -f option to grep, which will cause grep to output lines from foo.txt that match any line from db.dat.
grep -f /usr/local/caotoc/db.dat -F foo.txt
Instead of trying to workaround regexes, the simplest way is to turn off regular expressions using -F (or --fixed-strings) option, which makes grep act like a simple string search
-F, --fixed-strings PATTERN is a set of newline-separated strings
like this:
grep -F "'/ad/e','#'.base64_decode(\"ZXZhbA==\").'(\$zad)', 'add'" test
Note: because of the shell, you still need to escape:
double quotes
dollar sign or else $zad is evaluated as an environment variable
I have a file stv.txt containing some names
For example stv.txt is as follows:
hello
world
I want to generate another file by using these names and adding some extra text to them.I have written a script as follows
for i in `cat stvv.txt`;
do printf 'if(!strcmp("$i",optarg))' > my_file;
done
output
if(!strcmp("$i",optarg))
desired output
if(!strcmp("hello",optarg))
if(!strcmp("world",optarg))
how can I get the correct result?
This is a working solution.
1 All symbols inside single quotes is considered a string. 2 When using printf, do not surround the variable with quotes. (in this example)
The code below should fix it,
for i in `cat stvv.txt`;
printf 'if(!strcmp('$i',optarg))' > my_file;
done
basically, break the printf statement into three parts.
1: the string 'if(!strcmp('
2: $i (no quotes)
3: the string ',optarg))'
hope that helps!
To insert a string into a printf format, use %s in the format string:
$ for line in $(cat stvv.txt); do printf 'if(!strcmp("%s",optarg))\n' "$line"; done
if(!strcmp("hello",optarg))
if(!strcmp("world",optarg))
The code $(cat stvv.txt) will perform word splitting and pathname expansion on the contents of stvv.txt. You probably don't want that. It is generally safer to use a while read ... done <stvv.txt loop such as this one:
$ while read -r line; do printf 'if(!strcmp("%s",optarg))\n' "$line"; done <stvv.txt
if(!strcmp("hello",optarg))
if(!strcmp("world",optarg))
Aside on cat
If you are using bash, then $(cat stvv.txt) could be replaced with the more efficient $(<stvv.txt). This question, however, is tagged shell not bash. The cat form is POSIX and therefore portable to all POSIX shells while the bash form is not.
I want to check inside a file if it matches a binary pattern.
For that, I'm using clamAV signature database
Trojan.Bancos-166:1:*:3d415d736715ab5ee347238cacac61c7123fe35427224d25253c7b035558baf19e54e8d1a82742d6a7b37afc6d91015f751de1102d0a31e66ec33b74034b1ab471cc1381884dfdf0bb3e4233bd075fef235f342302ffd72ecabfa5aedf1b3dc99b3348346db4d9001026aef44c592fee61493f7262ad2bd1bce8a7ce60d81022533f6473ae184935f25cf6cc07c3aebfdf70a5a09139
I code this to retrieve the hex string representation signature
signature=$(echo "$line" |awk -F':' '{ print $4 }')
Moreover I change hex string to binary
printf -v variable $(sed 's/\(..\)/\\x\1/g;' <<< "$signature")
Until here It works perfectly.
Finally I would like to check if my file ( *$raw_file_path* ) matches my binary pattern (now in $variable)
I try this
test_var=$(grep -qU "$variable" "$raw_file_path")
or
test_var=$(grep -qU --regexp="$variable" "$raw_file_path")
I don't know why it doesn't work, Grep doesn't match anything
.
And sometimes some errors:
grep: Trailing backslash
grep: Invalid regular expression
I know it deals with pattern matching problems.
In my test I don't want use regular expression.
If you have any idea, or other bash tool.
Thanks.
You are currently using the --quiet option for grep by specifying q in -qU. This prevents grep from printing anything to stdout, therefore nothing will be saved to test_var.
Change your code to:
test_var=$(grep -UE "$variable" "$raw_file_path")
First the extra sub-shell can be avoided:
#!/bin/bash
signature="Trojan.Bancos-166:1:*:3d415d736715ab5ee347238cacac61c7123fe35427224d25253c7b035558baf19e54e8d1a82742d6a7b37afc6d91015f751de1102d0a31e66ec33b74034b1ab471cc1381884dfdf0bb3e4233bd075fef235f342302ffd72ecabfa5aedf1b3dc99b3348346db4d9001026aef44c592fee61493f7262ad2bd1bce8a7ce60d81022533f6473ae184935f25cf6cc07c3aebfdf70a5a09139"
variable=$(echo "${signature//*:/}" | sed 's/\(..\)/\\x\1/g;')
Require only confirmation of a match:
if grep -qU "$variable" "$raw_file_path"; then
# matches
fi
Or require the result for further processing:
test_var=$(grep -U "$variable" "$raw_file_path")
# contents of match in test_var
When returning to a variable, greps -q opt suppresses stdout
Edit
Tested working example
> signature="Trojan.Bancos-166:1:All_text before-the last : should be trimed:3d415d736715ab5ee347238cacac61c7123fe35427224d25253c7b035558baf19e54e8d1a82742d6a7b37afc6d91015f751de1102d0a31e66ec33b74034b1ab471cc1381884dfdf0bb3e4233bd075fef235f342302ffd72ecabfa5aedf1b3dc99b3348346db4d9001026aef44c592fee61493f7262ad2bd1bce8a7ce60d81022533f6473ae184935f25cf6cc07c3aebfdf70a5a09139" \
> hex_string=$( echo "${signature//*:/}" | sed 's/\(..\)/\\x\1/g;' ) \
> echo "$hex_string"
\x3d\x41\x5d\x73\x67\x15\xab\x5e\xe3\x47\x23\x8c\xac\xac\x61\xc7\x12\x3f\xe3\x54\x27\x22\x4d\x25\x25\x3c\x7b\x03\x55\x58\xba\xf1\x9e\x54\xe8\xd1\xa8\x27\x42\xd6\xa7\xb3\x7a\xfc\x6d\x91\x01\x5f\x75\x1d\xe1\x10\x2d\x0a\x31\xe6\x6e\xc3\x3b\x74\x03\x4b\x1a\xb4\x71\xcc\x13\x81\x88\x4d\xfd\xf0\xbb\x3e\x42\x33\xbd\x07\x5f\xef\x23\x5f\x34\x23\x02\xff\xd7\x2e\xca\xbf\xa5\xae\xdf\x1b\x3d\xc9\x9b\x33\x48\x34\x6d\xb4\xd9\x00\x10\x26\xae\xf4\x4c\x59\x2f\xee\x61\x49\x3f\x72\x62\xad\x2b\xd1\xbc\xe8\xa7\xce\x60\xd8\x10\x22\x53\x3f\x64\x73\xae\x18\x49\x35\xf2\x5c\xf6\xcc\x07\xc3\xae\xbf\xdf\x70\xa5\xa0\x91\x39