Does shell echo complete content without escape character - linux

My shell script is following:
#!/bin/bash
account0=0xf2de2e86b9b634f655e441a4e8353c9bf59352d7
passwd=123456
data={"jsonrpc":"2.0","method":"personal_unlockAccount","id":1,"params":[$account0,$passwd]}
echo $data
My expected is (NOTICE "):
{"jsonrpc":"2.0","method":"personal_unlockAccount","id":1,"params":[0xf2de2e86b9b634f655e441a4e8353c9bf59352d7,123456]}
NOT
{jsonrpc:2.0,method:personal_unlockAccount,id:1,params:[0xf2de2e86b9b634f655e441a4e8353c9bf59352d7,123456]}
:
And I do not want to use escape character, how ?
like content insert into xml tag "<![CDATA[..."..]]>"

The echo isn't your problem. (It's a problem, but not your immediate problem).
Your problem is that your quotes aren't being assigned to the variable at all. Quotes are syntax to bash; it reads them as part of the instructions on how to parse a string. Consequently, they're consumed by bash itself, and not assigned as a value unless they are themselves quoted or escaped.
To make the whole thing literal, you can put the entire line in single quotes:
data='{"jsonrpc":"2.0","method":"personal_unlockAccount","id":1,"params":[$account0,$passwd]}'
echo "$data"
...or you can generate it with a heredoc, at some cost to efficiency:
{ IFS= read -r -d '' data || [[ $data ]]; } <<'EOF'
{"jsonrpc":"2.0","method":"personal_unlockAccount","id":1,"params":[$account0,$passwd]}
EOF
echo "$data"
Assuming you want to perform expansions, replacing account0 with the name of the like-named shell variable, the wrong way to do it is to switch from a single-quoted context to a double-quoted context before your variables are referenced:
# BAD: Does not guarantee result is valid JSON
account0=exampleName; passwd=examplePassword
data='{"jsonrpc":"2.0","method":"personal_unlockAccount","id":1,"params":["'"$account0"'","'"$passwd"'"]}'
echo "$data"
...or to switch to an unquoted heredoc (using <<EOF, not <<'EOF'):
# BAD: Does not guarantee result is valid JSON
account0=exampleName; passwd=examplePassword
{ IFS= read -r -d '' data || [[ $data ]]; } <<EOF
{"jsonrpc":"2.0","method":"personal_unlockAccount","id":1,"params":["$account0","$passwd"]}
EOF
echo "$data"
And the right way to do it is to use jq to generate safely-escaped JSON containing your literal values:
# GOOD: Result will always be syntactically valid JSON.
account0=exampleName; passwd=examplePassword
data=$(jq -cn --arg account0 "$account0" --arg passwd "$passwd" '
{"jsonrpc":"2.0","method":"personal_unlockAccount","id":1,"params":[$account0,$passwd]}
')
echo "$data"
And remember where I said echo is a problem, even if it's not your immediate problem? See the APPLICATION USAGE section of its POSIX specification to understand why it's innately unreliable when handling arbitrary data, keeping in mind that bash can be configured at runtime to behave according to any of the variants described in that spec. Use printf '%s\n' "$foo" instead of echo "$foo" to get consistent and reliable behavior.

Related

Increment a variable name in ksh [duplicate]

Seems that the recommended way of doing indirect variable setting in bash is to use eval:
var=x; val=foo
eval $var=$val
echo $x # --> foo
The problem is the usual one with eval:
var=x; val=1$'\n'pwd
eval $var=$val # bad output here
(and since it is recommended in many places, I wonder just how many scripts are vulnerable because of this...)
In any case, the obvious solution of using (escaped) quotes doesn't really work:
var=x; val=1\"$'\n'pwd\"
eval $var=\"$val\" # fail with the above
The thing is that bash has indirect variable reference baked in (with ${!foo}), but I don't see any such way to do indirect assignment -- is there any sane way to do this?
For the record, I did find a solution, but this is not something that I'd consider "sane"...:
eval "$var='"${val//\'/\'\"\'\"\'}"'"
A slightly better way, avoiding the possible security implications of using eval, is
declare "$var=$val"
Note that declare is a synonym for typeset in bash. The typeset command is more widely supported (ksh and zsh also use it):
typeset "$var=$val"
In modern versions of bash, one should use a nameref.
declare -n var=x
x=$val
It's safer than eval, but still not perfect.
Bash has an extension to printf that saves its result into a variable:
printf -v "${VARNAME}" '%s' "${VALUE}"
This prevents all possible escaping issues.
If you use an invalid identifier for $VARNAME, the command will fail and return status code 2:
$ printf -v ';;;' '%s' foobar; echo $?
bash: printf: `;;;': not a valid identifier
2
eval "$var=\$val"
The argument to eval should always be a single string enclosed in either single or double quotes. All code that deviates from this pattern has some unintended behavior in edge cases, such as file names with special characters.
When the argument to eval is expanded by the shell, the $var is replaced with the variable name, and the \$ is replaced with a simple dollar. The string that is evaluated therefore becomes:
varname=$value
This is exactly what you want.
Generally, all expressions of the form $varname should be enclosed in double quotes, to prevent accidental expansion of filename patterns like *.c.
There are only two places where the quotes may be omitted since they are defined to not expand pathnames and split fields: variable assignments and case. POSIX 2018 says:
Each variable assignment shall be expanded for tilde expansion, parameter expansion, command substitution, arithmetic expansion, and quote removal prior to assigning the value.
This list of expansions is missing the parameter expansion and the field splitting. Sure, that's hard to see from reading this sentence alone, but that's the official definition.
Since this is a variable assignment, the quotes are not needed here. They don't hurt, though, so you could also write the original code as:
eval "$var=\"the value is \$val\""
Note that the second dollar is escaped using a backslash, to prevent it from being expanded in the first run. What happens is:
eval "$var=\"the value is \$val\""
The argument to the command eval is sent through parameter expansion and unescaping, resulting in:
varname="the value is $val"
This string is then evaluated as a variable assignment, which assigns the following value to the variable varname:
the value is value
The main point is that the recommended way to do this is:
eval "$var=\$val"
with the RHS done indirectly too. Since eval is used in the same
environment, it will have $val bound, so deferring it works, and since
now it's just a variable. Since the $val variable has a known name,
there are no issues with quoting, and it could have even been written as:
eval $var=\$val
But since it's better to always add quotes, the former is better, or
even this:
eval "$var=\"\$val\""
A better alternative in bash that was mentioned for the whole thing that
avoids eval completely (and is not as subtle as declare etc):
printf -v "$var" "%s" "$val"
Though this is not a direct answer what I originally asked...
Newer versions of bash support something called "parameter transformation", documented in a section of the same name in bash(1).
"${value#Q}" expands to a shell-quoted version of "${value}" that you can re-use as input.
Which means the following is a safe solution:
eval="${varname}=${value#Q}"
Just for completeness I also want to suggest the possible use of the bash built in read. I've also made corrections regarding -d'' based on socowi's comments.
But much care needs to be exercised when using read to ensure the input is sanitized (-d'' reads until null termination and printf "...\0" terminates the value with a null), and that read itself is executed in the main shell where the variable is needed and not a sub-shell (hence the < <( ... ) syntax).
var=x; val=foo0shouldnotterminateearly
read -d'' -r "$var" < <(printf "$val\0")
echo $x # --> foo0shouldnotterminateearly
echo ${!var} # --> foo0shouldnotterminateearly
I tested this with \n \t \r spaces and 0, etc it worked as expected on my version of bash.
The -r will avoid escaping \, so if you had the characters "\" and "n" in your value and not an actual newline, x will contain the two characters "\" and "n" also.
This method may not be aesthetically as pleasing as the eval or printf solution, and would be more useful if the value is coming in from a file or other input file descriptor
read -d'' -r "$var" < <( cat $file )
And here are some alternative suggestions for the < <() syntax
read -d'' -r "$var" <<< "$val"$'\0'
read -d'' -r "$var" < <(printf "$val") #Apparently I didn't even need the \0, the printf process ending was enough to trigger the read to finish.
read -d'' -r "$var" <<< $(printf "$val")
read -d'' -r "$var" <<< "$val"
read -d'' -r "$var" < <(printf "$val")
Yet another way to accomplish this, without eval, is to use "read":
INDIRECT=foo
read -d '' -r "${INDIRECT}" <<<"$(( 2 * 2 ))"
echo "${foo}" # outputs "4"

When should I use "" to quote a value in shell test and in echo?

I'm writing bash script like this:
VF_ETH=$(command)
if [ -n "$VF_ETH" ] ; then
echo "ixgbevf eth: "$VF_ETH
fi
command is a linux command, my question is:
$VF_ETH is to get value of VF_ETH, why use "" to quote it in line2 in shell test?
if I do not use "" to quote it, will test failed?
if use "" to quote a value is to make it into string, why not use in echo in line3?
Thank you
Assuming you get an actual command stored in VF_ETH variable, which contains spaces. Now if you use if [ -n $VF_ETH ] and when shell expands the variable, there will be multiple parameters to -n whereas it expects only one. Hence you might get something like binary operator expected error.
Also in the echo command, it is not mandatory to have only one parameter. Hence even if you are not using double quotes, it works.
Hence to avoid it, always use double quotes while expanding variables.
Also use https://www.shellcheck.net/ to check your script and it will give you correct information on where your script is wrong/not as per standard.
You should always double quote variables used in command line arguments and within [ ... ] tests, for example:
ls "$var"
echo "$var"
[ -f "$var" ]
test -f "$var"
In all the above examples the commands used will receive different values with and without the double quotes, when the value of $var starts with a bunch of spaces and contains some spaces. For example:
var=" an apple"
echo "$var" # prints " an apple"
echo $var # prints "an apple", without the leading space
You don't need to quote them in simple assignments, for example:
a=$var
a=$var$b
If the assignment contains spaces or other special characters, then you have to quote, or escape those special characters:
a="$var $b"
a=$var\ $b
a=$var" "$b
All the above are equivalent, but the first one is probably the easiest to read, and therefore recommended.
You don't need to quote special variables that never have unsafe values, for example:
test $? = 0
If you're unsure, or not yet confident, then a good role of thumb is to double quote always.
For 1. and 2. If you set $VF_ETH="x -a -z x" and test it with code:
if [ -n $VF_ETH ] ; then
echo yes
else
echo nope
fi
the output will be nope as the the inside of the square brackets would expand to -n x AND -z x (nonempty and empty). Adding quotes around the variable would fix that.
Set $VF_ETH="*". Now if you echo $foo bash would do wildcard expansion and echo would output the contents of your current directory.

Simple bash get part of a string

I have this text and more stored in a $hello:
"id":"1234"
How can I find id, and then get just the number?
There will be a lot of other text in the string.
Whole string is:
After making says:
{"self":"http://xxxxx:8081/rest/api/latest/version/1234","id":"1234","description":"desc","name":"vi1teast","archived":false,"released":false,"releaseDate":"2016-02-06","overdue":true,"userReleaseDate":"06/Feb/16","projectId":xxxxx}
I cannot use any JSON parsers, must be with string manipulation!
If you want to parse a JSON, use jq!
$ jq '.id' file
"1234"
And if you don't want to get the quotes, use -r:
$ jq -r '.id' file
1234
From the manual:
--raw-output / -r:
With this option, if the filter’s result is a string then it will be
written directly to standard output rather than being formatted as a
JSON string with quotes. This can be useful for making jq filters talk
to non-JSON-based systems.
Note I had to tweak your JSON and write "projectId":"xxxxx" so that it is a valid one.
If you just want to get the id and can make sure that it will only contain digits, you can use a pure bash solution like this. It will use bash regex and BASH_REMATCH for referencing the capturing groups.
#!/bin/bash
str='{"self":"http://xxxxx:8081/rest/api/latest/version/1234","id":"1234","description":"desc","name":"vi1teast","archived":false,"released":false,"releaseDate":"2016-02-06","overdue":true,"userReleaseDate":"06/Feb/16","projectId":xxxxx}'
if [[ "$str" =~ ^.*\"id\":\"([0-9]*)\".*$ ]];
then
echo ${BASH_REMATCH[1]} ;
else
echo "Not propper format";
fi
This will output the desired 1234.
You can also shorten it up:
#short version
[[ "$str" =~ ^.*\"id\":\"([0-9]*)\".*$ ]] && id=${BASH_REMATCH[1]}
echo $id
If you cannot use a parser, try this sed:
$ sed 's/.*"id":"\([0-9]\+\)".*/\1/' <<< "$hello"
1234

Why am I getting command not found error on numeric comparison?

I am trying to parse each line of a file and look for a particular string. The script seems to be doing its intended job, however, in parallel it tries to execute the if command on line 6:
#!/bin/bash
for line in $(cat $1)
do
echo $line | grep -e "Oct/2015"
if($?==0); then
echo "current line is: $line"
fi
done
and I get the following (my script is readlines.sh)
./readlines.sh: line 6: 0==0: command not found
First: As Mr. Llama says, you need more spaces. Right now your script tries to look for a file named something like /usr/bin/0==0 to run. Instead:
[ "$?" -eq 0 ] # POSIX-compliant numeric comparison
[ "$?" = 0 ] # POSIX-compliant string comparison
(( $? == 0 )) # bash-extended numeric comparison
Second: Don't test $? at all in this case. In fact, you don't even have good cause to use grep; the following is both more efficient (because it uses only functionality built into bash and requires no invocation of external commands) and more readable:
if [[ $line = *"Oct/2015"* ]]; then
echo "Current line is: $line"
fi
If you really do need to use grep, write it like so:
if echo "$line" | grep -q "Oct/2015"; then
echo "Current line is: $line"
fi
That way if operates directly on the pipeline's exit status, rather than running a second command testing $? and operating on that command's exit status.
#Charles Duffy has a good answer which I have up-voted as correct (and it is), but here's a detailed, line by line breakdown of your script and the correct thing to do for each part of it.
for line in $(cat $1)
As I noted in my comment elsewhere this should be done as a while read construct instead of a for cat construct.
This construct will wordsplit each line making spaces in the file separate "lines" in the output.
All empty lines will be skipped.
In addition when you cat $1 the variable should be quoted. If it is not quoted spaces and other less-usual characters appearing in the file name will cause the cat to fail and the loop will not process the file.
The complete line would read:
while IFS= read -r line
An illustrative example of the tradeoffs can be found here. The linked test script follows. I tried to include an indication of why IFS= and -r are important.
#!/bin/bash
mkdir -p /tmp/testcase
pushd /tmp/testcase >/dev/null
printf '%s\n' '' two 'three three' '' ' five with leading spaces' 'c:\some\dos\path' '' > testfile
printf '\nwc -l testfile:\n'
wc -l testfile
printf '\n\nfor line in $(cat) ... \n\n'
let n=1
for line in $(cat testfile) ; do
echo line $n: "$line"
let n++
done
printf '\n\nfor line in "$(cat)" ... \n\n'
let n=1
for line in "$(cat testfile)" ; do
echo line $n: "$line"
let n++
done
let n=1
printf '\n\nwhile read ... \n\n'
while read line ; do
echo line $n: "$line"
let n++
done < testfile
printf '\n\nwhile IFS= read ... \n\n'
let n=1
while IFS= read line ; do
echo line $n: "$line"
let n++
done < testfile
printf '\n\nwhile IFS= read -r ... \n\n'
let n=1
while IFS= read -r line ; do
echo line $n: "$line"
let n++
done < testfile
rm -- testfile
popd >/dev/null
rmdir /tmp/testcase
Note that this is a bash-heavy example. Other shells do not tend to support -r for read, for example, nor is let portable. On to the next line of your script.
do
As a matter of style I prefer do on the same line as the for or while declaration, but there's no convention on this.
echo $line | grep -e "Oct/2015"
The variable $line should be quoted here. In general, meaning always unless you specifically know better, you should double-quote all expansion--and that means subshells as well as variables. This insulates you from most unexpected shell weirdness.
You decclared your shell as bash which means you will have there "Here string" operator <<< available to you. When available it can be used to avoid the pipe; each element of a pipeline executes in a subshell, which incurs extra overhead and can lead to unexpected behavior if you try to modify variables. This would be written as
grep -e "Oct/2015" <<<"$line"
Note that I have quoted the line expansion.
You have called grep with -e, which is not incorrect but is needless since your pattern does not begin with -. In addition you have full-quoted a string in shell but you don't attempt to expand a variable or use other shell interpolation inside of it. When you don't expect and don't want the contents of a quoted string to be treated as special by the shell you should single quote them. Furthermore, your use of grep is inefficient: because your pattern is a fixed string and not a regular expression you could have used fgrep or grep -F, which does string contains rather than regular expression matching (and is far faster because of this). So this could be
grep -F 'Oct/2015' <<<"$line"
Without altering the behavior.
if($?==0); then
This is the source of your original problem. In shell scripts commands are separated by whitespace; when you say if($?==0) the $? expands, probably to 0, and bash will try to execute a command called if(0==0) which is a legal command name. What you wanted to do was invoke the if command and give it some parameters, which requires more whitespace. I believe others have covered this sufficiently.
You should never need to test the value of $? in a shell script. The if command exists for branching behavior based on the return code of whatever command you pass to it, so you can inline your grep call and have if check its return code directly, thus:
if grep -F 'Oct/2015` <<<"$line" ; then
Note the generous whitespace around the ; delimiter. I do this because in shell whitespace is usually required and can only sometiems be omitted. Rather than try to remember when you can do which I recommend an extra one space padding around everything. It's never wrong and can make other mistakes easier to notice.
As others have noted this grep will print matched lines to stdout, which is probably not something you want. If you are using GNU grep, which is standard on Linux, you will have the -q switch available to you. This will suppress the output from grep
if grep -q -F 'Oct/2015' <<<"$line" ; then
If you are trying to be strictly standards compliant or are in any environment with a grep that doesn't know -q the standard way to achieve this effect is to redirect stdout to /dev/null/
if printf "$line" | grep -F 'Oct/2015' >/dev/null ; then
In this example I also removed the here string bashism just to show a portable version of this line.
echo "current line is: $line"
There is nothing wrong with this line of your script, except that although echo is standard implementations vary to such an extent that it's not possible to absolutely rely on its behavior. You can use printf anywhere you would use echo and you can be fairly confident of what it will print. Even printf has some caveats: Some uncommon escape sequences are not evenly supported. See mascheck for details.
printf 'current line is: %s\n' "$line"
Note the explicit newline at the end; printf doesn't add one automatically.
fi
No comment on this line.
done
In the case where you did as I recommended and replaced the for line with a while read construct this line would change to:
done < "$1"
This directs the contents of the file in the $1 variable to the stdin of the while loop, which in turn passes the data to read.
In the interests of clarity I recommend copying the value from $1 into another variable first. That way when you read this line the purpose is more clear.
I hope no one takes great offense at the stylistic choices made above, which I have attempted to note; there are many ways to do this (but not a great many correct) ways.
Be sure to always run interesting snippets through the excellent shellcheck and explain shell when you run into difficulties like this in the future.
And finally, here's everything put together:
#!/bin/bash
input_file="$1"
while IFS= read -r line ; do
if grep -q -F 'Oct/2015' <<<"$line" ; then
printf 'current line is %s\n' "$line"
fi
done < "$input_file"
If you like one-liners, you may use AND operator (&&), for example:
echo "$line" | grep -e "Oct/2015" && echo "current line is: $line"
or:
grep -qe "Oct/2015" <<<"$line" && echo "current line is: $line"
Spacing is important in shell scripting.
Also, double-parens is for numerical comparison, not single-parens.
if (( $? == 0 )); then

Using a variable to replace lines in a file with backslashes

I want to add the string %%% to the beginning of some specific lines in a text file.
This is my script:
#!/bin/bash
a="c:\Temp"
sed "s/$a/%%%$a/g" <File.txt
And this is my File.txt content:
d:\Temp
c:\Temp
e:\Temp
But nothing changes when I execute it.
I think the 'sed' command is not finding the pattern, possibly due to the \ backslashes in the variable a.
I can find the c:\Temp line if I use grep with -F option (to not interpret strings):
cat File.txt | grep -F "$a"
But sed seems not to implement such '-F` option.
Not working neither:
sed 's/$a/%%%$a/g' <File.txt
sed 's/"$a"/%%%"$a"/g' <File.txt
I have found similar threads about replacing with sed, but they don't refer to variables.
How can I replace the desired lines by using a variable adding them the %%% char string?
EDIT: It would be fine that the $a variable could be entered via parameter when calling the script, so it will be assigned like:
a=$1
Try it like this:
#!/bin/sh
a='c:\\Temp' # single quotes
sed "s/$a/%%%$a/g" <File.txt # double quotes
Output:
Johns-MacBook-Pro:sed jcreasey$ sh x.sh
d:\Temp
e:\Temp
%%%c:\Temp
You need the double slash '\' to escape the '\'.
The single quotes won't expand the variables.
So you escape the slash in single quotes and pass it into the double quotes.
Of course you could also just do this:
#!/bin/sh
sed 's/\(.*Temp\)/%%%&/' <File.txt
If you want to get input from the command line you have to allow for the fact that \ is an escape character there too. So the user needs to type 'c:\\' or the interpreter will just wait for another character. Then once you get it, you will need to escape it again. (printf %q).
#!/bin/sh
b=`printf "%q" $1`
sed "s/\($b\)/%%% &/" < File.txt
The issue you are having has to do with substitution of your variable providing a regular expression looking for a literal c:Temp with the \ interpreted as an escape by the shell. There are a number of workarounds. Seeing the comments and having worked through the possibilities, the following will allow the unquoted entry of the search term:
#!/bin/bash
## validate that needed input is given on the command line
[ -n "$1" -a "$2" ] || {
printf "Error: insufficient input. Usage: %s <term> <file>\n" "${0//*\//}" >&2
exit 1
}
## validate that the filename given is readable
[ -r "$2" ] || {
printf "Error: file not readable '%s'\n" "$2" >&2
exit 1
}
a="$1" # assign a
filenm="$2" # assign filename
## test and fix the search term entered
[[ "$a" =~ '/' ]] || a="${a/:/:\\}" # test if \ removed by shell, if so replace
a="${a/\\/\\\\}" # add second \
sed -e "s/$a/%%%$a/g" "$filenm" # call sed with output to stdout
Usage:
$ bash sedwinpath.sh c:\Temp dat/winpath.txt
d:\Temp
%%%c:\Temp
e:\Temp
Note: This allows both single-quoted or unquoted entry of the dos path search term. To edit in place use sed -i. Additionally, the [[ operator and =~ operator are limited to bash.
I could have sworn the original question said replace, but to append, just as you suggest in the comments. I have updated the code with:
sed -e "s/$a/%%%$a/g" "$filenm"
Which provides the new output:
$ bash sedwinpath.sh c:\Temp dat/winpath.txt
d:\Temp
%%%c:\Temp
e:\Temp
Remember: If you want to edit the file in place use sed -i or sed -i.bak which will edit the actual file (and if -i.bak is given create a backup of the original in originalname.bak). Let me know if that is not what you intended and I'm happy to edit again.
Creating your script with a positional parameter of $1
#!/bin/bash
a="$1"
cat <file path>|sed "s/"$1"/%%%"$1"/g" > "temporary file"
Now whenever you want sed to find "c:\Temp" you need to use your script command line as follows
bash <my executing script> c:\\\\Temp
The first backslash will make bash interpret any backslashes that follows therefore what will be save in variable "a" in your executing script is "c:\\Temp". Now substituting this variable in sed will cause sed to interpret 1 backlash since the first backslash in this variable will cause sed to start interpreting the other backlash.
when you Open your temporary file you will see:
d:\Temp
%%%c:\Temp
e:\Temp

Resources