I have two variables var1 and var2 exported in my shell.
var1=root
var2=webserver
I want a jq query which can replace a JSON key with the given value in the JSON file. something like that,
jq -r --arg var1 $var1 --arg var2 $var2 '(.[].appId=$var1-$var2)' service.json
It is giving error that var1 and var2 can't be subtracted. But i want the variable to be replaced with $var1-$var2 as a string.
I tried with "-", But it's giving compilation error.
service.json content is following.
[ {
"appId": "Eternal Flame",
"age": 1000000,
"secretIdentity": "Unknown",
"powers": [
"Immortality",
"Heat Immunity",
"Inferno",
"Teleportation",
"Interdimensional travel"
]
} ]
appId value should be replaced with "root-webserver"
Using a - tries to do an arithmetic subtraction on the items. I recommend to use String interpolation. It will automatically cast input to a string:
jq -r --arg var1 $var1 --arg var2 $var2 '(.[].appId="\($var1)-\($var2)")' service.json
Related
I am trying to output the value for nested json key but for some reason I get null value.
SP=$(curl --proxy ...my apicall)
declare -A prop_map=(
["ID"] = "tagInfo.id"
)
for key in ${!prop_map[#]}; do
pn=${prop_map["$key"]}
val=$(jq -r --arg e "$pn" '.[$e]' <<<
"$SP")
echo $val
Curl response looks like this
{
tag info:{ id: "12" , name:...}}
Can anyone please guide ..been stuck on this for so long
One of the fundamental problems with your script is that it
assumes that jq can make sense of:
jq -r --arg e "tagInfo.id" '.[$e]' <<< '{"tagInfo":{ "id": "12" , "name": "aname"}}'
Here are two workarounds:
Use a shell variable:
e="tagInfo.id"
jq -r ".[$e]" <<< '{"tagInfo":{ "id": "12" , "name": "aname"}}'
Specify the path as an array:
jq -r --argjson e '["tagInfo", "id"]' 'getpath($e)' <<< '{"tagInfo":{ "id": "12" , "name": "aname"}}'
Since (1) above is somewhat fragile, it would probably be better to go with (2).
Below is my bash script.
#!/bin/bash
message=$1
hostname=$2
severity=$3
eventname=$4
tagpath=$5
appname=$6
data="{"action":"EventsRouter","method":"add_event","data":[{"summary":"$message"},"device":"$hostname","message": "$message $eventname $tagpath","component":"$appname","severity":"$severity","evclasskey":"nxlog","evclass":"/nxlog/perf","monitor":"localhost"}],"type":"rpc","tid":1}"
echo "Total number of args : $#"
echo "message = $message"
echo "hostname = $hostname"
echo "appname = $appname"
echo "data = $data"
curl -u uname:password -k https://myurlcom/zport/dmd/evconsole_router -d $data
and when i try to run with sh tcp.sh value value value value value value
host:
'host,component:host,severity:host,evclasskey:nxlog,evclass:/nxlog/perf,monitor:localhost}],type:rpc,tid:1}'
is not a legal name (unexpected end of input) Total number of args : 6
message = message hostname = test appname = host data = curl: option
-d: requires parameter
I see that data has no value included.
This json has to be sent in this order for it to be accepted in the endpoint. Help me correct this.
Using jq to safely generate the desired JSON:
#!/bin/bash
parameters=(
--arg message "$1"
--arg hostname "$2"
--arg severity "$3"
--arg eventname "$4"
--arg tagpath "$5"
--arg appname "$6"
)
data=$(jq -n "${parameters[#]}" '
{action: "EventsRouter",
method: "add_event",
data: [ {summary: $message,
device: $hostname,
message: "\($message) \($eventname\) \($tagpath)",
component: $appname,
severity: $severity,
evclasskey: "nxlog",
evclass: "/nxlog/perf",
monitor: "localhost"
}
],
type: "rpc",
tid: 1
}'
curl -u uname:password -k https://myurlcom/zport/dmd/evconsole_router -d "$data"
Assuming:
message="my_message"
hostname="my_host"
severity="my_severity"
eventname="my_event"
tagpath="my_path"
appname="my_app"
If you run:
data="{"action":"EventsRouter","method":"add_event","data":[{"summary":"$message"},"device":"$hostname","message": "$message $eventname $tagpath","component":"$appname","severity":"$severity","evclasskey":"nxlog","evclass":"/nxlog/perf","monitor":"localhost"}],"type":"rpc","tid":1}"
you will get an error, because there is a not escaped white space before the string "my_event"
my_event: command not found
What happened? Since your json input has a lot of words between double quotes, you will have to enclose the whole string into single quotes, in order to preserve the double quotes inside of the string. But between single quotes, the bash variables will not be replaced by their value. So you will need to close the single quotes before each variable and reopen these again immediately after.
So that line of your script must become:
data='{"action":"EventsRouter","method":"add_event","data":[{"summary":"'$message'"},"device":"'$hostname'","message": "'$message $eventname $tagpath'","component":"'$appname'","severity":"'$severity'","evclasskey":"nxlog","evclass":"/nxlog/perf","monitor":"localhost"}],"type":"rpc","tid":1}'
If you execute:
echo "$data"
you will get:
{"action":"EventsRouter","method":"add_event","data":[{"summary":"my_message"},"device":"my_host","message": "my_message my_event my_path","component":"my_app","severity":"my_severity","evclasskey":"nxlog","evclass":"/nxlog/perf","monitor":"localhost"}],"type":"rpc","tid":1}
which is correct, I assume: the double quotes didn't disappear from your json data structure.
I am trying to create automated test and I do not understand why this happens, the error is:
Test case: Verify that circuit breaker has status CLOSED -> Test FAILED, EXPECTED VALUE: CLOSED, ACTUAL VALUE: "CLOSED", WILL ABORT
Why does it not compare Strings even if they are? I am new to bash scripting so this is probably something pretty obvious.
Function which is calling Spring Boot Actuator
My code looks like:
function testCircuitBreaker() {
echo "Start Circuit Breaker Test"
EXEC="docker run --rm -it --network=my-network alpine"
#Verify that circuit breaker is closed via health endpoint
assertEqual "CLOSED" "$($EXEC wget movie-composite:8080/actuator/health -qO - | jq .components.movieCircuitBreaker.details.state)" "Verify that circuit breaker has status CLOSED"
#Three slow calls to get TimeoutException
for ((n = 0; n < 3; n++)); do
assertCurl 500 "curl -k https://$HOST:$PORT/movie-composite/MOV_ID_REVS_RECS?delay=3 $AUTH -s"
message=$(echo $RESPONSE | jq -r .message)
assertEqual "Did not observe any item or terminal signal within 2000ms" "${message:0:57}"
done
}
Assertion Function:
function assertEqual() {
local expected=$1
local actual=$2
local message=$3
printf "Test case: $message -> "
if [ "$actual" = "$expected" ]; then
echo "Test OK (actual value: $actual)"
return 0
else
echo "Test FAILED, EXPECTED VALUE: $expected, ACTUAL VALUE: $actual, WILL ABORT"
return 1
fi
}
jq outputs well-formed JSON by default, so strings will be quoted. As an example:
$ jq .foo <<<'{"foo":"bar"}'
"bar"
Therefore your assertEqual "CLOSED" "$(... | jq .components.movieCircuitBreaker.details.state)" ... command is comparing CLOSED with "CLOSED" (note the extra quotes). You could quote the expected string, as #rtx13 suggests, but I think it's clearer to have jq output just the contents of the field. You can do that with the --raw-output flag:
With this option, if the filter's result is a string then it will be written directly to standard output rather than being formatted as a JSON string with quotes. This can be useful for making jq filters talk to non-JSON-based systems.
$ jq -r .foo <<<'{"foo":"bar"}'
bar
You could try preserving the quotes around CLOSED by changing:
assertEqual "CLOSED" "$($EXEC wget movie-composite:8080/actuator/health -qO - | jq .components.movieCircuitBreaker.details.state)" "Verify that circuit breaker has status CLOSED"
to
assertEqual '"CLOSED"' "$($EXEC wget movie-composite:8080/actuator/health -qO - | jq .components.movieCircuitBreaker.details.state)" "Verify that circuit breaker has status CLOSED"
The single quotes around "CLOSED" preserve the quotation marks when passed to the assertEqual function.
I'm a few weeks into bash scripting and I haven't advanced enough yet to get my head wrapped around this problem. Any help would be appreciated!
I have a "script.conf" file that contains the following:
key1=value1
key2=${HOME}/Folder
key3=( "k3v1" "k3 v2" "k3v3")
key4=( "k4v1"
"k4 v2"
"k4v3"
)
key5=value5
#key6="Do Not Include Me"
In a bash script, I want to read the contents of this script.conf file into an array. I've learned how to handle the scenarios for keys 1, 2, 3, and 5, but the key4 scenario throws a wrench into it with it spanning across multiple lines.
I've been exploring the use of sed -n '/=\s*[(]/,/[)]/{/' which does capture key4 and its value, but I can't figure out how to mix this so that the other keys are also captured in the matches. The range syntax is also new to me, so I haven't figured out how to separate the key/value. I feel like there is an easy regex that would accomplish what I want... in plain-text: "find and group the pattern ^(.*)= (for the key), then group everything after the '=' char until another ^(.*)= match is found, rinse and repeat". I guess if I do this, I need to change the while read line to not handle the key/value separation for me (I'll be looking into this while I'm waiting for a response). BTW, I think a solution where the value of key4 is flattened (new lines removed) would be acceptable; I know for key3 I have to store the value as a string and then convert it to an array later when I want to iterate over it since an array element apparently can't contain a list.
Am I on the right path with sed or is this a job for awk or some other tool? (I haven't ventured into awk yet). Is there an easier approach that I'm missing because I'm too deep into the forest (like changing the while read line in the LoadConfigFile function)?
Here is the code that I have so far in script.sh for processing and capturing the other pairs into the $config array:
__AppDir=$(dirname $0)
__AppName=${__ScriptName%.*}
typeset -A config #init config array
config=( #Setting Default Config values
[key1]="defaultValue1"
[key2]="${HOME}/defaultFolder"
[QuietMode]=0
[Verbose]=0 #Ex. Usage: [[ "${config[Verbose]}" -gt 0 ]] && echo ">>>Debug print"
)
function LoadConfigFile() {
local cfgFile="${1}"
shopt -s extglob #Needed to remove trailing spaces
if [ -f ${cfgFile} ]; then
while IFS='=' read -r key value; do
if [[ "${key:0:1}" == "#" ]]; then
#echo "Skipping Comment line: ${key}"
elif [ "${key:-EMPTY}" != "EMPTY" ]; then
value="${value%%\#*}" # Delete in-line, right comments
value="${value%%*( )}" # Delete trailing spaces
value="${value%%( )*}" # Delete leading spaces
#value="${value%\"*}" # Delete opening string quotes
#value="${value#\"*}" # Delete closing string quotes
#Manipulate any variables included in the value so that they can be expanded correctly
# - value must be stored in the format: "${var1}". `backticks`, "$var2", and "doubleQuotes" are left as is
value="${value//\"/\\\"}" # Escape double quotes for eval
value="${value//\`/\\\`}" # Escape backticks for eval
value="${value//\$/\\\$}" # Escape ALL '$' for eval
value="${value//\\\${/\${}" # Undo the protection of '$' if it was followed by a '{'
value=$(eval "printf '%s\n' \"${value}\"")
config[${key}]=${value} #Store the value into the config array at the specified key
echo " >>>DBG: Key = ${key}, Value = ${value}"
#else
# echo "Skipped Empty Key"
fi
done < "${cfgFile}"
fi
}
CONFIG_FILE=${__AppDir}/${__AppName}.conf
echo "Config File # ${CONFIG_FILE}"
LoadConfigFile ${CONFIG_FILE}
#Print elements of $config
echo "Script Config Values:"
echo "----------------------------"
for key in "${!config[#]}"; do #The '!' char gets an array of the keys, without it, we would get an array of the values
printf " %-20s = %s\n" "${key}" "${config[${key}]}"
done
echo "------ End Script Config ------"
#To convert to an array...
declare -a valAsArray=${config[RequiredAppPackages]} #Convert the value from a string to an array
echo "Count = ${#valAsArray[#]}"
for itemCfg in "${valAsArray[#]}"; do
echo " item = ${itemCfg}"
done
As I mentioned before, I'm just starting to learn bash and Linux scripting in general, so if you see that I'm doing some taboo things in other areas of my code too, please feel free to provide feedback in the comments... I don't want to start bad habits early on :-).
*If it matters, the OS is Ubuntu 14.04.
EDIT:
As requested, after reading the script.conf file, I would like for the elements in $config[#] to be equivalent to the following:
typeset -A config #init config array
config=(
[key1]="value1"
[key2]="${HOME}/Folder"
[key3]="( \"k3v1\" \"k3 v2\" \"k3v3\" )"
[key4]="( \"k4v1\" \"k4 v2\" \"k4v3\" )"
[key5]="value5"
)
I want to be able to convert the values of elements 'key4' and 'key3' into an array and iterated over them the same way in the following code:
declare -a keyValAsArray=${config[keyN]} #Convert the value from a string to an array
echo "Count = ${#keyValAsArray[#]}"
for item in "${keyValAsArray[#]}"; do
echo " item = ${item}"
done
I don't think it matters if \n is preserved for key4's value or not... that depends on if declare has a problem with it.
A shell is an environment from which to call tools with a language to sequence those calls. It is NOT a tool to manipulate text. The standard UNIX tool to manipulate text is awk. Trying to manipulate text in shell IS a bad habit, see why-is-using-a-shell-loop-to-process-text-considered-bad-practice for SOME of the reasons why
You still didn't post the expected result of populating the config array so I'm not sure but I think this is what you wanted:
$ cat tst.sh
declare -A config="( $(awk '
{ gsub(/^[[:space:]]+|([[:space:]]+|#.*)$/,"") }
!NF { next }
/^[^="]+=/ {
name = gensub(/=.*/,"",1)
value = gensub(/^[^=]+=/,"",1)
n2v[name] = value
next
}
{ n2v[name] = n2v[name] OFS $0 }
END {
for (name in n2v) {
value = gensub(/"/,"\\\\&","g",n2v[name])
printf "[%s]=\"%s\"\n", name, value
}
}
' script.conf
) )"
declare -p config
$ ./tst.sh
declare -A config='([key5]="value5" [key4]="( \"k4v1\" \"k4 v2\" \"k4v3\" )" [key3]="( \"k3v1\" \"k3 v2\" \"k3v3\")" [key2]="/home/Ed/Folder" [key1]="value1" )'
The above uses GNU awk for gensub(), with other awks you'd use [g]sub() instead.
Is there a way in bash to understand the name of a variable that's passed to it?
Example:
var1=file1.txt
err_var=errorfile.txt
function func1 {
echo "func1: name of the variable is: " $1
echo "func1: value of variable is: " $1
echo
if [ ! -e $var1 ]
then
$1 = $err_val #is this even possible?
fi
func2 $1
}
function func2 {
echo "func2: name of the variable is: " $1
echo "func2: value of variable is: " $1
echo
}
func1 $var1
func1 $err_var
I was hoping to get the following output if file1.txt exists:
func1: name of the variable is: var1
func1: value of variable is: file1.txt
func2: name of the variable is: var1
func2: value of variable is: file1.txt
And when file1.txt does not exist:
func1: name of the variable is: var1
func1: value of variable is: file1.txt
func2: name of the variable is: err_var
func2: value of variable is: errorfile.txt
Any ideas?
No, the variable is expanded before the function sees it. The function only sees the value, not the variable name.
If you pass the variable name unexpanded and without the dollar sign, you can use indirection.
get_it () {
echo "${!1}"
}
Demo:
$ foo=bar
$ baz=qux
$ get_it foo
bar
$ get_it baz
qux
Like Dennis said, once you've expanded the variable using the dollar-sign, your function no longer has a way to get the variable name. But I think you also asked about a way to set the variable's value, and that part hasn't been answered yet. There are some non-portable ways to do that, like with declare -n (Google that if you're interested), but my answer is going to stick to a universal solution.
I'm a C++ programmer, so I like to mimic the "getter and setter" philosophy, where you use tiny little functions to get and set the value of a variable. The downside to using getters and setters is that you need to create a function (or two functions) for every value you want to manage. So... I made a "factory function" that handles creating getters/setters for you:
makeGetSet() {
. /dev/fd/0 <<END_FUNC
${1}Val() { [ "\${1}" ] && ${1}="\${1}" || echo "\${${1}}"; }
END_FUNC
}
It does not require any particular shell with special features like indirection or namerefs, or the declare utility. No eval, no aliases, just 100% POSIX. You just pass in your variable name to makeGetSet, and your getter/setter function has that same name with a "Val" at the end (e.g. myVariableVal). Tested with bash and dash. You're free to continue using the "normal" shell ways to read/write to your variable in combination with my function.
Usage:
Setup: makeGetSet myVariable
Set: myVariableVal newValue
Get: anotherVariable="`myVariableVal`"
Print: myVariableVal
I wasn't sure about a couple parts of your script, so I took some educated guesses. Where you have if [ ! -e $var1 ], I think you meant if [ ! -e $1 ]. And at the end, where you call the functions, you had func1 $var1 and func1 $err_var, but I think you meant to just use func1 $var1 or have a third variable. It looks like $err_var is a "default value for errors" rather than something you'd give as input, but maybe I'm not following your idea.
So my answer to your question would look like:
var1=file1.txt; makeGetSet var1
err_var=errorfile.txt; makeGetSet err_var
function func1 {
echo "func1: name of the variable is: " ${1%Val}
echo "func1: value of variable is: " `${1}`
echo
# Shorter than if..fi
[ ! -e `${1}` ] && ${1} ${err_val}
func2 ${1}
}
function func2 {
echo "func2: name of the variable is: " ${1%Val}
echo "func2: value of variable is: " `${1}`
echo
}
func1 ${var1}