Pass keys from array from bash into jq -- output all nulls - linux

I'm trying to do something akin to this:
jq -r '. | ."Time Series (Daily)"."2020-12-02" | ."1. open"' newdata.json
...but with the key coming from a variable, as in:
jq -r --arg key "$key" '. | ."Time Series (Daily)"."[$key]" | ."1. open"' newdata.json
The first one works just fine, but when I assign the date to a variable called key and then try to get the data, it fails.
I tried This answer and This answer. But did not work for me.
{
"Meta Data": {
"1. Information": "Daily Prices (open, high, low, close) and Volumes",
"2. Symbol": "AB",
"3. Last Refreshed": "2020-12-02",
"4. Output Size": "Compact",
"5. Time Zone": "US/Eastern"
},
"Time Series (Daily)": {
"2020-12-02": {
"1. open": "32.6700",
"2. high": "33.3300",
"3. low": "32.5000",
"4. close": "33.1200",
"5. volume": "273799"
},
"2020-12-01": {
"1. open": "32.1500",
"2. high": "32.8000",
"3. low": "32.0000",
"4. close": "32.6000",
"5. volume": "265086"
},
"2020-11-30": {
"1. open": "32.3800",
"2. high": "32.4900",
"3. low": "31.7500",
"4. close": "31.8700",
"5. volume": "251970"
}
}
}
The above is the newdata.json file.
What I want to get is the "1. open" value.
I am using a for loop to iterate over all the keys of "Time Series (Daily)" and the keys are generated correctly. There is no issue with that. I then want to use the $key variable in each iteration to
get the data I need.
readarray keys <<< "$(jq '."Time Series (Daily)" | keys[]' newdata.json)"
for key in "${keys[#]}"; do
jq -r --arg key "$key" '. | ."Time Series (Daily)" | .[$key] | ."1. open"' newdata.json
done

Focusing On The Immediate Issue
The problem isn't how you're passing key to jq; the problem is how you're populating the key variable in the first place.
Change:
readarray keys <<< "$(jq '."Time Series (Daily)" | keys[]' newdata.json)"
...to:
readarray -t keys <<< "$(jq -r '."Time Series (Daily)" | keys[]' newdata.json)"
There are two changes here:
We added the -t argument to readarray, so it no longer includes the newline ending each line in the variable itself.
We added the -r argument to jq, so it no longer adds literal quotes around the strings.
Sidebar: Retrieving both keys and values at the same time
There's no reason to do one pass to retrieve keys and another to retrieve values -- better to just get them all at once:
dates=( )
opening_prices=( )
while IFS=$'\t' read -r date opening_price; do
dates+=( "$date" )
opening_prices+=( "$opening_price" )
done < <(
jq -r '
."Time Series (Daily)" | to_entries[] | [.key, .value."1. open"] | #tsv
' <newdata.json
)
...after which, declare -p dates opening_prices emits:
declare -a dates=([0]="2020-12-02" [1]="2020-12-01" [2]="2020-11-30")
declare -a opening_prices=([0]="32.6700" [1]="32.1500" [2]="32.3800")
Original response (before population of keys was shown)
Here's a different approach that only calls jq once, instead of once per item, while still getting your keys from an array. It does this by using -R to read raw strings as input; . is then used to address those inputs (which we rename to $key to make it clear how this lines up with the old code).
keys=("2020-12-02" "2020-12-01" "2020-11-30")
readarray -t openingPrices < <(
jq -Rr --slurpfile indatas newdata.json '
$indatas[0] as $indata | . as $key |
$indata."Time Series (Daily)"[$key]["1. open"]
' < <(printf '%s\n' "${keys[#]}")
)
After running that, declare -p keys openingPrices (to show how both arrays are defined) emits:
declare -a keys=([0]="2020-12-02" [1]="2020-12-01" [2]="2020-11-30")
declare -a openingPrices=([0]="32.6700" [1]="32.1500" [2]="32.3800")
...so you have an output array that lines up with your input array (so long as the latter isn't sparse).

Use | .[$key] | to get the key from your $key variable;
key="2020-12-02"
jq -r --arg key "$key" '."Time Series (Daily)" | .[$key] | ."1. open"' newdata.json
# output: 32.6700
Or, combined with the for() (hardcoded keys, since we're not sure how you get those)
keys=("2020-12-02" "2020-12-01" "2020-11-30")
for key in "${keys[#]}"; do
jq -r --arg key "$key" '."Time Series (Daily)" | .[$key] | ."1. open"' newdata.json
done
32.6700
32.1500
32.3800

Related

Basic Quadratic formula calculator shell script trouble

This is my very first shell script for a Unix class, this is one of the scripts I hope to submit for my final. However there are a few kinks I cannot seem to clear up, it seems to be arithmetic operation errors, and I can't seem to figure it out. Please be kind! thank you so much for your time.
lightgreen=`echo -en "\e[92m"
echo What are the values of a, b \& c?
LIGHTRED=`echo -en "\e[101m"
echo a value:
read a
echo b value:
read b
echo c value:
read c
discrim=$(($b**2 - 4*$a*$c))
sqrtd=$((sqrt($discrim) | bc ))
echo test $sqrtd
echo ${lightgreen}The discriminant is:${discrim}
#xone=$((( -$b + sqrt$discrim) / (2 * $a) | bc ))
#xtwo=$((( -$b - sqrt$discrim) / (2 * $a) | bc ))
xone=$((echo (-1*$b + sqrt($discrim)) / (2*$a) | bc ))
xtwo=$((echo (-1*$b - sqrt($discrim)) / (2*$a) | bc ))
echo ${lightgreen}The discriminant is:${discrim}
#if [$discrim -lt 0 ]
# echo $LIGHTRED There are no real solutions.
#
#
#
echo The two solutions are $xone $xtwo
I have tried to mess with the syntax a good amount, I'm not sure if it's the parentheses that mess me up or the sqrt function, I have tried to incorporate | bc but to no avail. Any help is greatly appreciated! :)
Don't hesitate to call man bash, man bc manual pages.
Use https://www.shellcheck.net/ to check your shell scripts.
Shellcheck also exists on command line and in Visual Studio Code with extension.
#! /usr/bin/env bash
# The first line is very important to now the name of the interpreter
# Always close " , ' , or ` sequences with same character
# Do not use old `...` syntax, replaced by $(...)
# Here, use $'...', to assign value with \... sequences
lightgreen=$'\e[92m'
lightred=$'\e[101m'
normal=$'\e[0m'
# It's better to put phrase between "..." or '...'
echo "What are the values of a, b & c?"
# Use read -p option to specify prompt
# Use read -r option to not act backslash as an escape character
read -p "a value: " -r a
read -p "b value: " -r b
read -p "c value: " -r c
# With bash only, it's only possible to use integer values
# discrim=$(($b**2 - 4*$a*$c))
# use bc instead
discrim=$(bc -l <<<"$b^2 - 4*$a*$c")
# The syntax:
# bc <<<"..."
# is equivalent to:
# echo "..." | bc
# but without pipe (|)
# Close the color change with normal return
echo "${lightgreen}The discriminant is: ${discrim}${normal}"
if [[ "${discrim:0:1}" == "-" ]]; then
echo "${lightred}There are no real solutions${normal}"
# ... complex ...
else
sqrtd=$(bc -l <<<"sqrt($discrim)")
echo "sqrt($discrim)=$sqrtd"
xone=$(bc -l <<<"(-1*$b + $sqrtd) / (2*$a)")
xtwo=$(bc -l <<<"(-1*$b - $sqrtd) / (2*$a)")
echo "The two solutions are: $xone and $xtwo"
fi

Bash shell how to remove string from array with -= operator [duplicate]

I need to remove an element from an array in bash shell.
Generally I'd simply do:
array=("${(#)array:#<element to remove>}")
Unfortunately the element I want to remove is a variable so I can't use the previous command.
Down here an example:
array+=(pluto)
array+=(pippo)
delete=(pluto)
array( ${array[#]/$delete} ) -> but clearly doesn't work because of {}
Any idea?
The following works as you would like in bash and zsh:
$ array=(pluto pippo)
$ delete=pluto
$ echo ${array[#]/$delete}
pippo
$ array=( "${array[#]/$delete}" ) #Quotes when working with strings
If need to delete more than one element:
...
$ delete=(pluto pippo)
for del in ${delete[#]}
do
array=("${array[#]/$del}") #Quotes when working with strings
done
Caveat
This technique actually removes prefixes matching $delete from the elements, not necessarily whole elements.
Update
To really remove an exact item, you need to walk through the array, comparing the target to each element, and using unset to delete an exact match.
array=(pluto pippo bob)
delete=(pippo)
for target in "${delete[#]}"; do
for i in "${!array[#]}"; do
if [[ ${array[i]} = $target ]]; then
unset 'array[i]'
fi
done
done
Note that if you do this, and one or more elements is removed, the indices will no longer be a continuous sequence of integers.
$ declare -p array
declare -a array=([0]="pluto" [2]="bob")
The simple fact is, arrays were not designed for use as mutable data structures. They are primarily used for storing lists of items in a single variable without needing to waste a character as a delimiter (e.g., to store a list of strings which can contain whitespace).
If gaps are a problem, then you need to rebuild the array to fill the gaps:
for i in "${!array[#]}"; do
new_array+=( "${array[i]}" )
done
array=("${new_array[#]}")
unset new_array
You could build up a new array without the undesired element, then assign it back to the old array. This works in bash:
array=(pluto pippo)
new_array=()
for value in "${array[#]}"
do
[[ $value != pluto ]] && new_array+=($value)
done
array=("${new_array[#]}")
unset new_array
This yields:
echo "${array[#]}"
pippo
This is the most direct way to unset a value if you know it's position.
$ array=(one two three)
$ echo ${#array[#]}
3
$ unset 'array[1]'
$ echo ${array[#]}
one three
$ echo ${#array[#]}
2
This answer is specific to the case of deleting multiple values from large arrays, where performance is important.
The most voted solutions are (1) pattern substitution on an array, or (2) iterating over the array elements. The first is fast, but can only deal with elements that have distinct prefix, the second has O(n*k), n=array size, k=elements to remove. Associative array are relative new feature, and might not have been common when the question was originally posted.
For the exact match case, with large n and k, possible to improve performance from O(nk) to O(n+klog(k)). In practice, O(n) assuming k much lower than n. Most of the speed up is based on using associative array to identify items to be removed.
Performance (n-array size, k-values to delete). Performance measure seconds of user time
N K New(seconds) Current(seconds) Speedup
1000 10 0.005 0.033 6X
10000 10 0.070 0.348 5X
10000 20 0.070 0.656 9X
10000 1 0.043 0.050 -7%
As expected, the current solution is linear to N*K, and the fast solution is practically linear to K, with much lower constant. The fast solution is slightly slower vs the current solution when k=1, due to additional setup.
The 'Fast' solution: array=list of input, delete=list of values to remove.
declare -A delk
for del in "${delete[#]}" ; do delk[$del]=1 ; done
# Tag items to remove, based on
for k in "${!array[#]}" ; do
[ "${delk[${array[$k]}]-}" ] && unset 'array[k]'
done
# Compaction
array=("${array[#]}")
Benchmarked against current solution, from the most-voted answer.
for target in "${delete[#]}"; do
for i in "${!array[#]}"; do
if [[ ${array[i]} = $target ]]; then
unset 'array[i]'
fi
done
done
array=("${array[#]}")
Here's a one-line solution with mapfile:
$ mapfile -d $'\0' -t arr < <(printf '%s\0' "${arr[#]}" | grep -Pzv "<regexp>")
Example:
$ arr=("Adam" "Bob" "Claire"$'\n'"Smith" "David" "Eve" "Fred")
$ echo "Size: ${#arr[*]} Contents: ${arr[*]}"
Size: 6 Contents: Adam Bob Claire
Smith David Eve Fred
$ mapfile -d $'\0' -t arr < <(printf '%s\0' "${arr[#]}" | grep -Pzv "^Claire\nSmith$")
$ echo "Size: ${#arr[*]} Contents: ${arr[*]}"
Size: 5 Contents: Adam Bob David Eve Fred
This method allows for great flexibility by modifying/exchanging the grep command and doesn't leave any empty strings in the array.
Partial answer only
To delete the first item in the array
unset 'array[0]'
To delete the last item in the array
unset 'array[-1]'
To expand on the above answers, the following can be used to remove multiple elements from an array, without partial matching:
ARRAY=(one two onetwo three four threefour "one six")
TO_REMOVE=(one four)
TEMP_ARRAY=()
for pkg in "${ARRAY[#]}"; do
for remove in "${TO_REMOVE[#]}"; do
KEEP=true
if [[ ${pkg} == ${remove} ]]; then
KEEP=false
break
fi
done
if ${KEEP}; then
TEMP_ARRAY+=(${pkg})
fi
done
ARRAY=("${TEMP_ARRAY[#]}")
unset TEMP_ARRAY
This will result in an array containing:
(two onetwo three threefour "one six")
Here's a (probably very bash-specific) little function involving bash variable indirection and unset; it's a general solution that does not involve text substitution or discarding empty elements and has no problems with quoting/whitespace etc.
delete_ary_elmt() {
local word=$1 # the element to search for & delete
local aryref="$2[#]" # a necessary step since '${!$2[#]}' is a syntax error
local arycopy=("${!aryref}") # create a copy of the input array
local status=1
for (( i = ${#arycopy[#]} - 1; i >= 0; i-- )); do # iterate over indices backwards
elmt=${arycopy[$i]}
[[ $elmt == $word ]] && unset "$2[$i]" && status=0 # unset matching elmts in orig. ary
done
return $status # return 0 if something was deleted; 1 if not
}
array=(a 0 0 b 0 0 0 c 0 d e 0 0 0)
delete_ary_elmt 0 array
for e in "${array[#]}"; do
echo "$e"
done
# prints "a" "b" "c" "d" in lines
Use it like delete_ary_elmt ELEMENT ARRAYNAME without any $ sigil. Switch the == $word for == $word* for prefix matches; use ${elmt,,} == ${word,,} for case-insensitive matches; etc., whatever bash [[ supports.
It works by determining the indices of the input array and iterating over them backwards (so deleting elements doesn't screw up iteration order). To get the indices you need to access the input array by name, which can be done via bash variable indirection x=1; varname=x; echo ${!varname} # prints "1".
You can't access arrays by name like aryname=a; echo "${$aryname[#]}, this gives you an error. You can't do aryname=a; echo "${!aryname[#]}", this gives you the indices of the variable aryname (although it is not an array). What DOES work is aryref="a[#]"; echo "${!aryref}", which will print the elements of the array a, preserving shell-word quoting and whitespace exactly like echo "${a[#]}". But this only works for printing the elements of an array, not for printing its length or indices (aryref="!a[#]" or aryref="#a[#]" or "${!!aryref}" or "${#!aryref}", they all fail).
So I copy the original array by its name via bash indirection and get the indices from the copy. To iterate over the indices in reverse I use a C-style for loop. I could also do it by accessing the indices via ${!arycopy[#]} and reversing them with tac, which is a cat that turns around the input line order.
A function solution without variable indirection would probably have to involve eval, which may or may not be safe to use in that situation (I can't tell).
Using unset
To remove an element at particular index, we can use unset and then do copy to another array. Only just unset is not required in this case. Because unset does not remove the element it just sets null string to the particular index in array.
declare -a arr=('aa' 'bb' 'cc' 'dd' 'ee')
unset 'arr[1]'
declare -a arr2=()
i=0
for element in "${arr[#]}"
do
arr2[$i]=$element
((++i))
done
echo "${arr[#]}"
echo "1st val is ${arr[1]}, 2nd val is ${arr[2]}"
echo "${arr2[#]}"
echo "1st val is ${arr2[1]}, 2nd val is ${arr2[2]}"
Output is
aa cc dd ee
1st val is , 2nd val is cc
aa cc dd ee
1st val is cc, 2nd val is dd
Using :<idx>
We can remove some set of elements using :<idx> also. For example if we want to remove 1st element we can use :1 as mentioned below.
declare -a arr=('aa' 'bb' 'cc' 'dd' 'ee')
arr2=("${arr[#]:1}")
echo "${arr2[#]}"
echo "1st val is ${arr2[1]}, 2nd val is ${arr2[2]}"
Output is
bb cc dd ee
1st val is cc, 2nd val is dd
http://wiki.bash-hackers.org/syntax/pe#substring_removal
${PARAMETER#PATTERN} # remove from beginning
${PARAMETER##PATTERN} # remove from the beginning, greedy match
${PARAMETER%PATTERN} # remove from the end
${PARAMETER%%PATTERN} # remove from the end, greedy match
In order to do a full remove element, you have to do an unset command with an if statement. If you don't care about removing prefixes from other variables or about supporting whitespace in the array, then you can just drop the quotes and forget about for loops.
See example below for a few different ways to clean up an array.
options=("foo" "bar" "foo" "foobar" "foo bar" "bars" "bar")
# remove bar from the start of each element
options=("${options[#]/#"bar"}")
# options=("foo" "" "foo" "foobar" "foo bar" "s" "")
# remove the complete string "foo" in a for loop
count=${#options[#]}
for ((i = 0; i < count; i++)); do
if [ "${options[i]}" = "foo" ] ; then
unset 'options[i]'
fi
done
# options=( "" "foobar" "foo bar" "s" "")
# remove empty options
# note the count variable can't be recalculated easily on a sparse array
for ((i = 0; i < count; i++)); do
# echo "Element $i: '${options[i]}'"
if [ -z "${options[i]}" ] ; then
unset 'options[i]'
fi
done
# options=("foobar" "foo bar" "s")
# list them with select
echo "Choose an option:"
PS3='Option? '
select i in "${options[#]}" Quit
do
case $i in
Quit) break ;;
*) echo "You selected \"$i\"" ;;
esac
done
Output
Choose an option:
1) foobar
2) foo bar
3) s
4) Quit
Option?
Hope that helps.
There is also this syntax, e.g. if you want to delete the 2nd element :
array=("${array[#]:0:1}" "${array[#]:2}")
which is in fact the concatenation of 2 tabs. The first from the index 0 to the index 1 (exclusive) and the 2nd from the index 2 to the end.
POSIX shell script does not have arrays.
So most probably you are using a specific dialect such as bash, korn shells or zsh.
Therefore, your question as of now cannot be answered.
Maybe this works for you:
unset array[$delete]
What I do is:
array="$(echo $array | tr ' ' '\n' | sed "/itemtodelete/d")"
BAM, that item is removed.
This is a quick-and-dirty solution that will work in simple cases but will break if (a) there are regex special characters in $delete, or (b) there are any spaces at all in any items. Starting with:
array+=(pluto)
array+=(pippo)
delete=(pluto)
Delete all entries exactly matching $delete:
array=(`echo $array | fmt -1 | grep -v "^${delete}$" | fmt -999999`)
resulting in
echo $array -> pippo, and making sure it's an array:
echo $array[1] -> pippo
fmt is a little obscure: fmt -1 wraps at the first column (to put each item on its own line. That's where the problem arises with items in spaces.) fmt -999999 unwraps it back to one line, putting back the spaces between items. There are other ways to do that, such as xargs.
Addendum: If you want to delete just the first match, use sed, as described here:
array=(`echo $array | fmt -1 | sed "0,/^${delete}$/{//d;}" | fmt -999999`)
Actually, I just noticed that the shell syntax somewhat has a behavior built-in that allows for easy reconstruction of the array when, as posed in the question, an item should be removed.
# let's set up an array of items to consume:
x=()
for (( i=0; i<10; i++ )); do
x+=("$i")
done
# here, we consume that array:
while (( ${#x[#]} )); do
i=$(( $RANDOM % ${#x[#]} ))
echo "${x[i]} / ${x[#]}"
x=("${x[#]:0:i}" "${x[#]:i+1}")
done
Notice how we constructed the array using bash's x+=() syntax?
You could actually add more than one item with that, the content of a whole other array at once.
In ZSH this is dead easy (note this uses more bash compatible syntax than necessary where possible for ease of understanding):
# I always include an edge case to make sure each element
# is not being word split.
start=(one two three 'four 4' five)
work=(${(#)start})
idx=2
val=${work[idx]}
# How to remove a single element easily.
# Also works for associative arrays (at least in zsh)
work[$idx]=()
echo "Array size went down by one: "
[[ $#work -eq $(($#start - 1)) ]] && echo "OK"
echo "Array item "$val" is now gone: "
[[ -z ${work[(r)$val]} ]] && echo OK
echo "Array contents are as expected: "
wanted=("${start[#]:0:1}" "${start[#]:2}")
[[ "${(j.:.)wanted[#]}" == "${(j.:.)work[#]}" ]] && echo "OK"
echo "-- array contents: start --"
print -l -r -- "-- $#start elements" ${(#)start}
echo "-- array contents: work --"
print -l -r -- "-- $#work elements" "${work[#]}"
Results:
Array size went down by one:
OK
Array item two is now gone:
OK
Array contents are as expected:
OK
-- array contents: start --
-- 5 elements
one
two
three
four 4
five
-- array contents: work --
-- 4 elements
one
three
four 4
five
To avoid conflicts with array index using unset - see https://stackoverflow.com/a/49626928/3223785 and https://stackoverflow.com/a/47798640/3223785 for more information - reassign the array to itself: ARRAY_VAR=(${ARRAY_VAR[#]}).
#!/bin/bash
ARRAY_VAR=(0 1 2 3 4 5 6 7 8 9)
unset ARRAY_VAR[5]
unset ARRAY_VAR[4]
ARRAY_VAR=(${ARRAY_VAR[#]})
echo ${ARRAY_VAR[#]}
A_LENGTH=${#ARRAY_VAR[*]}
for (( i=0; i<=$(( $A_LENGTH -1 )); i++ )) ; do
echo ""
echo "INDEX - $i"
echo "VALUE - ${ARRAY_VAR[$i]}"
done
exit 0
[Ref.: https://tecadmin.net/working-with-array-bash-script/ ]
How about something like:
array=(one two three)
array_t=" ${array[#]} "
delete=one
array=(${array_t// $delete / })
unset array_t
#/bin/bash
echo "# define array with six elements"
arr=(zero one two three 'four 4' five)
echo "# unset by index: 0"
unset -v 'arr[0]'
for i in ${!arr[*]}; do echo "arr[$i]=${arr[$i]}"; done
arr_delete_by_content() { # value to delete
for i in ${!arr[*]}; do
[ "${arr[$i]}" = "$1" ] && unset -v 'arr[$i]'
done
}
echo "# unset in global variable where value: three"
arr_delete_by_content three
for i in ${!arr[*]}; do echo "arr[$i]=${arr[$i]}"; done
echo "# rearrange indices"
arr=( "${arr[#]}" )
for i in ${!arr[*]}; do echo "arr[$i]=${arr[$i]}"; done
delete_value() { # value arrayelements..., returns array decl.
local e val=$1; new=(); shift
for e in "${#}"; do [ "$val" != "$e" ] && new+=("$e"); done
declare -p new|sed 's,^[^=]*=,,'
}
echo "# new array without value: two"
declare -a arr="$(delete_value two "${arr[#]}")"
for i in ${!arr[*]}; do echo "arr[$i]=${arr[$i]}"; done
delete_values() { # arraydecl values..., returns array decl. (keeps indices)
declare -a arr="$1"; local i v; shift
for v in "${#}"; do
for i in ${!arr[*]}; do
[ "$v" = "${arr[$i]}" ] && unset -v 'arr[$i]'
done
done
declare -p arr|sed 's,^[^=]*=,,'
}
echo "# new array without values: one five (keep indices)"
declare -a arr="$(delete_values "$(declare -p arr|sed 's,^[^=]*=,,')" one five)"
for i in ${!arr[*]}; do echo "arr[$i]=${arr[$i]}"; done
# new array without multiple values and rearranged indices is left to the reader

Bash script requires parameter

Below is my bash script.
#!/bin/bash
message=$1
hostname=$2
severity=$3
eventname=$4
tagpath=$5
appname=$6
data="{"action":"EventsRouter","method":"add_event","data":[{"summary":"$message"},"device":"$hostname","message": "$message $eventname $tagpath","component":"$appname","severity":"$severity","evclasskey":"nxlog","evclass":"/nxlog/perf","monitor":"localhost"}],"type":"rpc","tid":1}"
echo "Total number of args : $#"
echo "message = $message"
echo "hostname = $hostname"
echo "appname = $appname"
echo "data = $data"
curl -u uname:password -k https://myurlcom/zport/dmd/evconsole_router -d $data
and when i try to run with sh tcp.sh value value value value value value
host:
'host,component:host,severity:host,evclasskey:nxlog,evclass:/nxlog/perf,monitor:localhost}],type:rpc,tid:1}'
is not a legal name (unexpected end of input) Total number of args : 6
message = message hostname = test appname = host data = curl: option
-d: requires parameter
I see that data has no value included.
This json has to be sent in this order for it to be accepted in the endpoint. Help me correct this.
Using jq to safely generate the desired JSON:
#!/bin/bash
parameters=(
--arg message "$1"
--arg hostname "$2"
--arg severity "$3"
--arg eventname "$4"
--arg tagpath "$5"
--arg appname "$6"
)
data=$(jq -n "${parameters[#]}" '
{action: "EventsRouter",
method: "add_event",
data: [ {summary: $message,
device: $hostname,
message: "\($message) \($eventname\) \($tagpath)",
component: $appname,
severity: $severity,
evclasskey: "nxlog",
evclass: "/nxlog/perf",
monitor: "localhost"
}
],
type: "rpc",
tid: 1
}'
curl -u uname:password -k https://myurlcom/zport/dmd/evconsole_router -d "$data"
Assuming:
message="my_message"
hostname="my_host"
severity="my_severity"
eventname="my_event"
tagpath="my_path"
appname="my_app"
If you run:
data="{"action":"EventsRouter","method":"add_event","data":[{"summary":"$message"},"device":"$hostname","message": "$message $eventname $tagpath","component":"$appname","severity":"$severity","evclasskey":"nxlog","evclass":"/nxlog/perf","monitor":"localhost"}],"type":"rpc","tid":1}"
you will get an error, because there is a not escaped white space before the string "my_event"
my_event: command not found
What happened? Since your json input has a lot of words between double quotes, you will have to enclose the whole string into single quotes, in order to preserve the double quotes inside of the string. But between single quotes, the bash variables will not be replaced by their value. So you will need to close the single quotes before each variable and reopen these again immediately after.
So that line of your script must become:
data='{"action":"EventsRouter","method":"add_event","data":[{"summary":"'$message'"},"device":"'$hostname'","message": "'$message $eventname $tagpath'","component":"'$appname'","severity":"'$severity'","evclasskey":"nxlog","evclass":"/nxlog/perf","monitor":"localhost"}],"type":"rpc","tid":1}'
If you execute:
echo "$data"
you will get:
{"action":"EventsRouter","method":"add_event","data":[{"summary":"my_message"},"device":"my_host","message": "my_message my_event my_path","component":"my_app","severity":"my_severity","evclasskey":"nxlog","evclass":"/nxlog/perf","monitor":"localhost"}],"type":"rpc","tid":1}
which is correct, I assume: the double quotes didn't disappear from your json data structure.

How to extract key value pairs from a file when values span multiple lines?

I'm a few weeks into bash scripting and I haven't advanced enough yet to get my head wrapped around this problem. Any help would be appreciated!
I have a "script.conf" file that contains the following:
key1=value1
key2=${HOME}/Folder
key3=( "k3v1" "k3 v2" "k3v3")
key4=( "k4v1"
"k4 v2"
"k4v3"
)
key5=value5
#key6="Do Not Include Me"
In a bash script, I want to read the contents of this script.conf file into an array. I've learned how to handle the scenarios for keys 1, 2, 3, and 5, but the key4 scenario throws a wrench into it with it spanning across multiple lines.
I've been exploring the use of sed -n '/=\s*[(]/,/[)]/{/' which does capture key4 and its value, but I can't figure out how to mix this so that the other keys are also captured in the matches. The range syntax is also new to me, so I haven't figured out how to separate the key/value. I feel like there is an easy regex that would accomplish what I want... in plain-text: "find and group the pattern ^(.*)= (for the key), then group everything after the '=' char until another ^(.*)= match is found, rinse and repeat". I guess if I do this, I need to change the while read line to not handle the key/value separation for me (I'll be looking into this while I'm waiting for a response). BTW, I think a solution where the value of key4 is flattened (new lines removed) would be acceptable; I know for key3 I have to store the value as a string and then convert it to an array later when I want to iterate over it since an array element apparently can't contain a list.
Am I on the right path with sed or is this a job for awk or some other tool? (I haven't ventured into awk yet). Is there an easier approach that I'm missing because I'm too deep into the forest (like changing the while read line in the LoadConfigFile function)?
Here is the code that I have so far in script.sh for processing and capturing the other pairs into the $config array:
__AppDir=$(dirname $0)
__AppName=${__ScriptName%.*}
typeset -A config #init config array
config=( #Setting Default Config values
[key1]="defaultValue1"
[key2]="${HOME}/defaultFolder"
[QuietMode]=0
[Verbose]=0 #Ex. Usage: [[ "${config[Verbose]}" -gt 0 ]] && echo ">>>Debug print"
)
function LoadConfigFile() {
local cfgFile="${1}"
shopt -s extglob #Needed to remove trailing spaces
if [ -f ${cfgFile} ]; then
while IFS='=' read -r key value; do
if [[ "${key:0:1}" == "#" ]]; then
#echo "Skipping Comment line: ${key}"
elif [ "${key:-EMPTY}" != "EMPTY" ]; then
value="${value%%\#*}" # Delete in-line, right comments
value="${value%%*( )}" # Delete trailing spaces
value="${value%%( )*}" # Delete leading spaces
#value="${value%\"*}" # Delete opening string quotes
#value="${value#\"*}" # Delete closing string quotes
#Manipulate any variables included in the value so that they can be expanded correctly
# - value must be stored in the format: "${var1}". `backticks`, "$var2", and "doubleQuotes" are left as is
value="${value//\"/\\\"}" # Escape double quotes for eval
value="${value//\`/\\\`}" # Escape backticks for eval
value="${value//\$/\\\$}" # Escape ALL '$' for eval
value="${value//\\\${/\${}" # Undo the protection of '$' if it was followed by a '{'
value=$(eval "printf '%s\n' \"${value}\"")
config[${key}]=${value} #Store the value into the config array at the specified key
echo " >>>DBG: Key = ${key}, Value = ${value}"
#else
# echo "Skipped Empty Key"
fi
done < "${cfgFile}"
fi
}
CONFIG_FILE=${__AppDir}/${__AppName}.conf
echo "Config File # ${CONFIG_FILE}"
LoadConfigFile ${CONFIG_FILE}
#Print elements of $config
echo "Script Config Values:"
echo "----------------------------"
for key in "${!config[#]}"; do #The '!' char gets an array of the keys, without it, we would get an array of the values
printf " %-20s = %s\n" "${key}" "${config[${key}]}"
done
echo "------ End Script Config ------"
#To convert to an array...
declare -a valAsArray=${config[RequiredAppPackages]} #Convert the value from a string to an array
echo "Count = ${#valAsArray[#]}"
for itemCfg in "${valAsArray[#]}"; do
echo " item = ${itemCfg}"
done
As I mentioned before, I'm just starting to learn bash and Linux scripting in general, so if you see that I'm doing some taboo things in other areas of my code too, please feel free to provide feedback in the comments... I don't want to start bad habits early on :-).
*If it matters, the OS is Ubuntu 14.04.
EDIT:
As requested, after reading the script.conf file, I would like for the elements in $config[#] to be equivalent to the following:
typeset -A config #init config array
config=(
[key1]="value1"
[key2]="${HOME}/Folder"
[key3]="( \"k3v1\" \"k3 v2\" \"k3v3\" )"
[key4]="( \"k4v1\" \"k4 v2\" \"k4v3\" )"
[key5]="value5"
)
I want to be able to convert the values of elements 'key4' and 'key3' into an array and iterated over them the same way in the following code:
declare -a keyValAsArray=${config[keyN]} #Convert the value from a string to an array
echo "Count = ${#keyValAsArray[#]}"
for item in "${keyValAsArray[#]}"; do
echo " item = ${item}"
done
I don't think it matters if \n is preserved for key4's value or not... that depends on if declare has a problem with it.
A shell is an environment from which to call tools with a language to sequence those calls. It is NOT a tool to manipulate text. The standard UNIX tool to manipulate text is awk. Trying to manipulate text in shell IS a bad habit, see why-is-using-a-shell-loop-to-process-text-considered-bad-pr‌​actice for SOME of the reasons why
You still didn't post the expected result of populating the config array so I'm not sure but I think this is what you wanted:
$ cat tst.sh
declare -A config="( $(awk '
{ gsub(/^[[:space:]]+|([[:space:]]+|#.*)$/,"") }
!NF { next }
/^[^="]+=/ {
name = gensub(/=.*/,"",1)
value = gensub(/^[^=]+=/,"",1)
n2v[name] = value
next
}
{ n2v[name] = n2v[name] OFS $0 }
END {
for (name in n2v) {
value = gensub(/"/,"\\\\&","g",n2v[name])
printf "[%s]=\"%s\"\n", name, value
}
}
' script.conf
) )"
declare -p config
$ ./tst.sh
declare -A config='([key5]="value5" [key4]="( \"k4v1\" \"k4 v2\" \"k4v3\" )" [key3]="( \"k3v1\" \"k3 v2\" \"k3v3\")" [key2]="/home/Ed/Folder" [key1]="value1" )'
The above uses GNU awk for gensub(), with other awks you'd use [g]sub() instead.

How to update book title?

ok.. i'm having trouble with updating new book title.. i've already searched for solutions on this website.. and ive tried them but none of them work..
function update_book
{
#echo "Title: "
read -p $'Title: ' updatetitle
#echo "Author: "
read -p $'Name: ' updatename
if grep -Fq "${updatetitle}:${updatename}" BookDB.txt
then
echo "Book found!"
if ! [ -f BookDB.txt ] ; then
touch BookDB.txt
fi
selection=0
until [ "$selection" = "f" ]; do
echo ""
echo ""
echo "Book Update System"
echo ""
echo ""
echo "a) Update title"
echo "b) Update Author"
echo "c) Update Price"
echo "d) Update Qty Available"
echo "e) Update Qty Sold"
echo "f) Back to main menu"
echo -n "Enter your option: "
read selection
echo ""
case $selection in
a) upd_title;press_enter;;
b) upd_author;press_enter;;
c) upd_price;press_enter;;
d) upd_qty_avail;press_enter;;
e) upd_qty_sold;press_enter;;
f) main_menu;press_enter;;
* ) tput setf 4;echo "Please enter a, b, c, d, e, or f";tput setf 7; press_enter
esac
done
else
echo "Error!! Book does not exist!" #not found
fi
}
ok i've made some changes to the codes for this function.. found out that i should use awk to update from old to new. and use grep to get line number..so it's best i stick with this..
function upd_title
{
read -p 'New title: ' title
#awk '/liine/{ print NR; exit }' BookDB.txt
grep -n 'regex' | sed 's/^\([0-9]\+\):.*$/\1/'
awk 'NR==n{$1=a}1' FS=":" OFS =":" n=$linenumber a = $title BookDB.txt
echo "New title successfully updated!!"
}
but after i tried that code.. this is what i got:
Advanced Book Inventory System
1) Add new book
2) Remove existing book info
3) Update book info and quantity
4) Search for book by title/author
5) Process a book sold
6) Inventory summary report
7) Quit
Enter your option: 3
Title: The Notebook
Name: Nicholas Sparks
Book found!
Book Update System
a) Update title
b) Update Author
c) Update Price
d) Update Qty Available
e) Update Qty Sold
f) Back to main menu
Enter your option: a
New title: Notebook
still doesn't update the title.. >.< anyone can help me point out what's the problem? am i missing something here
help me how to do this.. thanks! :D :D
$updatetitle and $updateauthor defined by the read statements in update_book() are not visible in the upd_title() and other functions you define.
Try exporting them after they are read in. If this doesn't work, pass them in as positional parameters to update_title():
upd_title $update_title $update_author
...
function upd_title
{
update_title=$1
update_author=$2
...
}

Resources