How to change json object using jq bash - linux

I have a json object that looks like below:
{
"CallerReference": "terraform-20210518124856158900000001",
"Aliases": {
"Quantity": 3,
"Items": [
"consumer-portal.mlb.effi.com.au",
"*.mlb.effi.com.au",
"coolvalue.mylocalbroker.com.au"
]
}
}
And I need to;
First get the value of Quantity
Increment it by 1
Then change that value so it looks like "Quantity": 4,
And I need to add another entry at the bottom of the Items array.
I am using jq and trying to first replace the "Quantity" value as below:
filename=cf.json
rm $filename
aws cloudfront get-distribution-config --id <ID> \
--query 'DistributionConfig' --output json > $filename
etag=$(aws cloudfront get-distribution-config --id <ID> |
jq -r '.ETag')
aws cloudfront update-distribution \
--distribution-config file://$filename --id <ID> --if-match $etag > /dev/null
jq --argjson i "10" '.Aliases.Quantity[$i]' $filename
But I am getting this error:
jq: error (at cf.json:136): Cannot index number with number

Sounds like += should to the trick [docs]
.Aliases.Quantity += 1 To increment Quantity by 1
.Aliases.Items += [ "Example" ] To add a new index to the Items array
Full command:
jq '.Aliases.Quantity += 1 | .Aliases.Items += [ "Example" ]'
Online demo
{
"CallerReference": "terraform-20210518124856158900000001",
"Aliases": {
"Quantity": 4,
"Items": [
"consumer-portal.mlb.effi.com.au",
"*.mlb.effi.com.au",
"coolvalue.mylocalbroker.com.au",
"Example"
]
}
}

Related

Accessing JQ Variables Using Arguments

I am trying to get values base on key of this JSON:
{
"streams": {
"vs-first": {
"version": "2.33.0",
"branch": "ewewew",
"hash": "ewewewewe",
"widgets": []
},
"vs-second": {
"version": "1.58.0",
"branch": "ewewew",
"hash": "ewewew",
"widgets": []
},
"vs-third": {
"version": "1.42.0",
"branch": "ewewew",
"hash": "ewewe",
"widgets": []
},
"vs-fourth": {
"version": "1.58.0",
"branch": "eewfwfef",
"hash": "vvfffsfsf",
"widgets": []
},
"vs-fifth": {
"version": "1.39.0",
"branch": "fvrvvsdvds",
"hash": "vvsvdsvds",
"widgets": [
"1",
"2",
"3",
"4"
]
}
}
}
This is my Script implementation:
jq -r '.streams|keys[]' $vsconfig | while read key ; do
if [ $key == "[" ] || [ $key == "]" ]; then
continue
fi
if [ $key == "vs-first" ]; then
version=$(jq -r '.streams.vs-first.version' $vsconfig)
branch=$(jq -r '.streams.vs-first.branch' $vsconfig)
hash=$(jq -r '.streams.vs-first.hash' $vsconfig)
filename="one_file-$version-$branch-$hash.zip"
createdUrl="$someurl/$version/$filename"
curl $createdUrl --output ./som/random/dir --create-dirs
...
else
version=$(jq -r --arg v keyvar $key 'streams.[$keyvar].branch' $vsconfig)
branch=`jq --arg keyvar "streams.$key.branch" '$keyvar' $vsconfig`
hash=`jq --arg keyvar "streams.$key.hash" '$keyvar' $vsconfig`
filename = "$key-$version"
if [ $branch == "some_branch" ]; then
filename="one_file-$version-$branch-$hash.zip"
else
filename="$filename.zip"
fi
curl $createdUrl --output ./som/random/dir --create-dirs
fi
echo "Version: $version Branch: $branch Hash: $hash"
done
I've tried multiple formats, i.e:
version=$(jq -r --arg v keyvar $key 'streams.[$keyvar].branch' $vsconfig)
And:
branch=`jq --arg keyvar "streams.$key.branch" '$keyvar' $vsconfig`
It gives this error:
jq: error: support/0 is not defined at <top-level>, line 1:
jq is quite sophisticated. The best bet is probably to do all of this inside a single jq invocation and have minimal or no bash scripting.
Values only
To start with, you can get each version/branch/hash object by applying [] to the .streams object. Using [] on an object—or key/value map—extracts the values and throws away the keys.
$ jq -c '.streams[]' vs.json
{"version":"2.33.0","branch":"ewewew","hash":"ewewewewe","widgets":[]}
{"version":"1.58.0","branch":"ewewew","hash":"ewewew","widgets":[]}
{"version":"1.42.0","branch":"ewewew","hash":"ewewe","widgets":[]}
{"version":"1.58.0","branch":"eewfwfef","hash":"vvfffsfsf","widgets":[]}
{"version":"1.39.0","branch":"fvrvvsdvds","hash":"vvsvdsvds","widgets":[]}
Then you can get the individual fields you're interested in by piping the above objects to a filter which grabs .version, .branch, and .hash and throws the three values into an array:
$ jq -c '.streams[] | [.version, .branch, .hash]' vs.json
["2.33.0","ewewew","ewewewewe"]
["1.58.0","ewewew","ewewew"]
["1.42.0","ewewew","ewewe"]
["1.58.0","eewfwfef","vvfffsfsf"]
["1.39.0","fvrvvsdvds","vvsvdsvds"]
To get it to format the results you can generate strings instead of lists and use \(...) to embed values. The -r flag tells it to print raw results: print the strings without quotes, in other words.
$ jq -r '.streams[] | "Version: \(.version) Branch: \(.branch) Hash: \(.hash)"' vs.json
Version: 2.33.0 Branch: ewewew Hash: ewewewewe
Version: 1.58.0 Branch: ewewew Hash: ewewew
Version: 1.42.0 Branch: ewewew Hash: ewewe
Version: 1.58.0 Branch: eewfwfef Hash: vvfffsfsf
Version: 1.39.0 Branch: fvrvvsdvds Hash: vvsvdsvds
Keys and values
To add the keys into the mix you can use to_entries, which extracts the key/value pairs from an object:
$ jq -c '.streams | to_entries[]' vs.json
{"key":"vs-first","value":{"version":"2.33.0","branch":"ewewew","hash":"ewewewewe","widgets":[]}}
{"key":"vs-second","value":{"version":"1.58.0","branch":"ewewew","hash":"ewewew","widgets":[]}}
{"key":"vs-third","value":{"version":"1.42.0","branch":"ewewew","hash":"ewewe","widgets":[]}}
{"key":"vs-fourth","value":{"version":"1.58.0","branch":"eewfwfef","hash":"vvfffsfsf","widgets":[]}}
{"key":"vs-fifth","value":{"version":"1.39.0","branch":"fvrvvsdvds","hash":"vvsvdsvds","widgets":[]}}
Pulling out the different fields then becomes:
$ jq -c '.streams | to_entries[] | [.key, .value.version, .value.branch, .value.hash]' vs.json
["vs-first","2.33.0","ewewew","ewewewewe"]
["vs-second","1.58.0","ewewew","ewewew"]
["vs-third","1.42.0","ewewew","ewewe"]
["vs-fourth","1.58.0","eewfwfef","vvfffsfsf"]
["vs-fifth","1.39.0","fvrvvsdvds","vvsvdsvds"]
Or equivalently, with the repeated .value lookups refactored out:
jq -c '.streams | to_entries[] | [.key, (.value | .version, .branch, .hash)]' vs.json
["vs-first","2.33.0","ewewew","ewewewewe"]
["vs-second","1.58.0","ewewew","ewewew"]
["vs-third","1.42.0","ewewew","ewewe"]
["vs-fourth","1.58.0","eewfwfef","vvfffsfsf"]
["vs-fifth","1.39.0","fvrvvsdvds","vvsvdsvds"]
Add bash processing
jq can't do everything, so if you do want to get the results out to bash to do additional processing—e.g., call curl—you could use -r to print each value on a separate line and use read to read the lines into variables. It would look something like this:
jq -r '.streams | to_entries[] | .key, (.value | .version, .branch, .hash)' vs.json |
while read -r key &&
read -r version &&
read -r branch &&
read -r hash
do
...
done

Terraform - How to convert lists into map (How to fetch AMI tags using terraform)

I am trying to fetch the tags of AMI using AWS CLI and want to reuse the values from the output.
I have a terraform code below which is returning outputs in string format(Maybe not sure of format) which I want to convert into a map object.
variable "ami" {
default = "ami-xxxx"
}
locals {
tags = {
"platform" = lookup(data.local_file.read_tags.content, "platform", "") #Expecting to get platform from Map of read_tags
}
}
data "template_file" "log_name" {
template = "${path.module}/output.log"
}
resource "null_resource" "ami_tags" {
provisioner "local-exec" {
command = "aws ec2 describe-tags --filters Name=resource-id,Values=${var.ami} --query Tags[*].[Key,Value] > ${data.template_file.log_name.rendered}"
}
}
data "local_file" "read_tags" {
filename = "${data.template_file.log_name.rendered}"
depends_on = ["null_resource.ami_tags"]
}
output "tags" {
value = local.tags
}
output "cli-output-tags" {
value = "${concat(data.local_file.read_tags.content)}"
}
output of cli-output-tags is below:
[
[
"ENV",
"DEV"
],
[
"Name",
"Base-AMI"
],
[
"platform",
"Linux"
]
]
How can I convert this output into Map as below using terraform/(jq command), or is there any other way to fetch required values directly from cli-output-tags output:
{
ENV = "DEV",
Name = "Base-AMI",
platform = "Linux"
}
I have also tried changing the CLI command a bit like below but still not able to fetch values as expected:
'Tags[].{Key:Key,Value:Value}'
Resulted below output:
[
{
"Key": "ENV",
"Value": "DEV"
},
{
"Key": "Name",
"Value": "Base-AMI"
},
{
"Key": "platform",
"Value": "Linux"
}
]
You could use zipmap:
output "cli-output-tags" {
value = zipmap(
jsondecode(data.local_file.read_tags.content)[*][0],
jsondecode(data.local_file.read_tags.content)[*][1]
)
}
The code first changes string data from your file to json, then
gets all first elements [*][0] (same for second elements [*][1]), and zips them into map.
How can I convert this output into Map as below
One way would be to use jq as follows (assuming cli-output-tags is the name of the file holding the JSON array of arrays):
jq -r -f '"{", (.[] | "\(.[0]) = \"\(.[1])\""), "}"' cli-output-tags

Is it possible with jq to use a deleted value in setting new ones?

I am using the bash to json parser jq
Considering the following command:
jq '. * .transitive | del(.transitive) | del(.scope,.scopedName)' package.json > package.github.json$$
And the following input:
{
"name": "navigation",
"transitive": {
"name": "navigation",
"scope": "bs",
"scopedName": "#bs/navigation"
}
}
I am trying to get the following output:
{
"name": "#bs/navigation"
}
Is there a way before doing the delete of .scopedName, to use it's value to set .name?
Transforming your input to your output is as simple as:
jq '{"name": .transitive.scopedName}'
...and of course you could just reorder things to set name before deleting transitive:
jq '.name=.transitive.scopedName | del(.transitive)'
That said, if you really want to use del() first, you can save content in a variable and use it later:
jq '
.transitive as $transitive |
del(.transitive) |
.name=$transitive.scopedName
'

sort output of describe-instances?

I saw the previous question on this topic, but the answer was just "pipe it to a scripting language!", which I find unsatisfying. I know that JMESPath has sort_by, and sort, but I can't figure out how to use them.
I have
aws ec2 describe-instances \
--filters "Name=tag:Group,Values=production" "Name=instance-state-name,Values=running" "Name=tag:Name,Values=prod-*-${CURRENT_SHA}-*" \
--query 'Reservations[*].Instances[*].[LaunchTime,InstanceId,PrivateIpAddress,Tags[?Key==`Name`] | [0].Value]' \
--output table
And it outputs the right data, just in a random order. I want to sort by the last column of the data, Tag Name, aka Tags[?Key==`Name`], which in raw form looks like this:
{
"Tags": [{
"Value": "application-server-ab3634b34364a-2",
"Key": "Name"
}, {
"Value": "production",
"Key": "Group"
}]
}
Thoughts?
short answer
add
[] | sort_by(#, &[3])
at the end of your expression. The brackets ([]) will flatten the structure, sort_by(...) will sort the result (which is a four-column table) by the fourth column. The full query will be:
--query 'Reservations[*].Instances[*].[LaunchTime,InstanceId,PrivateIpAddress,Tags[?Key==`Name`] | [0].Value][] | sort_by(#, &[3])'
long answer
inspecting your current query result
According to the describe-instances docs, the structure of the describe-instances output looks like this:
{
"Reservations": [
{
"Instances": [
{
"LaunchTime": "..LaunchTime..",
"InstanceId": "R1I1",
"PrivateIpAddress": "..PrivateIpAddress..",
"Tags": [{"Key": "Name", "Value": "foo"}]
},
{
"LaunchTime": "..LaunchTime..",
"InstanceId": "R1I2",
"PrivateIpAddress": "..PrivateIpAddress..",
"Tags": [{"Key": "Name", "Value": "baz"}]
}
]
},
{
"Instances": [
{
"LaunchTime": "..LaunchTime..",
"InstanceId": "R2I1",
"PrivateIpAddress": "..PrivateIpAddress..",
"Tags": [{"Key": "Name", "Value": "bar"}]
}
]
}
]
}
Using your original query
--query 'Reservations[*].Instances[*].[LaunchTime,InstanceId,PrivateIpAddress,Tags[?Key==`Name`] | [0].Value]'
will output
[
[
[
"..LaunchTime..",
"R1I1",
"..PrivateIpAddress..",
"foo"
],
[
"..LaunchTime..",
"R1I2",
"..PrivateIpAddress..",
"baz"
]
],
[
[
"..LaunchTime..",
"R2I1",
"..PrivateIpAddress..",
"bar"
]
]
]
flattening the query result
You can see in the above result of your query that you're getting a list of tables ([[{},{}],[{}]]). I suppose you instead want a single non-nested table ([{},{},{}]). To achieve that, simply add [] at the end of your query, i.e.
--query 'Reservations[*].Instances[*].[LaunchTime,InstanceId,PrivateIpAddress,Tags[?Key==`Name`] | [0].Value][]'
This will flatten the structure, resulting in
[
[
"..LaunchTime..",
"R1I1",
"..PrivateIpAddress..",
"foo"
],
[
"..LaunchTime..",
"R1I2",
"..PrivateIpAddress..",
"baz"
],
[
"..LaunchTime..",
"R2I1",
"..PrivateIpAddress..",
"bar"
]
]
Now it's time to sort the table.
sorting the table
When using sort_by you shouldn't forget to prepend the expression by & (ampersand). This way you specify a reference to that expression, which is then passed to sort_by.
example: data | sort_by(#, &#) is equivalent to data | sort(#).
The TagName in the table you create ([LaunchTime,InstanceId,PrivateIpAddress,TagName]) is the fourth column. You can get that column by piping the table to the expression [3]:
TableExpression | [3]
But instead, you want to sort the table by the fourth column. You can do so like this:
TableExpression | sort_by(#, &[3])
and the resulting query will be:
--query 'Reservations[*].Instances[*].[LaunchTime,InstanceId,PrivateIpAddress,Tags[?Key==`Name`][] | [0].Value] | sort_by(#, &[3])'
Query result:
[
[
"..LaunchTime..",
"R2I1",
"..PrivateIpAddress..",
"bar"
],
[
"..LaunchTime..",
"R1I2",
"..PrivateIpAddress..",
"baz"
],
[
"..LaunchTime..",
"R1I1",
"..PrivateIpAddress..",
"foo"
]
]
As an enhancement to #ColinK's answer, I wanted to sort a table that had custom column headers but struggled with the syntax. I eventually got it to work so I thought I'd share in case someone else wanted to do the same. I added a column for State and sorted by that column.
--query 'sort_by(Reservations[*].Instances[*].{LaunchTime:LaunchTime, ID:InstanceId,IP:PrivateIpAddress,State:State.Name,Name:Tags[?Key==`Name`] | [0].Value}[], &State)'
Here is an other example that works also:
aws ec2 describe-instances --query 'Reservations[*].Instances[*].{Name:Tags[?Key==`Name`]|[0].Value,Instance:InstanceId} | sort_by(#, &[0].Name)'
The answer is to add | sort_by(#, &#[0][3])
aws ec2 describe-instances \
--filters "Name=tag:Group,Values=production" "Name=instance-state-name,Values=running" "Name=tag:Name,Values=prod-*-${CURRENT_SHA}-*" \
--query 'Reservations[*].Instances[*].[LaunchTime,InstanceId,PrivateIpAddress,Tags[?Key==`Name`] | [0].Value]| sort_by(#, &#[0][3])' \
--output table

Iterate over a json using shell

I have a json in the following format. I want to iterate over this json file
{
"atest_engine": { "version": "96" },
"a_kdfvm": { "version": "68" },
"aseft_api": { "version": "" },
"push_psservice": { "version": "68" },
}
I tried jq utility and my script is as follows.
count=$( jq '. | length' test.json )
echo $count
for((i=0;i<$count;i++))
do
name=$(cat test.json | jq '.|keys['${i}']')
version=$(cat test.json | jq '.|keys['${i}'].version')
echo $name
echo $version
done
I am getting count and name properly but not able to fetch version information. How can I get it. I am new to scripting and any help in this regard is greatly appreciated.Thanks in Advance.
input.json
{
"atest_engine": { "version": "96" },
"a_kdfvm": { "version": "68" },
"aseft_api": { "version": "" },
"push_psservice": { "version": "68" }
}
command
jq -r 'to_entries[] | "\(.key)\t\(.value.version)"' input.json |
while read name version
do
echo "name:" $name
echo "version:" $version
done
result
name: atest_engine
version: 96
name: a_kdfvm
version: 68
name: aseft_api
version:
name: push_psservice
version: 68
First up your JSON example seems slightly malformed - the push_psservice line has a comma after it but this is most likely a typo.
You might find it easier to turn your object's fields into an array using jq's to_entries (see https://stackoverflow.com/a/24254365/4513656 ) e.g.:
to_entries | .[0].key
to_entries | .[0].value.version
Try this on https://jqplay.org/ .

Resources