I am trying to format json output and exclude an element when a condition is met.
1) In this case I'd like to exclude any element that contains "valueFrom" using jq
[{
"name": "var1",
"value": "var1value"
},
{
"name": "var2",
"value": "var2value"
},
{
"name": "var3",
"value": "var3value"
},
{
"name": "var4",
"value": "var4value"
},
{ # <<< exclude this element as valueFrom exists
"name": "var5",
"valueFrom": {
"secretKeyRef": {
"key": "var5",
"name": "var5value"
}
}
}
]
After excluding the element mentioned above I am trying to return a result set that looks like this.
var1: var1value
var2: var2value
var3: var3value
var4: var4value
Any feedback is appreciated. Thanks.
Select array items that doesn't have the valueFrom key using a combination of select/1, has/1, and not/0. Then format the objects as you please.
$ jq -r '.[] | select(has("valueFrom") | not) | "\(.name): \(.value)"' input.json
Related
Json file:
{
"A": "jack",
"B": [
{
"name": "jack/jil",
"version": "0.1"
},
{
"name": "went/well",
"version": "1.2"
}
]
}
now I need to update every objects version: "$version-dev" where name starts from jack while retaining rest of the json intact.
the closest I can get.
jq '.B[] | select(.name|test("jack.")) | .version += "-dev"' old.json > new.json
in the above command I'm only getting the that particular object with updated value in the new.json but I need the whole json too.
any suggestions
You need to put parantheses around the whole selection to be updated: (… | .version) += …
jq '(.B[] | select(.name|test("jack.")) | .version) += "-dev"' old.json > new.json
{
"A": "jack",
"B": [
{
"name": "jack/jil",
"version": "0.1-dev"
},
{
"name": "went/well",
"version": "1.2"
}
]
}
Demo
I have the below response payload and I just want to check the amount == 1000 if it's matching then I just want to get the entire column as output.
Sample Input:
{
"sqlQuery": "select SET_UNIQUE, amt as AMOUNT from transactionTable where SET_USER_ID=11651 ",
"message": "2 rows selected",
"row": [
{
"column": [
{
"value": "22621264",
"name": "SET_UNIQUE"
},
{
"value": "1000",
"name": "AMOUNT"
}
]
},
{
"column": [
{
"value": "226064213",
"name": "SET_UNIQUE"
},
{
"value": "916",
"name": "AMOUNT"
}
]
}
]
}
Expected Output:
"column": [
{
"value": "22621264",
"name": "SET_UNIQUE"
},
{
"value": "1000",
"name": "AMOUNT"
}
]
The above sample I just want to fetch the entire column if the AMOUNT value will be 1000.
I just tried below to achieve this but no luck.
1. row[*].column[?(#.value==1000)].column
2. row[*].column[?(#.value==1000)]
I don't want to do this by using index. Because It will be change.
Any ideas please?
I think you'd need nested expressions, which isn't something that's widely supported. Something like
$.row[?(#.column[?(#.value==1000)])]
The inner expression returns matches for value==1000, then the outer expression checks for existence of those matches.
Another alternative that might work is
$.row[?(#.column[*].value==1000)]
but this assumes some implicit type conversions that may or may not be supported.
This question already has answers here:
Extract a specific field from JSON output using jq
(3 answers)
Closed 4 years ago.
I'm new to shell scripting and started baby steps in it.
I've written shell script recently to call rest API and I was able to execute it without any issues.
I've stored the output in a variable like below
{
"id": 3184136,
"name": "XXX TEST API",
"url": "http://xxxxxxxxxxx/_apis/test/Runs/3184136",
"isAutomated": true,
"owner": {
"displayName": "XXXX",
"url": "http://xxxxxxxxxxx/_apis/Identities/dbf722a9-73b0-46d6-a2bd-9835c1f0c221",
"_links": {
"avatar": {
"href": "http://xxxxxxxxxxx/_api/_common/identityImage?id=dbf722a9-73b0-46d6-a2bd-9835c1f0c221"
}
},
"id": "dbf722a9-73b0-46d6-a2bd-9835c1f0c221",
"uniqueName": "xxxxxxxxxxx\\ServiceLaunchpadDev",
"imageUrl": "http://xxxxxxxxxxx/_api/_common/identityImage?id=dbf722a9-73b0-46d6-a2bd-9835c1f0c221"
},
"project": {
"id": "6d5e21e7-c75e-464a-9708-90fbff086902",
"name": "eDellPrograms"
},
"startedDate": "2018-10-11T06:36:50.627Z",
"completedDate": "2018-10-11T07:04:45.153Z",
"state": "Completed",
"plan": {
"id": "5299555",
"name": "Smoke Dashboard Peso - DIT",
"url": "http://xxxxxxxxxxx/_apis/test/Plans/5299555"
},
"postProcessState": "Complete",
"totalTests": 5,
"incompleteTests": 0,
"notApplicableTests": 0,
"passedTests": 0,
"unanalyzedTests": 5,
"createdDate": "2018-10-11T06:36:50.533Z",
"lastUpdatedDate": "2018-10-11T07:04:45.153Z",
"lastUpdatedBy": {
"displayName": "xxxxxxxxxxx",
"url": "http://xxxxxxxxxxx/_apis/Identities/8de2a654-063b-48bd-8101-87e4ec2f05e3",
"_links": {
"avatar": {
"href": "http://xxxxxxxxxxx/_api/_common/identityImage?id=8de2a654-063b-48bd-8101-87e4ec2f05e3"
}
},
"id": "8de2a654-063b-48bd-8101-87e4ec2f05e3",
"uniqueName": "xxxxxxxxxxx\\xxxxxxxxxxx",
"imageUrl": "http://xxxxxxxxxxx/_api/_common/identityImage?id=8de2a654-063b-48bd-8101-87e4ec2f05e3"
},
"controller": "xxxxxxxxxxx",
"revision": 5,
"comment": "Build Definition : xxxxxxxxxxx \nBuild Version : xxxxxxxxxxx_20180925.1\nConfiguration : DIT\nBatch type : Suite\nTest type : Parallel\nTest Controller Name : xxxxxxxxxxx\nPreferred Agents : ADPTAW10A618|ADPTAW10A619|ADPTAW10A621 \nRequested by : xxxxxxxxxxx\nEmail Request : Y\nEmail To : xxxxxxxxxxx\nEmailCc : xxxxxxxxxxx\nEnvironment : DIT\nTest Setting : DIT\nContinue On Failure : false\nDNS Setting : false",
"dropLocation": "\\\\xxxxxxxxxxx\\DropFolder\\xxxxxxxxxxx_20180925.1",
"runStatistics": [
{
"state": "Completed",
"outcome": "Failed",
"count": 5
}
],
"webAccessUrl": "http://xxxxxxxxxxx/_TestManagement/Runs#runId=3184136&_a=runCharts"
}
from the above output, I'm trying to find the "state" and its value. But I couldn't make that happen. kindly anyone helps me.
echo $result | grep -o 'state*'
with above command, I was able to print state. but I'm expecting both state and its value.
Appreciate your help. Thanks in Advance.
I tried storing your json in a file called n2.json.
cat n2.json
{
"id":3184136,
"name":"XXX TEST API",
"url":"http://xxxxxxxxxxx/_apis/test/Runs/3184136",
"isAutomated":true,
"owner":{
"displayName":"XXXX",
"url":"http://xxxxxxxxxxx/_apis/Identities/dbf722a9-73b0-46d6-a2bd-9835c1f0c221",
"_links":{
"avatar":{
"href":"http://xxxxxxxxxxx/_api/_common/identityImage?id=dbf722a9-73b0-46d6-a2bd-9835c1f0c221"
}
},
"id":"dbf722a9-73b0-46d6-a2bd-9835c1f0c221",
"uniqueName":"xxxxxxxxxxx\\ServiceLaunchpadDev",
"imageUrl":"http://xxxxxxxxxxx/_api/_common/identityImage?id=dbf722a9-73b0-46d6-a2bd-9835c1f0c221"
},
"project":{
"id":"6d5e21e7-c75e-464a-9708-90fbff086902",
"name":"eDellPrograms"
},
"startedDate":"2018-10-11T06:36:50.627Z",
"completedDate":"2018-10-11T07:04:45.153Z",
"state":"Completed",
"plan":{
"id":"5299555",
"name":"Smoke Dashboard Peso - DIT",
"url":"http://xxxxxxxxxxx/_apis/test/Plans/5299555"
},
"postProcessState":"Complete",
"totalTests":5,
"incompleteTests":0,
"notApplicableTests":0,
"passedTests":0,
"unanalyzedTests":5,
"createdDate":"2018-10-11T06:36:50.533Z",
"lastUpdatedDate":"2018-10-11T07:04:45.153Z",
"lastUpdatedBy":{
"displayName":"xxxxxxxxxxx",
"url":"http://xxxxxxxxxxx/_apis/Identities/8de2a654-063b-48bd-8101-87e4ec2f05e3",
"_links":{
"avatar":{
"href":"http://xxxxxxxxxxx/_api/_common/identityImage?id=8de2a654-063b-48bd-8101-87e4ec2f05e3"
}
},
"id":"8de2a654-063b-48bd-8101-87e4ec2f05e3",
"uniqueName":"xxxxxxxxxxx\\xxxxxxxxxxx",
"imageUrl":"http://xxxxxxxxxxx/_api/_common/identityImage?id=8de2a654-063b-48bd-8101-87e4ec2f05e3"
},
"controller":"xxxxxxxxxxx",
"revision":5,
"comment":"Build Definition : xxxxxxxxxxx \nBuild Version : xxxxxxxxxxx_20180925.1\nConfiguration : DIT\nBatch type : Suite\nTest type : Parallel\nTest Controller Name : xxxxxxxxxxx\nPreferred Agents : ADPTAW10A618|ADPTAW10A619|ADPTAW10A621 \nRequested by : xxxxxxxxxxx\nEmail Request : Y\nEmail To : xxxxxxxxxxx\nEmailCc : xxxxxxxxxxx\nEnvironment : DIT\nTest Setting : DIT\nContinue On Failure : false\nDNS Setting : false",
"dropLocation":"\\\\xxxxxxxxxxx\\DropFolder\\xxxxxxxxxxx_20180925.1",
"runStatistics":[
{
"state":"Completed",
"outcome":"Failed",
"count":5
}
],
"webAccessUrl":"http://xxxxxxxxxxx/_TestManagement/Runs#runId=3184136&_a=runCharts"
}
Then use jq on top of this:
jq -r '.state' n2.json
Completed
You are looking for 'state', 'statee', 'stateee', 'stateeee', etc.
The wildcard applies to the preceeding character.
Try this:
echo $result | grep -o '"state":[^,]*'
It looks for everything up to, but excluding, the next comma.
I have a JSON file with 12,166,466 of lines.
I want to remove quotes from values on keys:
"timestamp": "1538564256",and "score": "10", to look like
"timestamp": 1538564256, and "score": 10,.
Input:
{
"title": "DNS domain", ,
"timestamp": "1538564256",
"domain": {
"dns": [
"www.google.com"
]
},
"score": "10",
"link": "www.bit.ky/sdasd/asddsa"
"id": "c-1eOWYB9XD0VZRJuWL6"
}, {
"title": "DNS domain",
"timestamp": "1538564256",
"domain": {
"dns": [
"google.de"
]
},
"score": "10",
"link": "www.bit.ky/sdasd/asddsa",
"id": "du1eOWYB9XD0VZRJuWL6"
}
}
Expected output:
{
"title": "DNS domain", ,
"timestamp": 1538564256,
"domain": {
"dns": [
"www.google.com"
]
},
"score": 10,
"link": "www.bit.ky/sdasd/asddsa"
"id": "c-1eOWYB9XD0VZRJuWL6"
}, {
"title": "DNS domain",
"timestamp": 1538564256,
"domain": {
"dns": [
"google.de"
]
},
**"score": 10,**
"link": "www.bit.ky/sdasd/asddsa",
"id": "du1eOWYB9XD0VZRJuWL6"
}
}
I have tried:
sed -E '
s/"timestamp": "/"timestamp": /g
s/"score": "/"score": /g
'
the first part is quite straightforward, but how to remove ", at that the end of the line that contains "timestamp" and "score"? How do I access that using sed or even awk, or other tool with the mind that I have 12 million lines to process?
Assuming that you fix your JSON input file like this:
<file jq .
[
{
"title": "DNS domain",
"timestamp": "1538564256",
"domain": {
"dns": [
"www.google.com"
]
},
"score": "10",
"link": "www.bit.ky/sdasd/asddsa",
"id": "c-1eOWYB9XD0VZRJuWL6"
},
{
"title": "DNS domain",
"timestamp": "1538564256",
"domain": {
"dns": [
"google.de"
]
},
"score": "10",
"link": "www.bit.ky/sdasd/asddsa",
"id": "du1eOWYB9XD0VZRJuWL6"
}
]
You can use jq and its tonumber function to change the wanted strings to values:
<file jq '.[].timestamp |= tonumber | .[].score |= tonumber'
If the JSON structure matches roughly your example (e. g., there won't be any other whitespace characters between "timestamp", the colon, and the value), then this awk should be ok. If available, using jq for JSON transformation is the better choice by far!
awk '{print gensub(/("(timestamp|score)": )"([0-9]+)"/, "\\1\\3", "g")}' file
Be warned that tonumber can lose precision. If using tonumber is inadmissible, and if the output is produced by jq (or otherwise linearized vertically), then using awk as proposed elsewhere on this page is a good way to go. (If your awk does not have gensub, then the awk program can be easily adapted.) Here is the same thing using sed, assuming its flag for extended regex processing is -E:
sed -E -e 's/"(timestamp|score)": "([0-9]+)"/"\1": \2/'
For reference, if there's any doubt about where the relevant keys are located, here's a filter in jq that is agnostic about that:
walk(if type == "object"
then if has("timestamp") then .timestamp|=tonumber else . end
| if has("score") then .score|=tonumber else end
else . end)
If your jq does not have walk/1, then simply snarf its def from the web, e.g. from https://raw.githubusercontent.com/stedolan/jq/master/src/builtin.jq
If you wanted to convert all number-valued strings to numbers, you could write:
walk(if type=="object" then map_values(tonumber? // .) else . end)
This might work for you (GNU sed):
sed ':a;/"timestamp":\s*"1538564256",/{s/"//3g;:b;n;/timestamp/ba;/"score":\s*"10"/s/"//3g;Tb}' file
On encountering a line that contains "timestamp": "1538564256", remove the 3rd or more "'s. Then read on until another line containing timestamp and repeat or a line containing "score": "10 and remove the 3rd or more "'s.
I have a terms stats query very similar to this one:
Sum Query in Elasticsearch
However, my key_field is a date.
I was expecting to receive results grouped by the full key_field value ["2014-01-20", "2014-01-21", "2014-01-22"] but it appears to be splitting the key field when it encounters a "-". What I received is actually grouped by ["2014", "01", "20", "21", "22"].
Why is it splitting my key?
You probably have your key_field mapped with a string-type using the standard-analyzer.
That'll tokenize 2014-01-20 into 2014, 01, and 20.
You probably want to index your date as having type date. You can also have it as a string without analyzing it.
Here's a runnable example you can play with: https://www.found.no/play/gist/5eb6b8d176e1cc72c9b8
#!/bin/bash
export ELASTICSEARCH_ENDPOINT="http://localhost:9200"
# Create indexes
curl -XPUT "$ELASTICSEARCH_ENDPOINT/play" -d '{
"settings": {},
"mappings": {
"type": {
"properties": {
"date_as_a_string": {
"type": "string"
},
"date_as_nonanalyzed_string": {
"type": "string",
"index": "not_analyzed"
}
}
}
}
}'
# Index documents
curl -XPOST "$ELASTICSEARCH_ENDPOINT/_bulk?refresh=true" -d '
{"index":{"_index":"play","_type":"type"}}
{"date":"2014-01-01T00:00:00.000Z","date_as_a_string":"2014-01-01T00:00:00.000Z","date_as_nonanalyzed_string":"2014-01-01T00:00:00.000Z","x":42}
'
# Do searches
curl -XPOST "$ELASTICSEARCH_ENDPOINT/_search?pretty" -d '
{
"facets": {
"date": {
"terms_stats": {
"key_field": "date",
"value_field": "x"
}
},
"date_as_a_string": {
"terms_stats": {
"key_field": "date_as_a_string",
"value_field": "x"
}
},
"date_as_nonanalyzed_string": {
"terms_stats": {
"key_field": "date_as_nonanalyzed_string",
"value_field": "x"
}
}
},
"size": 0
}
'