Replace pom version using sed in Jenkinsfile - linux

I have a variable myStrthat contains the following value:
"app": {
"services": {
"app": [{
"groupID": "com.mycompany",
"artifactId": "myapp-versions",
"version": "1.0.0"
},
{
"groupID": "com.mycompany.xyz",
"artifactId": "car-stats",
"version": "1.0-master"
},
{
"groupID": "com.mycompany.service",
"artifactId": "my-differential-service",
"version": "1.0.0-master"
}
]
}
}
Now I want to replace the version of only my-differential-service artifactId to NEW_VERSION.
I tried using sed command on myStr variable but couldn't succeed as I am not much familiar with this command.
Can anyone please guide me on how should I proceed to achieve this?
Any help would be highly appreciated.

When you need sed for parsing this, look for the support of the option -z in your sed. This option ignores the special meaning of \n.
When you know that version is the first field after the artifactId you can try
new_version=1.0.1 # avoid uppercase variable names
echo "$myStr" |
sed -rz 's/("my-differential-service",[^:]*: ")[^"]*/\1'"${new_version}"'/'
When the order of fields can change, you might want to ask for jq in Jenkins or try awk.
When you want to use sed but don't have the -z option, you can translate first:
echo "$myStr" | tr '\n' '\r' | sed -r 's/..../' | tr '\r' '\n'

If jq is available, congrats! and please try the following:
echo "{ $myStr }" | jq '(.app.services.app[] | select(.artifactId == "my-differential-service") | .version) = "NEW_VERSION"'
which yields:
{
"app": {
"services": {
"app": [
{
"groupID": "com.mycompany",
"artifactId": "myapp-versions",
"version": "1.0.0"
},
{
"groupID": "com.mycompany.xyz",
"artifactId": "car-stats",
"version": "1.0-master"
},
{
"groupID": "com.mycompany.service",
"artifactId": "my-differential-service",
"version": "NEW_VERSION"
}
]
}
}
}
If you do not need the outermost curly braces, please remove them by bash's parameter expansion or something similar.
As a fallback, you can say with sed:
echo "$myStr" | sed '
:l
N
$!b l
s/\("my-differential-service"[^"]*"version": *\)"[^"]*"/\1"NEW_VERSION"/g'
Hope this helps.

Related

using grep commands to find a duplicate id within a json file

I am looking for a way to use grep on a linux server to find duplicate json records, is it possible to have a grep to search for duplicate id's in the example below ?
so the grep would return: 01
{
"book": [
{
"id": "01",
"language": "Java",
"edition": "third",
"author": "Herbert Schildt"
},
{
"id": "02",
"language": "Java",
"edition": "third",
"author": "Herbert Schildt"
},
{
"id": "03",
"language": "Java",
"edition": "third",
"author": "Herbert Schildt"
},
{
"id": "01",
"language": "Java",
"edition": "third",
"author": "Herbert Schildt"
},
{
"id": "04",
"language": "C++",
"edition": "second",
"author": "E.Balagurusamy"
}
]
}
use grep along with uniq.
grep '"id":' filename | sort | uniq -d
The -d option only prints duplicates.
However, this depends on the JSON being laid out neatly. To handle more general formatting, I recommend you use the jq utility.
A jq-based approach:
jq -r '.book[].id' < in.json | sort | uniq -d
01
This should work even for minified JSON files with no newlines.
OK, discarding any whitespace from the JSON strings I can offer this if awk is acceptable - hutch being the formatted chunk of JSON above in a file.
I use tr to remove any whitespace, use , as a field separator in awk; iterate over the one long lines elements with a for-loop, do some pattern-matching in awk to isolate ID fields and increment an array for each matched ID. At the end of processing I iterate over the array and print ID's that have more than one match.
Here your data:
$ cat hutch
{
"book": [
{
"id": "01",
"language": "Java",
"edition": "third",
"author": "Herbert Schildt"
},
{
"id": "02",
"language": "Java",
"edition": "third",
"author": "Herbert Schildt"
},
{
"id": "03",
"language": "Java",
"edition": "third",
"author": "Herbert Schildt"
},
{
"id": "01",
"language": "Java",
"edition": "third",
"author": "Herbert Schildt"
},
{
"id": "04",
"language": "C++",
"edition": "second",
"author": "E.Balagurusamy"
}
]
}
And here the finding of dupes:
$ tr -d '[:space:]' <hutch | awk -F, '{for(i=1;i<=NF;i++){if($i~/"id":/){a[gensub(/^.*"id":"([0-9]+)"$/, "\\1","1",$i)]++}}}END{for(i in a){if(a[i]>1){print i}}}'
01
Use a Perl one-liner to extract the numeric ids, then sort | uniq -d to print only the duplicates (as in the answer by Barmar):
This assumes that the id key/value pair is on the same line, but disregards whitespace (or lack of whitespace) anywhere on the line (leading, trailing, and in between):
perl -lne 'print for /"id":\s*"(\d+)"/' in.json | sort | uniq -d
This makes no assumptions (disregards whitespace and newlines). Note that it reads the entire json file into memory (using the -0777 command line switch):
perl -0777 -nE 'say for /"id":\s*"(\d+)"/g' in.json | sort | uniq -d
The Perl one-liner uses these command line flags:
-e : Tells Perl to look for code in-line, instead of in a file.
-E : Tells Perl to look for code in-line, instead of in a file. Also enables all optional features. Here, enables say.
-n : Loop over the input one line at a time, assigning it to $_ by default.
-l : Strip the input line separator ("\n" on *NIX by default) before executing the code in-line, and append it when printing.
-0777 : Slurp files whole.
The regex uses this modifier:
/g : Multiple matches.
SEE ALSO:
perldoc perlrun: how to execute the Perl interpreter: command line switches
perldoc perlre: Perl regular expressions (regexes)
perldoc perlre: Perl regular expressions (regexes): Quantifiers; Character Classes and other Special Escapes; Assertions; Capture groups
perldoc perlrequick: Perl regular expressions quick start

jq parsing and linux formatting to desired output

I am trying to format json output and exclude an element when a condition is met.
1) In this case I'd like to exclude any element that contains "valueFrom" using jq
[{
"name": "var1",
"value": "var1value"
},
{
"name": "var2",
"value": "var2value"
},
{
"name": "var3",
"value": "var3value"
},
{
"name": "var4",
"value": "var4value"
},
{ # <<< exclude this element as valueFrom exists
"name": "var5",
"valueFrom": {
"secretKeyRef": {
"key": "var5",
"name": "var5value"
}
}
}
]
After excluding the element mentioned above I am trying to return a result set that looks like this.
var1: var1value
var2: var2value
var3: var3value
var4: var4value
Any feedback is appreciated. Thanks.
Select array items that doesn't have the valueFrom key using a combination of select/1, has/1, and not/0. Then format the objects as you please.
$ jq -r '.[] | select(has("valueFrom") | not) | "\(.name): \(.value)"' input.json

Removing pattern from multiple lines using sed or awk in two places in the same line

I have a JSON file with 12,166,466 of lines.
I want to remove quotes from values on keys:
"timestamp": "1538564256",and "score": "10", to look like
"timestamp": 1538564256, and "score": 10,.
Input:
{
"title": "DNS domain", ,
"timestamp": "1538564256",
"domain": {
"dns": [
"www.google.com"
]
},
"score": "10",
"link": "www.bit.ky/sdasd/asddsa"
"id": "c-1eOWYB9XD0VZRJuWL6"
}, {
"title": "DNS domain",
"timestamp": "1538564256",
"domain": {
"dns": [
"google.de"
]
},
"score": "10",
"link": "www.bit.ky/sdasd/asddsa",
"id": "du1eOWYB9XD0VZRJuWL6"
}
}
Expected output:
{
"title": "DNS domain", ,
"timestamp": 1538564256,
"domain": {
"dns": [
"www.google.com"
]
},
"score": 10,
"link": "www.bit.ky/sdasd/asddsa"
"id": "c-1eOWYB9XD0VZRJuWL6"
}, {
"title": "DNS domain",
"timestamp": 1538564256,
"domain": {
"dns": [
"google.de"
]
},
**"score": 10,**
"link": "www.bit.ky/sdasd/asddsa",
"id": "du1eOWYB9XD0VZRJuWL6"
}
}
I have tried:
sed -E '
s/"timestamp": "/"timestamp": /g
s/"score": "/"score": /g
'
the first part is quite straightforward, but how to remove ", at that the end of the line that contains "timestamp" and "score"? How do I access that using sed or even awk, or other tool with the mind that I have 12 million lines to process?
Assuming that you fix your JSON input file like this:
<file jq .
[
{
"title": "DNS domain",
"timestamp": "1538564256",
"domain": {
"dns": [
"www.google.com"
]
},
"score": "10",
"link": "www.bit.ky/sdasd/asddsa",
"id": "c-1eOWYB9XD0VZRJuWL6"
},
{
"title": "DNS domain",
"timestamp": "1538564256",
"domain": {
"dns": [
"google.de"
]
},
"score": "10",
"link": "www.bit.ky/sdasd/asddsa",
"id": "du1eOWYB9XD0VZRJuWL6"
}
]
You can use jq and its tonumber function to change the wanted strings to values:
<file jq '.[].timestamp |= tonumber | .[].score |= tonumber'
If the JSON structure matches roughly your example (e. g., there won't be any other whitespace characters between "timestamp", the colon, and the value), then this awk should be ok. If available, using jq for JSON transformation is the better choice by far!
awk '{print gensub(/("(timestamp|score)": )"([0-9]+)"/, "\\1\\3", "g")}' file
Be warned that tonumber can lose precision. If using tonumber is inadmissible, and if the output is produced by jq (or otherwise linearized vertically), then using awk as proposed elsewhere on this page is a good way to go. (If your awk does not have gensub, then the awk program can be easily adapted.) Here is the same thing using sed, assuming its flag for extended regex processing is -E:
sed -E -e 's/"(timestamp|score)": "([0-9]+)"/"\1": \2/'
For reference, if there's any doubt about where the relevant keys are located, here's a filter in jq that is agnostic about that:
walk(if type == "object"
then if has("timestamp") then .timestamp|=tonumber else . end
| if has("score") then .score|=tonumber else end
else . end)
If your jq does not have walk/1, then simply snarf its def from the web, e.g. from https://raw.githubusercontent.com/stedolan/jq/master/src/builtin.jq
If you wanted to convert all number-valued strings to numbers, you could write:
walk(if type=="object" then map_values(tonumber? // .) else . end)
This might work for you (GNU sed):
sed ':a;/"timestamp":\s*"1538564256",/{s/"//3g;:b;n;/timestamp/ba;/"score":\s*"10"/s/"//3g;Tb}' file
On encountering a line that contains "timestamp": "1538564256", remove the 3rd or more "'s. Then read on until another line containing timestamp and repeat or a line containing "score": "10 and remove the 3rd or more "'s.

Replacing date within a JSON config file

I'm new but trying to get a new script running but I need it to call on todays date as a variable within the configuration file so the program can be run.
I'm sure sure the best way to implement it so far this line will replace the correct part of the configuration file I need but I can't figure out how to get it to use the "todays date" e.g. date +%F command.
sed -i 's/"to_date":.*/"to_date":"date +%F"/' /config/settings
config following:
{
"username":"admin",
"password":"redhat",
"assumeyes":true,
"to_date": "2011-10-01",
"skip_depsolve":false,
"skip_errata_depsolve":false,
"security_only":false,
"use_update_date":false,
"no_errata_sync":false,
"dry_run":false,
"errata": ["RHSA-2014:0043", "RHBA-2014:0085"],
"blacklist": {
},
"removelist": {
},
"channels":[
{
"rhel-x86_64-server-5": {
"label": "my-rhel5-x86_64-clone",
"existing-parent-do-not-modify": true
},
"rhn-tools-rhel-x86_64-server-5": {
"label": "my-tools-5-x86_64-clone",
"name": "My Clone's Name",
"summary": "This is my channel's summary",
"description": "This is my channel's description"
}
},
{
"rhel-i386-server-5": "my-rhel5-i386-clone"
}
]
}
Using a proper JSON parser jq with the --arg field to pass the current date,
jq --arg inputDate $(date +%F) '.to_date = $inputDate' /config/settings
{
"username": "admin",
"password": "redhat",
"assumeyes": true,
"to_date": "2017-01-27",
"skip_depsolve": false,
"skip_errata_depsolve": false,
"security_only": false,
"use_update_date": false,
"no_errata_sync": false,
"dry_run": false,
"errata": [
"RHSA-2014:0043",
"RHBA-2014:0085"
],
"blacklist": {},
"removelist": {},
"channels": [
{
"rhel-x86_64-server-5": {
"label": "my-rhel5-x86_64-clone",
"existing-parent-do-not-modify": true
},
"rhn-tools-rhel-x86_64-server-5": {
"label": "my-tools-5-x86_64-clone",
"name": "My Clone's Name",
"summary": "This is my channel's summary",
"description": "This is my channel's description"
}
},
{
"rhel-i386-server-5": "my-rhel5-i386-clone"
}
]
}
The jq download and usage instructions are pretty straight forward. Recommend using it for manipulating JSON, instead of depending upon regex.
jq does not edit the file in-place, save it to a temporary file and rename it back, using GNU mktemp
jsonTemp=$(mktemp)
jq --arg inputDate $(date +%F) '.to_date = $inputDate' /config/settings > "$jsonTemp"
mv "$jsonTemp" /config/settings
To include the output of a command inside some quoted text, you have to use a subshell and use double-quotes so the text will get expanded :
sed -i "s/\"to_date\":.*/\"to_date\":\"$(date +%F)\"/" /config/settings
Also I second Inian's comment : you should be using jq to manipulate JSON data.
For example, the following command should do the modification you need :
jq ".toDate = $(date +%F)" /config/settings

Iterate over a json using shell

I have a json in the following format. I want to iterate over this json file
{
"atest_engine": { "version": "96" },
"a_kdfvm": { "version": "68" },
"aseft_api": { "version": "" },
"push_psservice": { "version": "68" },
}
I tried jq utility and my script is as follows.
count=$( jq '. | length' test.json )
echo $count
for((i=0;i<$count;i++))
do
name=$(cat test.json | jq '.|keys['${i}']')
version=$(cat test.json | jq '.|keys['${i}'].version')
echo $name
echo $version
done
I am getting count and name properly but not able to fetch version information. How can I get it. I am new to scripting and any help in this regard is greatly appreciated.Thanks in Advance.
input.json
{
"atest_engine": { "version": "96" },
"a_kdfvm": { "version": "68" },
"aseft_api": { "version": "" },
"push_psservice": { "version": "68" }
}
command
jq -r 'to_entries[] | "\(.key)\t\(.value.version)"' input.json |
while read name version
do
echo "name:" $name
echo "version:" $version
done
result
name: atest_engine
version: 96
name: a_kdfvm
version: 68
name: aseft_api
version:
name: push_psservice
version: 68
First up your JSON example seems slightly malformed - the push_psservice line has a comma after it but this is most likely a typo.
You might find it easier to turn your object's fields into an array using jq's to_entries (see https://stackoverflow.com/a/24254365/4513656 ) e.g.:
to_entries | .[0].key
to_entries | .[0].value.version
Try this on https://jqplay.org/ .

Resources