I have a file which contains my json
{
"type": "xyz",
"my_version": "1.0.1.66~22hgde",
}
I want to edit the value for key my_version and everytime replace the value after third dot with another number which is stored in a variable so it will become something like 1.0.1.32~22hgde. I am using sed to replace it
sed -i "s/\"my_version\": \"1.0.1.66~22hgde\"/\"my_version\": \"1.0.1.$VAR~22hgde\"/g" test.json
This works but the issue is that my_version string doesn't remain constant and it can change and the string can be something like this 1.0.2.66 or 2.0.1.66. So how do I handle such case in bash?
how do I handle such case?
You write a regular expression to match any possible combination of characters that can be there. You can learn regex with fun with regex crosswords online.
Do not edit JSON files with sed - sed is for lines. Consider using JSON aware tools - like jq, which will handle any possible case.
A jq answer: file.json contains
{
"type": "xyz",
"my_version": "1.0.1.66~22hgde",
"object": "can't end with a comma"
}
then, replacing the last octet before the tilde:
VAR=32
jq --arg octet "$VAR" '.my_version |= sub("[0-9]+(?=~)"; $octet)' file.json
outputs
{
"type": "xyz",
"my_version": "1.0.1.32~22hgde",
"object": "can't end with a comma"
}
Related
I am trying to insert a variable value into file from Jenkinsfile using shell script in the section
Variable value is dynamic. I am using sed.
Sed is working fine but it is not retaining the white spaces that the variable have at the beginning.
ex:
The value of >> repoName is " somename"
stage('trying sed command') {
steps {
script {
sh """
#!/bin/bash -xel
repo='${repoName}'
echo "\$repo"
`sed -i "5i \$repo" filename`
cat ecr.tf
"""
}
}
}
current output:
names [
"xyz",
"ABC",
somename
"text"
]
Expected output:
names [
"xyz",
"ABC",
somename
"text"
]
How do i retain the spaces infront of the variable passing from sed
With
$ cat filename
names [
"xyz",
"ABC",
"text"
]
$ repo=somename
we can do:
sed -E "3s/^([[:blank:]]*).*/&\\n\\1${repo},/" filename
names [
"xyz",
"ABC",
somename,
"text"
]
That uses capturing parentheses to grab the indentation from the previous line.
if $repo might contain a value with slashes, you can tell the shell to escape them with this (eye-opening) expansion
repo='some/name'
sed -E "3s/^([[:blank:]]*).*/&\\n\\1${repo//\//\\\/},/" filename
names [
"xyz",
"ABC",
some/name,
"text"
]
I used 1 sed statement to add the content first to the file and then another sed statement for just adding spaces. This fixed my issue. All day i was trying to fit in one command did not work probably from Jenkins and shell usage. But using 2 sed commands as a workaround i was able to finish my task
The i command skips over spaces after the i command to find the text to insert. You can put the text on a new line, with a backslash before the newline, to have the initial whitespace preserved.
stage('trying sed command') {
steps {
script {
sh """
#!/bin/bash -xel
repo='${repoName}'
echo "\$repo"
`sed -i "5i \\\\\\
\$repo" filename`
cat ecr.tf
"""
}
}
}
I've tested this from a regular shell command line, I hope it will also work in the Jenkins recipe.
I have a huge json file, i will copy a part of it :
"panels": [
"targets": [
{
"alias": "First Prod",
"dimensions": {
"Function": "robot-support-dev-create-human"
},
}
{
"alias": "Second Prod",
"dimensions": {
"Function": "robot-support-prototype-dev-beta-activate-human"
},
}
{
"alias": "third Prod",
"dimensions": {
"Function": "robot-support-dev-jira-kill-human"
},
}
{
"alias": "Somehting",
"dimensions": {
"Robotalias": "default",
"RobotName": "Robot-prod-prototype",
"Operation": "Fight"
},
]
]
I want to perform a Regex on Function each time it contains the robot-support-dev to robot-support-prod-...
sed -i ' s/"robot-support-([a-zA-Z0-9_]+)-dev-([^"]+)"/robot-support-\\1-prod-\\2/g;'
This is what i did but there's something wrong with my regex maybe
You can't match a json with a regex. The "right way(TM)" would be to first extract Function using a json aware tool like jq, then modify it with sed, then insert it back using json aware tool.
Sed by default uses basic regex, so you need to change ( ) + into \( \) \{1,\} (the \+ is a GNU extension) or with GNU sed just use sed -E to use extended regex. Also the \\1 would be interpreted as 2 characters \ with 1, you want to use \1 with a single \. But anyway, your regex is just invalid and does not match what you want (I guess). Also the " are missing on the right side in the replacement string, so your command would just remove the ". Just substitute what you need, try:
sed 's/"robot-support-dev-/"robot-support-prod-/g;'
As #KamilCuk pointed out, there are some issues with the regex you are currently using. I think that if the occurrences you give as an example are the only possibilities, it would work if you match these groups:
sed -i 's/"robot-support\(-\|-.*-\)dev-\(.*\)"/"robot-support\1prod-\2"/g'
As pointed out elsewhere on this page, using sed for this type of problem is, at best, fraught with danger. Since the question has been tagged with jq, it should be pointed out that jq is an excellent match for this type of problem. In particular, a trivial solution can be obtained using the filter sub of arity 2 or 3, i.e. sub/3: sub("FROM"; "TO"; "g").
You might also wish to use walk so that you don't have to be concerned about where exactly the "Function" keys occur, e.g.
walk( if type == "object" and (.Function|type) == "string"
then .Function |= sub( "robot-support-(?<a>([^-]+-)?)dev-"; "robot-support-\(.a)prod-"; "g")
else . end)
Disclaimer: I used an extremely simple example thinking each argument had some hidden encoding I wasn't aware of. Turns out my
formatting was entirely wrong. As #miken32 said, I should be using
commas. I changed my format and it works perfectly. Valuable lesson
learned.
I've exported a csvfile from an xlsl with Excel 2013 (on Windows). I emailed myself the new csv file and am running these tests on Unix (MacOS Sierra).
Consider the following CSV file:
John
Adam
Cameron
Jordan
I'm trying to format each line to look like this:
{'operator':'EQ', 'property':'first_name', 'value':'John'},
{'operator':'EQ', 'property':'first_name', 'value':'Adam'},
{'operator':'EQ', 'property':'first_name', 'value':'Cameron'},
{'operator':'EQ', 'property':'first_name', 'value':'Jordan'}
So value is the only argument changing between each line.
Here is the awk file I wrote:
BEGIN { }
{
print "{'operator':'EQ', 'property':'first_name', 'value':'"$0"'},";
}
END { }
But after executing this is the output I get:
{'operator':'EQ', 'property':'first_name', 'value':'John
'},
{'operator':'EQ', 'property':'first_name', 'value':'Adam
'},
Notice how right after the argument ($0) is printed out, a newline is printed? This is messing with my JSON format. I have a feeling this has to do with the excel exporting (which was done by Save as .csv).
Any suggestions?
In awk, $0 represents the entire line, whereas $1, $2, $n represent the delimited fields in the line.
The sample provided isn't a CSV file, since there aren't any values separated by commas. If it were, you could do this:
awk -F, '{print "{'"'"'operator'"'"':'"'"'EQ'"'"', '"'"'property'"'"':'"'"'first_name'"'"', '"'"'value'"'"':'"'"'"$1"'"'"'},"}' foo.txt
Which gets a bit crazy with the shell-friendly quoting!
You should be aware that there are tools such as jq, which are designed to create and work with JSON data. If this is more than a one-off task you might be better served looking at those.
Edit using a suggestion by Ed Morton from a comment:
awk -F, '{print "{\047operator\047:\047EQ\047, \047property\047:\047first_name\047, \047value\047:\047"$1"\047},"}' foo.txt
(But from your original question it looks like you're using a separate script file anyway, so you won't have to worry about escaping quotes.)
As has been noted, your sample output with '-based quoting isn't valid JSON, where only " may be used.
Ensuring valid JSON output is a good reason to
use the jq CLI, which not only makes the task more robust, but also simplifies it:
jq -Rnc 'inputs | { operator: "EQ", property: "first_name", value: . }' <<EOF
John
Adam
Cameron
Jordan
EOF
yields:
{"operator":"EQ","property":"first_name","value":"John"}
{"operator":"EQ","property":"first_name","value":"Adam"}
{"operator":"EQ","property":"first_name","value":"Cameron"}
{"operator":"EQ","property":"first_name","value":"Jordan"}
Explanation:
-R reads Raw input (input that isn't JSON)
-n suppresses automatic reading of the input, so that special variables input and inputs can be used instead.
-c produces compact output (not pretty-printed)
inputs represents all input lines, and the expression after | sees each line as ., iteratively.
The output object can be specified using JavaScript syntax, which simplifies matters because the property names don't require quoting; the expanded value of { ... } is converted to JSON on output.
Perl:
perl -MJSON -nlE 'push #p,{operator=>"EQ",property=>"first_name",value=>$_}}{say JSON->new->pretty->encode(\#p)' file
output is valid, pretty-printed JSON:
[
{
"operator" : "EQ",
"property" : "first_name",
"value" : "John"
},
{
"operator" : "EQ",
"value" : "Adam",
"property" : "first_name"
},
{
"operator" : "EQ",
"property" : "first_name",
"value" : "Cameron"
},
{
"property" : "first_name",
"value" : "Jordan",
"operator" : "EQ"
}
]
more readble:
perl -MJSON -nlE '
push #p, { operator=>"EQ", property=>"first_name", value=>$_}
END {
say JSON->new->pretty->encode(\#p)
}' file
If you generating JSON, a final note: in the JSON the single quotes aren't allowed.
I have a file that includes, among other things, a json. The json contains passwords that need masking. The bash script responsible for the masking has no way of knowing the actual password itself, so its not a simple sed search-and-replace.
The passwords appear within the json under a constant key named "password" or "Password". Typically, the appearance is like -
...random content..."Password\":\"actualPWD\"...random content....
The bash script needs to change such appearances to -
...random content..."Password\":\"******\"...random content....
The quotes aren't important, so even ...random
content..."Password\":******...random content...
would work.
I reckon the logic would need to find the index of the ':' that appears after the text "Password"/"password" and then substring from that point on till the second occurrence of quote (") from there and replace the whole thing with *****. But I'm not sure how to do this with sed or awk. Any suggestion would be helpful.
Perl to the rescue!
perl -pe 's/("[Pp]assword\\":\\")(.*?)(\\")/$1 . ("." x length $2) . $3/ge'
/e interprets the replacement part as code, so you can use the repetition operator x and repeat the dot length $2 times.
Since JSON is structured, any approach based solely on regular expressions is bound to fail at some point unless the input is constrained in some way. It would be far better (simpler and safer) to use a JSON-aware approach.
One particularly elegant JSON-aware tool worth knowing about is jq. (Yes, the "j" is for JSON :-)
Assuming we have an input file consisting of valid JSON and that we want to change the value of every "password" or "Password" key to "******" (no matter how deeply nested the object having these keys may be), we could proceed as follows:
Place the following into a file, say mask.jq:
def mask(p): if has(p) then .[p] = "******" else . end;
.. |= if type == "object"
then mask("password") | mask("Password") else . end
Now suppose in.json has this JSON:
{"password": "secret", "details": [ {"Password": "another secret"} ]}
Then executing the command:
jq -f mask.jq in.json
produces:
{
"password": "******",
"details": [
{
"Password": "******"
}
]
}
More on jq at https://github.com/stedolan/jq
I have a file:
"tags": "['PNP']"
Clearly "[ is wrong, it must ot be "tags" : ['PNP']
So I wanna to replace with sed:
sed -i "1,$ s/"[/[/g" file.json
However it told me that it is not match
How can I do it?
You can do
sed 's/"\[/\[/; s/\]"/\]/' file.json
The brackets [] are special characters in basic regular expressions, so you need to escape them.
On input:
"tags": "['PNP']"
This outputs:
"tags": ['PNP']