I have a Jenkins console output that looks like this:
Started by remote host 10.16.17.13
Building remotely on ep9infrajen201 (ep9) in workspace d:\Jenkins\workspace\Tools\Provision
[AWS-NetProvision] $ powershell.exe -NonInteractive -ExecutionPolicy ByPass "& 'C:\Users\user\AppData\Local\Temp\jenkins12345.ps1'"
Request network range: 10.1.0.0/13
{
"networks": [
"10.1.0.0/24"
]
}
Finished: SUCCESS
I get this from a curl command that I run. to check the JENKINS_JOB_URL/lastBuild/consoleText
My question is, for the sake of some other automation I am doing, how do I get just "10.1.0.0/24" so I can assign it to a shell variable using LINUX tools?
Thank you
Since you listed jq among the tags of your duplicate question, I'll assume you have jq installed. You have to clean up your output to get JSON first, then get to the part of JSON you need. awk does the former, jq the latter.
.... | awk '/^{$/{p=1}{if(p){print}}/^}$/{p=0}' | jq -r .networks[0]
The AWK script looks for { on its own on a line to turn on a flag p; prints the current line if the flag is set; and switches off the flag when it encounters } all by itself.
EDIT: Since this output was generated on a DOS machine, it has DOS line endings (\r\n). To convert those before awk, additionally pipe through dos2unix.
Related
This question already has answers here:
Parsing JSON with Unix tools
(45 answers)
Closed 24 days ago.
I have json string like this
{"state":{"stateId":0,"nextPollingTime":null,"updateState":null},"config":{"privateIPs":null,"64Bit":false,"silent":false,"iid":14,"selfp":null,"sevlfp":null,"av":14,"aid":null,"aty":2,"sev":0,"seci":0,"sti":false,"scto":60000,"sci":5000,"stkd":5000,"sud":5000,"mpfa":3}}
I need to get "state" and "config" keys content to a shell custom variable or to a file.
tried with a command:
$RESPONSE is the API response, the above json string
echo $RESPONSE | sed 's/{"$state":"*\([0-9a-zA-Z]*\)"*,*.*}/\1/'
this prints nothing but suppose to get this output:
{"stateId":0,"nextPollingTime":null,"updateState":null}
Tried with saving the response to tmp file and executed this command
cat /tmp/a.json | grep -o -e "{"state":.*}"
this also print empty string but expected result:
{"stateId":0,"nextPollingTime":null,"updateState":null}
Am new to shell script and trying with various options available in the internet, please help me to write the command for the same.
This uses any sed:
$ sed 's/.*"state":\({[^}]*}\).*/\1/' file
{"stateId":0,"nextPollingTime":null,"updateState":null}
That will work for your posted input but fail given other json input (e.g. with "state" as a value or with } inside a value), just like any other solution that uses mandatory POSIX tools as none of them have a JSON parser (unless you write one with awk, but I expect that's more work than you or anyone here would put into this).
Trying to execute svn command on windows machine and capture the output for the same.
Code:
import subprocess
cmd = "svn log -l1 https://repo/path/trunk | grep ^r | awk '{print \$3}'"
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, shell=True)
'grep' is not recognized as an internal or external command,
operable program or batch file.
I do understand that 'grep' is not windows utility.
Getting error as "grep' is not recognized as an internal or external command,
operable program or batch file."
Is it only limited to execute on Linux?
Can we execute the same on Windows?
Is my code right?
For windows your command will look something like the following
svn log -l1 https://repo/path/trunk | find "string_to_find"
You need to use the find utility in windows to get the same effect as grep.
svn --version | find "ra"
* ra_svn : Module for accessing a repository using the svn network protocol.
* ra_local : Module for accessing a repository on local disk.
* ra_serf : Module for accessing a repository via WebDAV protocol using serf.
Use svn log --search FOO instead of grep-ing the command's output.
grep and awk are certainly available for Windows as well, but there is really no need to install them -- the code is easy to replace with native Python.
import subprocess
p = subprocess.run(["svn", "log", "-l1", "https://repo/path/trunk"],
capture_output=True, text=True)
for line in p.stdout.splitlines():
# grep ^r
if line.startswith('r'):
# awk '{ print $3 }'
print(line.split()[2])
Because we don't need a pipeline, and just run a single static command, we can avoid shell=True.
Because we don't want to do the necessary plumbing (which you forgot anyway) for Popen(), we prefer subprocess.run(). With capture_output=True we conveniently get its output in the resulting object's stdout atrribute; because we expect text output, we pass text=True (in older Python versions you might need to switch to the old, slightly misleading synonym universal_newlines=True).
I guess the intent is to search for the committer in each revision's output, but this will incorrectly grab the third token on any line which starts with an r (so if you have a commit message like "refactored to use Python native code" the code will extract use from that). A better approach altogether is to request machine-readable output from svn and parse that (but it's unfortunately rather clunky XML, so there's another not entirely trivial rabbithole for you). Perhaps as middle ground implement a more specific pattern for finding those lines -- maybe look for a specific number of fields, and static strings where you know where to expect them.
if line.startswith('r'):
fields = line.split()
if len(fields) == 13 and fields[1] == '|' and fields[3] == '|':
print(fields[2])
You could also craft a regular expression to look for a date stamp in the third |-separated field, and the number of changed lines in the fourth.
For the record, a complete commit message from Subversion looks like
------------------------------------------------------------------------
r16110 | tripleee | 2020-10-09 10:41:13 +0300 (Fri, 09 Oct 2020) | 4 lines
refactored to use native Python instead of grep + awk
(which is a useless use of grep anyway; see http://www.iki.fi/era/unix/award.html#grep)
i run the following function in (git-)bash under windows:
function config_get_container_values() {
local project_name=$1
local container_name=$2
#local container_name="gitea"
echo "###"
buildcmd="jq -r \".containers[]."
echo "$buildcmd"
buildcmd="${buildcmd}${container_name}"
echo "$buildcmd"
buildcmd="${buildcmd}foobar"
echo "$buildcmd"
echo "###"
}
The output of this is the following. Whyever, after using the variable to extend the string, he starts to overwrite $buildcmd. I tried this also with everything in one line as well with the append command (=+). Everytime the same result.
###
jq -r ".containers[].
jq -r ".containers[].gitea
foobar".containers[].gitea
###
The really strange thing is: When i enable the line local container_name="gitea" everything works as expected. The output is:
###
jq -r ".containers[].
jq -r ".containers[].gitea
jq -r ".containers[].giteafoobar
###
When i put this all into a news file, its also works as expected. So i think something goes wrong in the thousands of line before calling this function. Any idea, what could be cause of this behavior?
Regards
Dave
This is not how you should build up the command, DOS line endings aside. Use --arg to pass the name into the filter as a variable. For example,
config_get_container_values() {
local project_name=$1
local container_name=$2
jq -r --arg n "$container_name " '.containers[][$n+"foobar"]'
}
config_get_container foo gitea < some.json
If the function is invoked with
config_get_container_values proj gitea
it produces the "expected" output. If it is invoked with
config_get_container_values proj $'gitea\r'
it produces output that looks like the first output example. $'gitea\r' expands to a string that consists of 'gitea' followed by a Carriage return (CR) character.
One possible cause of the problem is that the container name (gitea) was read from a file that had Windows/DOS line endings (CR-LF). Problems like that are common. See the first question ("Check whether your script or data has DOS style end-of-line characters") in the "Before asking about problematic code" section of the Stack Overflow 'bash' Info page.
I have a small bash script
#!/bin/bash
echo $(cat /proc/$PPID/cmdline | strings -1)
I call this script from a perl script which is run through nginx.
my $output_string = `/tmp/my_bash_script.sh`;
print $output_string;
When I load this in a browser, the result is something like:
/mnt/my_working_d -d /etc/my_httpd -f /etc/my_httpd/conf/httpd.conf
The location of the perl script is indeed somewhere in /mnt/my_working_directory/.... but why is this truncated and is there anyting I can do to log the whole command. I don't think the cmdline limit of 4k characters (?) which seems hardcoded in the kernel applies here.
Ok, here I'm again, struggling with ssh. I'm trying to retrieve some data from remote log file based on tokens. I'm trying to pass multiple tokens in egrep command via ssh:
IFS=$'\n'
commentsArray=($(ssh $sourceUser#$sourceHost "$(egrep "$v" /$INSTALL_DIR/$PROP_BUNDLE.log)"))
echo ${commentsArray[0]}
echo ${commentsArray[1]}
commax=${#commentsArray[#]}
echo $commax
where $v is something like below but it's length is dynamic. Meaning it can have many file names seperated by pipe.
UserComments/propagateBundle-2013-10-22--07:05:37.jar|UserComments/propagateBundle-2013-10-22--07:03:57.jar
The output which I get is:
oracle#172.18.12.42's password:
bash: UserComments/propagateBundle-2013-10-22--07:03:57.jar/New: No such file or directory
bash: line 1: UserComments/propagateBundle-2013-10-22--07:05:37.jar/nouserinput: No such file or directory
0
Thing worth noting is that my log file data has spaces in it. So, in the code piece I've given, the actual comments which I want to extract start after the jar file name like : UserComments/propagateBundle-2013-10-22--07:03:57.jar/
The actual comments are 'New Life Starts here' but the logs show that we are actually getting it till 'New' and then it breaks at space. I tried giving IFS but of no use. Probably I need to give it on remote but I don't know how should I do that.
Any help?
Your command is trying to run the egrep "$v" /$INSTALL_DIR/$PROP_BUNDLE.log on the local machine, and pass the result of that as the command to run via SSH.
I suspect that you meant for that command to be run on the remote machine. Remove the inner $() to get that to happen (and fix the quoting):
commentsArray=($(ssh $sourceUser#$sourceHost "egrep '$v' '/$INSTALL_DIR/$PROP_BUNDLE.log'"))
You should use fgrep to avoid regex special interpretation from your input:
commentsArray=($(ssh $sourceUser#$sourceHost "$(fgrep "$v" /$INSTALL_DIR/$PROP_BUNDLE.log)"))