How do I provide arguments containing spaces to the execute method of strings in groovy? Just adding spaces like one would in a shell does not help:
println 'ls "/tmp/folder with spaces"'.execute().text
This would give three broken arguments to the ls call.
The trick was to use a list:
println(['ls', '/tmp/folder with spaces'].execute().text)
Sorry man, none of the tricks above worked for me.
This piece of horrible code is the only thing that went thru:
def command = 'bash ~my_app/bin/job-runner.sh -n " MyJob today_date=20130202 " '
File file = new File("hello.sh")
file.delete()
file << ("#!/bin/bash\n")
file << (command)
def proc = "bash hello.sh".execute() // Call *execute* on the file
One weird trick for people who need regular quotes processing, pipes etc: use bash -c
['bash','-c',
'''
docker container ls --format="{{.ID}}" | xargs -n1 docker container inspect --format='{{.ID}} {{.State.StartedAt}}' | sort -k2,1
'''].execute().text
Using a List feels a bit clunky to me.
This would do the job:
def exec(act) {
def cmd = []
act.split('"').each {
if (it.trim() != "") { cmd += it.trim(); }
}
return cmd.execute().text
}
println exec('ls "/tmp/folder with spaces"')
More complex example:
println runme('mysql "-uroot" "--execute=CREATE DATABASE TESTDB; USE TESTDB; \\. test.sql"');
The only downside is the need to put quotes around all your args, I can live with that!
did you tried escaping spaces?
println 'ls /tmp/folder\ with\ spaces'.execute().text
Related
For personal development and projects I work on, we use four spaces instead of tabs.
However, I need to use a heredoc, and I can't do so without breaking the indention flow.
The only working way to do this I can think of would be this:
usage() {
cat << ' EOF' | sed -e 's/^ //';
Hello, this is a cool program.
This should get unindented.
This code should stay indented:
something() {
echo It works, yo!;
}
That's all.
EOF
}
Is there a better way to do this?
Let me know if this belongs on the Unix/Linux Stack Exchange instead.
(If you are using bash 4, scroll to the end for what I think is the best combination of pure shell and readability.)
For heredocs, using tabs is not a matter of preference or style; it's how the language is defined.
usage () {
⟶# Lines between EOF are each indented with the same number of tabs
⟶# Spaces can follow the tabs for in-document indentation
⟶cat <<-EOF
⟶⟶Hello, this is a cool program.
⟶⟶This should get unindented.
⟶⟶This code should stay indented:
⟶⟶ something() {
⟶⟶ echo It works, yo!;
⟶⟶ }
⟶⟶That's all.
⟶EOF
}
Another option is to avoid a here document altogether, at the cost of having to use more quotes and line continuations:
usage () {
printf '%s\n' \
"Hello, this is a cool program." \
"This should get unindented." \
"This code should stay indented:" \
" something() {" \
" echo It works, yo!" \
" }" \
"That's all."
}
If you are willing to forego POSIX compatibility, you can use an array to avoid the explicit line continuations:
usage () {
message=(
"Hello, this is a cool program."
"This should get unindented."
"This code should stay indented:"
" something() {"
" echo It works, yo!"
" }"
"That's all."
)
printf '%s\n' "${message[#]}"
}
The following uses a here document again, but this time with bash 4's readarray command to populate an array. Parameter expansion takes care of removing a fixed number of spaces from the beginning of each lie.
usage () {
# No tabs necessary!
readarray message <<' EOF'
Hello, this is a cool program.
This should get unindented.
This code should stay indented:
something() {
echo It works, yo!;
}
That's all.
EOF
# Each line is indented an extra 8 spaces, so strip them
printf '%s' "${message[#]# }"
}
One last variation: you can use an extended pattern to simplify the parameter expansion. Instead of having to count how many spaces are used for indentation, simply end the indentation with a chosen non-space character, then match the fixed prefix. I use : . (The space following
the colon is for readability; it can be dropped with a minor change to the prefix pattern.)
(Also, as an aside, one drawback to your very nice trick of using a here-doc delimiter that starts with whitespace is that it prevents you from performing expansions inside the here-doc. If you wanted to do so, you'd have to either leave the delimiter unindented, or make one minor exception to your no-tab rule and use <<-EOF and a tab-indented closing delimiter.)
usage () {
# No tabs necessary!
closing="That's all"
readarray message <<EOF
: Hello, this is a cool program.
: This should get unindented.
: This code should stay indented:
: something() {
: echo It works, yo!;
: }
: $closing
EOF
shopt -s extglob
printf '%s' "${message[#]#+( ): }"
shopt -u extglob
}
geta() {
local _ref=$1
local -a _lines
local _i
local _leading_whitespace
local _len
IFS=$'\n' read -rd '' -a _lines ||:
_leading_whitespace=${_lines[0]%%[^[:space:]]*}
_len=${#_leading_whitespace}
for _i in "${!_lines[#]}"; do
printf -v "$_ref"[$_i] '%s' "${_lines[$_i]:$_len}"
done
}
gets() {
local _ref=$1
local -a _result
local IFS
geta _result
IFS=$'\n'
printf -v "$_ref" '%s' "${_result[*]}"
}
This is a slightly different approach which requires Bash 4.1 due to printf's assigning to array elements. (for prior versions, substitute the geta function below). It deals with arbitrary leading whitespace, not just a predetermined amount.
The first function, geta, reads from stdin, strips leading whitespace and returns the result in the array whose name was passed in.
The second, gets, does the same thing as geta but returns a single string with newlines intact (except the last).
If you pass in the name of an existing variable to geta, make sure it is already empty.
Invoke geta like so:
$ geta hello <<'EOS'
> hello
> there
>EOS
$ declare -p hello
declare -a hello='([0]="hello" [1]="there")'
gets:
$ unset -v hello
$ gets hello <<'EOS'
> hello
> there
> EOS
$ declare -p hello
declare -- hello="hello
there"
This approach should work for any combination of leading whitespace characters, so long as they are the same characters for all subsequent lines. The function strips the same number of characters from the front of each line, based on the number of leading whitespace characters in the first line.
The reason all the variables start with underscore is to minimize the chance of a name collision with the passed array name. You might want to rewrite this to prefix them with something even less likely to collide.
To use in OP's function:
gets usage_message <<'EOS'
Hello, this is a cool program.
This should get unindented.
This code should stay indented:
something() {
echo It works, yo!;
}
That's all.
EOS
usage() {
printf '%s\n' "$usage_message"
}
As mentioned, for Bash older than 4.1:
geta() {
local _ref=$1
local -a _lines
local _i
local _leading_whitespace
local _len
IFS=$'\n' read -rd '' -a _lines ||:
_leading_whitespace=${_lines[0]%%[^[:space:]]*}
_len=${#_leading_whitespace}
for _i in "${!_lines[#]}"; do
eval "$(printf '%s+=( "%s" )' "$_ref" "${_lines[$_i]:$_len}")"
done
}
I'm looking for a standard tool capable of taking all of its arguments and turning it into a single string suitable for use as multiple arguments in an automatically generated bash/sh/zsh script. Such a command is extremely useful in various disciplines of script-fu. An example of its usage:
% shsafe 'A big \nasty string '\'' $HOME $PATH' 'another string \\'
'A big \nasty string '\'' $HOME $PATH' 'another string \\'
Using it in another script:
% sshc host rm 'file/with spaces and $special chars'
where sshc contains
#!/bin/bash
# usage: sshc host command [arg ...]
# Escapes its arguments so that the command may contain special
# characters. Assumes the remote shell is sh-like.
host=$1
shift
exec ssh "$host" "$(shsafe "$#")"
Another example:
#!/bin/bash
# Run multiple commands in a single sudo session. The arguments of
# this script are passed as arguments to the first command. Useful if
# you don't want to have to type the password for both commands and
# the first one takes a while to run.
sudo bash -c "pacman -Syu $(shsafe "$#") && find /etc -name '*.pacnew'"
I couldn't find a suitable solution to this problem in the pre-existing commands, so I made up my own, called shsafe. It uses the fact that single quotes, '', turn off absolutely all shell expansion, except for ' itself.
shsafe:
#!/usr/bin/env python
from sys import *
n = len(argv)
if n == 1:
exit(0)
i = 1
while True:
stdout.write("'" + argv[i].replace("'", "'\\''") + "'")
i += 1
if i == n:
break
stdout.write(' ')
stdout.write('\n')
Is there any standard tool capable of doing this to its arguments?
Note that the printf command with a format string consisting of just the %q formatter is not good enough for this, because it won't keep multiple arguments separated:
% printf %q arg1 arg2
arg1arg2
I did eventually figure out a decent way of doing this:
% printf "$'%q' " 'crazy string \ $HOME' 'another\ string'
$'crazy\ string\ \\\ \$HOME' $'another\\\ string'
It's a little error prone what with the quotes everywhere, so it's not ideal, IMO, but it's a solid solution that should work anywhere. If it's being used a lot, you could always turn it into a shell function:
shsafe () {
printf "$'%q' " "$#"
}
I have something like this on a Jenkinsfile (Groovy) and I want to record the stdout and the exit code in a variable in order to use the information later.
sh "ls -l"
How can I do this, especially as it seems that you cannot really run any kind of groovy code inside the Jenkinsfile?
The latest version of the pipeline sh step allows you to do the following;
// Git committer email
GIT_COMMIT_EMAIL = sh (
script: 'git --no-pager show -s --format=\'%ae\'',
returnStdout: true
).trim()
echo "Git committer email: ${GIT_COMMIT_EMAIL}"
Another feature is the returnStatus option.
// Test commit message for flags
BUILD_FULL = sh (
script: "git log -1 --pretty=%B | grep '\\[jenkins-full]'",
returnStatus: true
) == 0
echo "Build full flag: ${BUILD_FULL}"
These options where added based on this issue.
See official documentation for the sh command.
For declarative pipelines (see comments), you need to wrap code into script step:
script {
GIT_COMMIT_EMAIL = sh (
script: 'git --no-pager show -s --format=\'%ae\'',
returnStdout: true
).trim()
echo "Git committer email: ${GIT_COMMIT_EMAIL}"
}
Current Pipeline version natively supports returnStdout and returnStatus, which make it possible to get output or status from sh/bat steps.
An example:
def ret = sh(script: 'uname', returnStdout: true)
println ret
An official documentation.
quick answer is this:
sh "ls -l > commandResult"
result = readFile('commandResult').trim()
I think there exist a feature request to be able to get the result of sh step, but as far as I know, currently there is no other option.
EDIT: JENKINS-26133
EDIT2: Not quite sure since what version, but sh/bat steps now can return the std output, simply:
def output = sh returnStdout: true, script: 'ls -l'
If you want to get the stdout AND know whether the command succeeded or not, just use returnStdout and wrap it in an exception handler:
scripted pipeline
try {
// Fails with non-zero exit if dir1 does not exist
def dir1 = sh(script:'ls -la dir1', returnStdout:true).trim()
} catch (Exception ex) {
println("Unable to read dir1: ${ex}")
}
output:
[Pipeline] sh
[Test-Pipeline] Running shell script
+ ls -la dir1
ls: cannot access dir1: No such file or directory
[Pipeline] echo
unable to read dir1: hudson.AbortException: script returned exit code 2
Unfortunately hudson.AbortException is missing any useful method to obtain that exit status, so if the actual value is required you'd need to parse it out of the message (ugh!)
Contrary to the Javadoc https://javadoc.jenkins-ci.org/hudson/AbortException.html the build is not failed when this exception is caught. It fails when it's not caught!
Update:
If you also want the STDERR output from the shell command, Jenkins unfortunately fails to properly support that common use-case. A 2017 ticket JENKINS-44930 is stuck in a state of opinionated ping-pong whilst making no progress towards a solution - please consider adding your upvote to it.
As to a solution now, there could be a couple of possible approaches:
a) Redirect STDERR to STDOUT 2>&1
- but it's then up to you to parse that out of the main output though, and you won't get the output if the command failed - because you're in the exception handler.
b) redirect STDERR to a temporary file (the name of which you prepare earlier) 2>filename (but remember to clean up the file afterwards) - ie. main code becomes:
def stderrfile = 'stderr.out'
try {
def dir1 = sh(script:"ls -la dir1 2>${stderrfile}", returnStdout:true).trim()
} catch (Exception ex) {
def errmsg = readFile(stderrfile)
println("Unable to read dir1: ${ex} - ${errmsg}")
}
c) Go the other way, set returnStatus=true instead, dispense with the exception handler and always capture output to a file, ie:
def outfile = 'stdout.out'
def status = sh(script:"ls -la dir1 >${outfile} 2>&1", returnStatus:true)
def output = readFile(outfile).trim()
if (status == 0) {
// output is directory listing from stdout
} else {
// output is error message from stderr
}
Caveat: the above code is Unix/Linux-specific - Windows requires completely different shell commands.
this is a sample case, which will make sense I believe!
node('master'){
stage('stage1'){
def commit = sh (returnStdout: true, script: '''echo hi
echo bye | grep -o "e"
date
echo lol''').split()
echo "${commit[-1]} "
}
}
For those who need to use the output in subsequent shell commands, rather than groovy, something like this example could be done:
stage('Show Files') {
environment {
MY_FILES = sh(script: 'cd mydir && ls -l', returnStdout: true)
}
steps {
sh '''
echo "$MY_FILES"
'''
}
}
I found the examples on code maven to be quite useful.
All the above method will work. but to use the var as env variable inside your code you need to export the var first.
script{
sh " 'shell command here' > command"
command_var = readFile('command').trim()
sh "export command_var=$command_var"
}
replace the shell command with the command of your choice. Now if you are using python code you can just specify os.getenv("command_var") that will return the output of the shell command executed previously.
How to read the shell variable in groovy / how to assign shell return value to groovy variable.
Requirement : Open a text file read the lines using shell and store the value in groovy and get the parameter for each line .
Here , is delimiter
Ex: releaseModule.txt
./APP_TSBASE/app/team/i-home/deployments/ip-cc.war/cs_workflowReport.jar,configurable-wf-report,94,23crb1,artifact
./APP_TSBASE/app/team/i-home/deployments/ip.war/cs_workflowReport.jar,configurable-temppweb-report,394,rvu3crb1,artifact
========================
Here want to get module name 2nd Parameter (configurable-wf-report) , build no 3rd Parameter (94), commit id 4th (23crb1)
def module = sh(script: """awk -F',' '{ print \$2 "," \$3 "," \$4 }' releaseModules.txt | sort -u """, returnStdout: true).trim()
echo module
List lines = module.split( '\n' ).findAll { !it.startsWith( ',' ) }
def buildid
def Modname
lines.each {
List det1 = it.split(',')
buildid=det1[1].trim()
Modname = det1[0].trim()
tag= det1[2].trim()
echo Modname
echo buildid
echo tag
}
If you don't have a single sh command but a block of sh commands, returnstdout wont work then.
I had a similar issue where I applied something which is not a clean way of doing this but eventually it worked and served the purpose.
Solution -
In the shell block , echo the value and add it into some file.
Outside the shell block and inside the script block , read this file ,trim it and assign it to any local/params/environment variable.
example -
steps {
script {
sh '''
echo $PATH>path.txt
// I am using '>' because I want to create a new file every time to get the newest value of PATH
'''
path = readFile(file: 'path.txt')
path = path.trim() //local groovy variable assignment
//One can assign these values to env and params as below -
env.PATH = path //if you want to assign it to env var
params.PATH = path //if you want to assign it to params var
}
}
Easiest way is use this way
my_var=`echo 2`
echo $my_var
output
: 2
note that is not simple single quote is back quote ( ` ).
I set a variable with spaces in a string to a new bash:
VAR='my variable with spaces' /bin/bash
And now if I want to start a new bash with the same environment, I would do something like:
ENV=$(cat /proc/self/environ | xargs -0 | grep =)
env -i - $ENV /bin/bash
But the thing is, in /proc/self/environ, this variable is without quotes. So the last command throws a: env: variable: No such file or directory
How can I work around this limitation?
PS: this is a simplified version of the following issue: https://github.com/jpetazzo/nsenter/issues/62
I think the answer here is to not use a shell script to set things up. Using a higher-level language makes it much easier to parse /proc/<PID>/environ into something useful. Here's a short example:
#!/usr/bin/python
import os
import sys
import argparse
def parse_args():
p = argparse.ArgumentParser()
p.add_argument('pid')
p.add_argument('command', nargs=argparse.REMAINDER)
return p.parse_args()
def main():
args = parse_args()
env = {}
with open('/proc/%s/environ' % args.pid) as fd:
for envspec in fd.read().split('\000'):
if not envspec:
continue
varname, varval = envspec.split('=', 1)
env[varname] = varval
print env
os.execvpe(args.command[0], args.command, env)
if __name__ == '__main__':
main()
Put this in a file called env-from, make it executable, and then you
can run:
env-from <pid> bash
And you'll get a shell using the environment variables from the
specified process.
Just add -L1 to xargs (max non-blank input lines per command line):
xargs -0 -L1 -a /proc/self/environ
This will give you each variable on a separate line, which makes it easier to process. Or simply use
strings /proc/self/environ
In Linux, say I have the following file (e.g. conf.properties):
HOST_URL=http://$HOSTNAME
STD_CONFIG=http://$HOSTNAME/config
USER_CONFIG=http://$HOSTNAME/config/$unconfigured
I want to create another file with all the environment variables replaced...e.g. say the environment variable $HOSTNAME is 'myhost' and $unconfigured is not set, a script should produce the following output:
HOST_URL=http://myhost
STD_CONFIG=http://myhost/config
USER_CONFIG=http://myhost/config/
I was thinking this could be done in a simple one-liner with some sort of sed/awk magic, but I'm no expert and my searches have been in vein, so appreciate any help.
Edit:
I should mention that the file can really be any format text file, for example xml. I just want to replace anything that looks like an env variable with whatever is currently set in the environment.
This is what envsubst is for.
echo 'Hello $USER'
Hello $USER
echo 'Hello $USER' | envsubst
Hello malvineous
You would probably use it more like this though:
envsubst < input.txt > output.txt
envsubst seems to be part of GNU gettext.
sed 's/$HOSTNAME/myhost/g;s/$unconfigured//g' yourfile.txt > another_file.txt
update:
Based on updates in your question, this won't be a good solution.
update2 :
This is based on an answer to a related question. I've hacked at it (I'm unfamiliar with perl) to remove undefined vars.
perl -p -e 's/\$\{([^}]+)\}/defined $ENV{$1} ? $ENV{$1} : $&/eg; s/\$\{([^}]+)\}//eg' yourfile.txt
Should work for any input text file, however you will need to define vars using the ${...} format which simplifies the string matching.
(rant regarding the evilness of eval moved to a separate post so as not to confuse readers)
"eval is evil"
This is not an answer, but a warning in response to using eval for this task. You really really really don't want to do that.
Exhibit 1: a malicious template file:
HOST_URL=http://$HOSTNAME
STD_CONFIG=http://$HOSTNAME/config
USER_CONFIG=http://$HOSTNAME/config/$unconfigured
&& cat /etc/redhat-release
An unsuspecting user:
[lsc#aphek]$ cat somefile | while read line; do echo $(eval echo `echo $line`); done
HOST_URL=http://xyz
STD_CONFIG=http://xyz/config
USER_CONFIG=http://xyz/config/
Red Hat Enterprise Linux WS release 4 (Nahant Update 9)
Note the last line!
Now, imagine the possibilities....
I'd do it like this:
# Set the $HOSTNAME and other variables
# Now evaluate the properties file as a shell script.
. config.properties
# Write the values
cat >somefile <<EOF
HOST_URL=$HOST_URL
STD_CONFIG=$STD_CONFIG
USER_CONFIG=$USER_CONFIG
EOF
Edit: Or this very nasty thing (I'm sure there's a better way)
for name in HOST_URL STD_CONFIG USER_CONFIG
echo "$name=$(eval echo `echo '$'$name`)" >>somefile
end
Thanks to #DarkDust I came up with this:
cat somefile | while read line; do echo $(eval echo `echo $line`); done > somefile.replaced
I used this oneliner to replace ${VARIABLE} style variables in a file:
TARGET_FILE=/etc/apache2/apache2.conf; for VARNAME in $(grep -P -o -e '\$\{\S+\}' ${TARGET_FILE} | sed -e 's|^\${||g' -e 's|}$||g' | sort -u); do sed -i "s|\${$(echo $VARNAME)}|${!VARNAME}|g" ${TARGET_FILE}; done
I'm pretty sure someone can do this in 1/3rd of the length using awk… feel challenged! ;)
Here is a snippet of Javascript that I like to have around for solving this exact problem:
// A Javascript version of envsubst for our builds
// Purpose: replace all ocurrences of ${VAR} with the equivalent var from the environment from stdin
var readline = require('readline');
var rl = readline.createInterface({
input: process.stdin,
output: process.stdout,
terminal: false
});
const environment = process.env;
rl.on('line', function(line) {
const newLine = line.replace(/\$\{([a-zA-Z0-9_]+)\}/g, function(_match, variable) {
const envVar = environment[variable];
return envVar ? envVar : '';
});
process.stdout.write(`${newLine}\n`);
});
Hopefully this helps somebody else.
Here's a short one-liner that uses python's curly brace formatting to safely do the magic:
contents=\"\"\"`cat $file`\"\"\"; python -c "import os;print $contents.format(**os.environ)"
avoids evil eval
allows outputting curly braces: use {{ instead of {
no need to specify vars explicitly when calling the script
For example, given properties file settings.properties:
# my properties file
someVar = {MY_ENV_VAR}
curlyBraceVar = has {{curly braces}}
Then, do the substitution with:
$ export MY_ENV_VAR="hello"
$ file=settings.properties
$ contents=\"\"\"`cat $file`\"\"\"; python -c "import os;print $contents.format(**os.environ)"
# my properties file
someVar = hello
curlyBraceVar = has {curly braces}
A script is here: https://raw.githubusercontent.com/aneilbaboo/machome/master/bin/substenv
if you have installed nodejs you can run
npx #utft/tt -e FOO=bar /path/to/input /path/to/output
or you can run it programmatically
https://github.com/utftufutukgyftryidytftuv/tt