I have the following Jenkinsfile, executing in a Linux container under Kubernetes. My Jenkins server is version 2.263.4 running on Windows 2012 R2. Any variable I define in my environments section shows up in the sh action with a newline at the end:
pipeline {
agent {
kubernetes {
label UUID.randomUUID().toString()
yaml """
# ..snip...
"""
}
}
environment {
VAR1 = 'VALUE 1'
VAR2 = 'VALUE 2'
}
stages {
stage('One') {
steps {
container('docker') {
sh 'echo -n "$PATH"'
sh 'echo -n "$VAR1"'
sh 'echo -n "$VAR2"'
}
}
}
}
}
Which results in this output:
[Pipeline] sh
+ echo -n /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
[Pipeline] sh
+ echo -n 'VALUE 1
'
VALUE 1
[Pipeline] sh
+ echo -n 'VALUE 2
'
VALUE 2
As you can see, the PATH environment variable has no newline in the shell command, but the two variables from the Jenkinsfile have newlines at the end of their command and are surrounded by single quotes, even though they are surrounded by double-quotes in my sh command.
The problem happens when I use these values as parameters to other commands. For example
sh 'git clone -b $BRANCH $REMOTE source'
Ends up running this command:
+ git clone -b 'BranchValue
' 'RemoteValue
+ ' source
How do I get my environment variables to not have newlines at the end of their values?
I had same issue even without docker with such a simple Jenkinsfile
pipeline {
agent any
stages {
stage('Demo') {
environment {
POM_VERSION = sh(script: 'echo "2.5-SNAPSHOT"', returnStdout: true)
}
steps {
echo "POM_VERSION '${POM_VERSION}'"
}
}
}
}
The sh command indeed adds new line at the end of shell script's output. Jenkins console log looks like this (see ending quote ' at the beginning of new line):
[Pipeline] echo
POM_VERSION '2.5-SNAPSHOT
'
During troubleshooting I came across article How to strip some form of new line character at end of parsed Jenkinsfile variable.
It turned out that is was enough to call trim() on sh's returned value like this:
POM_VERSION = sh(script: 'echo "2.5-SNAPSHOT"', returnStdout: true).trim()
Now the value of variable is the expected one
[Pipeline] echo
POM_VERSION '2.5-SNAPSHOT'
Jenkins version: 2.303.2
Related
Trying to add sql dbs created from azure portal to azure failover group.
Terraform block will call bash script :
data "external" "database_names" {
program = ["sh", "${path.module}/scripts/fetch_db_id.sh"]
query = {
db_rg = azurerm_resource_group.mssql.name
server_name = azurerm_mssql_server.mssqlserver.name
}
}
fetch_db_id.sh bash script:
#!/usr/bin/env bash
# This script will get the database names at runtime.
eval "$(jq -r '#sh "export DB_RG=\(.db_rg) SERVER_NAME=\(.server_name)"')"
if [[ -z $DB_RG || -z $SERVER_NAME ]]; then
echo "Required variables DB_RG & SERVER_NAME not set" 1>&2
exit 1
fi
db_id=$(az sql db list --resource-group $DB_RG --server $SERVER_NAME --query [*].id | grep -v master 2>/dev/null)
jq -n --arg db_id "$db_id" '{"db_id":$db_id}'
unset DB_RG SERVER_NAME NODE_RG db_id
exit 0
Bash script output : Ran it locally on a linux VM without terraform :
"db_id": "[\n "/subscriptions/my_subscription_id/resourceGroups/sql_rg/providers/Microsoft.Sql/servers/sql_server_name/databases/databaseprd-db1",\n "/subscriptions/my_subscription_id/resourceGroups/sql_rg/providers/Microsoft.Sql/servers/sql_server_name/databases/databaseprd-db2",\n "/subscriptions/my_subscription_id/resourceGroups/sql_rg/providers/Microsoft.Sql/servers/sql_server_name/databases/databaseprd-db3"\n]"
}
Terraform resource block to add database inside failover group:
resource "azurerm_sql_failover_group" "mssql_failover" {
count = (var.enable_read_replica && var.environment == "prd") ? 0 : 1
name = var.mssql_failover_group
resource_group_name = azurerm_resource_group.mssql.name
server_name = azurerm_mssql_server.mssqlserver.name
databases = toset(jsondecode(data.external.database_names.result["db_id"]))
partner_servers {
id = azurerm_mssql_server.replica[0].id
}
read_write_endpoint_failover_policy {
mode = "Automatic"
grace_minutes = 60
}
depends_on = [
azurerm_mssql_server.replica
]
}
terraform error code: when executed via terraform jenkins pipeline
[1mdata.external.database_names.result["db_id"][0m is "[\n "/subscriptions/my_subscription_id/resourceGroups/sql_rg/providers/Microsoft.Sql/servers/sql_server_name/databases/databaseprd-db1",\n "/subscriptions/my_subscription_id/resourceGroups/sql_rg/providers/Microsoft.Sql/servers/sql_server_name/databases/databaseprd-db2",\n "/subscriptions/my_subscription_id/resourceGroups/sql_rg/providers/Microsoft.Sql/servers/sql_server_name/databases/databaseprd-db3",\n]"
Call to function "jsondecode" failed: invalid character ']' looking for
beginning of value
Note: it is introducing an extra "," when we run it with terraform jenkins pipeline, which could lead to json error.
I don't think there's enough information here to be certain, but I can give a partial, speculative answer.
az sql db list --resource-group $DB_RG --server $SERVER_NAME --query [*].id | grep -v master 2>/dev/null
This looks suspicious: the az command outputs JSON, but you're filtering it with grep. What does the output from az look like here? Do you expect the result to be valid JSON?
You say that our output has a comma that you don't expect. This is what you'd expect to see if the az command spat out something like:
[
"/blah/blah/databaseprd-db1",
"/blah/blah/databaseprd-db2",
"/blah/blah/databaseprd-db3",
"/blah/blah/master"
]
The grep -v master would remove the line containing the term "master", leaving you with invalid JSON:
[
"/blah/blah/databaseprd-db1",
"/blah/blah/databaseprd-db2",
"/blah/blah/databaseprd-db3",
]
If you want to use jq, you could replace the grep with something like
jq 'map(select(index("master")|not))'
So I have a simple script running in an IF statement. I always get:
syntax error: unexpected end of file (expecting "fi")
I am wondering what could be the possible solution for this ?
def call(Map config) {
withCredentials([[$class: 'AmazonWebServicesCredentialsBinding', credentialsId: 'JENKINS_AWS'],
sshUserPrivateKey(credentialsId: 'JENKINS-SSH', keyFileVariable: 'SSH_KEY')]) {
sh """
#!/bin/bash
source add_ssh_key "${SSH_KEY}"
source init_env "${TARGET_STAGE}"
source create-bastion-pod "${PROMETHEUS_PUSHGATEWAY}" "${PROMETHEUS_PUSHGATEWAY_PORT}"
if [ ${TARGET_STAGE} == 'dev' ]; then
cat <<-EOF | curl --data-binary #- \${BASTION_URL}/metrics/job/sone_job
# TYPE some_metric counter
some_metric{label="val1"} 42
EOF
fi
delete-bastion-pod
"""
}
}
<<- only strips tabs from the here-document; your closing delimiter appears to be indented (according to what Groovy actually presents to the shell) with a couple of spaces. Try something like
sh """
#!/bin/bash
if [ ${TARGET_STAGE} == 'dev' ]; then
cat <<EOF | curl --data-binary #- \${BASTION_URL}/metrics/job/some_job
# TYPE some_metric counter
some_metric{label="val1"} 42
EOF
fi
"""
Note that as far as the shell executing the script is concerned, the here-document and the closing EOF aren't indented at all.
I have something like this on a Jenkinsfile (Groovy) and I want to record the stdout and the exit code in a variable in order to use the information later.
sh "ls -l"
How can I do this, especially as it seems that you cannot really run any kind of groovy code inside the Jenkinsfile?
The latest version of the pipeline sh step allows you to do the following;
// Git committer email
GIT_COMMIT_EMAIL = sh (
script: 'git --no-pager show -s --format=\'%ae\'',
returnStdout: true
).trim()
echo "Git committer email: ${GIT_COMMIT_EMAIL}"
Another feature is the returnStatus option.
// Test commit message for flags
BUILD_FULL = sh (
script: "git log -1 --pretty=%B | grep '\\[jenkins-full]'",
returnStatus: true
) == 0
echo "Build full flag: ${BUILD_FULL}"
These options where added based on this issue.
See official documentation for the sh command.
For declarative pipelines (see comments), you need to wrap code into script step:
script {
GIT_COMMIT_EMAIL = sh (
script: 'git --no-pager show -s --format=\'%ae\'',
returnStdout: true
).trim()
echo "Git committer email: ${GIT_COMMIT_EMAIL}"
}
Current Pipeline version natively supports returnStdout and returnStatus, which make it possible to get output or status from sh/bat steps.
An example:
def ret = sh(script: 'uname', returnStdout: true)
println ret
An official documentation.
quick answer is this:
sh "ls -l > commandResult"
result = readFile('commandResult').trim()
I think there exist a feature request to be able to get the result of sh step, but as far as I know, currently there is no other option.
EDIT: JENKINS-26133
EDIT2: Not quite sure since what version, but sh/bat steps now can return the std output, simply:
def output = sh returnStdout: true, script: 'ls -l'
If you want to get the stdout AND know whether the command succeeded or not, just use returnStdout and wrap it in an exception handler:
scripted pipeline
try {
// Fails with non-zero exit if dir1 does not exist
def dir1 = sh(script:'ls -la dir1', returnStdout:true).trim()
} catch (Exception ex) {
println("Unable to read dir1: ${ex}")
}
output:
[Pipeline] sh
[Test-Pipeline] Running shell script
+ ls -la dir1
ls: cannot access dir1: No such file or directory
[Pipeline] echo
unable to read dir1: hudson.AbortException: script returned exit code 2
Unfortunately hudson.AbortException is missing any useful method to obtain that exit status, so if the actual value is required you'd need to parse it out of the message (ugh!)
Contrary to the Javadoc https://javadoc.jenkins-ci.org/hudson/AbortException.html the build is not failed when this exception is caught. It fails when it's not caught!
Update:
If you also want the STDERR output from the shell command, Jenkins unfortunately fails to properly support that common use-case. A 2017 ticket JENKINS-44930 is stuck in a state of opinionated ping-pong whilst making no progress towards a solution - please consider adding your upvote to it.
As to a solution now, there could be a couple of possible approaches:
a) Redirect STDERR to STDOUT 2>&1
- but it's then up to you to parse that out of the main output though, and you won't get the output if the command failed - because you're in the exception handler.
b) redirect STDERR to a temporary file (the name of which you prepare earlier) 2>filename (but remember to clean up the file afterwards) - ie. main code becomes:
def stderrfile = 'stderr.out'
try {
def dir1 = sh(script:"ls -la dir1 2>${stderrfile}", returnStdout:true).trim()
} catch (Exception ex) {
def errmsg = readFile(stderrfile)
println("Unable to read dir1: ${ex} - ${errmsg}")
}
c) Go the other way, set returnStatus=true instead, dispense with the exception handler and always capture output to a file, ie:
def outfile = 'stdout.out'
def status = sh(script:"ls -la dir1 >${outfile} 2>&1", returnStatus:true)
def output = readFile(outfile).trim()
if (status == 0) {
// output is directory listing from stdout
} else {
// output is error message from stderr
}
Caveat: the above code is Unix/Linux-specific - Windows requires completely different shell commands.
this is a sample case, which will make sense I believe!
node('master'){
stage('stage1'){
def commit = sh (returnStdout: true, script: '''echo hi
echo bye | grep -o "e"
date
echo lol''').split()
echo "${commit[-1]} "
}
}
For those who need to use the output in subsequent shell commands, rather than groovy, something like this example could be done:
stage('Show Files') {
environment {
MY_FILES = sh(script: 'cd mydir && ls -l', returnStdout: true)
}
steps {
sh '''
echo "$MY_FILES"
'''
}
}
I found the examples on code maven to be quite useful.
All the above method will work. but to use the var as env variable inside your code you need to export the var first.
script{
sh " 'shell command here' > command"
command_var = readFile('command').trim()
sh "export command_var=$command_var"
}
replace the shell command with the command of your choice. Now if you are using python code you can just specify os.getenv("command_var") that will return the output of the shell command executed previously.
How to read the shell variable in groovy / how to assign shell return value to groovy variable.
Requirement : Open a text file read the lines using shell and store the value in groovy and get the parameter for each line .
Here , is delimiter
Ex: releaseModule.txt
./APP_TSBASE/app/team/i-home/deployments/ip-cc.war/cs_workflowReport.jar,configurable-wf-report,94,23crb1,artifact
./APP_TSBASE/app/team/i-home/deployments/ip.war/cs_workflowReport.jar,configurable-temppweb-report,394,rvu3crb1,artifact
========================
Here want to get module name 2nd Parameter (configurable-wf-report) , build no 3rd Parameter (94), commit id 4th (23crb1)
def module = sh(script: """awk -F',' '{ print \$2 "," \$3 "," \$4 }' releaseModules.txt | sort -u """, returnStdout: true).trim()
echo module
List lines = module.split( '\n' ).findAll { !it.startsWith( ',' ) }
def buildid
def Modname
lines.each {
List det1 = it.split(',')
buildid=det1[1].trim()
Modname = det1[0].trim()
tag= det1[2].trim()
echo Modname
echo buildid
echo tag
}
If you don't have a single sh command but a block of sh commands, returnstdout wont work then.
I had a similar issue where I applied something which is not a clean way of doing this but eventually it worked and served the purpose.
Solution -
In the shell block , echo the value and add it into some file.
Outside the shell block and inside the script block , read this file ,trim it and assign it to any local/params/environment variable.
example -
steps {
script {
sh '''
echo $PATH>path.txt
// I am using '>' because I want to create a new file every time to get the newest value of PATH
'''
path = readFile(file: 'path.txt')
path = path.trim() //local groovy variable assignment
//One can assign these values to env and params as below -
env.PATH = path //if you want to assign it to env var
params.PATH = path //if you want to assign it to params var
}
}
Easiest way is use this way
my_var=`echo 2`
echo $my_var
output
: 2
note that is not simple single quote is back quote ( ` ).
This question already has answers here:
How to increment version number in a shell script?
(14 answers)
Closed 8 years ago.
I use the following bash/shell script to semi-automate the git add/commit/push routine on my project:
git_push.sh
#!/bin/bash
# Mini Config
RColor='\e[0m'
Red='\e[0;31m';
Green='\e[0;32m';
Yellow='\e[0;33m'
# Change To Working Directory
clear;
cd $HOME/public_html/iwms_reboot;
# Get Git Commit Notes
echo -en "\r\n${Green}Enter commit notes: ${Yellow}";
read notes;
if [[ -z "$notes" ]]
then
echo -e "\r\n${Red}ERROR: You have not entered any git commit notes.${RColor}\r\n";
exit 0;
fi
echo -e "${RColor}";
# Git Add, Commit & Push
git add .;
git commit -m "${notes}";
echo -e "\r\n";
git push;
echo -e "\r\n";
This works perfectly fine.
I want to take this one step further. On my prject, there is a single file called version.php with the following lines of code:
<?php
// Script Version
$script_version = 'v1.0.5';
?>
My question is, is it possible to use bash/shell scripting to load this file's content and find the number after the 2nd period (.) and increment it by one?
i.e. v1.0.5 will become v1.0.6
This way, I can run this version number updating function before my (git add/commit/push) routine to implement an automatic minor version number update functionality on my project. I.e. script version number goes up automatically every time I commit.
If you want a 'pure-bash' solution, here it is...:
#!/bin/bash
new_version=''
increment_version_number () {
declare -a part=( ${1//\./ } )
declare new
declare -i carry=1
for (( CNTR=${#part[#]}-1; CNTR>=0; CNTR-=1 )); do
len=${#part[CNTR]}
new=$((part[CNTR]+carry))
[ ${#new} -gt $len ] && carry=1 || carry=0
[ $CNTR -gt 0 ] && part[CNTR]=${new: -len} || part[CNTR]=${new}
done
new="${part[*]}"
new_version="${new// /.}";
}
version=$(sed version.php -e "s/\$script_version = 'v//" | sed -e "s/';$//")
increment_version_number $version
echo $new_version;
UPDATE:
Code for a two digits version numbers (as requested in comment):
#!/bin/bash
new_version=''
increment_version_number () {
declare -a part=( ${1//\./ } )
declare new
declare -i carry=1
for (( CNTR=${#part[#]}-1; CNTR>=0; CNTR-=1 )); do
len=${#part[CNTR]}
new=$((part[CNTR]+carry))
[ ${#new} -gt $(($len+1)) ] && carry=1 || carry=0
part[CNTR]=${new}
done
new="${part[*]}"
new_version="${new// /.}";
}
version=$(grep "\$script_version" version.php | sed -e "s/\$script_version = 'v//" | sed -e "s/';$//")
increment_version_number $version
echo $new_version;
(warning: not fully tested code...)
Thanks to fredtantini and using the answer from How to increment version number in a shell script?, I have come up with the following solution to my original problem.
I first created a file called version.data and put the text 1.0.5 in it.
Then I updated my PHP script like so:
<?php
// Script Version
$script_version = 'v'. trim(file_get_contents(app_path() .'/version.data'));
?>
Then I have created the following gawk script called version_updater.sh (next to my git_push.sh script):
#!/usr/bin/gawk -f
BEGIN {
printf("%s", inc(ARGV[1]))
}
function inc(s, a, len1, len2, len3, head, tail)
{
split(s, a, ".")
len1 = length(a)
if(len1==0)
return -1
else if(len1==1)
return s+1
len2 = length(a[len1])
len3 = length(a[len1]+1)
head = join(a, 1, len1-1)
tail = sprintf("%0*d", len2, (a[len1]+1)%(10^len2))
if(len2==len3)
return head "." tail
else
return inc(head) "." tail
}
function join(a, x, y, s)
{
for(i=x; i<y; i++)
s = s a[i] "."
return s a[y]
}
Then I have updated my git_push.sh script like so:
#!/bin/bash
# Mini Config
RColor='\e[0m'
Red='\e[0;31m';
Green='\e[0;32m';
Yellow='\e[0;33m';
Source=$HOME/public_html/iwms_reboot;
# Increment Script Version
CurrentVersion=`cat "$Source/app/version.data"`;
NewVersion=`./version_updater.sh $CurrentVersion`;
# Change To Working Directory
clear;
cd $Source;
# Get Git Commit Notes
echo -en "\r\n${Green}Enter commit notes: ${Yellow}";
read notes;
if [[ -z "$notes" ]]
then
echo -e "\r\n${Red}ERROR: You have not entered any git commit notes.${RColor}\r\n";
exit 0;
fi
echo -e "${RColor}";
# Update Script Version
echo $NewVersion > $Source/app/version.data;
# Git Add, Commit & Push
git add .;
git commit -m "${notes}";
echo -e "\r\n";
git push;
echo -e "\r\n";
... and that's it, it works.
when the script runs, it reads the current version, passes it to the version_updater.sh to programatically increment it and put the return value into a variable.
Just before I commit, I update my version.data file with the updated version number. Now when I commit, I commit with the new version number.
file= input.txt
br=0
awk -v 'BEGIN{FS=";"}
{
for(i=1;i<=NF;i++)
{
print $i
br = NF
}
}'<$file
print "value of br " $br
I am storing the value of NF in br, so that i can use it further in script. In my case the value of NF is 10. but in br I am receiving 0.
You are confusing shell scripts and awk, which are different programs/interpreters.
What happens in awk stays in awk.
The best you can do is print a string to the shell.
I note that I did not need the -v tag, and substituted the -F tag for setting FS.
In your case you might use the END directive in awk, which only runs at the the end.
file= input.txt
br=0
gawk -F; '
{
for(i=1;i<=NF;i++)
{
print $i
br = NF
}
}
END { print "final value of br:"br } '<$file
If you need this number in a shell environment variable called br, you could do it this way:
file= input.txt
br=$( gawk -F; '
{
for(i=1;i<=NF;i++)
{
br = NF
}
}
END { print br } '<$file
)
How It Works
In bash shell
VARNAME=$( command )
runs the command and sets the environment variable VARNAME to the output from running the command.
Important: Use source instead of executing to set variables from shell scripts
Note that if you stick this in br.sh and chmod 700 br.sd, you might be tempted to run it with ./br.sh which will run it, but it will set shell variable $br in the resulting temporary child process, not the calling parent. The parent will have $br empty in that case. To get $br set in the parent, you would have to run the command file with source br.sh not by executing it directly.