Uploading a file into Nexus from Jenkins pipeline is not working - groovy

This doesn't work:
withCredentials([usernamePassword(credentialsId: 'xxxxxxxxxxx', passwordVariable: 'ABCD', usernameVariable: 'XYZ')]) {
dir("build") {
sh "curl -u $XYZ:$ABCD --upload-file xyz.tar.gz https://mnpqr/repository/abc/$BRANCH_DIR/xyz.tar.gz"
}
}
Error:
/var/lib/jenkins/workspace/xyz/build#tmp/durable-71c2368a/script.sh: line 1: VG7cJ: No such file or directory
But this works:
withCredentials([usernamePassword(credentialsId: 'xxxxxxxxxxx', passwordVariable: 'ABCD', usernameVariable: 'XYZ')]) {
dir("build") {
sh 'curl -u $XYZ:$ABCD --upload-file xyz.tar.gz https://mnpqr/repository/abc/$BRANCH_DIR/xyz.tar.gz'
}
}
I want to interpolate more data into the sh script but I cannot as its failing with double quotes.

You have to be careful to distinguish the variable scopes:
environment variables set in your script
environment variables set by Jenkins processes
local variables not available in forked shell process
They all have to be handled in a different way when replacing them in your double-quoted string:
node {
stage('My stage') {
// local variable scope
varContent = "Variable content"
// environment variable set up in script
env.envContent = "Environment content"
// environment variable available in shell script
withEnv(["BRANCH_NAME=myBranch"]) {
sh("echo ${varContent} XX ${env.envContent} XX \${envContent} XX \${BRANCH_NAME} XX ${env.BRANCH_NAME}")
}
}
}
In the example you see all of the three types. Let's have a closer look at the shell command:
sh("echo ${varContent} XX ${env.envContent} XX \${envContent} XX \${BRANCH_NAME} XX ${env.BRANCH_NAME}")
${varContent} is a variable from local script scope it is replaced before the string is written to a temporary shell script
${env.envContent} and ${env.BRANCH_NAME} handle environment variables that have already been set beforehand, as if they were a "local scope" variable
\${envContent} and \${BRANCH_NAME} are the actual environment variables. The backslash escapes the dollar sign, and the shell script will contain shell variable placeholders ${envContent} and ${BRANCH_NAME} that will be replaced at shell script run time.
Running the above script will show the following output:

Related

how to write the correct pipline jenkins docker grovy node

I am rewriting my pipline in node, I need to understand how to perform a step with a gait in node now an error is coming from stage('Deploy')
node {
checkout scm
def customImage = docker.build("python-web-tests:${env.BUILD_ID}")
customImage.inside {
sh "python ${env.CMD_PARAMS}"
}
stage('Deploy') {
post {
always {
allure([
includeProperties: false,
jdk: '',
properties: [],
reportBuildPolicy: 'ALWAYS',
results: [[path: 'report']]
])
cleanWs()
}
}
}
and this is the old pipeline
pipeline {
agent {label "slave_first"}
stages {
stage("Создание контейнера image") {
steps {
catchError {
script {
docker.build("python-web-tests:${env.BUILD_ID}", "-f Dockerfile .")
}
}
}
}
stage("Running and debugging the test") {
steps {
sh 'ls'
sh 'docker run --rm -e REGION=${REGION} -e DATA=${DATA} -e BUILD_DESCRIPTION=${BUILD_URL} -v ${WORKSPACE}:/tmp python-web-tests:${BUILD_ID} /bin/bash -c "python ${CMD_PARAMS} || exit_code=$?; chmod -R 777 /tmp; exit $exit_code"'
}
}
}
post {
always {
allure([
includeProperties: false,
jdk: '',
properties: [],
reportBuildPolicy: 'ALWAYS',
results: [[path: 'report']]
])
cleanWs()
}
}
}
I tried to transfer the method of creating an allure report, but nothing worked, I use the version above, almost everything turned out, you can still add environment variables to the build, for example, those that are specified -e DATA=${DATA} how do I add it
I don't recommend to switch from declarative to scriptive pipeline.
You are losing possibility to use multiple tooling connected with declarative approach like syntax checkers.
If you still want to use scriptive approach try this:
node('slave_first') {
stage('Build') {
checkout scm
def customImage = docker.build("python-web-tests:${env.BUILD_ID}")
customImage.inside {
sh "python ${env.CMD_PARAMS}"
}
}
stage('Deploy') {
allure([
includeProperties: false,
jdk: '',
properties: [],
reportBuildPolicy: 'ALWAYS',
results: [[path: 'report']]])
cleanWs()
}
}
There is no post and always directive in scriptive pipelines. It's on your head to catch all exceptions and set status of the job. I guess you were using this page: https://www.jenkins.io/doc/book/pipeline/syntax/, but it's a mistake.
This page only refers to declarative approach and in few cases you have hidden scriptive code as examples.
Also i don't know if you have default agent label set in your Jenkins config, but by looking at your declarative one I think you missed 'slave_first' arg in node object.
those that are specified -e DATA=${DATA} how do I add it
That's a docker question not a Jenkins. If you want to launch docker image and then also have access to some reports located in this container you should mount workspace/file where those output files landed. You should also pass location of those files to allure.
I suggest you to try this:
mount some subfolder in workspace to docker container
cat test report file if it's visible
add allure report with passing this file location to allure step

Jenkins pipeline - Set environment variable in nodejs code

I have a Jenkins pipeline which is executing few nodejs files like below -
stage('validate_paramters') {
steps {
sh 'node ${WORKSPACE}/file1.js'
}
}
stage('test') {
steps {
sh 'node ${WORKSPACE}/file2.js'
}
}
How can I set variables in file1 which can be accessed inside file2? I tried below approach but its giving undefined as value -
file1.js -
process.env['OPERATIONS'] = "10"
file2.js -
var operations = process.env.OPERATIONS

Unable to interpolate sensitive environment variables

I have a piece of code that runs like this
package core.jenkins
class Utils implements Serializable {
def script
Utils(script) {
this.script = script
}
def func() {
script.withCredentials([script.usernamePassword(credentialsId: 'chartmuseum-basic-auth', usernameVariable: 'USER', passwordVariable: 'PASSWORD')]) {
script.sh "helm repo add --username script.USER} --password ${script.PASSWORD} chartmuseum \"http://${chartmuseumHostname}:8080\""
}
}
The above works perfectly fine but I do not a warning
Warning: A secret was passed to "sh" using Groovy String interpolation, which is insecure.
Affected argument(s) used the following variable(s): [PASSWORD, USER]
See https://jenkins.io/redirect/groovy-string-interpolation for details.
+ helm repo add --username **** --password **** chartmuseum http://apps-chartmuseum.apps.svc.cluster.local:8080
So following the guide, Im doing the following
script.withCredentials([script.usernamePassword(credentialsId: 'chartmuseum-basic-auth', usernameVariable: 'USER', passwordVariable: 'PASSWORD')]) {
script.sh 'helm repo add --username $script.USER --password $script.PASSWORD chartmuseum "http://$chartmuseumHostname:8080"'
}
But running the variable values are not be properly substitured and I get
+ helm repo add --username .USER --password .PASSWORD chartmuseum http://:8080
Error: Looks like "http://:8080" is not a valid chart repository or cannot be reached: Get http://:8080/index.yaml: dial tcp :8080: connect: connection refused
So neither the credentials nor the value of the chartmuseumHostname variable is being substituted correctly. What am I missing here ?
Actuall withCredentials() creates a environment variable which you can access it from shell scripts.
See here: https://www.jenkins.io/doc/pipeline/steps/credentials-binding/
Try using directly the shell variables:
script.sh 'helm repo add --username $USER --password $PASSWORD chartmuseum "http://$chartmuseumHostname:8080"'
Just binding together the answers already on this post, the withCredentials makes it so that you should be able to use the variables directly (answer by #catalin), the single quotes make it so that jenkins should stop complaining about security and if you want to be extra careful, you can double quote the variable values as suggested in the docs for withCredentials.
This should give you something like this:
script.withCredentials([script.usernamePassword(credentialsId: 'chartmuseum-basic-auth',
usernameVariable: 'USER',
passwordVariable: 'PASSWORD')]) {
script.sh 'helm repo add --username "$USER" --password "$PASSWORD" chartmuseum "http://$chartmuseumHostname:8080"'
}
which still leaves us with the question of why you are calling things with the script. prefix as mentioned in the comments by #matt-schuchard.
I try using the suggestion by #Catalin (https://www.jenkins.io/doc/pipeline/steps/credentials-binding/ using directly the shell variables)
But for me adding double quotes inside single quotes doesn't work.
The only solution I found is taking the variables out of the single quotes like:
'myscript $secretvariable' + notsecretvariable
Examples:
Test1: Try using recommended solution (jenkins/#catalin)
Code:
sh label: 'Test1', script: 'echo this is a secret $docker_pwd this is not "$dockerRegistry"'
Result: variable dockerRegistry is not interpolated/resolved
15:50:43 [Pipeline] sh (Test1)
15:50:43 + echo this is a secret **** this is not ''
15:50:43 this is a secret **** this is not
Test2: Take non-sensitive variable out of the single quotes:
sh label: 'Test2', script: 'echo this is a secret $docker_pwd this is not' + dockerRegistry
Result: variable dockerRegistry is properlly resolved
15:50:44 [Pipeline] sh (Test2)
15:50:44 + echo this is a secret **** this is not my.repositories.xx
15:50:44 this is a secret **** this is not my.repositories.xx

How to access value of a jenkins groovy variable in shell script for loop

When i am passing value of a variable declared in jenkins Groovy script its value is not retained in for loop which is running on a remote server. Strange thing is i am able to access the same value outside the for loop.
Here is the sample code i am trying to use
#!/usr/bin/env groovy
def config
def COMMANDS_TO_CHECK='curl grep hello awk tr mkdir bc'
pipeline {
agent {
label "master"
}
stages {
stage ('Validation of commands') {
steps {
script {
sh """
#!/bin/bash
/usr/bin/sshpass -p passwrd ssh user#host << EOF
hostname
echo $COMMANDS_TO_CHECK ---> This is printed
for CURRENT_COMMAND in \$COMMANDS_TO_CHECK
do
echo ${CURRENT_COMMAND} ---> Why This is not printed?
echo \${CURRENT_COMMAND} ----> Why This is not printed?
done
hostname
EOF
exit
"""
}
}
}
}
}
Output
[workspace#3] Running shell script
+ /usr/bin/sshpass -p passwrd ssh user#host
Pseudo-terminal will not be allocated because stdin is not a terminal.
illinsduck01
curl grep hello awk tr mkdir bc
illinsduck01
+ exit
You can wrap sh in """ ... """ as below
#!/usr/bin/env groovy
def config
pipeline {
agent {
label "master"
}
stages {
stage ('Validation of commands') {
steps {
script {
sh """#!/bin/sh
/usr/bin/sshpass -p password ssh username#hostname << EOF
COMMANDS_TO_CHECK="curl grep hello awk tr mkdir bc"
hostname
echo \$COMMANDS_TO_CHECK
for CURRENT_COMMAND in \$COMMANDS_TO_CHECK
do
echo \$CURRENT_COMMAND
which \$CURRENT_COMMAND
status=\$?
if [ \${status} -eq 0 ]
then
echo \${CURRENT_COMMAND} command is OK
else
echo "Failed to find the \${CURRENT_COMMAND} command"
fi
done
hostname
EOF
exit
"""
}
}
}
}
}

replacing file variables by envsubst in jenkins pipeline

I want to replace some variables in a file having $variablename, at runtime from jenkins pipeline script. It seems envsubst is the best for my use case. When i execute by command line on linux server its working fine but when i'm executing through jenkins pipeline in sh script, nothing happens.
sonar-scanner.properties:
sonar.projectKey=Project:MavenTest$BRANCHNAME
sonar.projectName=MavenTest$BRANCHNAME
Example of Command line on linux box:
$ export BRANCHNAME=develop
$ envsubst '$BRANCHNAME'
Output:
sonar.projectKey=Project:MavenTestdevelop
sonar.projectName=MavenTestdevelop
But when i'm executing through jenkins file as a script, nothing is changed in file.
jenkins script:
node {
stage('checkout'){
checkout([$class: 'GitSCM', branches: [[name: ':^(?!origin/master$|origin/develop$).*']], doGenerateSubmoduleConfigurations: false, extensions: [], submoduleCfg: [], userRemoteConfigs: [[credentialsId: 'c0ce73db-3864-4360-9c17-d87caf8a9ea5', url: 'http://172.16.4.158:17990/scm/ctoo/testmaven.git']]])
}
stage('initialize variables'){
// Configuring BRANCH_NAME variable
sh 'git name-rev --name-only HEAD > GIT_BRANCH'
sh label: '', script: 'cut -d \'/\' -f 3 GIT_BRANCH > BRANCH'
branchname = readFile('BRANCH').trim()
env.BRANCHNAME = branchname
}
stage('build & SonarQube analysis') {
withSonarQubeEnv('Sonar') {
sh "envsubst '$BRANCHNAME' <sonar-scanner.properties"
}
}
}
Output:
[Pipeline] sh (hide)
envsubst repotest
sonar.projectKey=Project:MavenTest$BRANCHNAME
sonar.projectName=MavenTest$BRANCHNAME
Can someone please help me
Hi I don't have idea about the envbust but this can be achieved by passing sonar parameters via command line to the sonar see the below example:
withSonarQubeEnv('Sonar') {
sh "<sonarscanner path> -Dsonar.projectKey=Project:MavenTest$BRANCHNAME"
}
I had this problem and solved it by using his escape character
\,
for example:
sh "envsubst '\${SERVER_NAME}' < ./config/nginx/nginx.conf.template > ./config/nginx/nginx.conf"

Resources