GitLab CI syntax to write FOR loop statement? - gitlab

Below is the script mentioned in the gitlab-ci.yml file. This GitLab CI configuration is valid. But, when the CI/CD build is run, the job fails. Is it something to do with the FOR loop syntax?
deploy_dv:
stage: deploy_dv
variables:
GIT_STRATEGY: none
script:
- echo "Deploying Artifacts..."
- echo "Configure JFrog CLI with parameters of your Artifactory instance"
- 'c:\build-tools\JFROG-CLI\jfrog rt config --url %ARTIFACTORY_WEBSITE% --user %ARTIFACTORY_USER% --apikey %APIKEY%'
- 'cd ..\artifacts'
- 'SETLOCAL ENABLEDELAYEDEXPANSION'
- FOR %%i in (*) do (
'c:\build-tools\curl\bin\curl.exe --header "PRIVATE-TOKEN:%HCA_ACCESS_TOKEN%" --insecure https://code.example.com/api/repository/tags/%CI_COMMIT_TAG% | c:\build-tools\jq\jq-win64.exe ".release.description" > temp.txt'
'set /p releasenote=<temp.txt'
'rem del temp.txt'
'set mydate=%DATE:~6,4%-%DATE:~3,2%-%DATE:~0,2%'
'c:\build-tools\JFROG-CLI\jfrog rt u "%%i" %ARTIFACTORY_ROOT_PATH%/%PROJECT_NAME%/%%i --build-name=%%i --build-number=%BUILDVERSION% --props releasenote=%releasenote%;releaseversion=%BUILDVERSION%;releasedate=%mydate% --flat=false'
)
- '%CURL% -X POST -F token=%REPOSITORY_TOKEN% -F ref=master -F "variables[RELEASE]=false" -F "variables[PROGRAM]=test" --insecure https://code.example.com/api/repository/trigger'
only:
- /^(dv-)(\d+\.)(\d+\.)(\d+)$/
I get this below error:
$ echo "Deploying Artifacts..."
"Deploying Artifacts..."
$ echo "Configure JFrog CLI with parameters of your Artifactory instance"
"Configure JFrog CLI with parameters of your Artifactory instance"
$ c:\build-tools\JFROG-CLI\jfrog rt config --url %ARTIFACTORY_WEBSITE% --user %ARTIFACTORY_USER% --apikey %APIKEY%
Artifactory server ID [Default-Server]: $ cd ..\artifacts
$ SETLOCAL ENABLEDELAYEDEXPANSION
$ FOR %%i in (*) do ( 'c:\build-tools\curl\bin\curl.exe --header "PRIVATE-TOKEN:%HCA_ACCESS_TOKEN%" --insecure https://code.example.com/api/repository/tags/%CI_COMMIT_TAG% | c:\build-tools\jq\jq-win64.exe ".release.description" > temp.txt' 'set /p releasenote=<temp.txt' 'rem del temp.txt' 'set mydate=%DATE:~6,4%-%DATE:~3,2%-%DATE:~0,2%' 'c:\build-tools\JFROG-CLI\jfrog rt u "%%i" %ARTIFACTORY_ROOT_PATH%/%PROJECT_NAME%/%%i --build-name=%%i --build-number=%BUILDVERSION% --props releasenote=%releasenote%;releaseversion=%BUILDVERSION%;releasedate=%mydate% --flat=false' )
The filename, directory name, or volume label syntax is incorrect.
ERROR: Job failed: exit status 255

Since there is still no good answer to this question, I will give it a try. I used this snippet to start multiple Docker builds for every directory in my repository. Notice the |+ and the > characters, which lets you put multi-line commands in YAML and are part of GitLab syntax.
Linux example:
build:
stage: loop
script:
- |+
for i in $(seq 1 3)
do
echo "Hello $i"
done
Windows example:
build:
stage: loop
script:
- >
setlocal enabledelayedexpansion
for %%a in ("C:\Test\*.txt") do (
set FileName=%%~a
echo Filename is: !FileName!
)
endlocal

Here is a working example of a job in a .gitlab-ci with a loop running on GNU/Linux OS and using Sh/Bash shell :
edit:
stage: edit
script:
- for file in $(find ${CI_PROJECT_DIR} -type f -name deployment.yml)
do
CURRENT_IMAGE=$(grep "image:" $file | cut -d':' -f2- | tr -d '[:space:]' | cut -d':' -f3)
sed -ie "s/$CURRENT_IMAGE/$VERSION/g" "$file"
done
only:
- master
I'm not an expert on Gitlab-Runner on Windows but Windows Batch is default shell used but you can also use Powershell.

In .gitlab.yml anything you write under "script" is shell. Thus for loop will be same as it works in shell script.
for var in ${NAME_1} ${NAME_2} ${NAME_3} ; do
*----computations----*
done

Related

Use multiline string in Gitlab release-cli description

I want to create a release in the gitlab-cli with a multiline description but when add one by yaml specs I get the following error:
Executing "step_release" stage of the job script
/bin/sh: eval: line 150: syntax error: unexpected newline
$ release-cli create --name "Release $CI_COMMIT_TAG" --description "Release created using the release-cli. # collapsed multi-line command
Cleaning up project directory and file based variables
ERROR: Job failed: exit code 2
Job:
create release:
...
release:
tag_name: $CI_COMMIT_TAG
name: 'Release $CI_COMMIT_TAG'
description: |-
Release created using the release-cli.
Maven dependency
```xml
<dependency>
<groupId>group</groupId>
<artifactId>artifact-id</artifactId>
<version>$CI_COMMIT_TAG</version>
</dependency>
```
(also tried with > and |)
I tried it manually with \n but that is literally added in the release as
Release created using the release-cli.\n\n ...
How do I create a release via the release-cli with a multiline (markdown) description?
I found a dirty unreadable workaround but it works.
(Note that multiline echo doesn't work either)
create release:
...
image: registry.gitlab.com/gitlab-org/release-cli:latest
rules:
script:
- echo "Running the release job for version '$CI_COMMIT_TAG'."
- set -e
- |
echo "Release created using the release-cli." > release_notes.md
echo "" >> release_notes.md
echo "Maven dependency:" >> release_notes.md
echo "\`\`\`xml" >> release_notes.md
echo "<dependency>" >> release_notes.md
echo " <groupId>group</groupId>" >> release_notes.md
echo " <artifactId>artifact-id</artifactId>" >> release_notes.md
echo " <version>$CI_COMMIT_TAG</version>" >> release_notes.md
echo "</dependency>" >> release_notes.md
echo "\`\`\`" >> release_notes.md
release:
tag_name: $CI_COMMIT_TAG
name: 'Release $CI_COMMIT_TAG'
description: release_notes.md
Still searching for a better solution.

Gitlab CI CD Pipeline Variable Concat Two Variables Not Working

The Problem: I am trying to concat two variables for a copy cmd in a before script for a gitlab ci/cd pipeline job.
What I expect: myfile_filesuffix
What I get: _filesuffix
Can anyone see what I am doing wrong? When I run this for loop on my local CLI I have no problems. Thank you!
before_script:
- rm -rf .terraform
- terraform --version
- mkdir ~/.aws
- echo "[default]" > ~/.aws/credentials
- echo "aws_access_key_id=$AWS_ACCESS_KEY_ID" >> ~/.aws/credentials
- echo "aws_secret_access_key=$AWS_SECRET_ACCESS_KEY" >> ~/.aws/credentials
- mkdir ./deployments
- ls common
- common_files=$(find common -type f)
- echo $common_files
- prefix_common=$(echo $common_files | cut -d"/" -f 1)
- echo $prefix_common
- for f in $common_files;
do
common_file="$(basename $f)"
cp $f ./deployments/""${common_file}"_"${prefix_common}"";
done
you can used GitLab repo settings -> CI/CD -> Variables to add FILE type variable and use mv command move to your folder.
ex: File Type variable is ANSIBLE_VARIABLES_FILE
script:
- mv $ANSIBLE_VARIABLES_FILE ./deployments/variables_common.tf

How to set system path variable in github action workflow

I was wondering how I can set the system path variables in the GitHub actions workflow.
export "$PATH:$ANYTHING/SOMETHING:$AA/BB/bin"
You can use the following run command to set a system path variable in your actions workflow.
Syntax:
echo "{path}" >> $GITHUB_PATH
- run: |
echo "$AA/BB/bin" >> $GITHUB_PATH
Additionally, if you have downloaded some binaries and trying to set its path, GitHub uses a special directory called $GITHUB_WORKSPACE as your current directory. You may need to specify this variable in your path in that case.
- run: |
echo "$GITHUB_WORKSPACE/BB/bin" >> $GITHUB_PATH
If you are using Bash shell
- name: Add to PATH
shell: bash
run: |
echo "Folder PATH" >> $GITHUB_PATH
For Powershell as a shell:
- name: Add to PATH
shell: pwsh
run: |
echo "Folder PATH" | Out-File -FilePath $env:GITHUB_PATH -Encoding utf8 -Append

Curl Command in Shell Script in Azure Pipeline not working

I'm trying to run a script which provides the status code of some url, using azure pipeline.
My Azure Yaml file:
pool:
vmImage: ubuntu-16.04
steps:
- script: echo Hello
displayName: ' Welcome'
- script: cat webpages.txt
displayName: 'display file'
- script: curl -s -w '%{http_code}\n' -o /dev/null https://www.google.co.in
displayName: 'Checking Curl Code'
- script: cat -v script.sh
displayName: 'Cariage retrun'
- task: ShellScript#2
inputs:
scriptPath: script.sh
My Script.sh file
#!/bin/sh
while read line ; do echo "$line - `curl -s -w '%{http_code}\n' -o /dev/null $line`" ;done < webpages.txt
webpages.txt file
https://www.vodafone.co.uk/good-stuff
https://www.vodafone.co.uk/help-and-information/cancel-your-account
https://www.vodafone.co.uk/help-and-information/complaints
https://www.vodafone.co.uk/help-and-information/complaints/code-of-practice
https://www.vodafone.co.uk/help-and-information/costs-and-charges
https://www.vodafone.co.uk/help-and-information/costs-and-charges/call-and-text-charges
https://www.vodafone.co.uk/help-and-information/costs-and-charges/data-charges
Problem
When I run my pipeline, the curl command is not working
and output comes as
2020-11-17T09:59:05.4094324Z ##[section]Starting: ShellScript
2020-11-17T09:59:05.4102534Z ==============================================================================
2020-11-17T09:59:05.4102904Z Task : Shell script
2020-11-17T09:59:05.4103204Z Description : Run a shell script using Bash
2020-11-17T09:59:05.4103453Z Version : 2.165.2
2020-11-17T09:59:05.4103701Z Author : Microsoft Corporation
2020-11-17T09:59:05.4104086Z Help : https://learn.microsoft.com/azure/devops/pipelines/tasks/utility/shell-script
2020-11-17T09:59:05.4104506Z ==============================================================================
2020-11-17T09:59:05.7637654Z [command]/bin/bash /home/vsts/work/1/s/script.sh
2020-11-17T09:59:05.7692023Z https://www.vodafone.co.uk/good-stuff
2020-11-17T09:59:05.7700780Z - 000
2020-11-17T09:59:05.7797276Z https://www.vodafone.co.uk/help-and-information/cancel-your-account
2020-11-17T09:59:05.7798378Z - 000
2020-11-17T09:59:05.7851183Z https://www.vodafone.co.uk/help-and-information/complaints
2020-11-17T09:59:05.7866944Z - 000
2020-11-17T09:59:05.7908420Z https://www.vodafone.co.uk/help-and-information/complaints/code-of-practice
2020-11-17T09:59:05.7909144Z - 000
2020-11-17T09:59:05.7967261Z https://www.vodafone.co.uk/help-and-information/costs-and-charges
2020-11-17T09:59:05.7967920Z - 000
2020-11-17T09:59:05.8023329Z https://www.vodafone.co.uk/help-and-information/costs-and-charges/call-and-text-charges
2020-11-17T09:59:05.8024443Z - 000
2020-11-17T09:59:05.8095527Z https://www.vodafone.co.uk/help-and-information/costs-and-charges/data-charges
but if I replace my curl with
curl -s -w '%{http_code}\n' -o /dev/null https://www.vodafone.co.uk/good-stuff
it gives the output at 200.
When I'm running this locally it works perfectly:
$ while read line ; do echo "$line - `curl -s -w '%{http_code}\n' -o /dev/null $line`" ;done < webpages.txt
https://www.vodafone.co.uk/good-stuff - 200
https://www.vodafone.co.uk/help-and-information/cancel-your-account - 200
(...)
I notice that a \n character is printed in your echo $line, that's probably causing the issue. What could solve it, is:
replacing $line by ${line},
replacing echo by echo -n to omit newlines.
So this is what I did,
I replaced the whole curl command with a simple curl $line
It gave me an error : curl: (3) Illegal characters found in URL
So, I figured my URL is adding some unrequired values.
I replaced my do with
line=${line%$'\r'}
while read line ; do line=${line%$'\r'} ; echo "$line - `curl -s -w '%{http_code}\n' -o /dev/null $line`"; done < webpages.txt
and Voila it works!

How do I check if a string containing variables is present in a specific file in bash?

Background
I have two files:
~/.bash.local # $LOCAL_BASH_CONFIG_FILE
~/.fish.local # $LOCAL_FISH_CONFIG_FILE
I am dynamically adding configurations to each file within a bash script. Each file is parsed by either ~/.bash_profile if it's for bash or ~/.config/fish/config.fish if it's for fish. to implement these configurations.
However, I do not want to add the lines that configure a binary to either file every time I run my bash script. I would like it to do the following:
Check if the string (e.g. BASH_CONFIG or FISH_CONFIG) is contained within each specified file
If it isn't contained within the file, printf it into the designated file.
If it is contained within the file, then do not add it again.
My below script's purpose is to install the npm package n using n-install and add the proper configuration to both bash and fish.
execute is a function that I defined to run the task in the background and display a message on the screen with a spinner.
Current Bash Script Snippet
add_n_configs() {
# bash
declare -r BASH_CONFIGS="
# n - Node version management.
export N_PREFIX=\"\$HOME/n\";
[[ :\$PATH: == *\":\$N_PREFIX/bin:\"* ]] || PATH+=\":\$N_PREFIX/bin\"
"
execute \
"printf '%s\n' '$BASH_CONFIGS' >> $LOCAL_BASH_CONFIG_FILE \
&& . $LOCAL_BASH_CONFIG_FILE" \
"n (update $LOCAL_BASH_CONFIG_FILE)"
# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
# fish
declare -r FISH_CONFIGS="
# n - Node version management.
set -xU N_PREFIX \"\$HOME/n\"
set -U fish_user_paths \"\$N_PREFIX/bin\" \$fish_user_paths
"
execute \
"printf '%s\n' '$FISH_CONFIGS' >> $LOCAL_FISH_CONFIG_FILE" \
"n (update $LOCAL_FISH_CONFIG_FILE)"
}
After a bit of research it seems that this task can be accomplished using grep as so:
if ! grep -q "$BASH_CONFIGS" "$LOCAL_BASH_CONFIG_FILE"; then
execute \
"printf '%s\n' '$BASH_CONFIGS' >> $LOCAL_BASH_CONFIG_FILE \
&& . $LOCAL_BASH_CONFIG_FILE" \
"n (update $LOCAL_BASH_CONFIG_FILE)"
fi
if ! grep -q "$FISH_CONFIGS" "$LOCAL_FISH_CONFIG_FILE"; then
execute \
"printf '%s\n' '$FISH_CONFIGS' >> $LOCAL_FISH_CONFIG_FILE" \
"n (update $LOCAL_FISH_CONFIG_FILE)"
fi
grep is a command-line utility that can search and filter text using a common regular expression syntax.
To search for an exact string: grep search_string path/to/file
To search for an exact string in quiet mode: grep -q search_string path/to/file
grep -q will only search a file until a match has been found.

Resources