I have a gitlab ci job, which runs the same command twice, but the command runs first time and second time differs in one parameter. How to create a parametrized command?
I am looking for something like this:
.long_terminal_command: &long_terminal_command
- echo $variable
main_job:
script:
- *long_terminal_command "One"
- *long_terminal_command "Two"
Related
I have a simple pipeline with one job to test bash scripts. The pipeline as follow:
image: alpine/git
stages:
- test_branching
test_branch:
stage: test_branching
before_script:
- mkdir -p .common
- wget https://x.x.x.x/branching.sh > .common/test.sh && chmod +x .common/test.sh
- source .common/test.sh
script:
- test_pipe
- echo "app version is ${app_version}"
The bash script as follow:
#!/bin/sh
function test_pipe () {
app_version="1.0.0.0-SNAPSHOT"
}
The problem is that the pipeline for whatever reason does not recognize the function inside the script. The logs are:
...
$ test_pipe
/scripts-1050-417479/step_script: eval: line 180: test_pipe: not found
Does anybody know what happend with this?? I miss a lot Jenkins shared libraries, gitlab does not have it, also gitlab does not have the function to include scripts inside yml files.
I dont want to use multiproject pipeline, I need to do it at this way. This is only an example of a more complicated pipeline logic.
Thanks in advance
As the documentation states before_script is just concatenated together with script and run on a single shell. The script you are downloading does not define test_pipe.
... gitlab does not have the function to include scripts inside yml
files.
It does, just use the YAML multiline literal syntax with |, e.g.:
script:
- |
echo "this"
echo "is"
echo "an \
example"
Here's what I am trying to do.
in .gitlab-ci:
Check schedules pass:
stage: check
image: ${myimage}
script:
- MY_CI_VAR=aVeryLongVariable
- echo "$MY_SCHEDULE_VAR"
In schedules:
Which is not working.
The reason I want to do this is for picking different variable (out of many in the job) on each schedule.
Yes, it is possible to use variables within other variables. This feature was released in GitLab 14.3.
However, since you are using GitLab 13.x, this feature won't be available to you.
You may be able to get around this limitation by using a static value and altering your job script accordingly.
myjob:
before_script: |
if [[ "$SCHEDULE_VAR" == "abc" ]]; then
export FOO="$MY_CI_VAR"
fi
# ...
In versions of GitLab < 14.3 you can still make use of other variables within variables, but instead by using $$ to preserve variables from evaluation.
Example from the docs:
variables:
FLAGS: '-al'
LS_CMD: 'ls "$FLAGS" $$TMP_DIR'
script:
- 'eval "$LS_CMD"' # Executes 'ls -al $TMP_DIR'
i am wondering whether or not one can set a title or name for an execution element in a bitbucket pipeline:
pipelines:
default:
- step:
script: # Modify the commands below to build your repository.
- "Configure": ./configure
- "Build": make
- "Test": make test
- "Long Script": |
make whatever1
make whatever2
make whatever3
I'd expect the output to be:
Configure
Build
Test
Long Script
within the titles, and seeing the script only, if I unfolding the execution elements in the UI, just like with github:
Any ideas? :-)
The only one I found was to put everthing in bash scripts, but then I do not see the executed command, which I still want.
Thanks.
Any step of your pipeline can have its own name property, a good example can be found here.
In case you'd like to assign names to individual commands of your step's script, I reckon echo would be a good option:
echo "Test" && make test
variable big_var_01 defined with value '3q4w#V$X3q4w#V$X' by following azure pipeline yaml file gets corrupted to become value '3q4w#V#V' when read back in azure pipeline template
cat parent_scott.yaml
variables:
- name: big_var_01
value: ${{ parameters.big_var_01 }}
parameters:
- name: big_var_01
displayName: "this var wants to get written to then read by templates"
type: string
default: '3q4w#V$X3q4w#V$X'
# CI Triggers
trigger:
branches:
exclude:
- '*'
pool:
vmImage: 'ubuntu-latest'
# Release Stages
stages:
- template: child_scott_one.yaml
following azure pipeline template variable big_var_01 is read back however its value is corrupted and does not match above assignment
cat child_scott_one.yaml
# Release Stages
stages:
- stage: A
jobs:
- job: JA
steps:
- script: |
echo "here is big_var_01 -->$(big_var_01)<-- "
local_var_01=$(big_var_01)
echo
echo "here is local_var_01 -->$local_var_01<-- "
echo
echo "length of local_var_01 is ${#local_var_01}"
echo
name: DetermineResult
see run of above pipeline
https://dev.azure.com/sekhemrekhutawysobekhotep/public_project/_build/results?buildId=525&view=logs&j=54e3124b-25ae-54f7-b0df-b26e1988012b&t=52fad91f-d6ac-51fb-b63d-00fda7898bb6&l=13
see code at https://github.com/sekhemrekhutawysobekhotep/shared_variables_across_templates
How to make the string variable big_var_01 get treated as a literal evidently its somehow getting evaluated and thus corrupted ... above code is a simplification of my actual azure pipeline where I get same variable corruption issue even when setting up a Key Value secret with value 3q4w#V$X3q4w#V$X which gets corrupted when read back in a pipeline template
here is another pipeline run which explicitly shows this issue https://dev.azure.com/sekhemrekhutawysobekhotep/public_project/_build/results?buildId=530&view=logs&j=ed5db508-d8c1-5154-7d4e-a21cef45e99c&t=a9f49566-82d0-5c0a-2e98-46af3de0d6e9&l=38 on this run I check marked ON pipeline run option: "Enable system diagnostics" ... next I will try to single quote my shell assignment from the azure variable
At some step Azure DevOps or Ubuntu replaced part of your string. So you have:
3q4w#V$X3q4w#V$X = 3q4w#V + $X3q4w + #V + $X
and this part $X3q4w and this $X was replaced with empty string giving you 3q4w#V + #V.
If you run this with \ before $ like here 3q4w#V\$X3q4w#V\$X
This is job Foo.
here is big_var_01 -->3q4w#V$X3q4w#V$X<--
here is local_var_01 -->3q4w#V$X3q4w#V$X<--
length of local_var_01 is 16
I got an error running this on windows-latest however I got correct string:
"This is job Foo."
"here is big_var_01 -->3q4w#V$X3q4w#V$X<-- "
'local_var_01' is not recognized as an internal or external command,
operable program or batch file.
ECHO is off.
"here is local_var_01 -->$local_var_01<-- "
ECHO is off.
"length of local_var_01 is ${#local_var_01}"
ECHO is off.
##[error]Cmd.exe exited with code '9009'.
so it looks like ubuntu replaces it with env variables however, since there is not variables like $X3q4w and $X it replaces with empty string.
Found a solution ... it works if I single quote around the azure variable during bash shell assignment
local_var_01=$(big_var_01) # bad results with value 3q4w#V#V
local_var_02="$(big_var_01)" # bad results with value 3q4w#V#V
local_var_03='$(big_var_01)' # good this gives value 3q4w#V$X3q4w#V$X
my instincts said to not use single quotes knowing its not the bash shell way however once I accepted the fact azure performs some magic ~~helping~~ interstigal layer pre processing in between pipeline source code and fully resolved bash shell execution I took a chance and tried this ... come to find out this is how bash shell blocks variable expansion
I am trying to build a Gitlab pipeline that is made up of 4 jobs. The stages I have are:
stages:
- compare
- build
- test
- deploy
The compare stage is taking a dump from an API on another server, comparing it to the same dump from the last successful pipeline run (it's made available as an artifact) then comparing the two.
If there is any difference I would like the pipeline to move onto the next stage, if there is no difference then I would like it to exit gracefully.
I have it working but rather than exiting gracefully if there are no differences it fails and the pipeline is marked as failed, here is how it looks.
Here is the important code from my .gitlab-ci.yaml (with some identifying information removed )
Get_inventory_dump:
stage: compare
only:
- schedules
script:
- 'curl -k --output "previous-inventory.json" --header "PRIVATE-TOKEN: $user_token" "https://url/to/get/artifact/from/last/successful/run"'
- python3 auto_config_scripts/dump_device_inventory_api_to_json.py -p $pass -o /inventory.json -u https://url/for/inventory/dump -y
- /usr/bin/cmp previous-inventory.json inventory.json && echo "No Change in inventory since last successful run" && exit 1 || echo "Inventory has changed since last run, continue" && exit 0
artifacts:
when: on_success
expire_in: 4 weeks
paths:
- inventory.json
Generate_icinga_config:
stage: build
only:
- schedules
when: on_success
script:
Everything is behaving as I would expect but I feel like there is a better way to do this.
Is there a way, if the comparison is the same to simply skip the next stages of the pipeline but still have the pipeline completed as 'passed' rather than 'failed'?
There are two solutions I can think of. Unfortunately, they either come slightly confusing UI behavior or you have to adapt all jobs.
Job attributes like only or changes are only concerned with the state of or the files of the git repository (see https://docs.gitlab.com/ee/ci/yaml/) and therefore not of use here as the file is only created during CI and not part of the repository.
Solution 1: You can allow_failure: true to the first job. This will mark the pipeline as successful despite the job failing and subsequent jobs will not be executed as the first job did not succeed. The drawback is that when you investigate the pipeline there will be an exclamation mark instead of a green check for this job.
Solution 2: Instead of failing the first job when there are no changes the inventory.json file is removed. And all subsequent jobs directly terminate with exit code 0 when the file doesn't exist. Note that this only works because inventory.json is marked as an artifact.
Based on Fzgregors suggestion, this is how I solved my problem:
If there was a difference and I wanted my second stage to actually do some work I created a file called "continue" and made it available as an artifact.
The second stage will look for that file and use an IF statement to decide if it should do something or just exit nicely
Get_inventory_dump:
stage: compare
only:
- schedules
script:
- 'curl -k --output "previous-inventory.json" --header "PRIVATE-TOKEN: $user_token" "https://url/to/get/artifact/from/last/successful/run"'
- python3 auto_config_scripts/dump_device_inventory_api_to_json.py -p $pass -o /inventory.json -u https://url/for/inventory/dump -y
- /usr/bin/cmp previous-inventory.json inventory.json && echo "No Change in inventory since last successful run" || echo "Inventory has changed since last run, continue" && touch continue
artifacts:
when: on_success
expire_in: 4 weeks
paths:
- inventory.json
- continue
Generate_icinga_config:
stage: build
only:
- schedules
when: on_success
script:
- if [[ -f continue ]]; then
do some stuff;
else
echo "No Change in inventory, nothing to do";
fi
This allowed me to keep my inventory artifact but at the same time let the next stage know if it needed to do some work or just do nothing and exit with code 0
I have a substantially similar construction and I'm looking for essentially the same solution.
When I allow_failure: true my subsequent jobs DO execute.
If I use a stamp file, all of the subsequent jobs also execute taking up queue, runners, etc, even though they aren't needed.
I was hoping for an easier solution but I think I'm going to have to go with generated yaml files. That seems to be the only way to inject dynamic information like decisions into a pipeline.