Problem
I have a GitLab job in a stage that will prompt for a manual trigger. I only want this job to be able to be manually run if an automatically run job shows that migrations are needed
Example
As an example, here job1 looks for a word using grep. I want to have job2 be able to be manually triggered only if the grep finds results but skipped otherwise.
job1:
stage: .pre
script:
- grep "bob_and_alice" file.txt
job2:
stage: run-after
script:
- echo "Bob and Alice were not in this story"
when: manual
Related
Consider this .gitlab-ci.yml:
variables:
var1: "bob"
var2: "bib"
job1:
script:
- "[[ ${var1} == ${var2} ]]"
job2:
script:
- echo "hello"
after_script:
- "[[ ${var1} == ${var2} ]]"
In this example, job1 fails as expected but job2 succeeds, incomprehensibly. Can I force a job to fail in the after_script section?
Note: exit 1 has the same effect as "[[ ${var1} == ${var2} ]]".
The status of a job is determined solely by its script:/before_script: sections (the two are simply concatenated together to form the job script).
after_script: is a completely different construct -- it is not part of the job script. It is mainly for taking actions after a job is completed. after_script: runs even when jobs fail beforehand, for example.
Per the docs: (emphasis added on the last bullet)
Scripts you specify in after_script execute in a new shell, separate from any before_script or script commands. As a result, they:
Have the current working directory set back to the default (according to the variables which define how the runner processes Git requests).
Don’t have access to changes done by commands defined in the before_script or script, including:
Command aliases and variables exported in script scripts.
Changes outside of the working tree (depending on the runner executor), like software installed by a before_script or script script.
Have a separate timeout, which is hard-coded to 5 minutes.
Don’t affect the job’s exit code. If the script section succeeds and the after_script times out or fails, the job exits with code 0 (Job Succeeded).
I have written a github workflow file. I want to run a python program in github actions to validate few changes. I have one environment.yml file which contains all conda environment dependencies required by this program. The thing is, actual program is not running at all, and my workflow is completed with success.
Following is jobs section of workflow.yml file
jobs:
build-linux:
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout#v2
with:
ref: refs/pull/${{ github.event.pull_request.number }}/merge
- name: Set up Python 3.8
uses: actions/setup-python#v2
with:
python-version: 3.8
- name: Cache conda
uses: actions/cache#v2
env:
# Increase this value to reset cache if etc/example-environment.yml has not changed
CACHE_NUMBER: 0
with:
path: ~/conda_pkgs_dir
key:
${{ runner.os }}-conda-${{ env.CACHE_NUMBER }}-${{hashFiles('**/environment.yml') }}
- uses: conda-incubator/setup-miniconda#v2
with:
activate-environment: test-env
environment-file: environment.yml
use-only-tar-bz2: true # IMPORTANT: This needs to be set for caching to work properly!
- name: Test
run: |
export PATH="./:$PATH"
conda init bash
exec bash
conda activate test-env
echo "Conda prefix: $CONDA_PREFIX"
python test.py
shell: bash
I also tried removing shell:bash in the last step, but this is also giving me the same result.
The logs in last step looks like this:
Run export PATH="./:$PATH"
export PATH="./:$PATH"
conda init bash
exec bash
conda activate test-env
echo "Conda prefix: $CONDA_PREFIX"
python test.py
shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
env:
pythonLocation: /opt/hostedtoolcache/Python/3.8.11/x64
LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.8.11/x64/lib
CONDA_PKGS_DIR: /home/runner/conda_pkgs_dir
no change /usr/share/miniconda/condabin/conda
no change /usr/share/miniconda/bin/conda
no change /usr/share/miniconda/bin/conda-env
no change /usr/share/miniconda/bin/activate
no change /usr/share/miniconda/bin/deactivate
no change /usr/share/miniconda/etc/profile.d/conda.sh
no change /usr/share/miniconda/etc/fish/conf.d/conda.fish
no change /usr/share/miniconda/shell/condabin/Conda.psm1
no change /usr/share/miniconda/shell/condabin/conda-hook.ps1
no change /usr/share/miniconda/lib/python3.9/site-packages/xontrib/conda.xsh
no change /usr/share/miniconda/etc/profile.d/conda.csh
modified /home/runner/.bashrc
==> For changes to take effect, close and re-open your current shell. <==
As we can clearly see, the line echo "Conda prefix: $CONDA_PREFIX" is not getting executed at all, and the workflow terminates with success. We should expect it to either run or fail the job, but nothing happens. The workflow simply ignores these commands and marks the workflow as success.
Your CI script contains the line:
exec bash
When this line is executed, the shell process is replaced with a new one, and the new shell process has no idea it should continue executing the script the previous process was: all the execution state is lost. GitHub Actions passes the script to execute as a command-line argument to the initial shell process and sets standard input to /dev/null; as the new shell process is started with an empty command line and an empty file on standard input, it simply exits immediately. The fact that this works well with an interactive shell is something of a lucky coincidence.
The reason the installer directs you to restart your shell is to apply the environment variable changes added to the shell’s initialisation file. As such, it should probably be enough to replace the exec bash line with
source "$HOME/.bashrc"
However, even with this line the environment modifications will not be applied in subsequent steps, as the documentation of the setup-miniconda action warns:
Bash shells do not use ~/.profile or ~/.bashrc so these shells need to be
explicitely declared as shell: bash -l {0} on steps that need to be properly
activated (or use a default shell). This is because bash shells are executed
with bash --noprofile --norc -eo pipefail {0} thus ignoring updated on bash
profile files made by conda init bash. See
Github Actions Documentation
and
thread.
Based on this advice, I think the best course of action is to end the actions step at the point where you would put exec bash, and apply the shell: setting to all further steps (or at least those which actually need it):
- name: Set up Conda environment
run: |
export PATH="./:$PATH"
conda init bash
- name: Perform Conda tests
shell: bash -l {0}
run: |
export PATH="./:$PATH"
conda activate test-env
echo "Conda prefix: $CONDA_PREFIX"
python test.py
As #user3840170 mentioned, bash shells do not use ~/.profile or ~/.bashrc. Then, a way to make it work would be to run what conda initialization would run. On GitHub Actions the path to the conda installation is on the variable $CONDA, so you can use it to run the initialization on each step needing conda activate. The following code worked for me on GitHub Actions (the one provided above didn't work in my case).
- name: Set up Conda environment
run: |
echo "${HOME}/$CONDA/bin" >> $GITHUB_PATH
conda init --all --dry-run
- name: On the steps you want to use
shell: bash
run: |
source $CONDA/etc/profile.d/conda.sh
conda activate test-env
python test.py
I try to write a shell script (bash).
The aim of the script is
to get the message of the last git commit
grasp any content inside []-parenthesis of the last git commit message
export that content into an environment variable called GIT_COMMIT_MESSAGE_CONTEXT
Example:
Last git commit message = "[Stage] - gitlab trial"
Working example: The environment variable exported should be
echo $GIT_COMMIT_MESSAGE_CONTEXT
Stage
I found the following, to get the message of the last git commit:
echo $(git log -1 --pretty=%B)
[Stage] - gitlab trial
I am new to bash-scripts and therefore my trial (see below) is somewhat poor so far.
Maybe somebody has more experience to help out here.
My bash script (my-bash-script.sh) looks as follows:
#!/usr/bin/bash
# get last git commit Message
last_git_commit_message="$(git log -1 --pretty=%B)"
export LAST_GIT_COMMIT_MESSAGE="$last_git_commit_message"
I run the bash-script in a terminal as follows:
bash my-bash-script.sh
After closing/re-opening Terminal, I type:
echo $LAST_GIT_COMMIT_MESSAGE
Unfortunately without any result.
Here my questions:
Why do I not get any env-variable echo after running the bash script ?
How to deduct the content of []-parenthis of the last git commit message ?
How to re-write my script ?
The script seems fine, but the approach is flawed. Bash can only export variables to subshells but not vice versa. When you call a script a new shell is started. All variables in that shell, even the exported ones, will be lost after the script exits. See also here.
Some possible ways around that problem:
Source the script.
Let the script print the value and capture its output: variable=$(myScript)
Write the script as a bash function inside your .bashrc.
Depending on what you want to do I recommend 2. or 3. To do 3., put the following in your ~/.bashrc (or ~/.bash_profile if you are using Mac OS) file, start a new shell and use the command extractFromLastCommit as if it were a script.
extractFromLastCommit() {
export LAST_GIT_COMMIT_MESSAGE=$(
git log -1 --pretty=%B |
grep -o '\[[^][]\]' |
tr -d '\[\]' |
head -n1 # only take the first "[…]" – remove this line if you want all
)
}
bash my-bash-script.sh
Starts a new bash process into which your var is exported, then it exits and takes its environment with it.
source my-bash-script.sh
Executes the script in the current shell context and will have the desired effect.
The documentation isn't clear on whether the after_script is executed for cancelled jobs:
after_script is used to define the command that will be run after for all jobs, including failed ones.
I'm doing potentially critical cleanup in the after_script and while cancelled jobs should be rare, I'd like to know that my clean up is guaranteed to happen.
No, I ran some tests and here are the behaviours I exhibited:
after_script:
- echo "This is not executed when a job is cancelled."
- echo "A failing command, like this one, doesn't fail the job." && false
- echo "This is not executed because the previous command failed."
1. after_script is not executed when a job is cancelled
There's an open issue for this on gitlab.com, so if this is affecting you, head over there and make some noise.
2. If a command in the after_script fails, the rest aren't executed
This is quite easy to work around:
after_script:
- potentially failing command || true
- next command
Replace potentially failing command with your command and the next command will execute regardless of whether potentially failing command passed or failed.
One could argue that this behaviour is actually desired, as it gives some flexibility to the user, but it might be counterintuitive to some.
I have written a few cucumber features and also tagged some scenarios with #wip and #test.
I am running cucumber features and want to skip the scenarios which are tagged with #wip and #test so I used the below mentioned line to skip the same.
1. --format pretty --quiet --color --tags ~#wip,~#test
But at the time of executing scenarios, I found all the scenarios tagged with #wip and #test are still getting executed. Then I changed the above line as:
2. --format pretty --quiet --color --tags ~#wip, --tags ~#test
Then it is skipping the scenarios tagged with #wip and #test.
So, I want to know is it possible to skip two or multiple tags in single --tag i.e mentioned in point 1?
Your solution is
1.--format pretty --quiet --color --tags ~#wip,~#test
The comma means that the conditions are OR-ed.
It says: all scenarios that are not #wip OR are not #test
So scenarios with only #wip are executed
The solution is to use quotes, than the conditions are AND-ed
1.--format pretty --quiet --color --tags "~#wip","~#test"