Context
After creating a general post-receive for a GitLab server, I noticed it gets triggered directly after a new commit is detected in any repository. However, I would like the post-receive script to do something with the build status of the GitLab Runner CI on the commit that triggered the post-receive script.
Approach
Based on this question and answer, I wrote a post-receive script that gets the commit and repository, and I tried to get the build status from that commit from within the GitLab docker:
#!/bin/bash
read oldrev newrev refname
echo "Previous Commit: $oldrev"
echo "New/latest Commit: $newrev"
echo "Repository name: $refname"
# Get build status of $newrev
curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/17/$refname/commits/$newrev/statuses
However, that API call does not work from within the Docker environment (which is from where the post-receive script runs).
Docker GitLab Build Status File locations
I also found the build status badges inside the Docker, they are located in: /opt/gitlab/embedded/service/gitlab-rails/public/assets/. However I do not (yet) know how to decode their filenames. For example, the build status badge accompanying Job #3, of commit: 9514d16aafc1d741ba6a9ff47718d632fa8d435b has filename: icons-6d7d4be41eac996c72b30eac2f28399ac8c6eda840a6fe8762fc1b84b30d5a2d.svg. Basically I do not know to which commit/repository that build status badge belongs.
On the other hand, I have found the location of the job logs in the hashed path of the repository:
/var/opt/gitlab/gitlab-rails/shared/artifacts/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35/2021_10_09/1/1/job.log
/var/opt/gitlab/gitlab-rails/shared/artifacts/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35/2021_10_14/3/3/job.log
Which each in turn contain their respective commit and branch as:
Checking out 9514d16a as master...
So in principle I could scan the repository path and accompanying job logs until I found the job.log that contains the commit of the post-receive script (for e.g. 5 minutes, to account for the delay between the commit and the starting of the GitLab Runner CI), and then search for the build status output in that job.log (e.g. Job succeeded) (for e.g. 60 minutes to allow for long jobs). However, that seems like a convoluted work-around.
Question
Hence, I was wondering, *Is there a better/faster/robuster method to get the GitLab Runner CI build status of the commit that triggered the general post-receive script of a GitLab server, inside that triggered instance/run of the post-receive script?
Related
I search the method to copy files to "test" host when I launch the "git push" command.
git push ------- TO REPO --> REPO_SERVER
\
\_________ TO DIR --> Host_TEST
Git Version: 2.20.1
It sounds like you are trying to (re) invent CI/CD.
If you are using GitHub or GitLab as a remote server you can use Pipelines (or Actions in GitHub).
In there you can define (almost) anything you want to happen after a push, I am assuming your Host_TEST is accessible online.
In case you are running your own git server
You can implement "push to deploy" using the post-receive hook. Hooks are scripts that are placed inside .git/hooks and executed at a precise phase of a git command. you can find several example implementations in .git/hook. See here for more information: Setting up Push-to-Deploy with git
In case you don't have access to your own git server
You can use the pre-push script on your local machine, BUT THIS IS A BAD IDEA
This hooks is executed after you execute git push but before git actually pushes anything. If your script fails (i.e non-zero return code) it will not push.
Also, if your script manages to copy but then git fails to push you will end up testing code that's not in your repo.
If all this sound way too complicated
You can create a bash function that does both operations and add it to your .bashrc.
Here is an example:
push_copy() {
if git push
then
# Copy for command here: scp ...
else
echo "Failed..."
fi
}
I'm writing GitLab CI/CD pipeline script in .gitlab-ci.yml
I want to check if a specific file changed in another repo and if so I would like to copy the file, commit and push to the current repo.
everything works until I get to the 'git push' part
I tried several ways to fixed it:
stages:
- build
build:
stage: build
script:
- echo "Building"
- git checkout -b try
- git remote add -f b https://gitlab-ci-token:${CI_JOB_TOKEN}#gitlab.{otherRepo}.git
- git remote update
- CHANGED=$(git diff try:mobile_map.conf b/master:mobile_map.conf)
- if [ -n "${CHANGED}" ]; then
echo 'changed';
FILE=$(git show b/master:mobile_map.conf > mobile_map.conf);
git add mobile_map.conf;
git commit -m "updating conf file";
git push;
else
echo 'not changed';
fi
- git remote rm b
for this code I get :
fatal: unable to access 'https://gitlab-ci-token:[MASKED]#gitlab.{curr_repo}.git/': The requested URL returned error: 403
also I tried to add this line in the beginning :
git remote set-url origin 'https://{MY_USER_NAME}:"\"${PASSWORD}\""#gitlab.{curr_repo}.git'
and I get this error message:
fatal: Authentication failed for 'https://{MY_USER_NAME}:"\"${PASSWORD}\""#{curr_repo}.git/'
also I added:
- git config --global user.name {MY_USER_NAME}
- git config --global user.email {MY_EMAIL}
please help me,
Thanks
Job-tokens only have read-permission to your repository.
A unique job token is generated for each job and provides the user read access all projects that would be normally accessible to the user creating that job. The unique job token does not have any write permissions, but there is a proposal to add support.
You can't use deploy-tokens because they can't have write-access to a repository (possible tokens).
You could use a project-access-token with read-write-access to your repository.
You can use project access tokens:
On GitLab SaaS if you have the Premium license tier or higher. Project
access tokens are not available with a trial license.
On self-managed instances of GitLab, with any license tier. If you
have the Free tier: [...]
Then you can use your project-access-token as an environment variable in the url.
git push "https://gitlab-ci-token:$PROJECT_ACCESS_TOKEN#$CI_SERVER_HOST/$CI_PROJECT_PATH.git"
At least that's how we use it in our pipelines.
I hope this helps you further.
Complete example of simple GitLab CI stage, that commits back to its own repo
ci_section_name:
# N.B. This stage produces an extra commit to the repo!
stage: stage_name
script:
- apt-get update && apt-get install -y git
- echo "hello" >> file_to_be_modified.txt # your real job may do smth else
after_script:
- git config user.name "Name On Your Choice"
- git config user.email "email_on_your_choice#$CI_SERVER_HOST"
- git pull "https://project_access_token_name:$PROJECT_VARIABLE_WITH_ACCESS_TOKEN_VALUE#$CI_SERVER_HOST/$CI_PROJECT_PATH.git" $CI_COMMIT_BRANCH --rebase --autostash
- git commit -a -m "Message on your choice"
- git push "https://project_access_token_name:$PROJECT_VARIABLE_WITH_ACCESS_TOKEN_VALUE#$CI_SERVER_HOST/$CI_PROJECT_PATH.git" HEAD:$CI_COMMIT_BRANCH
I want to stop on few important aspects:
file_to_be_modified.txt - any modified files in the example are supposed to already exist in the repo, if you need add smth new, you will need
include at least git add command
project_access_token_name - is the name of used Project Access Token, not the token value itself (see screenshots below); you should create it by hands for your GitLab project if necessary
PROJECT_VARIABLE_WITH_ACCESS_TOKEN_VALUE - is the name of Project Variable (see screenshots below), you should create this variable by hands for your GitLab project and populate it with value of chosen Project Access Token; btw, you can add literal value of the token into stage code instead, but it is probably bad for security reasons
--rebase --autostash - note that autostashing your changes instead of simply trying to pull-push them will help you to guarantee the push in case of conflicts (even our new GitLab stage may conflict between its launches) but files would be left with conflict trace instead of correct content; because it is hard to resolve conflicts automatically, it is supposed that you control such situation with another tools (for example, further build on conflict state will simply fail)
This simple example will lead to an infinite flow of commits, probably real stage should contain some only conditions, etc.
Screenshots
GitLab Project Access Tokens page
GitLab Project CI/CD Settings menu, where to find Variables settings
Add project Variable pop-up
I followed below article to push gitlab repository code to Google cloud source repository but I'm getting an error on this command
git push -f google master
error: src refspec master does not match any.
error: failed to push some refs to 'https://source.developers.google.com/p/project/r/test/'
Article followed:
https://medium.com/#bamnet/cloud-source-repositories-gitlab-2fdcf1a8e50c
Is there anything , I'm doing wrong 😜? Any thoughts as to how I can avoid this error message?
src refspec master does not match any
The issue is the date of the article you are following: Aug. 2018.
GitLab Runner has changed since then, more precisely in May 2019.
The problem is described in this thread from May 2019:
Since we are using refspec to clone/fetch the repository, we checkout a specific commit and not checking out a specific branch.
When the script does git push master, the branch is nowhere to be found so git doesn’t know what to push.
That was because of, on GitLab side, MR 1203:
Basically, GitLab CE/EE sends refspecs parameter to GitLab Runner gitlab-org/gitlab-foss app/presenters/ci/build_runner_presenter.rb: this parameter is to used in GitLab Runners for fetching branch/tag refs from remote repository.
This change was introduced because we wanted GitLab Rails side to leverage respecs in order for issue 7380 "Combined ref pipelines (source+target branch)" though, there should not be a big difference between git clone $URL or mkdir $REPO_DIR && git remote add origin $URL && git fetch +refs/heads/branch_name:refs/remotes/origin/branch_name.
In fact, the new behavior has already run on our development project
https://gitlab.com/gitlab-org/gitlab-ce/pipelines and has no issues so far.
Issue 4097 was opened at the time
Workaround
Use HEAD when you want to push this to another remote.
deploy:
stage: deploy
script:
- git remote add heroku https://heroku:$HEROKU_API_KEY#git.heroku.com/<project>.git
- git push -f heroku HEAD:master
So don't push master. Push HEAD.
The OP Adam uses another workaround and add:
before_script:
- git checkout master
We need to generate commit history file automatically when user commits code into git repository.
It can be done using Jenkins, Gitlab Webhooks and Jenkins Git Changelog plugin. Also, it can be created using below git command.
$ git log --pretty=format:'At %ci, %cN committed %h : %s' --decorate --graph >log.log
But, is there anyway we can generate commit history file using Gitlab CI/CD operations. The file can be saved in git repositry or local storage.
Sample Commit History File
* At 2018-11-16 18:02:21, kRiZ committed 1714a95 : Commit 4
* At 2018-11-15 16:06:06, kRiZ committed bab5c0c : Commit 3
* At 2018-11-14 18:05:09, kRiZ committed b3c9d86 : Commit 2
* At 2018-11-14 06:47:34, kRiZ committed 8e6ee30 : Add README.md
I'm sure there are multiple ways of doing this in GitLab. Here's one:
Create a .gitlab-ci.yaml file at the root of your repository. You can do this locally or using GitLab's web UI.
Paste this snippet into your .gitlab-ci.yaml file:
changelog:
image: docker:git
script:
- git log --pretty=format:'At %ci, %cN committed %h - %s' --decorate --graph >log.log
artifacts:
paths: [log.log]
Either commit and push locally or commit on GitLab's Web UI. The changelog job will be triggered.
After the job has finished successfully, your log.log file will be available as an artifact of the changelog job
Essentially, with this snippet you are setting up GitLab's CI/CD system to:
Use the Docker executor with a Docker image with git preinstalled
Define a changelog job that will run your git command
Define a log.log artifact that will be generated as part of the job and stored, so that you can download it afterwards.
I'd also recommend checking out the GitLab CI/CD quickstart
The library used by the Jenkins plugin also has a command line tool that can be used in anywhere:
npx git-changelog-command-line -std -tec "
# Changelog
Changelog for {{ownerName}} {{repoName}}.
{{#tags}}
## {{name}}
{{#issues}}
{{#hasIssue}}
{{#hasLink}}
### {{name}} [{{issue}}]({{link}}) {{title}} {{#hasIssueType}} *{{issueType}}* {{/hasIssueType}} {{#hasLabels}} {{#labels}} *{{.}}* {{/labels}} {{/hasLabels}}
{{/hasLink}}
{{^hasLink}}
### {{name}} {{issue}} {{title}} {{#hasIssueType}} *{{issueType}}* {{/hasIssueType}} {{#hasLabels}} {{#labels}} *{{.}}* {{/labels}} {{/hasLabels}}
{{/hasLink}}
{{/hasIssue}}
{{^hasIssue}}
### {{name}}
{{/hasIssue}}
{{#commits}}
**{{{messageTitle}}}**
{{#messageBodyItems}}
* {{.}}
{{/messageBodyItems}}
[{{hash}}](https://github.com/{{ownerName}}/{{repoName}}/commit/{{hash}}) {{authorName}} *{{commitTime}}*
{{/commits}}
{{/issues}}
{{/tags}}
"
I'm trying to use a remote trigger for (re)building in ci.gitlab. For explaining this, I made up this scenario:
2 repository, "lib" and "app1"
app1 will successfully build only if lib is included (solved simply by .gitlab-ci.yml)
I need to trigger the build of app1 (only for the master branch, in best-case) on commit (or merge request) of lib
I tried to figure it out using web hooks, but I wasn't able to find a url for ci.gitlab.com. Is this possible in a gitlab environment?
You can do this with newly added triggers functionality.
In your CI's project, find the section "Triggers". Add a trigger and use its token like this:
curl -X POST \
-F token=TOKEN \
https://ci.gitlab.com/api/v1/projects/{project_id}/refs/REF_NAME/trigger
(https://about.gitlab.com/2015/08/22/gitlab-7-14-released/)
Obsolete:
we have the same problem, and the way we solved it is by pushing and subsequently deleting a tag.
The assumption is that you manage the machine with Gitlab-CI runner. First, clone the main repository, app1 for you. And in lib's .gitlab-ci.yml add the steps:
- cd /path/to/app1_repository
- git pull
- git tag ci-trigger master
- git push origin ci-trigger
- git push --delete origin ci-trigger
- git tag -d ci-trigger
Make sure that you have the option Tag push events checked in your Gitlab Services settings for Gitlab-CI.
This solution has drawbacks:
Gitlab-CI runner must have write permissions to the repository, so it won't work for shared runners
git history will be bloated with all this tagging (especially Gitlab UI)
I opened an issue for this (https://gitlab.com/gitlab-org/gitlab-ci/issues/223) so let's hope they add this functionality to the API (http://doc.gitlab.com/ci/api/README.html).