I'm trying to configure gitlab-runner for my project (baseProject) using shell executor. My project has two submodules: submoduleOne and submoduleTwo.
This is how my .gitmodules looks like:
[submodule "lib/submoduleOne"]
path = lib/submoduleOne
url = ../submoduleOne.git
[submodule "third_party/submoduleTwo"]
path = third_party/submoduleTwo
url = ../submoduleTwo.git
And this is how relevant part of my .gitlab-ci.yml looks like:
stages:
- preparation
clean:
stage: preparation
script:
- eval $(ssh-agent -s)
- ssh-add ~/.ssh/submoduleOne
- ssh-add -l
- git submodule update --init -- lib/submoduleOne
- ssh-add -D
- ssh-add ~/.ssh/submoduleTwo
- ssh-add -l
- git submodule update --init -- third_party/submoduleTwo
- ./setup_project.sh
Public keys ~/.ssh/submoduleTwo and ~/.ssh/submoduleOne are added to the respective projects in gitlab as "deploy keys" and the fingerprints match. Both projects seem to be configured in the same way.
Yet, the job fails and the output is:
Running with gitlab-runner 11.0.0 (5396d320)
on wawsrvbuild70 3295e1f6
Using Shell executor...
Running on wawsrvbuild70...
Fetching changes...
HEAD is now at 5e3be99 CI test
From https://gitlab01.mydomain.pl/Group/baseProject
5e3be99..0029318 p.czechows2/configure-gitlab-ci#1038 -> origin/p.czechows2/configure-gitlab-ci#1038
Checking out 00293180 as p.czechows2/configure-gitlab-ci#1038...
Skipping Git submodules setup
$ eval $(ssh-agent -s)
Agent pid 18755
$ ssh-add ~/.ssh/submoduleOne
Identity added: /home/gitlab-runner/.ssh/submoduleOne (/home/gitlab-runner/.ssh/submoduleOne)
$ ssh-add -l
4096 27:f5:e5:86:af:a7:7a:aa:52:03:b1:6d:05:72:56:29 /home/gitlab-runner/.ssh/submoduleOne (RSA)
$ git submodule update --init -- lib/submoduleOne
$ ssh-add -D
All identities removed.
$ ssh-add ~/.ssh/submoduleTwo
Identity added: /home/gitlab-runner/.ssh/submoduleTwo (/home/gitlab-runner/.ssh/submoduleTwo)
$ ssh-add -l
4096 89:aa:68:f9:88:72:4c:cf:8a:a3:c7:7e:34:92:91:6b /home/gitlab-runner/.ssh/submoduleTwo (RSA)
$ git submodule update --init -- third_party/submoduleTwo
Cloning into 'third_party/submoduleTwo'...
fatal: Authentication failed for 'https://gitlab-ci-token:yXkvnfXZadABF4FZWTSb#gitlab01.mydomain.pl/Group/submoduleTwo.git/'
Clone of 'https://gitlab-ci-token:yXkvnfXZadABF4FZWTSb#gitlab01.mydomain.pl/Group/submoduleTwo.git' into submodule path 'third_party/submoduleTwo' failed
ERROR: Job failed: exit status 1
This token yXkvnfXZadABF4FZWTSb is I think the token I gave to gitlab-runner while configuring it for the baseProject. But I'm not sure where to check this.
Also, changing the order of submodules initialization won't help: the problem is still with submoduleTwo.
So my question is: why wouldn't gitlab-runner use the private key stored in ssh-agent for cloning this repository? or maybe I've made a mistake somewhere? Suggestions for how to configure submodules initialization in gitlab-runner differently are also welcome.
Related
I have the following CI/CD
build-job:
stage: build
script:
- echo "Hello, $GITLAB_USER_LOGIN!"
- ls -al
- mkdir django_app
- cd django_app
- git clone git#gitlab.com:xxxxxx/yyyyyy.git .
It says
$ git clone git#gitlab.com:xxxxx/yyyyy.git .
Cloning into '.'...
Host key verification failed.
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
Cleaning up project directory and file based variables
How to clone the repository
You need to add ssh key with access to gitlab.com:xxxxxx/yyyyyy.git to your container before you try to clone.
I am new to gitlab CI/CD and I'm struggling to figure this out.
All I want to do is when I push to dev branch I want my react app to be built and the folder ./build to be pushed through SSH to my dev server.
Here is what I did so far, including a screenshot of the error message I get.
This is my gitlab-ci.yml
image: node:latest
cache:
paths:
- node_modules/
build_dev:
stage: build
environment: Development
only:
- dev
script:
- ls
- npm install
- npm run build
artifacts:
paths:
- build/
- ecosystem.config.js
deploy_dev:
stage: deploy
environment: Development
only:
- dev
script:
- rsync -r -a -v -e ssh --delete "./build" root#dev.teledirectasia.com:/var/www/gitlab/${CI_PROJECT_NAME}
- rsync -r -a -v -e ssh --delete "./ecosystem.config.js" root#dev.teledirectasia.com:/var/www/gitlab/${CI_PROJECT_NAME}/
- ssh root#dev.teledirectasia.com "cd /var/www/gitlab/${CI_PROJECT_NAME} && pm2 start ecosystem.config.js"
I don't know why I am getting this output with job failed
This is a DNS problem. Your runner cannot resolve the hostname of the GitLab server - gitlab.teledirectgroup.com. Did you set the GitLab hostname if your local workstation's host file manually, or did you set it up in a DNS server as a 'proper' hostname?
If you set up the hostname in a DNS server then the solution may be as simple as adding the DNS server to /etc/resolv.conf on the runner. However, if you just set the GitLab hostname in your workstation's hosts file then you'll need to set it in the runner's /etc/hosts file, too. It's hard to say what the exact solution is without knowing how you set up the GitLab hostname in the first place.
the solution that’s so applied to GitLab?
Use the git clone by ssh, I don’t have a good goal that’s so I can up to push that’s changes over a submodule from runner Shell by GitLab CI. The pipeline ever fails and prints this error.
ERROR PIPELINE JOB
In the local repo as a project the file config contains that line with the URL, more don’t have login with this about the pipeline.
.git/config
Some help or walkthrough of reference to culminate with that challenge in troubleshooting!
This is my code over the file ".gitlab-ci.yml"
variables:
TEST_VAR: "Update Git Submoudel in all Etecnic projects."
job1:
variables: {}
script:
- echo "$TEST_VAR"
job2:
variables: {}
script:
- echo "OK" >> exito.txt
- git add --all
- git commit -m "Update Submodule"
- git push origin HEAD:master
Versions:
GitLab:
gitlab-ce is already the newest version (15.6.0-ce.0).
Runner:
Version: 15.5.1
Git revision: 7178588d
Git branch: 15-5-stable
GO version: go1.18.7
Built: 2022-11-11T09:45:25+0000
OS/Arch: linux/amd64
Thanks so much for your attention.
I need my Gitlab CI to update submodules with --remote flag so that the HEAD is set to the remote's HEAD. After a bit of Googling I found that I need to set GIT_SUBMODULE_STRATEGY to none and run git submodule update --recursive --remote --init manually:
variables:
GIT_STRATEGY: clone
GIT_SUBMODULE_STRATEGY: none
before_script:
- apk add git || ( apt-get update && apt-get -y install git )
- git submodule update --recursive --remote --init
test:build:
services:
- docker:dind
image: ubuntu
variables:
DOCKER_HOST: tcp://docker:2375
DOCKER_DRIVER: overlay2
script:
- echo "done
Unfortunately I'm getting a CI failure (names edited):
$ git submodule update --recursive --remote --init
Submodule 'current_project_name/submodule_project_name' (ssh://git#gitlab.someserver.net:9931/someorg/submodule_project_name.git) registered for path 'current_project_name/submodule_project_name'
Cloning into '/builds/someorg/current_project_name/current_project_name/submodule_project_name'...
Host key verification failed.
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
fatal: clone of 'ssh://git#gitlab.someserver.net:9931/someorg/submodule_project_name.git' into submodule path '/builds/someorg/current_project_name/current_project_name/submodule_project_name' failed
Failed to clone 'current_project_name/submodule_project_name'. Retry scheduled
Cloning into '/builds/someorg/current_project_name/current_project_name/submodule_project_name'...
Host key verification failed.
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
fatal: clone of 'ssh://git#gitlab.someserver.net:9931/someorg/submodule_project_name.git' into submodule path '/builds/someorg/current_project_name/current_project_name/submodule_project_name' failed
Failed to clone 'current_project_name/submodule_project_name' a second time, aborting
I can see that the CI does have permissions to clone that submodule_project_name because if I set GIT_SUBMODULE_STRATEGY e.g. to recursive, CI manages to pull it (but it's not --remote, so it doesn't work the way I want). Unfortunately when my before_script tries to do it, I'm getting the error. How can I bypass it?
I mentioned before updating the ~/.ssh/.known_hosts file, as in here.
This is not needed when fetching the submodules before the script (which is not what you are doing with GIT_SUBMODULE_STRATEGY set to NONE)
With dind (Docker In Docker), consider also this thread, regarding ssh-add for private keys, and .dockerini / .dockerenv SSH directives.
The OP d33tah confirms in the comments:
I actually didn't add any key, assuming that since Gitlab CI's defaults can pull the key, I should be able to as well.
Then I found that docs say that I needed a deploy key and I added one
Yes: adding the public key on Gitlab side is mandatory.
I am using gitlab CI runner to test my code and generating some files. I just want to push the generated files to gitlab repository via CI runner. Is there any way to do that ?
I have resolved this issue by doing this:
Note: If you want to git push to a non protected branch do not set the runner variable as protected
Generate new gitlab access token with api scope: User Settings > Access Tokens
Add a protected CI variable into your project settings with the new token: Your project > Settings > Secret variable using variable name CI_PUSH_TOKEN
Add another protected CI variable with your username using variable name CI_USERNAME
Then you can use this token instead of the default in you gitlab-ci script. for example:
before_script:
- git remote set-url origin https://${CI_USERNAME}:${CI_PUSH_TOKEN}#gitlab.com/${CI_PROJECT_NAME}.git
- git config --global user.email '${GITLAB_USER_EMAIL}'
- git config --global user.name '${GITLAB_USER_ID}'
...
- git checkout -B branch
- # do the file changes here
- git commit -m '[skip ci] commit from CI runner'
- git push --follow-tags origin branch
Generated a SSH Key in gitlab
--> Profile Settings --> SSH Keys --> Generate It
After generating the SSH Key store that in the gitlab variables named SSH
--> Project Settings --> Variables --> Add Variable
In the .gitlab-ci.yml add the below lines.
before_script:
- mkdir -p ~/.ssh
- echo "$SSH" | tr -d '\r' > ~/.ssh/id_rsa
- chmod 600 ~/.ssh/id_rsa
- ssh-keyscan -H 'Git_Domain' >> ~/.ssh/known_hosts
After that pushed the files to the repository using this below js code.
var child_process = require("child_process");
child_process.execSync("git checkout -B 'Your_Branch'");
child_process.execSync("git remote set-url origin Your_Repository_Git_Url");
child_process.execSync("git config --global user.email 'Your_Email_ID'");
child_process.execSync("git config --global user.name 'Your_User_Name'");
for (var i=0;i<filesToBeAdded.length;i++) {
child_process.execSync("git add "+filesToBeAdded[i]);
}
var ciLog = child_process.execSync("git commit -m '[skip ci]Automated commit for CI'");
var pushLog = child_process.execSync("git push origin Your_Branch");
[skip ci] is most important in commit message. Otherwise it will start a infinity loop of CI process.
You could use of course SSH keys but you could also provide user and password (user with write access) as secret variables and use them.
Example:
before_script:
- git remote set-url origin https://$GIT_CI_USER:$GIT_CI_PASS#gitlab.com/$CI_PROJECT_PATH.git
- git config --global user.email 'myuser#mydomain.com'
- git config --global user.name 'MyUser'
You have to define GIT_CI_USER and GIT_CI_PASS as secret variables (you could always create dedicated user for this purpose).
With this configuration you could normally work with git. I'm using this approach to push the tags after the release (with Axion Release Gradle Pluing - http://axion-release-plugin.readthedocs.io/en/latest/index.html)
Example release job:
release:
stage: release
script:
- git branch
- gradle release -Prelease.disableChecks -Prelease.pushTagsOnly
- git push --tags
only:
- master
Another solution using Gitlab API to commit back a file .terraform.lock.hcl in terraform/ directory on $CI_COMMIT_BRANCH with [skip ci] :
script:
- 'STATUS=$(curl -Ss --head --header "JOB-TOKEN: $CI_JOB_TOKEN" "$CI_API_V4_URL/projects/$CI_PROJECT_ID/repository/files/terraform%2F%2Eterraform%2Elock%2Ehcl?ref=$CI_COMMIT_BRANCH" | grep "HTTP/1.1" | cut -d " " -f2)'
- if [[ $STATUS == "404" ]]; then ACTION="create"; else ACTION="update"; fi
- 'curl --request POST --form "branch=$CI_COMMIT_BRANCH" --form "commit_message=[skip ci] terraform.lock.hcl from pipeline" --form "actions[][action]=$ACTION" --form "actions[][file_path]=terraform/.terraform.lock.hcl" --form "actions[][content]=<.terraform.lock.hcl" --header "JOB-TOKEN: $CI_JOB_TOKEN" "$CI_API_V4_URL/projects/$CI_PROJECT_ID/repository/commits"'
The Feature you are looking for is called Artifacts. Artifacts are files which are attached to a build when they are successful.
To enable an Artifact put this in your .gitlab-ci.yml:
artifacts:
paths:
- dir/
- singlefile
This will upload the dir directory and the file singlefile back to GitLab.
My goal :
Continuous integration, then deploy if the build is on a specific branch.
The push should be to a server hosting a pod instance.
It is basically a git bare repository where I can push. When I push a hook triggered and voilà.
My Problem :
At the end of the build, ssh is asking for a password.
My configuration :
.travis.yml :
before_install:
- openssl aes-256-cbc -K $encrypted_9bbc0c90c60c_key -iv $encrypted_9bbc0c90c60c_iv
-in key.enc -out key -d
addons:
ssh_known_hosts: dev.ogdabou.ninja
after_success:
- if [[ $TRAVIS_BRANCH == "dev" ]]; then chmod 750 deploy.sh; ./deploy.sh; fi
where keyis a private ssh key having ssh password-less authentification to the server.
deploy.sh
#!/bin/bash
eval "$(ssh-agent -s)"
chmod 600 key
mv key ~/.ssh/id_rsa
cd dist;
pwd;
git init;
git config --global user.name "travis"
git config --global user.email "travis#github.com"
git remote add deploy $DEV_DEPLOY_REPO;
git add .;
git commit -m "Build $TRAVIS_BUILD_NUMBER";
git push deploy master;
Thanks for your help :).
The first time I tried I used the server hosting the POD service.
It is now working, created a user in cygwin, new ssh key then configured password-less ssh.
Finally, encode travis keys and follow travis tutorial.