I have a GitLab project that utilises GitLab CI.
The project also uses submodules, both the project and it's submodules are under the same GitLab account.
Here is my .gitmodules file
[submodule "proto_contracts"]
path = proto_contracts
url = https://gitlab.com/areller/proto_contracts.git
I also have this piece in the .gitlab-ci.yml file
variables:
GIT_SUBMODULE_STRATEGY: recursive
However, when i run the CI I get this error
fatal: could not read Username for 'https://gitlab.com': No such device or address
Both the project and the submodules are in a private repository so you would expect to be prompted for authentication, but as I've mentioned, the project and the submodule are under the same account and one of the runner's jobs is to clone the original repository
So it's odd that it's unable to reach the submodule
Is there a way around it?
You must use relative URLs for submodules. Update your .gitmodules as follow:
[submodule "proto_contracts"]
path = proto_contracts
url = ../../areller/proto_contracts.git
Further reading: Using Git submodules with GitLab CI | GitLab Docs
Related
I'm using a shared runner that has access to the entire project group. In this one particular project on my GitLab server whose visibility is set to "private", when the runner attempts to clone the repository, it presents an error message noting that it could not clone the repository with a 403 response.
I searched high and low in the documentation but couldn't find the explanation for this nor a solution to this problem. I noticed when I switched the project's visibility to internal, everything started working.
Does anyone know why GitLab runners cannot access private repositories? If I wish to grant the runner access and the ability to execute the repo's CICD pipeline, how can I do so?
Here are the logs:
Fetching changes with git depth set to 50...
hint: Using 'master' as the name for the initial branch. This default branch name
hint: is subject to change. To configure the initial branch name to use in all
hint: of your new repositories, which will suppress this warning, call:
hint:
hint: git config --global init.defaultBranch <name>
hint:
hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
hint: 'development'. The just-created branch can be renamed via this command:
hint:
hint: git branch -m <name>
Initialized empty Git repository in /builds/4q6GE3ka/0/mysecretproject/myproject/.git/
Created fresh repository.
remote: You are not allowed to download code from this project.
fatal: unable to access 'https://example.com/mysecretproject/project.git/': The requested URL returned error: 403
I have a two private repositories: MyProject, MyProjetUtils.
My project uses the MyProjectUtils as a submodule.
My .gitsubmodules looks like this:
[submodule "MyProjetUtils"]
path = MyProjetUtils
url = git#gitlab.com:MyCompany/MyProjetUtils.git
My .gitlab-ci.yml file looks like this:
default:
image: python:latest
variables:
GIT_SUBMODULE_STRATEGY: recursive
all_test:
stage: test
script:
- apt-get update
- pip install -r requirements.txt
- python tests/run_tests.py
The error I'm getting during the job run:
Updating/initializing submodules recursively with git depth set to 50...
Submodule 'MyProjetUtils' (git#gitlab.com:MyCompany/MyProjetUtils.git) registered for path 'MyProjetUtils'
Cloning into '/builds/MyCompany/MyProject/MyProjetUtils'...
error: cannot run ssh: No such file or directory
fatal: unable to fork
fatal: clone of 'git#gitlab.com:MyCompany/MyProjetUtils.git' into submodule path '/builds/MyCompany/MyProject/MyProjetUtils' failed
Failed to clone 'MyProjetUtils'. Retry scheduled
This error occur before the test stage. I've looked for answer here, and here but could not find an answer.
The first link you posted has the solution you are looking for:
When your submodule is on the same GitLab server, you should use relative URLs in your .gitmodules file. Then you can clone with HTTPS in all your CI/CD jobs. You can also use SSH for all your local checkouts.
Assuming that your submodules are in the same group, update your .gitmodules to use a relative URL.
ie:
[submodule "MyProjetUtils"]
path = MyProjetUtils
url = ../../MyCompany/MyProjetUtils.git
May need to update ../../ to work for your groups.
I am trying to get gitlab’s CI to work properly with an external submodule.
I have a submodule in ANOTHER repository, so no relative path.
I do NOT want to use a SSH key solution.
I want to use the CI token (CI_JOB_TOKEN).
Documentation is NOT clear because what is possible, or not, has changed and there are texts all over the place with many different approaches and, yet, nothing that fits the basic criteria.
It used to not be possible to pull submodules, with CI, if they had an absolute path, so people came up with various solutions. Then it became possible and there are a few solutions regarding authentication issues.
But they all involve doing a clone which is not needed anymore because now we can set the CI to do a recursive pool.
This means that most of the online posts have become irrelevant and outdated and answers are one liners here and there but after a few hours none has worked for me.variables:
GIT_SUBMODULE_STRATEGY: recursive
What has worked is to make a global admin key, giving access to everything but this is not a good solution at all.
What I have now is:
variables:
GIT_SUBMODULE_STRATEGY: recursive
before_script:
- git config --global url."https://gitlab-ci-token:${CI_JOB_TOKEN}#gitlab.com/".insteadOf "git#gitlab.com:"
- git submodule sync && git submodule update --init
Which, according to Getting GitLab CI to clone private repositories should work.
But it fails with:
fatal: could not read Username for 'https://gitlab.com': No such device or address
I set GIT_SUBMODULE_STRATEGY: none and rewrote submodule URLs to use HTTPS with access/deploy tokens created to only allow reading of the repositories of the submodules. Something like this in .gitlab-ci.yaml:
image: docker:stable
variables:
GIT_STRATEGY: clone
# don't clone submodules by default
GIT_SUBMODULE_STRATEGY: none
someJob:
tags:
- docker-priv
before_script:
- apk update
- apk add git
- git --version
- git config --global url."https://${ACCESS_TOKEN_XYZ_NAME}:${ACCESS_TOKEN_XYZ}#GITLAB-HOST/GROUP/XYZ.git".insteadOf git#GITLAB-HOST/GROUP/XYZ.git
- git config --global url."https://${ACCESS_TOKEN_XYZ_NAME}:${ACCESS_TOKEN_XYZ}#GITLAB-HOST/GROUP/XYZ.git".insteadOf https://GITLAB-HOST/GROUP/XYZ.git
- git submodule sync --recursive
- git submodule update --init --recursive
Please mind, that default names of deploy tokens in gitlab use a + sign and thus need to be url encoded when stored as CI variable for above usage. You may use one variable (gitlab%2Breadonly-token-name-123:randomtokenvalue) instead of two to make it a bit easier to read.
I have a repository repoB on a private server serverB, and get a deploy token (user + password) from it.
This repository is used as a submodule in the project I'm trying to configure.
In this project, I want to be able to init this submodule during the Gitlab-CI.
My .gitmodules is:
[submodule "repoB"]
path = repoB
url = https://serverB/groupB/repoB.git
And I have in my .gitlab-ci.yml:
test_build:
variables:
GIT_SUBMODULE_STRATEGY: recursive
Actually, the error I get is the following (group: japan7, project: nanachan, repoB: karaneko, serverB: git.inpt.fr)
I suppose that I should put the deploy token user and password in the project secrets (in Settings −> CI / CD −> Variables) but I could not find the name of these variables, nor any help to solve this particular case.
How should I proceed?
Thank you in advance for your help!
I solved same issue using deploy token. Just added submodule with deploy token and everything on CI runs ok.
http://<username>:<deploy_token>#gitlab.example.com/tanuki/awesome_project.git
And .gitmodules looks like:
[submodule "tanuki/awesome_project"]
path = tanuki/awesome_project
url = https://<username>:<deploy_token>#gitlab.example.com/tanuki/awesome_project.git
https://docs.gitlab.com/ee/user/project/deploy_tokens/
I've got an account with OpenShift which provides a git repository for each project you to work in. I've got one main project and two smaller projects resulting in a total of 3 git repositories.
I have just started to use the two smaller projects in the main project and seen that git can use the submodule command. I've added them both to the custom directory like so:
git submodule add ssh://...#app.rhcloud.com/~/git/app.git/
git submodule add ssh://...#api.rhcloud.com/~/git/api.git/
which has produced a directory structure like so:
node_modules
custom
app ------ full of tasty files
api ------ full of tasty files
index.js
.gitmodules
Within the .gitmodules file I have
[submodule "custom/api"]
path = custom/api
url = ssh://...#api.rhcloud.com/~/git/api.git/
[submodule "custom/app"]
path = custom/app
url = ssh://...#app.rhcloud.com/~/git/app.git/
Which is exactly what I want. It all works locally.
git add --all
git commit -m "new submodules"
git push
The issue is when I run the git push, it comes back with this error:
remote: Host key verification failed.
remote: fatal: Could not read from remote repository.
remote:
remote: Please make sure you have the correct access rights
remote: and the repository exists.
It looks like I need to add the ssh key of the main project to the two smaller projects so that I can ssh. Does anyone know how I can do this or have a better way of including submodules in OpenShift?
Thanks for the help in advance.
It looks like I need to add the ssh key of the main project to the two smaller projects so that I can ssh
At the parent (main) repo level, this shouldn't be the case: all the push does is to push the .gitmodules, and 2 gitlinks (special entries in the index) representing those 2 submodules.
It is more likely that the push doesn't find the proper .ssh/known_hosts, or the public/private ssh keys in order to push back to the app.rhcloud.com upstream repo: See "SSH connection problem with “Host key verification failed…” error" and "The authenticity of host can't be established".