I have a two private repositories: MyProject, MyProjetUtils.
My project uses the MyProjectUtils as a submodule.
My .gitsubmodules looks like this:
[submodule "MyProjetUtils"]
path = MyProjetUtils
url = git#gitlab.com:MyCompany/MyProjetUtils.git
My .gitlab-ci.yml file looks like this:
default:
image: python:latest
variables:
GIT_SUBMODULE_STRATEGY: recursive
all_test:
stage: test
script:
- apt-get update
- pip install -r requirements.txt
- python tests/run_tests.py
The error I'm getting during the job run:
Updating/initializing submodules recursively with git depth set to 50...
Submodule 'MyProjetUtils' (git#gitlab.com:MyCompany/MyProjetUtils.git) registered for path 'MyProjetUtils'
Cloning into '/builds/MyCompany/MyProject/MyProjetUtils'...
error: cannot run ssh: No such file or directory
fatal: unable to fork
fatal: clone of 'git#gitlab.com:MyCompany/MyProjetUtils.git' into submodule path '/builds/MyCompany/MyProject/MyProjetUtils' failed
Failed to clone 'MyProjetUtils'. Retry scheduled
This error occur before the test stage. I've looked for answer here, and here but could not find an answer.
The first link you posted has the solution you are looking for:
When your submodule is on the same GitLab server, you should use relative URLs in your .gitmodules file. Then you can clone with HTTPS in all your CI/CD jobs. You can also use SSH for all your local checkouts.
Assuming that your submodules are in the same group, update your .gitmodules to use a relative URL.
ie:
[submodule "MyProjetUtils"]
path = MyProjetUtils
url = ../../MyCompany/MyProjetUtils.git
May need to update ../../ to work for your groups.
Related
in my project i have 2 submodules (both are maintained by us), for some reason when pushing our code into the CI the runners are all failing because:
Running with gitlab-runner 14.4.0 (4b9e985a)
on bv1 runner docker DcUNN4JN
Resolving secrets
00:00
Preparing the "docker" executor
Using Docker executor with image registry.gitlab.com/visionary.ai/brightervision/gitlab-cuda-trt-v2 ...
Authenticating with credentials from job payload (GitLab Registry)
Pulling docker image registry.gitlab.com/<company-name>/<repository>/gitlab-cuda-trt-v2 ...
Using docker image sha256:ASDFASDFSADFSDAF for registry.gitlab.com/<company-name>/<repository>/gitlab-cuda-trt-v2 with digest registry.gitlab.com/<company-name>/<repository>/gitlab-cuda-trt-v2#sha256:fgsdgsdfgsdfgdsfgsdfg ...
Preparing environment
00:07
Running on runner-dcunn4jn-project-30307801-concurrent-0 via bv1...
Getting source from Git repository
00:16
Fetching changes with git depth set to 1...
Reinitialized existing Git repository in /ci_builds/DcUNN4JN/0/<company-name>/<repository>/.git/
Checking out 09e4b628 as ci-add-bokeh...
Updating/initializing submodules recursively with git depth set to 1...
Synchronizing submodule url for 'jetson/isp-arc-implementations'
Synchronizing submodule url for 'models'
Entering 'jetson/isp-arc-implementations'
Entering 'models'
Entering 'jetson/isp-arc-implementations'
HEAD is now at b9480b7 cleanup
Entering 'models'
HEAD is now at 9c17371 Merge branch 'mar-22' into 'main'
fatal: refusing to merge unrelated histories
Unable to merge 'c90d7c8a3564ff09bb5e02513f28e64a688b325b' in submodule path 'jetson/isp-arc-implementations'
Uploading artifacts for failed job
00:09
Uploading artifacts...
WARNING: untracked: no files
ERROR: No files to upload
Cleaning up project directory and file based variables
00:06
ERROR: Job failed: exit code 1
our .gitmodules looks like this:
[submodule "models"]
path = models
url =../models.git
branch = main
[submodule "jetson/isp-arc-implementations"]
path = jetson/isp-arc-implementations
url = ../isp-arc-implementations.git
branch = main
we are all using linux OS (ubuntu18.04 distribution), couldn't find anything in the gitlab documentations nor anywhere else so i wanted to try my luck here,
if anyone has encountered this and knows how to solve it, it would be much appreciated
a few notes:
for some reason the submodule commit sha's don't match the ones we have on our repositories, but as far as i understand it should fetch the most recent one?
when going to the directory where the runner builds everything, i can't manually pull and update the submodules there (getting authentication errors, i read that the runner can't pull\push in gitlab)
I'm writing GitLab CI/CD pipeline script in .gitlab-ci.yml
I want to check if a specific file changed in another repo and if so I would like to copy the file, commit and push to the current repo.
everything works until I get to the 'git push' part
I tried several ways to fixed it:
stages:
- build
build:
stage: build
script:
- echo "Building"
- git checkout -b try
- git remote add -f b https://gitlab-ci-token:${CI_JOB_TOKEN}#gitlab.{otherRepo}.git
- git remote update
- CHANGED=$(git diff try:mobile_map.conf b/master:mobile_map.conf)
- if [ -n "${CHANGED}" ]; then
echo 'changed';
FILE=$(git show b/master:mobile_map.conf > mobile_map.conf);
git add mobile_map.conf;
git commit -m "updating conf file";
git push;
else
echo 'not changed';
fi
- git remote rm b
for this code I get :
fatal: unable to access 'https://gitlab-ci-token:[MASKED]#gitlab.{curr_repo}.git/': The requested URL returned error: 403
also I tried to add this line in the beginning :
git remote set-url origin 'https://{MY_USER_NAME}:"\"${PASSWORD}\""#gitlab.{curr_repo}.git'
and I get this error message:
fatal: Authentication failed for 'https://{MY_USER_NAME}:"\"${PASSWORD}\""#{curr_repo}.git/'
also I added:
- git config --global user.name {MY_USER_NAME}
- git config --global user.email {MY_EMAIL}
please help me,
Thanks
Job-tokens only have read-permission to your repository.
A unique job token is generated for each job and provides the user read access all projects that would be normally accessible to the user creating that job. The unique job token does not have any write permissions, but there is a proposal to add support.
You can't use deploy-tokens because they can't have write-access to a repository (possible tokens).
You could use a project-access-token with read-write-access to your repository.
You can use project access tokens:
On GitLab SaaS if you have the Premium license tier or higher. Project
access tokens are not available with a trial license.
On self-managed instances of GitLab, with any license tier. If you
have the Free tier: [...]
Then you can use your project-access-token as an environment variable in the url.
git push "https://gitlab-ci-token:$PROJECT_ACCESS_TOKEN#$CI_SERVER_HOST/$CI_PROJECT_PATH.git"
At least that's how we use it in our pipelines.
I hope this helps you further.
Complete example of simple GitLab CI stage, that commits back to its own repo
ci_section_name:
# N.B. This stage produces an extra commit to the repo!
stage: stage_name
script:
- apt-get update && apt-get install -y git
- echo "hello" >> file_to_be_modified.txt # your real job may do smth else
after_script:
- git config user.name "Name On Your Choice"
- git config user.email "email_on_your_choice#$CI_SERVER_HOST"
- git pull "https://project_access_token_name:$PROJECT_VARIABLE_WITH_ACCESS_TOKEN_VALUE#$CI_SERVER_HOST/$CI_PROJECT_PATH.git" $CI_COMMIT_BRANCH --rebase --autostash
- git commit -a -m "Message on your choice"
- git push "https://project_access_token_name:$PROJECT_VARIABLE_WITH_ACCESS_TOKEN_VALUE#$CI_SERVER_HOST/$CI_PROJECT_PATH.git" HEAD:$CI_COMMIT_BRANCH
I want to stop on few important aspects:
file_to_be_modified.txt - any modified files in the example are supposed to already exist in the repo, if you need add smth new, you will need
include at least git add command
project_access_token_name - is the name of used Project Access Token, not the token value itself (see screenshots below); you should create it by hands for your GitLab project if necessary
PROJECT_VARIABLE_WITH_ACCESS_TOKEN_VALUE - is the name of Project Variable (see screenshots below), you should create this variable by hands for your GitLab project and populate it with value of chosen Project Access Token; btw, you can add literal value of the token into stage code instead, but it is probably bad for security reasons
--rebase --autostash - note that autostashing your changes instead of simply trying to pull-push them will help you to guarantee the push in case of conflicts (even our new GitLab stage may conflict between its launches) but files would be left with conflict trace instead of correct content; because it is hard to resolve conflicts automatically, it is supposed that you control such situation with another tools (for example, further build on conflict state will simply fail)
This simple example will lead to an infinite flow of commits, probably real stage should contain some only conditions, etc.
Screenshots
GitLab Project Access Tokens page
GitLab Project CI/CD Settings menu, where to find Variables settings
Add project Variable pop-up
I am trying to get gitlab’s CI to work properly with an external submodule.
I have a submodule in ANOTHER repository, so no relative path.
I do NOT want to use a SSH key solution.
I want to use the CI token (CI_JOB_TOKEN).
Documentation is NOT clear because what is possible, or not, has changed and there are texts all over the place with many different approaches and, yet, nothing that fits the basic criteria.
It used to not be possible to pull submodules, with CI, if they had an absolute path, so people came up with various solutions. Then it became possible and there are a few solutions regarding authentication issues.
But they all involve doing a clone which is not needed anymore because now we can set the CI to do a recursive pool.
This means that most of the online posts have become irrelevant and outdated and answers are one liners here and there but after a few hours none has worked for me.variables:
GIT_SUBMODULE_STRATEGY: recursive
What has worked is to make a global admin key, giving access to everything but this is not a good solution at all.
What I have now is:
variables:
GIT_SUBMODULE_STRATEGY: recursive
before_script:
- git config --global url."https://gitlab-ci-token:${CI_JOB_TOKEN}#gitlab.com/".insteadOf "git#gitlab.com:"
- git submodule sync && git submodule update --init
Which, according to Getting GitLab CI to clone private repositories should work.
But it fails with:
fatal: could not read Username for 'https://gitlab.com': No such device or address
I set GIT_SUBMODULE_STRATEGY: none and rewrote submodule URLs to use HTTPS with access/deploy tokens created to only allow reading of the repositories of the submodules. Something like this in .gitlab-ci.yaml:
image: docker:stable
variables:
GIT_STRATEGY: clone
# don't clone submodules by default
GIT_SUBMODULE_STRATEGY: none
someJob:
tags:
- docker-priv
before_script:
- apk update
- apk add git
- git --version
- git config --global url."https://${ACCESS_TOKEN_XYZ_NAME}:${ACCESS_TOKEN_XYZ}#GITLAB-HOST/GROUP/XYZ.git".insteadOf git#GITLAB-HOST/GROUP/XYZ.git
- git config --global url."https://${ACCESS_TOKEN_XYZ_NAME}:${ACCESS_TOKEN_XYZ}#GITLAB-HOST/GROUP/XYZ.git".insteadOf https://GITLAB-HOST/GROUP/XYZ.git
- git submodule sync --recursive
- git submodule update --init --recursive
Please mind, that default names of deploy tokens in gitlab use a + sign and thus need to be url encoded when stored as CI variable for above usage. You may use one variable (gitlab%2Breadonly-token-name-123:randomtokenvalue) instead of two to make it a bit easier to read.
I have a GitLab project that utilises GitLab CI.
The project also uses submodules, both the project and it's submodules are under the same GitLab account.
Here is my .gitmodules file
[submodule "proto_contracts"]
path = proto_contracts
url = https://gitlab.com/areller/proto_contracts.git
I also have this piece in the .gitlab-ci.yml file
variables:
GIT_SUBMODULE_STRATEGY: recursive
However, when i run the CI I get this error
fatal: could not read Username for 'https://gitlab.com': No such device or address
Both the project and the submodules are in a private repository so you would expect to be prompted for authentication, but as I've mentioned, the project and the submodule are under the same account and one of the runner's jobs is to clone the original repository
So it's odd that it's unable to reach the submodule
Is there a way around it?
You must use relative URLs for submodules. Update your .gitmodules as follow:
[submodule "proto_contracts"]
path = proto_contracts
url = ../../areller/proto_contracts.git
Further reading: Using Git submodules with GitLab CI | GitLab Docs
I tried to deploy the Hexo on my GithubPage.
The generate process looks fine, but error happens when I deploy it on my GithubPage.
Here's the deployment part in _config.yml:
# Deployment
## Docs: https://hexo.io/docs/deployment.html
deploy:
type: git
repo: https://github.com/ZhangYuef/ZhangYuef.github.io.git
# branch: Hexo
Generate
Deployment
So what's going on there?
Thx for help! :)
The context you provided in the question is not sufficient...
But according to invalid chars on the screenshot, I suppose that your Chinese file path may be the cause.
References:
Node JS Error: ENOENT
Why does ENOENT mean "No such file or directory"?
try to update the _config.yml like this:
deploy:
type: git
repository: https://github.com/fakeYanss/fakeYanss.github.io.git
branch: master
yaml is very very very strict, and indent is important!
Not sure what reason causing this error.
Check your environment whether these things have been set up.
I think it might be your config type is wrong.
npm install hexo-deployer-git --save
git repository settings like
deploy:
- type: git
repo: git#github.com:xxx.git
branch: master
- type: git
repo: git#github.com:xxx.git
branch: src
extend_dirs: /
ignore_hidden: false
ignore_pattern:
public: .
By this way, you can not only deploy your blog, but also backup your blog files, which you can use the command git pull to get the blog files on another machine.
- set up your ssh
ssh-keygen -t rsa -C "yourEmail#icloud.com"
ssh-agent -s
chmod id_rsa 600
ssh-add id_rsa
(you need to add the id_rsa.pub to the github's deployer key)
ssh -T git#github.com
sometimes it maybe you have several gits, make deployer confused.
Try delete .git directory and make sure there is no any git in other directories.
encoding. It could be the encoding is different. In my case, I make all the files belong to UTF-8.
By the way, it could be your files' error.try npm install hexo-server --save and hexo server to detect whether the website can be deployed.
(http://localhost:4000/xx)