I have a nested project structure such that there is a submodule within one of my submodules. The project structure is like this:
project A
|_project B
|_project C
|_project D
Where A is a superproject, B and C are submodules of A, and D is a submodule of C. Project's C's file structure does not put project D at the top level of the directory, though this isn't a problem when I clone and update recursively manually. The problem I am encountering occurs only for project D, and only on my gitlab runner. Specifically, when the pipeline runs, it fails to instantiate project D with the following error:
fatal: could not read Username for 'the-gitlab-server': No such device or address
Eventually leading to:
Failed to recurse into submodule path 'Project C'
Projects B and C appear to instantiate correctly according to the job output. Something to do with the nesting appears to be affecting this, and only on the runner, but I cannot figure out what.
I have tried various configurations of .gitlab-ci.yml, both with variables and before_script. For example, in .gitlab-ci.yml:
variables:
GIT_SUBMODULE_STRATEGY: recursive
GIT_STRATEGY: clone
and on the triggering job I have instead tried:
<the-job-that-causes-the-failure>:
stage: smoke
script:
- <job>
before_script:
- git submodule sync --recursive
- git submodule update --init --recursive
Those exact commands correctly instantiate the completed project structure, including D, when I clone the directory personally and then run them, which makes me think the .gitmodules are fine for each project. Both the variables and before_script approach result in the same error.
The url field for each module is using a relative path as suggested in another answer elsewhere that did not resolve my problem.
The versions of git between the runner and my personal VM are the same (2.25).
Appreciate any help you can give; thanks!
I have the same problem, tried the same things and nothing.
I think I will remove recursive submodule strategy an add the inner submodules to the root .gitmodule.
But is a poor solution if works.
Related
I'm struggling to make work a simple CI/CD pipeline with dependencies between stages. This is my .gitlab-ci.yml file:
image: gcc
stages:
- build
- test
build_app:
stage: build
script:
- make
- make install
test_app:
stage: test
script:
- run app ...
dependencies:
- build_app
So the build stage will compile the application and install it under /usr/local/bin/. The above example will fail because the test stage does not find the executable, even when it is states as dependent on the build step (it seems it is not attached by default).
If I define /usr/local/bin/app as an artifact it will also fail, because these must be relative and child of $CI_PROJECT_DIR (reference).
So I'm now trying to do some modifications on it but I feel I do not really understand what is going on there. I finally tried to attach the file compiled (APP) in the repository directory as an artifact (without using make install), and calling that binary for testing (eg. ./APP instead of APP). This way, it works, but I feel like I had to renounce to the make install instruction, and also that there is probably a much better and straightforward way to implement this.
Is there a recommended way to perform this task?
cache won't work either. It is restricted (though not documented) to the same limitations as artifacts.
These are both security measures since it would otherwise provide access to the entire filesystem the gitlab-ci-runner is installed.
The solution would be to install the binary in a path relative to your build directory instead of /usr/local/bin/app and use artifacts.
I'd like Gitlab CI to fetch source code of another project. Is there a better way than adding a read-only deploy key and setting it up in .gitlab-ci.yml?
You can also use GIT SUBMODULES within your project A to refer to project B and then add
GIT_SUBMODULE_STRATEGY: recursive
to the gitlab-ci.yml file in project A.
This also enables you to specifically include a specific branch or commit from your subproject.
https://docs.gitlab.com/ce/ci/git_submodules.html
I have 2 maven projects hosted on Gitlab. Let's call them A and B. Project A depends on project B.
I want to use gitlab-ci to build A.
Here is the gitlab-ci.yml file for project B:
image: maven:3-jdk-8
build:
script: "mvn install -B"
What should the gitlab-ci in project A look like?
Use GIT SUBMODULES with your project A to refer to project B and then add
GIT_SUBMODULE_STRATEGY: recursive
to the gitlab-ci.yml file in project A. Further project B needs also a CI configuration file in the project root.
https://docs.gitlab.com/ce/ci/git_submodules.html
I have my .gitlab-ci.yml file set up in the typical three stages: test, build, deploy. During the build stage, I run a command that compiles my project and puts it in a tarball. The build stage appears to execute successfully because it moves on to the deploy stage, but the deploy stage then says it can't find the tarball. Is it in another directory? What happened to it? Thanks.
For each test gitlab-ci clean the build folder, therefore the output files of the build stage are not available in the deploy stage.
You need to rebuild your project also in the deploy stage.
The "stages" are only useful to order your tests, i.e. avoid to try to do a deploy test if a build test failed.
EDIT:
Since Gitlab 8.6, it is possible using dependencies feature
I was surprised to see the same behaviour (on GitLab 8.4).
I use cmake to create makefiles, then make to build, and then make test to run the test. I run all these in a build/ directory.
I don't want to repeat myself and identify easily which steps are failing. As such, I've created different gitlab-ci stages: cmake, make, test, etc. I then tell gitlab-ci to keep the build directory using the cache option:
cache:
key: "$CI_BUILD_REF_NAME"
untracked: true
paths:
- build/
I think that the key option will keep the same build directory for all stages acting on the same branch. See the gitlab-ci doc here: http://doc.gitlab.com/ce/ci/yaml/README.html#cache
EDIT: Don't use the cache for this! GitLab implemented reusable artifacts between stages in 8.4: https://gitlab.com/gitlab-org/gitlab-ce/issues/3423
The CI runners will have to be adapted to support this. See: https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/issues/336
In our environment we have a node app with a dependency on a submodule which is also a node module. We ran into issues using npm link so we tried using local dependencies (i.e. setting the value of the dependency in package.json to file:./path/to/dep). The problem with this is that when you make a change in the submodule you have to then bump the version & update it in the parent. Is there a better way of dealing with this type of dependency so that I can just make changes in my submodule and have it just propagate to the parent?
If you want changes you make in your submodule to be reflected immediately in your main module, the only way I know to achieve that is to create a symbolic link from your main module's node_modules/ directory to your submodule's directory. I really recommend finding out why npm link doesn't work for you, because it's the nicest way to acieve this. However, if you want to, you can create the link manually too.
For instance, if your submodule's package name is 'wonderful' and your file structure looks like this:
main-module/
sub-module/
Then you can create a symbolic link main-module/node_modules/wonderful pointing to main-module/sub-module by running the following command from the root of the main module:
ln -s ../sub-module ./node_modules/wonderful
And then whatever change you make in the submodule will be immediately used in the main module.
Two notes on this:
Make sure main-module/node_modules/wonderful doesn't exist as an npm install-ed directory before creating the link, or it won't work.
When you npm install again, it will overwrite your symbolic link, so put the above command in a shell script if you want to execute it often.
When you make a change in the submodule, that means you have to make a commit in the parent repo (in order to record the new gitlink, a special entry in the index which memorize the new SHA1 of your submodule)
Why not use that commit opportunity to execute automatically a script which will modify the package.json file with the right information?
That is called a 'clean' script from a content filter driver:
(image shown in "Customizing Git - Git Attributes", from "Pro Git book")
The clean script is:
declared with git config filter.<filtername>.clean ./<filterscript>
used in a .gitattributes file.
Its function will be to fetch the version of the npm submodule and update the package.json file appropriately.