We have a set of CI pipelines running in Gitlab v15.3.
We have an intermittent issue, where some build targets hang. If we cancel and retry they run to completion. There seems to be no distinct pattern; any targets might be affected, a set of parallel tasks will generally work but one or two jobs hang.
The symptom we see is
Checking out e18f49c1 as XYZ ...
Skipping Git submodules setup
and then no further action, hangs indefinitely. Cancel and retry always seems to work.
We have a relatively large mono-repository and a large gitlab-ci.yml file.
Suggestions please
We have a relatively large mono-repository
Then consider updating your gitlab-ci.yml, optimizing GitLab for large repositories.
At least check if a shallow clone would help:
variables:
GIT_DEPTH: 3
test:
script:
- ls -al
If not, try and split your gitlab-ci jobs into multiple files, to test each of them in isolation and to check if one of them would triggers the issue.
Related
BitBucket Pipelines have the ability to add custom caches which are reused between pipeline runs. ParcelJS makes use of its own .parcel-cache file cache to speed up builds between runs.
I have a team running several CI branches at the same time and we're looking to move to ParcelJS.
As far as I can tell BitBucket creates the same cache regardless of the branch you're on - so I was wondering if it is possible and whether it makes sense to create a custom cache pointing to the .parcel-cache?
As far as I understand, .parcel-cache is only good for the development server
If you restart the dev server, Parcel will only rebuild files that have changed since the last time it ran.
https://parceljs.org/features/development/#caching
I reckon you are running build, lint and test steps in Bitbucket Pipelines so, unless I misunderstand parceljs' cache purpose, you should better not save it. Otherwise you would be wasting build time by uploading and downloading a bundle that would not speed-up your scripts in return.
You should test this in your workstation nonetheless. Like, with a clean clone of your repo, does the presence of this cache speed up the instructions that run in the pipeline? If it does, then you can face the issues of cache usefulness across different branches.
I have three different stages, each should be running in a same container as there are many customizations and installations which can't be processed or mapped to a new container.
install the application and run the main logic(deploy command).
Undeploy command.
Clean up
If Stage 1 fails/success, then Stage 2 should still execute.
And If Stage1/Stage2 fails then only stage 3 should execute.
As there is no option to use same container for three different stages, I thought to club all of them in a single stage also. But once the stage 1 portion failed, it is not proceeding to next steps.
If I keep allow_failure: true, even with necessary portion also, it is not failing.
How to implement this?
As far as I understood, jobs and stages don't normally share the same container.
One workaround for that is to put all your steps and logic in a script file (PowerShell, bash, etc...) and just run it in one job, not making full use of gitlab CI but will do.
I have a few independent scheduled CI jobs.
Check that the infrastructure matches Terraform code.
Check for npm vulnerabilities.
Check that external APIs pass tests.
These are not hermetic. They do not test the fitness of the code in isolation. They could succeed at commit 12345 and then fail at commit 12345 and then succeed again.
I run these daily.
Gitlab lacks the ability to have multiple pipeline types (unlike, say, Buildkite), so I use a variable to control which steps run.
However, I am left with the problem that these checks interfere with the main passed/failed status of commits.
For example, if the Terraform infra check fails, then it's broken and people are notified and whoever is the next to push is informed they fixed it.
These kinds of checks can't be that uncommon, right? How should thse be managed?
It sounds like you're looking for the allow_failure keyword.
https://docs.gitlab.com/ce/ci/yaml/#allow_failure
When allow_failure is set to true and the job fails, the job shows an orange warning in the UI. However, the logical flow of the pipeline considers the job a success/passed, and is not blocked.
job1:
stage: test
script:
- execute_script_that_will_fail
allow_failure: true
I have set up a PR Pipeline in Azure. As part of this pipeline I run a number of regression tests. These run against a regression test database - we have to clear out the database at the start of the tests so we are certain what data is in there and what should come out of it.
This is all working fine until the pipeline runs multiple times in parallel - then the regression database is being written to multiple times and the data returned from it is not what is expected.
How can I stop a pipeline running in parallel - I've tried Google but can't find exactly what I'm looking for.
If the pipeline is running, the the next build should wait (not for all pipelines - I want to set it on a single pipeline), is this possible?
Depending on your exact use case, you may be able to control this with the right trigger configuration.
In my case, I had a pipeline scheduled to kick off every time a Pull Request is merged to the main branch in Azure. The pipeline deployed the code to a server and kicked off a suite of tests. Sometimes, when two merges occurred just minutes apart, the builds would fail due to a shared resource that required synchronisation being used.
I fixed it by Batching CI Runs
I changed my basic config
trigger:
- main
to use the more verbose syntax allowing me to turn batching on
trigger:
batch: true
branches:
include:
- main
With this in place, a new build will only be triggered for main once the previous one has finished, no matter how many commits are added to the branch in the meantime.
That way, I avoid having too many builds being kicked off and I can still use multiple agents where needed.
One way to solve this is to model your test regression database as an "environment" in your pipeline, then use the "Exclusive Lock" check to prevent concurrent "deployment" to that "environment".
Unfortunately this approach comes with several disadvantages inherent to "environments" in YAML pipelines:
you must set up the check manually in the UI, it's not controlled in source code.
it will only prevent that particular deployment job from running concurrently, not an entire pipeline.
the fake "environment" you create will appear in alongside all other environments, cluttering the environment view if you happen to use environments for "real" deployments. This is made worse by this view being a big sack of all environments, there's no grouping or hierarchy.
Overall the initial YAML reimplementation of Azure Pipelines mostly ignored the concepts of releases, deployments, environments. A few piecemeal and low-effort aspects have subsequently been patched in, but without any real overarching design or apparent plan to get to parity with the old release pipelines.
You can use "Trigger Azure DevOps Pipeline" extension by Maik van der Gaag.
It needs to add to you DevOps and configure end of the main pipeline and point to your test pipeline.
Can find more details on Maik's blog.
According to your description, you could use your own self-host agent.
Simply deploy your own self-hosted agents.
Just need to make sure your self host agent environment is the same as your local development environment.
Under this situation, since your agent pool only have one available build agent. When multiple builds triggered, only one build will be running simultaneously. Others will stay in queue with a specific order for agents. Unless the prior build finished, it will not run with next build.
For other pipeline, just need to keep use the host agent pool.
I have a job that build a project and a downstream job that use some scripts to test that.
Is there any way to change result of a build from a downstream build?
I tried using groovy script as below but did not work:
Hudson.instance.items[10].getLastBuild().setResult(hudson.model.Result.UNSTABLE)
You can use parametrised build plugin. It allows you to have your downstream builds as build task. Your upstream build can fail if any of the downstream builds fail.
In the job configuration, section "Post build actions", there's an "Aggregate downstream test results" option.
According to the help:
Because tests often dominates the execution time, a Hudson best practice involves splitting test executions into different jobs, possibly in multiple different jobs.
When you do that, setting test aggregation is a convenient way of collecting all the test results from such downstream test jobs and display it along with the build that they are testing. In this way, people can see the overall test status of the given build quickly.
That should do what you need.