I'm having some troubles understanding how to activate Secrets Detection in Gitlab CI/CD.
I created a new NodeJS Express from template and then i activated auto-devops from Settings > CI/CD and checked the checkbox Default to Auto Devops pipeline under the Auto Devops Menu. After that i opened the app.js file in the project folder and inserted a variable that looks like a key-value. Here's the piece of code where i inserted the line:
...
var app = express();
var key = "api-12321321321321321";
// view engine setup
app.set('views', path.join(__dirname, 'views'));
...
After committing the changes i expected the pipeline to fail because of the leak of the secret. Here's an images that shows that secret dection passed.
Can anyone tell me how to make so that the pipeline reports the error?
This behavior confused me as well when I first tried it out.
However, it seems that GitLab did this on purpose. Here is what the official documentation says:
So with a free or premium account, you can just use this reporter, but you won't see any results, unless you download the JSON report.
Also, there is NO mention that the job will fail. This is just our expectation.
However, there are reasons why this makes sense, especially for a paid version of GitLab. In an MR, you want to know if there are any vulnerabilities in your code. However, not all reports are accurate. So you may get what is known as a false positive. Or, the reported issue is very generic and your software is not affected.
However, if you are on a free/premium account, this feature is almost useless, as nobody will go to the job and manually inspect it.
The only workaround would be to override the secret_detection job, parse the gl-secret-detection-report.json, check if it passed or failed and decide to PASS or FAIL the JOB.
This should be easier with GitLab 13.12 (May 2021, Ultimate only):
Configuration tool for Secret Detection
Following in the footsteps of the GitLab SAST configuration tool we are adding support for Secret Detection on the Security Configuration page.
We believe that security is a team effort and this configuration experience makes it easier for non-CI experts to get started with GitLab Secret Detection.
The tool helps a user create a merge request to enable Secret Detection scanning while leveraging best configuration practices like using the GitLab-managed SAST.gitlab-ci.yml template.
The Configuration tool can create a new .gitlab-ci.yml file if one does not exist or update existing simple GitLab CI files, allowing the tool to be used with projects that already have GitLab CI setup.
See Documentation and Epic.
See GitLab 14.5 (November 2021, for all tiers)
Additional Secret Detection pattern support
We’ve updated the GitLab Secret Detection scanner to detect 47 new ‘well-identifiable’ secret patterns for widely used applications. This brings the GitLab Secret Detection detection up to over 90 detectable patterns.
If you are a SaaS application vendor and your app generates secret tokens with well-identifiable patterns, and you’d like GitLab to be able to detect them, please add your regex pattern and a few invalid sample tokens in a comment on this issue and we’ll get them added to GitLab Secret Detection.
See Documentation and Issue.
GitLab has a full post about setting this up in a pipeline: https://docs.gitlab.com/ee/user/application_security/secret_detection/
EDIT:
The given instructions are a bit unclear:
You need to add the include tag at "root-level" of your configuration.
Example
stages:
- build
- test
image: node:latest
build:
stage: build
script:
- echo "Building"
- npm install typescript
- yarn run build
test:
stage: test
script:
- echo "Testing"
include:
- template: Security/Secret-Detection.gitlab-ci.yml
The secret detection will run in the test stage:
Pipeline
Related
I have notified the complexity of my configuration for Gitlab CI/CD grows fairly fast. Normally, in a either a programming language or infrastructure code I could write automated test for the code.
Is it possible to write automated test for the gitlab-ci.yml file itself?
Does it exist any library or testing framework for it?
Ideally I would like to setup the different environments variables, execute the gitlab-ci.yml file, and do assertions on the output.
I am not aware of any tool currently to really test behaviour. But i ended up with two approaches:
1. testing in smaller chunks aka unit testing my building blocks.
I extracted my builds into smaller chunks which i can include, and test seperately. This works fine for script blocks etc. but is not really sufficient for rule blocks - but offers great value, eg i use the templates for some steps as verification for the generated docker image, which will be used by those steps.
2. verification via gitlab-ci-local
https://github.com/firecow/gitlab-ci-local is a tool which allows you to test it locally, and which allows you to provide environment variables. There are some minor issues with branch name resolution in pr pipelines but besides that it works great. I use it to test gitlab ci files in GitHub Actions.
I hope that this helps somehow
Gitlab have a linter for gitlab-ci.yaml
To access the CI Lint tool, navigate to CI/CD > Pipelines or CI/CD > Jobs in your project and click CI lint.
Docs
If I recall correctly, Azure Devops was set to add a feature to allow release pipelines to be placed into yaml , similar to build pipeline. Is this feature available yet?
Im hoping to get some guidance on the migration of legacy classic release pipeline, to the new format?
UPDATE: It looks like I'm only about 4 months behind. Posted on May 6, 2019
With our new updates, development teams can now leverage the same YAML documents to build multi-stage pipelines-as-code for both Continuous Integration and Continuous Delivery. This was one of the biggest requests from our customers
here are some helpful links for those that are like me : have many classic release pipelines manually created, interested in the new format.
https://azure.microsoft.com/en-us/updates/unified-pipelines/
https://azure.microsoft.com/en-us/blog/accelerating-devops-with-github-and-azure/
For migrating release classic pipelines, Azure DevOps still lacks some more straightforward support.
But there is an external tool, yamlizr, for helping migrate many release pipelines.
In my case I had to hack a bit the tool (see its issues if they are still opened).
The result is not production ready, but is still a good starting point from my viewpoint, and helps a lot.
I've discovered that Microsoft provides some built-in help to migrate a release pipeline from a classic GUI-based Release pipeline to a YAML-based pipeline.
If you edit your classic GUI-based Release pipeline and drill down to a task, there is a link at top right that says View YAML:
Clicking the link generates the YAML associated with the task. Unfortunately it doesn't include the field values. Instead it uses variables and doesn't specify the values of those variables:
steps:
- task: IISWebAppManagementOnMachineGroup#0
displayName: 'IIS Web App Manage'
inputs:
IISDeploymentType: '$(Parameters.IISDeploymentType)'
ActionIISWebsite: '$(Parameters.ActionIISWebsite)'
WebsiteName: '$(Parameters.WebsiteName)'
WebsitePhysicalPath: '%SystemDrive%\inetpub\wwwroot\AspNetDemo'
AddBinding: '$(Parameters.AddBinding)'
Bindings: '$(Parameters.Bindings)'
ParentWebsiteNameForVD: '$(Parameters.WebsiteName)'
VirtualPathForVD: '$(Parameters.VirtualPathForApplication)'
ParentWebsiteNameForApplication: '$(Parameters.WebsiteName)'
VirtualPathForApplication: '$(Parameters.VirtualPathForApplication)'
AppPoolName: '$(Parameters.AppPoolName)'
Luckily the task in the GUI has links against most fields, with field-specific help. That help mentions which variable maps to the field value:
So you can go through each field in the task determining which variable corresponds to each field, to work out what values to add to the YAML snippet.
It's a bit labour intensive but it's quicker than trying to figure it all out from scratch.
Unfortunately there doesn't seem to be any way to view the YAML at the job or stage level, only at the task level. So you'll have to repeat the process for each task. You'll also have to work out the YAML that defines the stage and the jobs yourself.
Create a new multi-stage pipeline-as-code.
https://azure.microsoft.com/en-us/updates/unified-pipelines/
https://azure.microsoft.com/en-us/blog/accelerating-devops-with-github-and-azure/
Today, while trying to setup CodeQuality in my .gitlab-ci.yml (on gitlab-ee 11.10, gitlab-runner 11.10) I encountered the following issue:
First thing the GitLab documentation tells you is it is possible to set this with help of a DockerInDocker gitlab-runner and a single configuration line:
include:
- template: Code-Quality.gitlab-ci.yml
There is also disclaimer that this is supported in gitlab 11.11 or later, which is strange because as of April 23rd 2019, the latest version is 11.10. Not sure if this a typo or if they publish documentation before the releases are actually available.
I tried following this instructions but many things are unclear:
I realized that the include statement should be added after the stages definition for the syntax check to pass.
A code_quality job showed up and passed:
(source: cozyo.io)
However, later I learned from this answer that I need to create a .codeclimate.yml file and somehow add it .gitlab-ci.yml. That answer shares two links that can be used to understand how to work with CodeClimate but I'm haven't figured a way to add to gitlab-ci.yml. I found some example in this gitlab related page BUT this is not using the include statement described in the docs.
I'm unable to find the report of the code_quaity job that passed. In this answer someone pointed out that the report is only available for download on the merge request for gitlab-ee. However this is not practical since then devs would have to start spamming mock merge request just to see if they code suffered down grades.
The gitlab-ci.yml I use looks something like this:
image: docker:stable
variables:
ARTEFACT: my_app
VERSION: 0.1
DOCKER_HOST: tcp://docker:2375/
DOCKER_DRIVER: overlay2
services:
- docker:dind
before_script:
- docker info
stages:
- build
- test
- deploy
build:
stage: build
script:
- docker build -t $ARTEFACT:$VERSION-DEV .
test:
stage: test
script:
- docker run --rm --env MODE=DEV $ARTEFACT:$VERSION-DEV ./my_test.sh
include:
- template: Code-Quality.gitlab-ci.yml
Ideally, this should be a simple as figuring out how to use CodeClimate for python applications and then adding its configuration to the repo and referencing in gitlab-ci.yml correct? But how to make the reference. Is there any clear documentation available somewhere?
EDIT: I now know that jobs are independent and that I should modify the build stage in the .gitlab-ci.yml above to push the build image somewhere other jobs can pull from. Still this doesn't help to solve the CodeQuality problem I think.
First, yes, GitLab often has docs for features of the coming release before the official release. The docs often apply to pre-release versions which are often running on GitLab.com.
The docs could probably be clearer, so I encourage you to file an issue with specifics on how the docs could be improved or even submit a MR if you feel comfortable with that.
Do note that code quality is a paid feature so requires a Starter license for a self-managed install (or Bronze on GitLab.com).
The code quality process will store information based on the first run in a codeclimate.json file. This is merely reference for future runs and is not the code quality report itself. Upon subsequent runs, the code climate report will be viewable in the Merge Request which kicked off the pipeline.
In order for the code climate report to appear in the Merge Request, the following conditions must be met:
You've defined a job in your .gitlab-ci.yml which produces the Code Quality report artifact
The job must not be the first run. If the Code Quality report doesn’t have anything to compare to, no information will be displayed in the merge request area. That is the case when you add the Code Quality job in your .gitlab-ci.yml for the very first time. Consecutive merge requests will have something to compare to and the Code Quality report will be shown properly.
A code error must be present. There has to be a degradation/error in your code as this is what will be displayed in the merge request page. If the code is correct there will be nothing displayed.
The .json file must not be too large. There is an existing issue which prevents the code report from displaying when the related .json file is very large.
We are trying to get a clean YAML build going, and ran into a quirk: The build has an extra "Get Sources" step at the end, which is not in our YAML file, and can't be removed using the UI.
I created an azure-pipelines.yml file in the root of a new Azure Git (not GitHub) repo. A build definition was automatically created under OurRepo/OurRepo CI in the Builds pane on Azure DevOps.
The build works, but note the extra step at the end:
When I edit the job in Azure's UI via Pipeline Settings, I notice a "Get Sources" task that cannot be removed:
While this non-removable step makes sense for GUI-defined builds, I'm trying to go "pure YAML". The extra pull doesn't take long, so it's not a big deal, just annoying.
Apparently other users have this extra step in YAML builds as well: try googling "Post-job: Get sources".
Am I doing something wrong, or is this just a quirk with Azure Git repos using YAML builds? (The MS Docs tutorial uses a regular GitHub repo, I noticed).
Edit: I have also tried creating a build definition from YAML via New Build Definition > Azure Git Repo > YAML. The resulting page fails to detect the azure-pipeilnes.yml file (whether that file is empty or has a known working build definition when I committed it--tried both), so I ended up in the same place.
I doubt you can since it appears built-in to the pipeline. Is the output of that task have similar to my post task. Although it is labeled "Post-job Checkout". It looks like a clean-up step to me.
2019-01-30T21:39:38.1940431Z ##[section]Starting: Checkout
2019-01-30T21:39:38.2032443Z ==============================================================================
2019-01-30T21:39:38.2032500Z Task : Get sources
2019-01-30T21:39:38.2032550Z Description : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.
2019-01-30T21:39:38.2032583Z Version : 1.0.0
2019-01-30T21:39:38.2032794Z Author : Microsoft
2019-01-30T21:39:38.2032822Z Help : [More Information](https://go.microsoft.com/fwlink/?LinkId=798199)
2019-01-30T21:39:38.2032852Z ==============================================================================
2019-01-30T21:39:38.5783539Z Cleaning any cached credential from repository: Sandbox (Git)
2019-01-30T21:39:38.5854582Z ##[section]Finishing: Checkout
The post job step is to wipe out the downloaded source content from the hosted agent machine. There is no way that the user has control over it and it is a built-in feature.
I am wondering what is the best way to push code from a CI/CD job in Gitlab. For example,
project1: where you run the gitlab-ci
project2: where you want to push your code
Both projects are in Gitlab, and the idea is that we push code from the CI at proejct1 to project2. The idea is that based on project1 we compile and build some artifacts that later are pushed to project2.
One quick solution would be to use a PERSONAL_ACCESS_TOKEN:
git push https://${GITLAB_USER_LOGIN}:${PERSONAL_ACCESS_TOKEN}#<your-gitlab-server>/<your-project-2-name>.git HEAD:master
To configure the personal token:
Your user profile - Settings - Access Token - "Create personal access token"
Project 1 - Settings - CI/CD - Secret variables
PERSONAL_ACCESS_TOKEN:
But this solution has some major problems:
All members of the project will be able to see the PERSONAL_ACCESS_TOKEN
If you enable the CI_DEBUG_TRACE flag, the personal token will be in the output of your job
I guess it can be a temporal solution. But I am wondering what is the best solution.
Thanks!
Project mirroring seems to be the feature you are looking for :
Go to your project page
Settings->Repository Settings
There, you can configure pushong to another repository, only protected branches or everything, you choose.
I hope it'll help you.
If not, please provide more info.