I haven't found any way to pass secret variables in GitLab CI pipelines except with so-called «protected» variables. Any other variables can be revealed by any committer as every commit/branch goes throw a pipeline and the code can be modified.
I don't like protected variables because they are too complicated. I need to grant access to some variable to certain people like I do in SQL-databases or Linux filesystems. Instead, I have to make a protected variable, a protected branch, a protected environment (premium feature). And I have to add the maintainer permission level to some users. And then (maybe) they will the only people to access my secret variables.
Also, I have no idea how are those variables stored. Usually, I use Hashicorp Vault and now GitLab is the weakest security point.
Is it safe enough?
Are there more reliable methods to keep secrets in CI pipelines?
issue 13784 refers to an encryption at REST, so the security is not... optimal
There is an epic opened to improve that, and you can setup an Vault integration, but there is not one by default.
Issue 61053 is about solving that: "Vault integration for key/value secrets MVC"
More and more teams are starting to store their secrets in Vault.
We should provide a secure way to fetch short-lived tokens from Vault that can be used at runtime by a job in a CI/CD pipeline.
This is for GitLab 12.3, Sept. 2019.
Just to add to the answer of #VonC, here is the general vision expressed by GitLab with regards to Secrets Management and various scenarios of integrating with Vault, including fully embedding it inside: https://about.gitlab.com/direction/release/secrets_management/
Related
I am using bitbucket pipeline to run test cases. In order for test cases to succeed, I need secrets which is stored in google secret manager. Is there any way I can access those secrets within bitbucket pipeline environment ?
There are a couple of options.
In case if these secrets are static, the easiest solution would be adding them to your Repository or Deployment variables. Make sure that they're marked as Secured, so that they will be masked, i.e hidden, in the logs.
Alternatively, if your secrets are rotated and must be fetched from the secrets manager on every build in order to stay up-to-date, you'll need to use corresponding CLI commands in the build script. In order for this to work you will have to give Bitbucket Pipelines access to the secrets in your cloud. For details, check out, for example, this page.
In my Gitlab Repo, I have to run a scheduled JOB which triggers a Pipeline. And this pipeline deletes the old JOB Logs using Gitlab API.
But this API calls needs the Gitlab AccessToken to perform the operation. Initially I though of using CI_JOB_TOKEN variable, which is auto-generated token, but it has no access to Gitlab APIs.
Alternatively I can store Project AccessToken as a Variable in my Schedule Job. But it will be visible to other people also in Project with Maintainer or Owners roles.
Is there any other way, where either I can store my tokens without reveling it to others? Or some mechanism where I can make it run without passing my Project AccessTokens?
Your best bet would be to store the secret in a vault/cloud service, such as HashiCorp Vault, AWS Secrets Manager, Azure Vault, etc. GitLab has the CI_JOB_JWT_V2 token, which can be used to authenticate to cloud services. With this method, you do not need to store any secrets in GitLab at all.
You can also see the Vault integration as another option.
The only other option might be to use a runner that has the secret on the system and lock that runner to your project.
We have a hosted GitLab instance internally and a Nexus repository hosted internally (neither of which touches the open internet). The Nexus repository uses client certificates for authentication. We have a repository in GitLab that is accessed by many developers and we need a way to get the user's client certificate in the runner so we can access Nexus.
Is there a way to specify in the .gitlab-ci.yml a user-specific mount? Putting the user's certificate information in the repository's "variables" is not an option because we have many developers accessing the same project. We (as developers) also don't have access to the runners. I can, however, create a new container/image that the GitLab runner can execute. Any thoughts on how to get the CI pipeline to recognize the user's certificate in the pipeline would be greatly appreciated!
After reading the GitLab documentation and realizing how far behind we were in releases (a major version) I discovered that GitLab now integrates with Vault. This appears to work for exactly our use case.
https://docs.gitlab.com/ee/ci/examples/authenticating-with-hashicorp-vault/
I'm working on an asp.net core project and I'm trying to figure out how to keep my source and my pipelines 100% secret free.
I've got a VM running the azure agent and an azure dev ops pipelines for build and release.
If i delete the site on the VM, the release pipeline will auto-magically recreate it for me and deploy the latest build.
Super cool.
Now I read up on best practices for configuring a .Net core app and I found this article: https://www.humankode.com/asp-net-core/asp-net-core-configuration-best-practices-for-keeping-secrets-out-of-source-control
So its a bad idea to keep secrets in code, that makes perfect sense.
But if i apply the same security principals to Yaml, then surely I shouldn't place secrets in my pipelines either.
But I need the pipelines to be able to just recreate the site from scratch and it should just work. Somehow the site needs to know where its default sql connection is, or it needs to have a key to the azure app config service. I shouldn't have to log onto the VM and create an appsettings.json manually after every release!
So whatever the site needs to operate needs to be included in the pipeline, therefore some artifact, or included in the code.
I've googled for days, but I can't seem to find any info on how to fully automate this.
I've considered creating a custom configuration provider that reads from the actual VM registry, but that feels wrong too.
I basically need a config option that is NOT hosted in the site itself. So i set it up once on the VM and never again.
The approach that Lex Li lists in the comments is the Microsoft recommended way of securing "secrets" in pipelines.
Ben Smith's answer in my opinion is just as good, maybe slightly less secure.
I use this approach in our organization. All of our release pipelines do the final configuration transformation with the appropriate settings based on the environment they are being deployed to.
i.e db connections are transformed at the dev, test and UAT and production deployment stages.
I keep the relevant secrets in the pipeline variables as protected secrets. I do this for 2 reasons:
Only a select number of trusted personnel have access to the release pipeline definitions.
Even if someone does have access to those definitions - you cannot see a secured variable. Even you you "undo the padlock" on the variable tab - you cannot see what the setting is.
Our actual secrets are then stored in our enterprise secret vault.
Using the Azure Key Vault is definitely a good approach. However we already have a centralized place to keep our stuff; I don't want it in 3 spots.
I would be remiss to not include Variable Groups as part of the pipeline process. Same concept as the build / release variables - the difference is you can now share them in one spot.
These are all opinions of course. This is just one way of doing this; which I feel is a pretty good balance of security and flexibility.
In addition to the suggestions in your questions comments, you can also store secrets in the pipeline "Variables" section.
In here you can add variables and then mark them as secret by selecting "Keep this value secret". Once you've saved a secret its value is then obfuscated i.e. you can make use of it but you can no long see its original value within Azure Devops (which admittedly can be rather frustrating if you want to revisit the variable to check it!).
You can then reference the secret variable in your pipeline YAML using the syntax:
$(variable-name)
So this approach keeps secrets safe within Azure Devops until they need to be resolved by the pipeline YAML script.
Take the following case as example.
You've a RESTful API layer secured using OAuth2. In the other hand, to let users authenticate against your APIs, you need to request an access token (i.e. grant_type=password).
In order to request a password access token, client app requires an OAuth Client (key+secret pair).
Now you've configured everything to use continuous integration and continuous deployment.
During a development build, the build script creates test data, including OAuth clients. Obviously, if a build creates test data, it previously drops all data created during automated tests.
So you'll want your client app to use one of OAuth clients and you want to avoid hardcoding one of them, because they're created using the API infrastructure, so they're re-created from scratch on each build.
Think that front-end and back-end are built by different build scripts.
Conclusion & question
What would be a good approach to share secrets between the server and client infrastructure, so both get up and running synchronized with the same security secrets?
Some ideas
Operating system environment variables. I could store those secrets on build machine environment variables. That is, client infrastructure will be always built and deployed with most up-to-date secrets.
Same as #1, but storing those secrets on a shared directory in the build machine.
Regarding TFS/VSTS build (TFS 2015 or later)/release (TFS 2017 or VSTS) system, you just need to check Allow Scripts to Access OAuth Token option in Options/General tab of build definition or release environment, then you can fetch Access OAuth Token by using $(System.AccessToekn) in each task.
Regarding other system, the better approach is to store the access token in system environment variables and remove it at the end, which is similar to the share variable value for other build/release tasks by using "##vso[task.setvariable variable=testvar;]testvalue" (PowerShell) in TFS or VSTS.
On the other hand, you can store encrypted Access Token in system environment for security, then decrypt and use it.
Finally I've ended up with the common build directory to store a JSON file with latest credentials approach where both builds can access it. Each backend build run persists a JSON file containing the whole credentials, and frontend build relies on the whole file.
Anyway, I tried the environment variables' approach, but since both builds are running on the same TFS build agent, client build couldn't see environment variables' changes unless the whole agent service would be restarted.