I am using bitbucket pipeline to run test cases. In order for test cases to succeed, I need secrets which is stored in google secret manager. Is there any way I can access those secrets within bitbucket pipeline environment ?
There are a couple of options.
In case if these secrets are static, the easiest solution would be adding them to your Repository or Deployment variables. Make sure that they're marked as Secured, so that they will be masked, i.e hidden, in the logs.
Alternatively, if your secrets are rotated and must be fetched from the secrets manager on every build in order to stay up-to-date, you'll need to use corresponding CLI commands in the build script. In order for this to work you will have to give Bitbucket Pipelines access to the secrets in your cloud. For details, check out, for example, this page.
Related
In my Gitlab Repo, I have to run a scheduled JOB which triggers a Pipeline. And this pipeline deletes the old JOB Logs using Gitlab API.
But this API calls needs the Gitlab AccessToken to perform the operation. Initially I though of using CI_JOB_TOKEN variable, which is auto-generated token, but it has no access to Gitlab APIs.
Alternatively I can store Project AccessToken as a Variable in my Schedule Job. But it will be visible to other people also in Project with Maintainer or Owners roles.
Is there any other way, where either I can store my tokens without reveling it to others? Or some mechanism where I can make it run without passing my Project AccessTokens?
Your best bet would be to store the secret in a vault/cloud service, such as HashiCorp Vault, AWS Secrets Manager, Azure Vault, etc. GitLab has the CI_JOB_JWT_V2 token, which can be used to authenticate to cloud services. With this method, you do not need to store any secrets in GitLab at all.
You can also see the Vault integration as another option.
The only other option might be to use a runner that has the secret on the system and lock that runner to your project.
I'm working on an asp.net core project and I'm trying to figure out how to keep my source and my pipelines 100% secret free.
I've got a VM running the azure agent and an azure dev ops pipelines for build and release.
If i delete the site on the VM, the release pipeline will auto-magically recreate it for me and deploy the latest build.
Super cool.
Now I read up on best practices for configuring a .Net core app and I found this article: https://www.humankode.com/asp-net-core/asp-net-core-configuration-best-practices-for-keeping-secrets-out-of-source-control
So its a bad idea to keep secrets in code, that makes perfect sense.
But if i apply the same security principals to Yaml, then surely I shouldn't place secrets in my pipelines either.
But I need the pipelines to be able to just recreate the site from scratch and it should just work. Somehow the site needs to know where its default sql connection is, or it needs to have a key to the azure app config service. I shouldn't have to log onto the VM and create an appsettings.json manually after every release!
So whatever the site needs to operate needs to be included in the pipeline, therefore some artifact, or included in the code.
I've googled for days, but I can't seem to find any info on how to fully automate this.
I've considered creating a custom configuration provider that reads from the actual VM registry, but that feels wrong too.
I basically need a config option that is NOT hosted in the site itself. So i set it up once on the VM and never again.
The approach that Lex Li lists in the comments is the Microsoft recommended way of securing "secrets" in pipelines.
Ben Smith's answer in my opinion is just as good, maybe slightly less secure.
I use this approach in our organization. All of our release pipelines do the final configuration transformation with the appropriate settings based on the environment they are being deployed to.
i.e db connections are transformed at the dev, test and UAT and production deployment stages.
I keep the relevant secrets in the pipeline variables as protected secrets. I do this for 2 reasons:
Only a select number of trusted personnel have access to the release pipeline definitions.
Even if someone does have access to those definitions - you cannot see a secured variable. Even you you "undo the padlock" on the variable tab - you cannot see what the setting is.
Our actual secrets are then stored in our enterprise secret vault.
Using the Azure Key Vault is definitely a good approach. However we already have a centralized place to keep our stuff; I don't want it in 3 spots.
I would be remiss to not include Variable Groups as part of the pipeline process. Same concept as the build / release variables - the difference is you can now share them in one spot.
These are all opinions of course. This is just one way of doing this; which I feel is a pretty good balance of security and flexibility.
In addition to the suggestions in your questions comments, you can also store secrets in the pipeline "Variables" section.
In here you can add variables and then mark them as secret by selecting "Keep this value secret". Once you've saved a secret its value is then obfuscated i.e. you can make use of it but you can no long see its original value within Azure Devops (which admittedly can be rather frustrating if you want to revisit the variable to check it!).
You can then reference the secret variable in your pipeline YAML using the syntax:
$(variable-name)
So this approach keeps secrets safe within Azure Devops until they need to be resolved by the pipeline YAML script.
I haven't found any way to pass secret variables in GitLab CI pipelines except with so-called «protected» variables. Any other variables can be revealed by any committer as every commit/branch goes throw a pipeline and the code can be modified.
I don't like protected variables because they are too complicated. I need to grant access to some variable to certain people like I do in SQL-databases or Linux filesystems. Instead, I have to make a protected variable, a protected branch, a protected environment (premium feature). And I have to add the maintainer permission level to some users. And then (maybe) they will the only people to access my secret variables.
Also, I have no idea how are those variables stored. Usually, I use Hashicorp Vault and now GitLab is the weakest security point.
Is it safe enough?
Are there more reliable methods to keep secrets in CI pipelines?
issue 13784 refers to an encryption at REST, so the security is not... optimal
There is an epic opened to improve that, and you can setup an Vault integration, but there is not one by default.
Issue 61053 is about solving that: "Vault integration for key/value secrets MVC"
More and more teams are starting to store their secrets in Vault.
We should provide a secure way to fetch short-lived tokens from Vault that can be used at runtime by a job in a CI/CD pipeline.
This is for GitLab 12.3, Sept. 2019.
Just to add to the answer of #VonC, here is the general vision expressed by GitLab with regards to Secrets Management and various scenarios of integrating with Vault, including fully embedding it inside: https://about.gitlab.com/direction/release/secrets_management/
I have a wonderful terraform plan that perfectly describes my infrastructure in Google Cloud Platform, however, I have a problem: since my repository isn't perfectly private, some steps of my plan are encrypted and must be decrypted using Google Key Management Service.
This means my plan must be broken down into two terraform phases:
Setup the Google Cloud Project and create a Key Ring and Key (after this, I encrypt secrets and put them in a variables.tf file)
Apply the entire plan.
Does Terraform support a way to break down my plan into phases? How should I go around implementing this?
Though terraform enables us to automate the resources creation, some preliminary steps need to be done manually, like account creation, billing setup, etc. Similarly for Google cloud setup, the project needs to be created prior running terrform scripts since terraform google provider requires the project details.
The project creation and terraform variables for the keys (as environment variables) can be generated through shell scripts. Then the shell script and the terraform scripts can be sequenced in execution using a make file.
The below link might be helpful for you to create GCP project through shell scripts.
https://medium.com/google-cloud/how-to-automate-project-creation-using-gcloud-4e71d9a70047
What I want to accomplish:
I want to deploy an Azure Cloud Service via Release Management. I managed to get this working by following the steps outlined in this post. In the post the Azure publishsettings file is added to the project and used in Release Management to deploy the Azure package to a Cloud Service. So far so good.
What is the issue:
The Azure publishsettings file will also contain information about the production environment. I don't want that information to be available to all the developers and therefor I would like to have a more secure alternative.
What did I try:
I created a custom action which takes 3 arguments: subscription id, subscription name and certificate key. This way the Azure information stays in Release Management and can be passed to a script. This didn't work because the action is not shown in the Release Template Toolbox.
What is my question:
What is the best way to pass Azure credentials to a deployment script via Release Management on a secure manner?
We have a solution for Build today that will work for RM in the future.
Publish Settings file is an important one with which anybody can get access to certain activities. And once how ever the way you pass on the publish settings file, it can be misused (if tried).
So along with the publish settings file, you need to add a bit of process to the deployment like -
Inactive or remove the management certificate which will in turn invalidate the given publish settings and anyone should request for a new set of publish settings file before they actually start any release procedures.
Even though it adds a rough edge to your smooth flow of deployment process, as it is a live or production system, it is always better to tight the process and make it idiot proof.