We have a Gitlab Project with multiple developers, and the repo itself is a conan Project.
When creating a release tag, I want to setup a pipeline which creates the conan package and then uploads it to artifactory. Uploading to the Artifactory requires a username and password login. This is similar to many other deployment jobs where a user+pass authentication is required.
I already found a solution to define secret variables for the project (project level) and use a single account for the whole project to upload to artifactory. This is security-wise an issue, as we want to know who uploaded the conan package, i.e., which user.
Is it somehow in Gitlab possible to define secrets on the user level?
I.e., if User1 creates the tag and has his own Artifactory Account User+Pass secrets set up, the pipeline successfuly pushes the conan package.
If now User2 creates a tag but did not setup secrets, the push should fail.
The following Gitlab issue is a similar description of the problem, but does not contain any solution:
https://gitlab.com/gitlab-org/gitlab/-/issues/15815
Also related: gitlab credentials for specific user (but handles a shared secret with specific user access).
CI secrets are currently only project level, but you might be able to do something similar with one of the predefined env variables. There are four variables that hold info about the Gitlab user who started the pipeline (either by trigger, schedule, push, etc). $GITLAB_USER_EMAIL, $GITLAB_USER_ID, $GITLAB_USER_LOGIN, $GITLAB_USER_NAME. Then, in your projects secret variables, you can store credentials for each of your users, and in your job grab the correct one based on the USER variables.
Related
We have an open source project in GitHub. And we use Azure DevOps pipelines for our CI.
We publish our artefacts to S3 and Maven after successful tests, so all the credentials are stored as secret variables.
It's nice that export and echo $top_secret are conveniently obfuscated with ***, but unfortunately literally any user on GitHub can create a pull request against our repo, and as part of the changes, they can edit our azure-pipelines.yml and call a curl (or similar) to read the credentials from environmental variables and send them to their own server.
In other CI providers (Travis CI) secret variables are not accessible from PR branches.
How can I prevent PRs from touching my CI configuration file and do anything with it?
How can I prevent PRs from touching my CI configuration file and do anything with it?
You CI configuration file is save in the GitHub open source and you want to restrict users from changing this file, right? Since we cannot set file permission in the GitHub. we cannot prevent PRs from touching your CI configuration file.
As a workaround, we could create classic editor pipeline in the Azure DevOps and set the CI Trigger, such as below. If users do not have permission to change the build definition, they cannot change your CI build definition.
Update1
Let's say I want to make a variable with the value from Vault.
variables:
$SSH_PRIVATE_KEY: `vault kv get -field=private_key project/production`
before_script:
- echo "$SSH_PRIVATE_KEY"
Is it possible?
Is there another way to use Vault secrets inside pipelines?
Original answer Jul 2019:
You can see it used in before/after script steps, with a revoked token at the end.
See gitlab.eng.cleardata.com pub/pipelines/gcp-ci.yml as an example:
# Obtains credentials via vault (the gitlab-runner authenticates to vault using its AWS credentials)
# Configures the `gcloud` sdk and `kubectl` to authenticate to our *production* cluster
#
# Note: Do not override the before_script or the after_script in your job
#
.auth-prod: &auth-prod
image: cleardata/bionic
before_script:
- |
export CLUSTER_NAME=production
export CLUSTER_LOCATION=us-central1
export CLUSTER_PROJECT_ID=cleardata-production-cluster
- vault login -method=aws -path=gitlab-ci -no-print header_value=gitlab.eng.cleardata.com
- GCP_CREDS=$(vault read -field=private_key_data gitlab-ci/gcp/cleardata-production-cluster/key/deployment-key)
- gcloud auth activate-service-account --key-file=<(base64 -d <<<$GCP_CREDS)
- gcloud auth configure-docker
- gcloud beta container clusters get-credentials $CLUSTER_NAME --region $CLUSTER_LOCATION --project $CLUSTER_PROJECT_ID
after_script:
- vault token revoke -self
Update March 2020: This is supported with GitLab 12.9
HashiCorp Vault GitLab CI/CD Managed Application
GitLab wants to make it easy for users to have modern secrets management. We are now offering users the ability to install Vault within a Kubernetes cluster as part of the GitLab CI managed application process.
This will support the secure management of keys, tokens, and other secrets at the project level in a Helm chart installation.
See documentation and issue.
April 2020: GitLab 12.10:
Retrieve CI/CD secrets from HashiCorp Vault
In this release, GitLab adds support for lightweight JSON Web Token (JWT) authentication to integrate with your existing HashiCorp Vault.
Now, you can seamlessly provide secrets to CI/CD jobs by taking advantage of HashiCorp’s JWT authentication method rather than manually having to provide secrets as a variable in GitLab.
See documentation and issue.
See GitLab 13.4 (September 2020)
For Premium/Silver only:
Use HashiCorp Vault secrets in CI jobs
In GitLab 12.10, GitLab introduced functionality for GitLab Runner to fetch and inject secrets into CI jobs. GitLab is now expanding the JWT Vault Authentication method by building a new secrets syntax in the .gitlab-ci.yml file. This makes it easier for you to configure and use HashiCorp Vault with GitLab.
https://about.gitlab.com/images/13_4/vault_ci.png -- Use HashiCorp Vault secrets in CI jobs
See Documentation and Issue.
See GitLab 13.9 (February 2021)
Vault JWT (JSON Web Token) supports GitLab environments.
To simplify integrations with HashiCorp Vault, we’ve shipped
Vault JWT token support. From the launch, you could restrict access based on
data in the JWT. This release gives you a new dimension for restricting
access to credentials: the environment a job targets.
This release extends the existing Vault JWT token to support environment-based
restrictions too. As the environment name could be supplied by the user running
the pipeline, we recommend you use the new environment-based restrictions with the
already-existing ref_type values for maximum security.
See Documentation and Issue.
We have a helper script baked into our builder images that can convert GitLab CI/CD job variables pointing to secrets into job env vars containing Vault secrets. In our case, we're also using the appRole auth method to limit the validity of the temporary Vault access token.
An example use case would be:
I want a job env var "MY_SECRET_TOKEN" with a value from a Vault secret.
So I add a CI/CD variable called V_MY_SECRET_TOKEN="secret/<path>/<key>"
Then I insert a job step to retrieve the secret value and populate
the MY_SECRET_TOKEN with the value associated with the key.
Variables added to the CICD job setup in GitLab.
VAULT_ADDR=https://vault.example.com:8200
VAULT_ROLE_ID=db02de05-fa39-4855-059b-67221c5c2f63
VAULT_SECRET_ID=6a174c20-f6de-a53c-74d2-6018fcceff64
VAULT_VAR_FILE=/var/tmp/vault-vars.sh
Steps added to the .gitlab-ci.yml job definition.
script:
- get-vault-secrets-by-approle > ${VAULT_VAR_FILE}
- source ${VAULT_VAR_FILE} && rm ${VAULT_VAR_FILE}
Here is a reference to the get-vault-secrets-by-approle helper script we use. Here is a writeup of the thinking behind the design.
The 'before_script' option didn't fit our workflows as we define a combination of privileged and non-privledged stages in our gitlab-ci.yml definition. The non-privileged jobs build and QA code while the privileged jobs package and release code. The VAULT_ROLE_ID and VAULT_SECRET_ID job variables should only be visible to the privileged package and release jobs.
I also experimented with using include's, extend's, and yaml anchors but I wanted to merge items into existing yaml maps (script: {} or before_script: {}) as opposed to replacing all the items in a map with the template.
GitLab now has nice feature called "Releases". You can define "release" as combination of "tag + some description + some URLs" and it will be shown on "Releases" and "Tags" pages of your project. GitLab doc says:
we recommend doing this as one of the last steps in your CI/CD release pipeline
But, wait! CI/CD job by default has no access to API calls or write to git repository. We can configure "deploy token" or "deploy key" for access to repository and use them (via "secret variables") in build scripts. But neither "deploy token", nor "deploy key" give access to API.
So, we can't create release from CI/CD job using its environment variables, we can't use deploy tokens, we can't use deploy keys. So, what exactly GitLab suggests us to do when it says: "we recommend doing this as one of the last steps in your CI/CD release pipeline" ?
This previous question highlighted the same issue, pointing out you need to access in your CI/CD release pipeline to (from doc)
either OAuth2 tokens
Personal access tokens
Session cookie
This is not limited to release.
As seen in gitlab-ce issue 61108: "Allow tags to be managed with CI_JOB_TOKEN"
However, it turns out that tags cannot be removed by simply using the CI_JOB_TOKEN.
Instead I would need to have create an access token and pass this as CI variable to be able to call this API from within the CI jobs.
Other examples:
gitlab-ce issue 60643: "Download releases using deploy tokens."
gitlab-ce issue 58235: "Allow JOB_TOKEN to access Releases API"
However, it turns out the call to this REST API does not work with the JOB_TOKEN header but only with the PRIVATE_TOKEN.
Is this limitation intended?
I don't want to maintain extra Private tokens just for manipulating the assets of the release.
That means for now (June 2019), maintaining an extra Private token, and passing it as CI variable might be the only available workaround, pending those issues to be resolved.
That would use, I supposed, a masked variable (GitLab 11.0+)
2019, I commented:
for now there is no direct way to use the GitLab Releases API from a CI/CD release pipeline.
Update 2022:
See GitLab 15.5 (October 2022)
Update a release using the Release CLI
In this milestone we added the ability to update a release using the Release CLI. You can use this to automate releases by updating any of the release attributes directly from the .gitlab-ci.yml file, and leveraging the CI/CD pipeline to do so.
See Documentation and Issue.
So it is now possible.
Plus (still GitLab 15.5 (October 2022))
Access release description from tag in CI/CD pipeline variable
In past releases, there was no easy way to configure a pipeline that refers to a release or to release notes associated with a tag.
Now, you can refer to this information using two predefined environment variables: $CI_COMMIT_TAG_MESSAGE and $CI_RELEASE_DESCRIPTION.
See Documentation and Issue.
In my project's settings I see this:
Public deploy keys available to any project (15)
Rewind
CFMM Ansible Deployment
LRM Puppet Test
gitlab-runner (lion)
deploy#jasmine
deployer#stridsberg.nu
test-server
gitlab-runner
kijkmijnhuis#SensioLabsInsight
And many more... what are these things for? I know that if I enable one, that key then could clone my repo... but why are these things showing to me? Is there any benefit?
See "Deploy Keys":
Deploy keys allow read-only or read-write (if enabled) access to one or multiple projects with a single SSH key pair.
This is really useful for cloning repositories to your Continuous Integration (CI) server. By using deploy keys, you don't have to setup a dummy user account.
I use them with Jenkins: easy to setup, easy to revoke if needed.
And I use a read-write deploy key for a maven release task to be able to push back to any repo where that key is deployed.
I would like to deploy my Node.js app via GitHub to Azure.
I intend to make the app open source, thus no private info would be published in the repo; however, I still need to push the necessary credentials, API keys, etc. for the app to connect to other services.
How can I deploy the app without resorting to the private Git endpoint, and then awkward copy-pasting between the repos?
Typically you'll want to utilize an npm module like nconf to load environment variables from either a file or environment variables.
config.json is just a JSON document listing your key:value pairs. You'll want to add config.json to your .gitignore file to ensure you don't share your credentials publically.
Within the Azure Portal, you'll want to add your credentials as key:value pairs under Application Settings.
Note: You may be wondering what will happen if config.json is not found. nconf will simply move on to the next chained option. You could continue to chain config options together as in the following code snippet:
var nconf = require('nconf');
// Create nconf environtment
nconf
.file({ file: 'config.json' }) // Committed to repo; public settings
.file({file: 'local_config.json'}) // Not committed to repo; private or dev environment settings
.env();
Persistent data can be stored under d:\home, so I would recommend placing your private customizations there. If they need to be applied to the site in some way, you should do this by writing a deployment hook.
Set configuration as environment variables found in the "App Settings" section under Settings->Application Settings. Rationale here.
Your issue seems to be continuous deployment for Web App via Git from GitHub repo.
So I think #Dark Falcon 's answer is correct.
Azure continuous deployment support GitHub just need to do OAuth authentication in Azure Portal.
Find out the link "set up deployment from source control" at Azure WebApp Dashboard page and do it step by step, as the pictures below.
There is some blogs and vedio tutorials for details of helping you.
The blog explains how to use continuous deployment support for repo hosted on GitHub http://azure.microsoft.com/en-us/blog/using-app-service-web-apps-continuous-deployment-with-github-organizations/.
You also can follow these vedio tutorials to try to do it, as the references below.
http://azure.microsoft.com/en-us/documentation/videos/create-a-nodejs-site-deploy-from-github/
http://azure.microsoft.com/en-us/documentation/videos/deploying-to-azure-from-github/
https://channel9.msdn.com/Series/Windows-Azure-Web-Sites-Tutorials/Github-Continuous-Delivery-in-the-Preview-Portal.
Best Regards.