I have an Azure pipeline that needs to access a secret token to contact another service. I've been following the documentation, but it does not seem to work as I'd expect. As a minimal example, I'm trying to write the variable in cachix_token to a file.
- bash: |
set -ex
mkdir -p packages
echo $CACHIX_AUTH_TOKEN > packages/token
env:
CACHIX_AUTH_TOKEN: $(cachix_token)
However, when I download the resulting the token file, the contents are a literal
$(cachix_token)
How do I get yaml to substitute in the secret variable?
Update
Below is a screenshot of where I've defined the secret variable for the pipeline.
As I eventually found, Azure Pipelines doesn't allow forks to access secret variables. So, even though the secret variable is defined for the pipeline, if you're performing a pull request off of a fork, instead of the main repo, then the secret variable is not defined and you'll see the behavior that I explained above.
As explained in the documentation, this can be bypassed through the Make secrets available to builds of forks checkbox in the gui for the pipeline. However, this does open a massive security hole where anyone can craft a malicious PR that gives them a verbatim copy of all your secrets.
If you have literal$(cachix_token) in file it means that Azure Pipeline was not able to replace that variable. As this it means that you don't have it defined anywhere. You may also confirm this using this extension - Print all variables.
Here you have a documentation how to set secret variable. However you can use also Azure Key Vault to store variables and then fetch values from it. Using built-in extension it also load them as secrets.
I repeat your steps:
steps:
- bash: |
set -ex
mkdir -p packages
echo $CACHIX_AUTH_TOKEN > packages/token
cat packages/token
env:
CACHIX_AUTH_TOKEN: $(cachix_token)
and got this:
+ mkdir -p packages
+ echo ***
+ cat packages/token
***
Which means that variable was correctly replaced.
Related
I'm quite new to CI/CD and basically I'm trying to add this job to Gitlab CI/CD that will run through the repo looking for secret leaks. It requires some API key to be passed there. I was able to directly insert this key into .gitlab-ci.yml and it worked as it was supposed to - failing the job and showing that this happened due to this key being in that file.
But I would like to have this API key to be stored in .env file that won't be pushed to a remote repo and to pull it somehow into .gitlab-ci.yml file from there.
Here's mine
stages:
- scanning
gitguardian scan:
variables:
GITGUARDIAN_API_KEY: ${process.env.GITGUARDIAN_API_KEY}
image: gitguardian/ggshield:latest
stage: scanning
script: ggshield scan ci
The pipeline fails with this message: Error: Invalid API key. so I assume that the way I'm passing it into variables is wrong.
CI variables should be available in gitlab-runner(machine or container) as environment variables, they are either predefined and populated by Gitlab like the list of predefined variables here, or added by you in the settings of the repository or the gitlab group Settings > CI/CD > Add Variable.
After adding variables you can use the following syntax, you can test if the variable has the correct value by echoing it.
variables:
GITGUARDIAN_API_KEY: "$GITGUARDIAN_API_KEY"
script:
- echo "$GITGUARDIAN_API_KEY"
- ggshield scan ci
I just installed Libsecret and pointed it to be where my git credentials get saved:
git config --global credential.helper /usr/share/doc/git/contrib/credential/libsecret/git-credential-libsecret
But I really don't understand how to use it at all.. like at ALL.
It's been a pretty bleak experience to work with this, and actually seems like the only good solution on Linux at this point (its only been 2 years since the last update, rather than like 3+ for other options).
Is there a way to revoke a username/password stored on Libsecret? Like I have 0 clue how to wipe it other than to do --unset credential.helper, which just wipes everything. Can I not narrow it down by the repo/link the password being stored is related to? The Credential Manager on Windows makes this rediculously straightforward via the UI
Sorry to complain and talk about Windows' equivalent, but can anyone shine a light on that?
By all means, not set on using Libsecret if there are better alternatives to what I'm trying to do here. Please, any advance would be so greatly appreciated
It is not clear at all how to do this and the libsecret documentation -- https://developer.gnome.org/libsecret/0.18/ -- is just API/library documentation. Which is great, if you are programming an interface into libsecret. But is not great if you are an end user and want to update or remove an entry.
Also, I found that unsetting the git global config entry credential.helper just reverts git to using un-cached credentials. But when I pointed that setting back to git-credential-libsecret, my old password was still saved.
So, the answer to removing or updating a single entry turns out to be relatively simple. But NOT OBVIOUS.
Install Seahorse (https://wiki.gnome.org/Apps/Seahorse) if it isn't already installed. It will show up in your app menu as "Passwords and Keys"
Run Seahorse
Login (keychain) -> https://#github.com | Network Password
Double click or Right-click on it and edit, copy, or delete
As the other answer mentions, Seahorse is a GUI frontend to the same keyring, but there is also a CLI frontend called secret-tool that can access the same things.
No need to fiddle with what is behind a git credential helper.
Any git credential helper supports an "erase" method, as shown, for instance, in the sources of contrib/credential/libsecret/git-credential-libsecret.c:
/*
* Table with helper operation callbacks, used by generic
* credential helper main function.
*/
static struct credential_operation const credential_helper_ops[] = {
{ "get", keyring_get },
{ "store", keyring_store },
{ "erase", keyring_erase },
CREDENTIAL_OP_END
};
So, to erase a credential entry, you would need to type:
printf "protocol=https\nhost=github.com\nusername=<me>" | \
git-credential-libsecret erase
Replace "github.com" and <me> with the actual remote site and remote account username.
If your credential helper is not libsecret but "manager-core" (using the Microsoft GCM which is cross-platform) , that would be:
printf "protocol=https\nhost=github.com\nusername=<me>" | \
git-credential-manager-core erase
If your credential helper is not libsecret but "xxx" (any other helper, "store", "cache", ...):
printf "protocol=https\nhost=github.com\nusername=<me>" | \
git-credential-xxx erase
Simply type your credential helper command: it will display its commands.
In my case:
git-credential-manager-core
Required command was not provided.
Usage:
git-credential-manager-core [options] [command]
Options:
--version Show version information
-?, -h, --help Show help and usage information
Commands:
get [Git] Return a stored credential
store [Git] Store a credential
erase [Git] Erase a stored credential
configure Configure Git Credential Manager as the Git credential helper
unconfigure Unconfigure Git Credential Manager as the Git credential helper
azure-repos Commands for interacting with the Azure Repos host provider
Just make sure it is in your $PATH (it should be in /usr/bin, if not: /usr/lib/git-core)
Older helpers do not display all "action" commands, and use older synonyms for erase (remove or delete)
To check the erase/remove/delete has worked, display your stored password first ("get"), then "erase", then try and display it again, using again the "get" action:
printf "protocol=https\nhost=github.com\nusername=<me>" | \
git-credential-xxx get
If it prompts for you to enter your username/password, that means you have succeeded in deleting your cached entry.
I have also ran into this issue, and did a little bit of research. From what I can understand, under the hood, libsecret saves credentials to the local user keyring (like gnome-keyring). As the other answer mentions, Seahorse is a GUI frontend to the same keyring, but there is also a CLI frontend called secret-tool that can access the same things.
See https://ece.engr.uvic.ca/~frodo/courses/cpp/documents/github_authentication.pdf and http://manpages.ubuntu.com/manpages/bionic/man1/secret-tool.1.html for more details.
I am new to working with Azure DevOps and I am trying to set up build pipelines for multiple projects and share a yml template between them. I will demonstrate more clearly what I want to achieve but first let me show you our projects' structure:
proj0-common/
|----src/
|----azure-pipelines.yml
|----pipeline-templates/
|----build-project.yml
|----install-net-core
proj1/
|----src/
|----azure-pipelines.yml
proj2/
|----src/
|----azure-pipelines.yml
proj3/
|----src/
|----azure-pipelines.yml
The first folder is our Common project in which we want to put our common scripts and packages and use them in the projects. The rest of the folders (proj1-proj3) are .net core projects and act as microservice projects. As you can see, each project has its own azure-pipelines.yml pipeline file and each project resides in its own repository in Github. Then there are the template pipeline files (build-project.yml and install-net-core) which reside in the common project.
All the projects have the same build steps, therefore I would like to use the build-project.yml template for all the three projects (instead of hardcoding every step in every file).
My problem is that since they reside in distinct projects, I cannot access the template files simply, let's say from project3, by just addressing it like this:
.
.
.
- template: ../proj0-common/pipeline-templates/build-project.yml
.
.
.
And [I believe] the reason is that each project will have its own isolated build pool(please do correct me on this if I am wrong).
I was thinking if Azure DevOps had similar functionality to the variable groups but for pipeline templates, that could solve my problem, however, I cannot find such a feature. Could someone suggest a solution to this problem?
Could you copy this use case? I experimented a bit after checking out some of the docs. It had some gaps though, like most of Microsoft's other docs around Azure DevOps.
Say you have azdevops-settings.yml that specifies the pipeline in one of your service branches. In the example below it has two task steps that runs an external template in another repository, but in one of them I supply a parameter that is otherwise set to some default in the template.
Notice I had to use the endpoint tag, otherwise it will complain. Something that could be further specified in the docs.
# In ThisProject
# ./azdevops-settings.yml
resources:
repositories:
- repository: templates
type: bitbucket
name: mygitdomain/otherRepo
endpoint: MyNameOfTheGitServiceConnection
steps:
- template: sometemplate.yml#templates
parameters:
Param1: 'Changed Param1'
- template: sometemplate.yml#templates
In the template I first have the available parameters that I want to pipe to the template. I tried out referencing parameters without pipeing them, like build id and other predefined variables and they worked fine.
I also tried using an inline script as well as a script path reference. The 'test.ps1' just prints a string, like the output below.
# otherRepo/sometemplate.yml
parameters:
Param1: 'hello there'
steps:
- powershell: |
Write-Host "Your parameter is now: $env:Param"
Write-Host "When outputting standard variable build id: $(Build.BuildId)"
Write-Host "When outputting standard variable build id via env: $env:BuildNumber"
Write-Host "The repo name is: $(Build.Repository.Name)"
Write-Host "The build definition name is: $(Build.DefinitionName)"
env:
Param: ${{parameters.Param1}}
BuildNumber: $(Build.BuildId)
- powershell: './test.ps1'
And the separate powershell script:
# otherRepo/test.ps1
Write-Host "Running script from powershell specification"
Output:
========================== Starting Command Output ===========================
Your parameter is now: Changed Param1
When outputting standard variable build id: 23
When outputting standard variable build id via env: 23
The repo name is: mygitdomain/thisRepo
The build definition name is: ThisProject
Finishing: PowerShell
========================== Starting Command Output ===========================
Running script from powershell specification
Finishing: PowerShell
..and so on..
I found only one solution to actually do that. You can reference the parent directory by using an absolute path. The key was to populate the root path using a system variable. The solution for your example:
- template: ${{variables['System.DefaultWorkingDirectory']}}/proj0-common/pipeline-templates/build-project.yml
I have a case that marking variables as secrets loses their value in Release task, please allow me to elaborate further.
Please find below screenshot of Terraform Service principal
The above one works as variables are available in pipeline.
Take scenario, where they are secret and locked.
Now, run the pipeline and it reports required variable not set.
I have added a step to echo these variables, just to see if I can see them, here is the Release task:
I "assume" *** means actual echo, so they word in echo statement.
Not able to understand why the behavior is different:
When in plain text, they are available in pipeline
When marked as secret they are not available.
How to make them available in pipeline?
Updates
doing something like this:
Terraform plan -out main.plan -var "ARM_SUBSCRIPTION_ID=$(TF_VAR_ARM_SUBSCRIPTION_ID)" "ARM_CLIENT_ID=$(TF_VAR_ARM_CLIENT_ID)" "ARM_CLIENT_SECRET=$(TF_VAR_ARM_CLIENT_SECRET)" "ARM_TENANT_ID=$(TF_VAR_ARM_TENANT_ID)" It reports: 2019-03-07T00:21:19.7692360Z ##[command]"terraform" plan -out main.plan -var "ARM_SUBSCRIPTION_ID=***" "ARM_CLIENT_ID=***" "ARM_CLIENT_SECRET=***" "ARM_TENANT_ID=***" -input=false -no-color
get error
2019-03-07T00:21:19.8504985Z Too many command line arguments. Configuration path expected.
So to follow up with this, if make a variable a secret, you cannot access it from any script directly. What I do is that task that I need it decrypted is go to the environment variables section of the task and enter the following.
What this does is decrypts the variable and sets a variable of the same name so tools like Terraform can access it.
I'm setting up my production environment and would like to secure my environment-related variables.
For the moment, every environment has its own application parameters file, which works well, but I don't want every dev in my team knowing the production connection strings and other sensitive stuffs that could appear in there.
So I'm looking for every possibility available.
I've seen that in Azure DevOps, which I'm using at the moment for my CI/CD, there is some possible variable substitution (xml transformation). Is it usable in a SF project?
I've seen in another project something similar through Octopus.
Are there any other tools that would help me manage my variables by environment safely (and easily)?
Can I do that with my KeyVault eventually?
Any recommendations?
Thanks
EDIT: an example of how I'd like to manage those values; this is a screenshot from octopus :
so something similar to this that separates and injects the values is what I'm looking for.
You can do XML transformation to the ApplicationParameter file to update the values in there before you deploy it.
The other option is use Powershell to update the application and pass the parameters as argument to the script.
The Start-ServiceFabricApplicationUpgrade command accept as parameter a hashtable with the parameters, technically, the builtin task in VSTS\DevOps transform the application parameters in a hashtable, the script would be something like this:
#Get the existing parameters
$app = Get-ServiceFabricApplication -ApplicationName "fabric:/AzureFilesVolumePlugin"
#Create a temp hashtable and populate with existing values
$parameters = #{ }
$app.ApplicationParameters | ForEach-Object { $parameters.Add($_.Name, $_.Value) }
#Replace the desired parameters
$parameters["test"] = "123test" #Here you would replace with your variable, like $env:username
#Upgrade the application
Start-ServiceFabricApplicationUpgrade -ApplicationName "fabric:/AzureFilesVolumePlugin" -ApplicationParameter $parameters -ApplicationTypeVersion "6.4.617.9590" -UnmonitoredAuto
Keep in mind that the existing VSTS Task also has other operations, like copy the package to SF and register the application version in the image store, you will need to replicate it. You can copy the full script from Deploy-FabricApplication.ps1 file in the service fabric project and replace it with your changes. The other approach is get the source for the VSTS Task here and add your changes.
If you are planning to use KeyVault, I would recommend the application access the values direct on KeyVault instead of passing it to SF, this way, you can change the values in KeyVault without redeploying the application. In the deployment, you would only pass the KeyVault credentials\configuration.