How to retrieve certificate from a keyvault and import it into another without saving it? - azure

In an Azure pipeline there are following tasks
AzureResourceManagerTemplateDeployment#3 deploys a Key Vault from an ARM template
Then a AzurePowerShell#5 checks if the Key vault contains a "my-self-signed-cert" and if not - imports it into the Key Vault
Finally another AzureResourceManagerTemplateDeployment#3 deploys a Service Fabric cluster and configures the SF cluster and its VMSS to use the certificate
Here are the tasks:
- task: AzureResourceManagerTemplateDeployment#3
displayName: 'Deploy Keyvault'
inputs:
deploymentScope: 'Resource Group'
subscriptionId: '${{ parameters.SubscriptionId }}'
azureResourceManagerConnection: '${{ parameters.ArmConnection }}'
action: 'Create Or Update Resource Group'
resourceGroupName: '${{ parameters.resourceGroupName }}'
location: 'West Europe'
templateLocation: 'Linked artifact'
csmFile: '$(Build.SourcesDirectory)/pipelines/templates/keyvault.json'
csmParametersFile: '$(Build.SourcesDirectory)/pipelines/templates/keyvault-params.json'
deploymentMode: 'Incremental'
- task: ARM Outputs#5
displayName: 'Collect Keyvault output'
inputs:
ConnectedServiceNameSelector: 'ConnectedServiceNameARM'
ConnectedServiceNameARM: '${{ parameters.ArmConnection }}'
resourceGroupName: '${{ parameters.resourceGroupName }}'
whenLastDeploymentIsFailed: 'fail'
- task: AzurePowerShell#5
displayName: 'Import certificate'
inputs:
azureSubscription: '${{ parameters.ArmConnection }}'
ScriptType: 'InlineScript'
azurePowerShellVersion: '3.1.0'
Inline: |
$Cert = Get-AzKeyVaultCertificate -VaultName my-kv -Name my-self-signed-cert
if (!$Cert) {
$Base64 = 'MIIWMgIBA___3000_chars_here____o7WqDoWm5I7fg=='
$Cert = Import-AzKeyVaultCertificate -VaultName my-kv -Name my-self-signed-cert -CertificateString $Base64
}
# set the pipeline variables Thumbprint and SecretId - needed for SF deployment
echo "##vso[task.setvariable variable=Thumbprint]$($Cert.Thumbprint)"
echo "##vso[task.setvariable variable=SecretId]$($Cert.SecretId)"
# deploy SF cluster by ARM template and use the SF Cluster certificate thumbsprint as admin cert
- task: AzureResourceManagerTemplateDeployment#3
displayName: 'Deploy SF cluster'
inputs:
deploymentScope: 'Resource Group'
subscriptionId: '${{ parameters.SubscriptionId }}'
azureResourceManagerConnection: '${{ parameters.ArmConnection }}'
action: 'Create Or Update Resource Group'
resourceGroupName: '${{ parameters.resourceGroupName }}'
location: 'West Europe'
templateLocation: 'Linked artifact'
csmFile: '$(Build.SourcesDirectory)/pipelines/templates/sfcluster.json'
csmParametersFile: '$(Build.SourcesDirectory)/pipelines/templates/sfcluster-params.json'
overrideParameters: '-certificateThumbprint $(Thumbprint) -sourceVaultResourceId $(KeyvaultId) -certificateUrlValue $(SecretId)'
deploymentMode: 'Incremental'
This works well, but now I am trying to replace the self-signed certificate by a real certificate, hosted at another Key Vault.
My plan is to download the new certificate contents (including the key) from the other Key Vault, then Base64-encode it (to avoid creating any temporary files) - and finally Import-AzKeyVaultCertificate ... -CertificateString $Base64 into my Key Vault (see the "Step 2" in my list of tasks).
My problem is that I am stuck in retrieving the certificate contents.
I am able to retrieve the "real" certificate with the following PowerShell commands:
$Cert = Get-AzKeyVaultCertificate -VaultName the-company-kv -Name the-real-cert
$Secret = Get-AzKeyVaultSecret -VaultName the-company-kv -Name the-real-cert
They above commands return some metadata, but there is nothing resembling the contents that I would be able to (if not already base64-encoded):
$Base64 = [System.Convert]::ToBase64String($Bytes)
Import-AzKeyVaultCertificate -VaultName my-kv -Name my-self-signed-cert -CertificateString $Base64

Here a solution for how to copy a certificate from one Key Vault to another (here: the-company-kv -> my-kv) without saving it into a temporary file:
$Cert = Get-AzKeyVaultCertificate -VaultName my-kv -Name the-real-cert
if (!$Cert) {
$OrigCert = Get-AzKeyVaultCertificate -VaultName the-company-kv -Name the-real-cert
$Secret = Get-AzKeyVaultSecret -VaultName the-company-kv -Name $OrigCert.Name
$Cert = Import-AzKeyVaultCertificate -VaultName my-kv -Name $OrigCert.Name -CertificateString $Secret.SecretValueText
}
I didn't realize, that PowerShell is not showing all properties, when I enter $Secret at the command prompt and thus I didn't see the $Secret.SecretValueText at first.

Related

Azure SQL Server CICD Bicep and Key Vault

I have built a CICD Pipeline to deploy an Azure SQL Server and DB and part of this process is to obtain the username and password from secrets in a Key Vault. The Key Vault task in YAML works and can access the Vault, but when it calls an Azure CLI Task to execute the Bicep, it fails with the following error:
ERROR: Unable to parse parameter: **
My code:
steps:
- task: AzureKeyVault#2
displayName: 'Download Key Vault Secrets'
inputs:
connectedServiceName: ${{ parameters.AzureResourceManagerConnection }}
keyVaultName: ${{ parameters.keyVaultName }}
secretsFilter: '*'
- task: AzureCLI#2
displayName: '${{ parameters.deploymentType }}: ${{ parameters.targetEnvironment }} ${{ parameters.product }}'
inputs:
azureSubscription: ${{ parameters.AzureResourceManagerConnection }}
scriptType: 'ps'
scriptLocation: 'inlineScript'
inlineScript: |
$tags = "${{ parameters.releaseFolder }}/${{ parameters.targetEnvironment }}/tags.json"
$products = "${{ parameters.releaseFolder }}/${{ parameters.targetEnvironment }}/products.json"
$productDeploymentFile = "${{ parameters.releaseFolder }}/SQLServer.bicep"
az --version
az deployment group ${{ parameters.deploymentType }} --name ${{ parameters.targetEnvironment }}-${{ parameters.product }}-products-deployment --resource-group ${{ parameters.resourceGroup }} --template-file $productDeploymentFile --parameters $tags $products "$(sql-admin-username)" "$(sql-admin-password)" --mode ${{ parameters.deploymentMode }}
And the Bicep file:
param tags object
param sqlServers array
param sqlDatbases array
param sqlAdminUserName string
#secure()
param sqlAdminPassword string
resource sqlServer 'Microsoft.Sql/servers#2021-08-01-preview' = [for sql in sqlServers: {
name: sql.Name
location: sql.location
properties: {
administratorLogin: sqlAdminUserName
administratorLoginPassword: sqlAdminPassword
administrators: {
azureADOnlyAuthentication: false
administratorType: sql.administratorType
principalType: sql.principalType
login: sql.login
sid: sql.sid
tenantId: sql.tenantId
}
}
tags: tags
}]
Is there anything obvious that is incorrect?
Looking at your yaml file, the way you are passing inline parameters is not correct (see documentation), you would need to specify the parameter names as well:
az deployment group `
...
--parameters `
sqlAdminUserName="$(sql-admin-username)" `
sqlAdminPassword="$(sql-admin-password)" `
...

How to pass KeyVault secrets to a template or a script file in Azure Pipelines?

I have this YAML file:
steps:
- task: AzureKeyVault#2
displayName: Get secret from AzureVault
inputs:
azureSubscription: 'subName'
KeyVaultName: 'vaultName'
SecretsFilter: 'mySecret'
RunAsPreJob: true
- template: \pipelines\templates\vm_setup.yml
parameters:
os_pass: $(mySecret)
How do I use mySecret inside myScript.ps1 or inside myTemplate.yml?
I tried to pass it as an argument, or map it to an env variable then pass that env variable as an argument but neither worked!
My myTemplate.yml looks like this:
parameters:
- name: os_pass
type: string
steps:
- task: PowerShell#2
displayName: Trial
inputs:
targetType: 'filepath'
filePath: '${{ parameters.workingDirectory }}\myScript.ps1'
arguments: >-
- OS_Pass ${{ parameters.os_pass }}
And this is myScript.ps1
param (
[Parameter(Mandatory = $true)]
[string]$OS_Pass
)
$password = ConvertTo-SecureString -String $OS_Pass -AsPlainText -Force
$cred = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList 'admin', $password
Write-Host '---------'
Write-Host $OS_Pass
Doing so the secret is now a string! How do I pass it without changing its type?
Take a look at the official documentation here: https://learn.microsoft.com/en-us/azure/devops/pipelines/process/set-secret-variables?view=azure-devops&tabs=yaml%2Cbash.
There are a few different ways to accomplish what you want to do. Without knowing exactly what you want to do, it is hard to recommend a specific method to use. Take a look above and let us know if that resolves your issue.

Only SPN credential auth scheme is supported for non windows agent

I have a linux agent where I want to run a PowerShell script with an azcli command, using Azure Resource Manager service connection.
This is the task I am using :
- task: AzurePowerShell#5
displayName: 'Add webapp OutboundIPs into SA FW'
inputs:
azureSubscription: ${{ parameters.serviceConnection }}
ScriptType: 'FilePath'
ScriptPath: '$(path)/update-SA-firewall.ps1'
ScriptArguments: '-webappOutboundIPs "$(webappOutboundIPs)" -SAName $(SAName) -RG ${{ parameters.resourceGroupName }}'
azurePowerShellVersion: 'LatestVersion'
And this is the script:
Param(
[string] [Parameter(Mandatory=$true)] $webappOutboundIPs,
[string] [Parameter(Mandatory=$true)] $SAName,
[string] [Parameter(Mandatory=$true)] $RG
)
# get the Array of IPs from the given string
$IPs = $webappOutboundIPs.Split(",")
# Add these IPs into the SA Firewall
foreach ($ip in $IPs) {
az storage account network-rule add -g $RG --account-name $SAName --ip-address $ip | out-null
}
The error I get is :
Line | 106 | throw ("Only SPN credential auth scheme is
supported for non wind …
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| Only SPN credential auth scheme is supported for non windows
| agent.
##[error]PowerShell exited with code '1'.
Am I missing something here :/ ?

Delete the Azure Resources if any of the task in azure pipeline fails

I have Azure Pipeline which creates VMs, Storage , NICs etc. And i want that these resources should be deleted if any of the task in the pipeline got failed. How this can be achieved, do I need to use script in my YAML or is there any extension available?
A sample for you. You just need to output the vars and then use condition to check:
trigger:
- none
pool:
vmImage: windows-latest
steps:
- task: AzurePowerShell#5
displayName: Create Storage1
name: createstorage1
inputs:
azureSubscription: 'xxx'
ScriptType: 'InlineScript'
Inline: |
$resourceGroup = "xxx"
$location = "westus"
$accountName = "bowman08191"
Write-Host "##vso[task.setvariable variable=resourceGroup;isoutput=true]$resourceGroup"
Write-Host "##vso[task.setvariable variable=location;isoutput=true]$location"
Write-Host "##vso[task.setvariable variable=accountName;isoutput=true]$accountName"
New-AzStorageAccount -ResourceGroupName $resourceGroup `
-Name $accountName `
-Location $location `
-SkuName Standard_RAGRS `
-Kind StorageV2
azurePowerShellVersion: 'LatestVersion'
- task: AzurePowerShell#5
displayName: Create Storage2
name: createstorage2
inputs:
azureSubscription: 'xxx'
ScriptType: 'InlineScript'
Inline: |
$resourceGroup = "xxx"
$location = "westus"
$accountName = "bowman08192"
Write-Host "##vso[task.setvariable variable=resourceGroup;isoutput=true]$resourceGroup"
Write-Host "##vso[task.setvariable variable=location;isoutput=true]$location"
Write-Host "##vso[task.setvariable variable=accountName;isoutput=true]$accountName"
New-AzStorageAccount -ResourceGroupName $resourceGroup `
-Name $accountName `
-Location $location `
-SkuName Standard_RAGRS `
-Kind StorageV2
azurePowerShellVersion: 'LatestVersion'
- task: AzurePowerShell#5
displayName: This will be failed
inputs:
azureSubscription: 'xxx'
ScriptType: 'InlineScript'
Inline: |
xxx
azurePowerShellVersion: 'LatestVersion'
- task: AzurePowerShell#5
displayName: Create Storage1
name: createstorage3
inputs:
azureSubscription: 'xxx'
ScriptType: 'InlineScript'
Inline: |
xxx
$resourceGroup = "xxx"
$location = "westus"
$accountName = "bowman08193"
Write-Host "##vso[task.setvariable variable=resourceGroup;isoutput=true]$resourceGroup"
Write-Host "##vso[task.setvariable variable=location;isoutput=true]$location"
Write-Host "##vso[task.setvariable variable=accountName;isoutput=true]$accountName"
New-AzStorageAccount -ResourceGroupName $resourceGroup `
-Name $accountName `
-Location $location `
-SkuName Standard_RAGRS `
-Kind StorageV2
azurePowerShellVersion: 'LatestVersion'
- task: AzurePowerShell#5
condition: failed()
continueOnError: true
inputs:
azureSubscription: 'xxx'
ScriptType: 'InlineScript'
Inline: |
Remove-AzStorageAccount -Name $(createstorage1.accountName) -ResourceGroupName $(createstorage1.resourceGroup) -Force
Remove-AzStorageAccount -Name $(createstorage2.accountName) -ResourceGroupName $(createstorage2.resourceGroup) -Force
Remove-AzStorageAccount -Name $(createstorage3.accountName) -ResourceGroupName $(createstorage3.resourceGroup) -Force
azurePowerShellVersion: 'LatestVersion'
The above is Storage service, other service are similar.
By the way, you can deploy all of the services to a new resource group, if failed, just delete the whole group.
You haven't mentioned which scripting language you are using to deploy these resources. It depends on both scripting language and logic. In this case YAML pipeline can be more useful like having stages or jobs. Also use 'continue on error' as true
Set a variable to find your job executed successfully or not
echo "##vso[task.setvariable variable=ISVALIDBUILD;isOutput=true]True"
Create a new job if previous job fails. here InfraBuild is previous job.
- job: RunOnceifFailed
dependsOn: InfraBuild
variables:
PrintResults: $[ dependencies.InfraBuild.outputs['DetermineResult.PrintResults'] ]
condition: eq(dependencies.InfraBuild.outputs['DetermineResult.ISVALIDBUILD'], 'False')
Write the tasks to delete the resources in the new job.
Likewise you can also have a job if 'Infrabuild' Job executed successfully.
Please refer MS documents to get to know more about this, hope it helps you get started.

How to use Azure Key Vault to pass secure parameter value during deployment?

I have an ARM template for deploying a Service Fabric cluster in Azure pipeline:
- task: AzureResourceManagerTemplateDeployment#3
displayName: 'Deploy SF cluster'
inputs:
deploymentScope: 'Resource Group'
subscriptionId: '${{ parameters.SubscriptionId }}'
azureResourceManagerConnection: '${{ parameters.ArmConnection }}'
action: 'Create Or Update Resource Group'
resourceGroupName: '${{ parameters.ResourceGroupName }}'
location: 'West Europe'
templateLocation: 'Linked artifact'
csmFile: '$(Build.SourcesDirectory)/pipelines/templates/sfcluster.json'
csmParametersFile: '$(Build.SourcesDirectory)/pipelines/templates/sfcluster-params.json'
overrideParameters: '-certificateThumbprint $(Thumbprint) -sourceVaultResourceId $(KeyvaultId) -certificateUrlValue $(SecretId)'
deploymentMode: 'Incremental'
The ARM template works well (since many days and in 5 different pipelines) when I use:
"osProfile": {
"adminUsername": "RdpUsername",
"adminPassword": "RdpPassword",
"computernamePrefix": "[variables('vmNodeType0Name')]",
"secrets": [
{
"sourceVault": {
"id": "[parameters('sourceVaultResourceId')]"
},
"vaultCertificates": [
{
"certificateStore": "My",
"certificateUrl": "[parameters('certificateUrlValue')]"
}
]
}
]
},
Having hardcoded RDP credentials is however a security issue, so I would like to Use Azure Key Vault to pass secure parameter value during deployment
So I add 2 random secret strings to the Keyvault, which is deployed earlier in the pipeline and then try:
"osProfile": {
"adminUsername": {
"reference": {
"keyVault": {
"id": "[parameters('sourceVaultResourceId')]"
},
"secretName": "RdpUsername"
}
},
"adminPassword": {
"reference": {
"keyVault": {
"id": "[parameters('sourceVaultResourceId')]"
},
"secretName": "RdpPassword"
}
},
"computernamePrefix": "[variables('vmNodeType0Name')]",
"secrets": [
{
"sourceVault": {
"id": "[parameters('sourceVaultResourceId')]"
},
"vaultCertificates": [
{
"certificateStore": "My",
"certificateUrl": "[parameters('certificateUrlValue')]"
}
]
}
]
},
Unfortunately this produces the error:
Unexpected character encountered while parsing value: {. Path 'properties.virtualMachineProfile.osProfile.adminUsername', line 1, position 3178.
My question is: why is the bracket { unexpected there?
My ARM template is based on the azure-quickstart-templates/service-fabric-secure-cluster-5-node-1-nodetype and as written above works with the hardcoded values.
UPDATE:
Here is my sfcluster-params.json (but also please note the overrideParameters in the pipeline task listed below and being filled with pipeline variables set by Keyvault deployment):
{
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentParameters.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"clusterName": {
"value": "sfcluster"
}
}
}
And here a bigger part of my Azure pipeline:
# deploy KeyVault by ARM template and output VinHashKey and SF Cluster certificate thumbsprint
- task: AzureResourceManagerTemplateDeployment#3
displayName: 'Deploy Keyvault'
inputs:
deploymentScope: 'Resource Group'
subscriptionId: '${{ parameters.SubscriptionId }}'
azureResourceManagerConnection: '${{ parameters.ArmConnection }}'
action: 'Create Or Update Resource Group'
resourceGroupName: '${{ parameters.ResourceGroupName }}'
location: 'West Europe'
templateLocation: 'Linked artifact'
csmFile: '$(Build.SourcesDirectory)/pipelines/templates/keyvault.json'
csmParametersFile: '$(Build.SourcesDirectory)/pipelines/templates/keyvault-params.json'
deploymentMode: 'Incremental'
# collect outputs from the above ARM task and put them into pipeline vars
- task: ARM Outputs#5
displayName: 'Collect Keyvault output'
inputs:
ConnectedServiceNameSelector: 'ConnectedServiceNameARM'
ConnectedServiceNameARM: '${{ parameters.ArmConnection }}'
resourceGroupName: '${{ parameters.ResourceGroupName }}'
whenLastDeploymentIsFailed: 'fail'
# import the certificate my-self-signed-cert into the Keyvault if it is missing there
- task: AzurePowerShell#5
displayName: 'Import certificate'
inputs:
azureSubscription: '${{ parameters.ArmConnection }}'
ScriptType: 'InlineScript'
azurePowerShellVersion: '3.1.0'
Inline: |
$Cert = Get-AzKeyVaultCertificate -VaultName $(KeyVaultName) -Name my-self-signed-cert
if (!$Cert) {
$Pwd = ConvertTo-SecureString -String MyPassword -Force -AsPlainText
$Base64 = 'MII....gfQ'
$Cert = Import-AzKeyVaultCertificate -VaultName $(KeyVaultName) -Name my-self-signed-cert -CertificateString $Base64 -Password $Pwd
}
# set the pipeline variables Thumbprint and SecretId - needed for SF deployment
echo "##vso[task.setvariable variable=Thumbprint]$($Cert.Thumbprint)"
echo "##vso[task.setvariable variable=SecretId]$($Cert.SecretId)"
# deploy SF cluster by ARM template and use the SF certificate thumbsprint as admin cert
- task: AzureResourceManagerTemplateDeployment#3
displayName: 'Deploy SF cluster'
inputs:
deploymentScope: 'Resource Group'
subscriptionId: '${{ parameters.SubscriptionId }}'
azureResourceManagerConnection: '${{ parameters.ArmConnection }}'
action: 'Create Or Update Resource Group'
resourceGroupName: '${{ parameters.ResourceGroupName }}'
location: 'West Europe'
templateLocation: 'Linked artifact'
csmFile: '$(Build.SourcesDirectory)/pipelines/templates/sfcluster.json'
csmParametersFile: '$(Build.SourcesDirectory)/pipelines/templates/sfcluster-params.json'
overrideParameters: '-certificateThumbprint $(Thumbprint) -sourceVaultResourceId $(KeyvaultId) -certificateUrlValue $(SecretId)'
deploymentMode: 'Incremental'
Finally, here the sfcluster.json - it is too big to put at Stackoverflow.
A keyVault reference can only be used as a parameter value - so in a param file or the parameters property of a deployment. You can't use it directly as a property value on a resource.

Resources