I have a question regarding Azure DevOps, It seems it is not possible to set in pipeline's jobs Azure subscription field using variables. On attempt to do this I'm getting the error message:
The pipeline is not valid.
I was trying to use in variable:
Subscription ID
Service Connection Name
Service Connection ID
But with no luck - the same issue again and again. Since I need to change subscriptions for pipeline very often, it's very critical problem for me. Could you please elaborate the issue?
If you use YAML pipelines you could configure your subscription as variable as follows:
- task: AzureCLI#2
inputs:
azureSubscription: '$(SubscriptionId)'
scriptType: 'ps'
scriptLocation: 'inlineScript'
inlineScript: |
az --version
az account show
Variable settings
And during every build you can override your azure subscription.
Related
We use Azure DevOps YAML pipeline for managed group scoped deployments. In the first stage we validate the ARM deployment via YAML task AzureResourceManagerTemplateDeployment#3 (validation mode). Since today it fails with the following error: Template validation failed. Error: "Creating the deployment 'xxx-20221221-071859-a2d8' would exceed the quota of '800'. The current deployment count is '800'. Please delete some deployments before creating a new one, or see https://aka.ms/800LimitFix for information on managing deployment limits.
The YAML code snippet:
- task: AzureResourceManagerTemplateDeployment#3
displayName: Validate ARM Template
inputs:
azureResourceManagerConnection: ${{ variables.TestAzureResourceManagerConnection }}
deploymentScope: 'Management Group'
location: $(location)
templateLocation: 'Linked artifact'
csmFile: '${{ variables.MainTemplateFilePath }}'
overrideParameters: '-parTargetManagementGroupID $(managementGroupID)'
deploymentMode: 'Validation'
I already checked the deployments in the corresponding managementgroup and other managementgroups and deleted them without any success.
Template validation failed. Error: "Creating the deployment 'xxx-20221221-071859-a2d8' would exceed the quota of '800'. The current deployment count is '800'.
From the error message, the cause of the issue is that the Management Group has more than 800 deployments in its deployment history.
Refer to this doc: Resolve error when deployment count exceeds 800
Each resource group is limited to 800 deployments in its deployment history.
To solve this issue, I suggest that you can use the Azure CLIs to list all management group deployments and delete the deployments. You can delete deployments that are 5 days old or older to check if it works.
startdate=$(date +%F -d "-5days")
deployments=$(az deployment mg list --management-group-id --query "[?properties.timestamp<'$startdate'].name" --output tsv)
for deployment in $deployments
do
az deployment mg delete --management-group-id --name $deployment
done
Refer to this doc about azure cli: az deployment mg
First we used Terraform to create our resources along with proper tagging.
But later we identified that the Admin team performed some manual modifications to the resource as part of some issues quick fixes and all. At the end the terraform state file of these resources are out of sync.
Now we have requirement to update the tags of the already provisioned resources (terraform used) with new additional tags to them. When we tried by adding the terraform manifests with the changes for the tagging, and executed terraform plan, and could see that some resources are showing to replace and we are not advised to perform that .
We tried to manually import the changes what performed manually from portal, to the terraform state so that we can apply the tag changes from the terraform itself. But we are facing concerns like cant update the resources in minute level from the portal to the state file?
is there any automated way to tag a list of azure resources as per the tags specified for each resources. and we can update the state file so easily ?
On way that would work is using Azure PowerShell's Update-AzTag to update or add new tags to already existing resources in your subscription. I have few resources that are tagged as Key1:Value1 in my tenant where I'm trying to change them to SampleKey:SampleValue. Below is the script that worked for me to update all the tags that are in my tenant.
$SubscriptionId = Get-AzSubscription -TenantId (Get-AzContext).Tenant
$replacedTags = #{"key1"="value1"; "SampleKey"="SampleValue";}
foreach($SubId in $SubscriptionId) {
$ResourceId = "/subscriptions/"+$SubId.Id
Update-AzTag -ResourceId $ResourceId -Tag $replacedTags -Operation Replace
}
RESULTS
#SwethaKandikonda has given the script that can be used in Azure CLI. On the base of his answer, i want to make additional information.
If you want to automate this using azuredevops pipeline, here is a YAML pipeline example:
# Starter pipeline
trigger:
- master
pool:
vmImage: ubuntu-latest
steps:
- task: AzureCLI#2
inputs:
azureSubscription: 'your subscription'
scriptType: 'pscore'
scriptLocation: 'inlineScript'
inlineScript: 'az tag update --resource-id /subscriptions/{ sub-id }/resourceGroups/{ rg }/providers/Microsoft.Storage/storageAccounts/{your saName} --operation replace --tags key1=value1 key3=value3'
Running result
It will automatically tag the resource after running the pipeline successfully.
I'm trying to access an azure blob storage account to write some files from my pipeliens running with azure devops hosted agent.
We can't use yet the azure devops service tag with azure devops hosted agent
And I was wondering if there is a smart solution to access my blob storage from my hosted agents without opening it to all the internet.
Thank you in advance guys
Based on your requirement, you need to access Private Storage account with Microsoft-hosted agent.
As far as I know, service tag is not currently supported by Azure Storage account when setting Firewall.
To meet your requirements, you can use script to get the current Microsoft-hosted agent IP and add it to Azure Storage account firewall whitelist with Azure CLI or Azure PowerShell.
For example:
steps:
- task: AzurePowerShell#5
displayName: 'Azure PowerShell script: Set Rule'
inputs:
azureSubscription: kevin0215
ScriptType: InlineScript
Inline: |
$IP= Invoke-RestMethod http://ipinfo.io/json | Select -exp ip
$IP
Add-AzStorageAccountNetworkRule -ResourceGroupName "ResourceGroup" -AccountName "kevin0204" -IPAddressOrRange "$IP"
preferredAzurePowerShellVersion: ' 3.1.0'
- task: AzureFileCopy#4
displayName: 'AzureBlob File Copy'
inputs:
SourcePath: test
azureSubscription: kevin0322
Destination: AzureBlob
storage: test
ContainerName: 1
- task: AzurePowerShell#5
displayName: 'Azure PowerShell script: Remove Rule'
inputs:
azureSubscription: kevin0215
ScriptType: InlineScript
Inline: |
$IP= Invoke-RestMethod http://ipinfo.io/json | Select -exp ip
$IP
Remove-AzStorageAccountNetworkRule -ResourceGroupName "ResourceGroup" -AccountName "kevin0204" -IPAddressOrRange "$IP"
preferredAzurePowerShellVersion: ' 3.1.0'
Explanation：
You can add the IP to the firewall whitelist before uploading the file. After uploading, you can delete this IP.
Note:：The current azure storage account has a known limitation. Refer to this doc: Limitations of Azure Storage Account IP network rules.
When your Azure Devops Service organization and Azure Storage Account are in the same region, they will be accessed through private ip. This can cause intermittent access issues.
I have automated the deployment of a logic app standard via Azure Devops Pipeline using an arm template.
I have another pipeline that uses the Azure Devops zip deployment task to deploy the workflows (as recommended by Microsoft documentation).
My current struggle is when I have workflows that call other workflows.
When I deploy the zip file across different logic app standard instances the workflow url referenced is always the same.
How can I reference/call the workflow in a way that is not hardcoded and dynamically changes in the deploy? Can I use workflow() to reference other workflows?
As the access key is a property of the workflow and not the logic app standard I'm not able to set it as an app setting or parameter to be consumed inside the workflow.
Any ideas on how to bypass this issue?
What I ended up doing was the following.
I have created key vault secrets. In those key vault secrets I store the workflow url containing the authorization secret.
As I've created the other workflows pointing to the key vault secret name instead of a hardcoded url the logic app at run time will query the key vault, retrieve the url from the workflow i want to authenticate to and use it as input. As it already contains the signature it authenticates correctly.
It's probably a workaround but it was the only way I was able to achieve success in this operation.
For those with the same problem as me here are the steps:
First I have developed the workflow to obtain the secret containing the url from the keyvault
Get keyvault secret
Then it calls the url using the secret as input.
Secret as input for the url
When I have my workflows ready to deploy. I export them and put the code on Azure Devops.
Then in a build pipeline I use the following tasks
task: ArchiveFiles#2
displayName: "Archive Functions"
inputs:
rootFolderOrFile: "$(Build.Repository.LocalPath)/LogicApps"
includeRootFolder: false
archiveFile: "$(Build.ArtifactStagingDirectory)/LogicApps.zip"
task: AzureFunctionApp#1
displayName: "Deploy Functions"
inputs:
azureSubscription: "${ { parameters.Subscription }}"
appName: "mylogicappstandard"
package: "$(Agent.BuildDirectory)/${ { parameters.ArtifactName}}/LogicApps.zip"
task: AzureCLI#2
displayName: 'Update Signature url in ${ { parameters.KeyvaultName}}'
inputs:
azureSubscription: "${ { parameters.Subscription }}"
scriptType: 'ps'
scriptLocation: 'inlineScript'
inlineScript: "$(Agent.BuildDirectory)/${ { parameters.ArtifactName}}/Scripts/Get-WorkflowUrlSignature.ps1 $(AzureSubscriptionId) ${ { parameters.ResourceGroup }} mylogicappstandard ${ { parameters.KeyvaultName}}"
You can find the details for the script here Get-WorkflowUrlSignature.ps1
[CmdletBinding()]
param (
[Parameter(Mandatory)][string]$SubscriptionId,
[Parameter(Mandatory)][string]$ResourceGroup,
[Parameter(Mandatory)][string]$LogicAppName,
[Parameter(Mandatory)][string]$KeyVaultName
)
$json = az rest --method get --uri "https://management.azure.com/subscriptions/$SubscriptionId/resourceGroups/$ResourceGroup/providers/Microsoft.Web/sites/$LogicAppName/hostruntime/runtime/webhooks/workflow/api/management/workflows?api-version=2018-11-01"
$workflows = $json | convertfrom-json
foreach ($workflow in $workflows.Name){
$uri ="https://management.azure.com/subscriptions/$SubscriptionId/resourceGroups/$ResourceGroup/providers/Microsoft.Web/sites/$LogicAppName/hostruntime/runtime/webhooks/workflow/api/management/workflows/$workflow/triggers/manual/listCallbackUrl?api-version=2018-11-01"
if (az rest --method post --uri $uri){
# Gets the url with signature
$sigurl = az rest --method post --uri $uri | convertfrom-json
$secret = $sigurl.value.Replace('&','"&"')
$workflowName = $workflow.Replace("_","")
#Creates or updates secret in the keyvault
Write-Output "Updating secret $workflowName in the keyvault"
az keyvault secret set --name $workflowName --vault-name $KeyVaultName --value $secret
}else{
Write-Output "The workflow $workflow does not have any trigger url"
}
}
I hope this helps other people automate the process. Please let me know if you have an easier way to do it or to query the access key or url sig.
Is there a way to deploy complete variable group variables to Azure Function app settings using Yaml.
Below is the yaml code and how I am deploying variable :-
variables:
- group: MyDevVariable #This is my variable group name
#Place where I am referring my variables from variable groups
deploy:
steps:
-task: AzureFunctionApp#1
inputs:
appSetting: '-Key1 $(Key1) -Key2 $(Key2)' #Key1 and Key2 are stored in variable group.
but if I have many variables, I have to mentions every variable in appSettings as key $(key).
So Is there a way to deploying all my variables from variable group to azure function app setting.
Any response is greatly appreciated.
Thanks in advance.
I don't believe that you can iterate the group's variables in the YAML directly. There's no syntax to reach into the group from what I've seen. What you can do is to write a script that gets the variables out of the group (if you have Azure CLI installed on the agent) and then reference it later:
deploy:
steps:
- task: AzureCLI#2
inputs:
name: GetSettings
azureSubscription: Your Service Connection
scriptType: ps
scriptLocation: inlineScript
inlineScript: |
$env:AZURE_DEVOPS_EXT_PAT = '$(System.AccessToken)'
$groupVars = (az pipelines variable-group list --group-name "My Group Name") | ConvertFrom-JSON
$settingsList = ""
$groupVars.variables.PSObject.Properties | ForEach-Object {
$settingsList += "-$($_.Name) `"$($_.Value.value)`" "
}
Write-Host "##vso[task.setvariable variable=settingsList;isOutput=true]$settingsList"
Write-Host "Using settings $settingsList"
- task: AzureFunctionApp#1
inputs:
appSetting: $(GetSettings.settingsList)
NOTE: This may encounter issues with secret variables, in which case I'd store them in Azure Key Vault and use the key vault secrets task to read them into variables.