I have two parameter list and i want to pass these parameter values in sql task and webapp deploy task on azure pipeline yml.
parameters:
- name: db
type: object
default: [db1, db2, db3.......]
- name: apps
type: object
default: [app1, app2, app3......]
steps:
- ${{ each dblist in parameters.db && each applist in parameters.apps}}:
- task: SqlAzureDacpacDeployment#1
displayName: 'Azure SQL SqlTask ${{ db }}'
inputs:
azureSubscription: '$(Parameters.connectedServiceName)'
ServerName: xxxx.xxxx.windows.net
DatabaseName: ${{ dblist }}
SqlUsername: xxxxx
SqlPassword: xxxxx
deployType: SqlTask
SqlFile: 'Table.sql'
- task: AzureWebApp#1
displayName: 'Azure Web App Deploy ${{ applist }}'
continueOnError: true
inputs:
azureSubscription: '$(Parameters.connectedServiceName)'
appType: webApp
ResourceGroupName: $(group)
appName: ${{ applist }}
package: '$(build.artifactstagingdirectory)/**/*.zip'
When i run this pipeline, i got error and sequence that i want to run might not be right.
How can i run tasks like this loop:
db1
app1
db2
app2
db3
app3...... and so on
How the issue can be solved?
So you have an array of app + db that you would like to deploy in sequence if i understand.
Array are just objects so you should be able to do something like that
parameters:
- name: configurations
type: object
default:
- app: app1
db: db1
- app: app2
db: db2
steps:
- ${{ each configuration in parameters.configurations }}:
- pwsh: Write-Host Hello ${{ configuration.db }}
displayName: Deploy database
- pwsh: Write-Host Hello ${{ configuration.app }}
displayName: Deploy webapp
Related
So I have the following variable comdition defined in my YAML
${{ if endsWith( variables['Build.SourceBranchName'], 'main' ) }}:
dFile: 'Dockerfile'
${{ if endsWith( variables['Build.SourceBranchName'], 'development' ) }}:
dFile: 'Development.Dockerfile'
I then use this in a build image task like this:
- task: Docker#0
displayName: 'Build an image'
inputs:
azureSubscription: 'Azure Registry'
azureContainerRegistry: 'redacted'
dockerFile: $[variables.dFile]
imageName: '$(Build.Repository.Name)-$(build.sourceBranch):$(Build.BuildNumber)'
includeLatestTag: true
But for some reason this is the output I get:
##[error]Unhandled: No Docker file matching /home/vsts/work/1/s/$[variables.dockerFile] was found.
I also tried '$(dFile)' and got the same error
Defines the variables as name-value pair.
Here is what your variables block would look like:
variables:
- ${{ if endsWith( variables['Build.SourceBranchName'], 'main' ) }}:
- name: dFile
value: 'Dockerfile'
- ${{ if endsWith( variables['Build.SourceBranchName'], 'development' ) }}:
- name: dFile
value: 'Development.Dockerfile'
When I run the pipeline below I get an error within jobs.yml - it commplains about an "unexpected value" for the environment parameter... and within stages.yml "unexpected value: parameters" - what am I doing wrong here? The idea is to get the environments and the associated variableGroup - loop thru the environments array in Stages.yml and create a stage per environment... insert correct variableGroup per stage... use variablesGroup values to perform jobs in jobs.yml - each variable group contains the same vars.. with different values.
This is main.yml
#main.yml
trigger: none
extends:
template: /Build/Stages.yml
parameters:
environments:
- environment: Dev
variableGroup: Project-Dev-VarGrp
- environment: QA
variableGroup: Project-QA-VarGrp
dependsOn: Dev
- environment: UAT
variableGroup: Project-UAT-VarGrp
dependsOn: QA
- environment: UAT2
variableGroup: Project-UAT2-VarGrp
dependsOn: UAT
Then here is the next bit... Stages.yml
parameters:
- name: dataFolder
displayName: 'Data folder to process'
type: string
default: '/DataMigrations/Master_Data/'
- name: dataFiles
displayName: List of Data Files or Folder names
type: string
default: 'Dev_Data.zip'
- name: environments
type: object
default: []
stages:
- ${{ each environment in parameters.environments }}:
- stage: '${{ parameters.environment }}'
jobs:
- template: ./jobs.yml
variables:
- group: ${{ parameters.variableGroup }}
parameters:
environment: '${{ parameters.environment }}'
crmURL: $(crmURL)
oauthAppId: $(ClientID)
ClientPass: $(ClientPass)
dataFolder: '${{ parameters.dataFolder }}'
env: '${{ parameters.environment }}'
and here is jobs.yml
jobs:
- deployment: DeployData
displayName: 'Deploying Data to ${{ parameters.environment }}'
environment: ${{ parameters.environment }}
pool:
vmImage: 'windows-latest'
strategy:
runOnce:
deploy:
steps:
- checkout: self
clean: false
- powershell: |
Write-Host "##vso[task.setvariable variable=crmConnectionString]'AuthType=ClientSecret;Url=$(crmURL);ClientId=$(ClientID);ClientSecret=$(ClientPass)'"
displayName: 'PreDeploy configuration'
- task: PowerShell#2
displayName: 'Powershell: Run update-data.ps1'
inputs:
targetType: 'filePath'
filePath: $(System.DefaultWorkingDirectory)/DataMigrations/Scripts/update-data.ps1
arguments: -folderName '${{ parameters.dataFolder }}' -environment '${{ parameters.env }}'
workingDirectory: $(System.DefaultWorkingDirectory)/DataMigrations
- task: PowerShell#2
displayName: 'Powershell: Run zip-import-data.ps1'
inputs:
targetType: 'filePath'
filePath: $(System.DefaultWorkingDirectory)/DataMigrations/Scripts/zip-import-data.ps1
arguments: -folderName '${{ parameters.dataFolder }}' -dataMigrationFilenames '${{ parameters.dataFiles }}' -connectionString $(crmConnectionString)
workingDirectory: $(System.DefaultWorkingDirectory)/DataMigrations
Few things here:
When you're using ${{ each environment in parameters.environments }} then the nested environment and variableGroup are available using this syntax ${{ environment.environment }} and ${{ environment.variableGroup}}
In your Stages.yml file, you're trying to invoke the ./jobs.yml template but the associated parameters are defined after the - group: ${{ parameters.variableGroup }}. A valid syntax should looks like this:
stages:
- ${{ each environment in parameters.environments }}:
- stage: '${{ environment.environment }}'
variables:
- group: ${{ environment.variableGroup }}
jobs:
- template: ./jobs.yml
parameters:
environment: '${{ environment.environment }}'
crmURL: $(crmURL)
oauthAppId: $(ClientID)
ClientPass: $(ClientPass)
dataFolder: '${{ parameters.dataFolder }}'
env: '${{ environment.environment }}'
You also have few space typos. I know it's annoying but you need to have the exact yaml syntax otherwise the files can't be parsed.
I am trying to get my main Terraform pipeline to deploy to multiple subscriptions, using the same service principle. Yet I keep getting errors that state resource group not found when it tries to deploy to the subscription. Both subscriptions are in the same tenant.
Here is my YAML Code:
parameters:
- name: terraformWorkingDirectory
type: string
default: '$(System.DefaultWorkingDirectory)/Terraform/'
- name: serviceConnection
type: string
default: 'JasonTestEnvManagmentGroup'
- name: azureSubscription
type: string
default: 'JasonTestEnvManagmentGroup'
- name: appconnectionname
type: string
default: 'JasonTestEnvManagmentGroup'
- name: RG
type: string
default: 'Jason_Testing_Terraform'
- name: azureLocation
type: string
default: 'UK South'
- name: terraformVersion
type: string
default: '1.0.4'
- name: artifactName
type: string
default: 'Website'
#- name: authartifactName
# type: string
#default: 'AuthServer'
# Only run against develop
#trigger:
# branches:
# include:
# - main
#pool:
#vmImage: "ubuntu-latest"
# Don't run against PRs
#pr: none
#stages:
#- stage: terraformStage
# displayName: Detect Drift
# jobs:
#- job: terraform_plan_and_apply
steps:
- checkout: self
- task: TerraformInstaller#0
displayName: "install"
inputs:
terraformVersion: ${{ parameters.terraformVersion }}
- task: TerraformTaskV2#2
displayName: "init"
inputs:
provider: "azurerm"
command: "init"
backendServiceArm: ${{ parameters.serviceConnection }}
backendAzureRmResourceGroupName: "TerraformBackendForCICTesting"
backendAzureRmStorageAccountName: "nsterraformstatestorage"
backendAzureRmContainerName: "devopsterraformstatefile"
backendAzureRmKey: "terraform.tfstate"
workingDirectory: ${{ parameters.terraformWorkingDirectory }}
- task: TerraformTaskV1#0
displayName: "plan"
inputs:
provider: "azurerm"
command: "plan"
commandOptions: "-input=false"
environmentServiceNameAzureRM: ${{ parameters.serviceConnection }}
workingDirectory: ${{ parameters.terraformWorkingDirectory }}
- task: TerraformTaskV1#0
displayName: "apply"
inputs:
provider: "azurerm"
command: "apply"
commandOptions: "-input=false -auto-approve"
environmentServiceNameAzureRM: ${{ parameters.serviceConnection }}
workingDirectory: ${{ parameters.terraformWorkingDirectory }}
#- stage: put_pipelines_files_in_place
# displayName: Putting Pipeline Files In Place
#- jobs:
#- job: apply_artifiact_to_web_app
# displayName: Putting Files In Place
# dependsOn: terraform_plan_and_apply
# Download Artifact File
#- download: none
- task: DownloadPipelineArtifact#2 # Website Artifact
displayName: 'Download Build Artifacts'
inputs:
artifact: ${{ parameters.artifactName }}
patterns: '/website/**/*.zip'
path: '$(Build.ArtifactStagingDirectory)/website/'
# deploy to Azure Web App
- task: AzureWebApp#1
displayName: 'Azure Web App Deploy: nsclassroom-dgyn27h2dfoyojc' #Website Deploy Artifact
inputs:
package: $(Build.ArtifactStagingDirectory)/website/**/*.zip
azureSubscription: ${{ parameters.azureSubscription }}
ConnectedServiceName: ${{ parameters.appconnectionname}}
appName: 'nsclassroom-dgyn27h2dfoyojc'
ResourceGroupName: ${{ parameters.RG}}
- task: DownloadPipelineArtifact#2 # Authentication Server Artifact
displayName: 'Download Build Artifacts'
inputs:
artifact: ${{ parameters.artifactName}}
patterns: '/authsrv/**/*.zip'
path: '$(Build.ArtifactStagingDirectory)/authsrv/'
# deploy to Azure Web App
- task: AzureWebApp#1
displayName: 'Azure Web App Deploy: nsclassroomauthentication-dgyn27h2dfoyojc' #Authentication Server Deploy Artifact
inputs:
package: $(Build.ArtifactStagingDirectory)/authsrv/**/*.zip
azureSubscription: ${{ parameters.azureSubscription }}
ConnectedServiceName: ${{ parameters.appconnectionname}}
appName: 'nsclassroomauthentication-dgyn27h2dfoyojc'
ResourceGroupName: ${{ parameters.RG}}
The agent is uses is the default Ubuntu agent.
The service connection I have tried two, one mapped to both of the subscriptions doesn't work. Another mapped and scoped to the Management group.
The original service principle did work until I brought in the second subscription.
Say I have a main azure devops pipeline azure-pipelines.yml in which I call into my template deploy.yml.
In the main pipeline I'd like to be able to declare either a variable or a parameter of type hashset / map / dictionary or any other key value structure that I can then pass to the template.
I can see that it's possible to pass in an object type, but I can't wrap my head it's usage. How could I achieve the following?
Note the appSettings: {"key1":"value1","key2":"value2"} in azure-pipelines.yml is a fantasy, but showcases pretty well how I'd like this to work.
azure-pipelines.yml:
trigger:
- main
- job: deploy
pool:
vmImage: ${{ parameters.poolVmImage }}
steps:
- template: deploy.yml
parameters:
azureServiceConnection: ${{ parameters.azureServiceConnection }}
resourceGroupName: 'foo'
appServiceName: 'bar'
appSettings: {"key1":"value1","key2":"value2"}
deploy.yml:
parameters:
- name: azureServiceConnection
- name: resourceGroupName
- name: appServiceName
- name: appSettings
steps:
- task: AzureCLI#2
displayName: Deploy zip
name: deployZip
inputs:
azureSubscription: ${{ parameters.azureServiceConnection }}
scriptType: 'bash'
scriptLocation: 'inlineScript'
inlineScript: |
az webapp deployment source config-zip \
-g ${{ parameters.resourceGroupName }} \
-n ${{ parameters.appServiceName }} \
--src ./deployment.zip
az webapp config appsettings set \
-g ${{ parameters.resourceGroupName }} \
-n ${{ parameters.appServiceName }} \
--settings ${{ parameters.appSettings }}
How could I achieve the following?
You can indeed use object type parameters.
But in the template, you cannot use multiple object parameters at once, so you need to use each expression to loop through each object.
For example: - ${{ each setting in parameters.appSettings }}:
Here is my sample, you could refer to:
Deploy.yml:
parameters:
- name: azureServiceConnection
- name: resourceGroupName
- name: appServiceName
- name: appSettings
type: object
default: []
steps:
- task: AzureCLI#2
displayName: Deploy zip
name: deployZip
inputs:
azureSubscription: ${{ parameters.azureServiceConnection }}
scriptType: 'bash'
scriptLocation: 'inlineScript'
inlineScript: |
az webapp deployment source config-zip \
-g ${{ parameters.resourceGroupName }} \
-n ${{ parameters.appServiceName }} \
--src ./deployment.zip
- ${{ each setting in parameters.appSettings }}:
- task: AzureCLI#2
displayName: Deploy settings
inputs:
azureSubscription: ${{ parameters.azureServiceConnection }}
scriptType: 'bash'
scriptLocation: 'inlineScript'
inlineScript: |
az webapp config appsettings set \
-g ${{ parameters.resourceGroupName }} \
-n ${{ parameters.appServiceName }} \
--settings ${{ setting }}
azure-pipelines.yml:
trigger:
- none
parameters:
- name: InstanceArgs
type: object
default: [key2=value2,key3=value3]
jobs:
- job: deploy
pool:
vmImage: windows-latest
steps:
- template: deploy.yml
parameters:
azureServiceConnection: '${{ parameters.azureServiceConnection }}'
resourceGroupName: 'foo'
appServiceName: 'bar'
appSettings: ${{ parameters.InstanceArgs }}
Note: Since the parameters in the template are obejct type , the same type of parameters need to be set in the main yaml file to pass the object.
Workflow: Pass the object type parameters to the template. The template will run the deploy zip command first. Then it will traverse each object passed, and use the config settings command to pass them to the webapp one by one.
Result:
I'm trying to create a multi-stage pipeline which has Variable Groups defined for each stage of the pipeline. The goal is to pass values from the group as parameters to a sub template. It seems the value of the Group, defined at the stage, is not getting passed in to the sub template. It overrides the "DEFAULTVALUE" with an empty string.
pipeline.yml
trigger:
- none
pool:
name: 'Azure Pipelines'
vmImage: windows-latest
stages:
- stage: DEV
variables:
- group: my-group-dev
jobs:
- template: sub-template.yml
parameters:
env: 'dev'
subscriptionName: '$(SubscriptionName)' # This reference from the variable group doesn't get passed in
subscriptionId: '$(SubscriptionId)'
- stage: TEST
variables:
- group: my-group-test
jobs:
- template: sub-template.yml
parameters:
env: 'test'
subscriptionName: '$(SubscriptionName)' # This reference from the variable group doesn't get passed in
subscriptionId: '$(SubscriptionId)'
sub-template.yml
parameters:
env: 'DEFAULTVALUE'
subscriptionName: 'DEFAULTVALUE'
subscriptionId: 'DEFAULTVALUE'
jobs:
- deployment: ResourceDeployment
displayName: Deploy Resources ${{ parameters.env }}
environment: ${{ parameters.env }}
strategy:
runOnce:
deploy:
steps:
- task: AzureFileCopy#4
displayName: 'Upload ARM Templates'
inputs:
sourcePath: '$(Pipeline.Workspace)/templates'
azureSubscription: '${{ parameters.subscriptionName }}'
destination: 'azureBlob'
storage: 'my-storage-account'
containerName: 'arm'
name: AzureFileCopy
- task: AzureResourceManagerTemplateDeployment#3
inputs:
deploymentScope: 'Resource Group'
azureResourceManagerConnection: '${{ parameters.subscriptionName }}'
subscriptionId: '${{ parameters.subscriptionId }}'
action: 'Create Or Update Resource Group'
resourceGroupName: 'my-resource-group'
location: 'eastus2'
templateLocation: 'URL of the file'
csmFileLink: '$(AzureFileCopy.StorageContainerUri)${{ parameters.env }}/templates/main.json$(AzureFileCopy.StorageContainerSasToken)'
I have also tried adding the variable group within the sub-template but that also doesn't parse correctly...
Does anyone know if this is possible?
It is a known issue that the service connection endpoint cannot be referenced in variable groups defined under stage level.
You can fix this issue by below workarounds:
1, Define the variable groups in the global level instead of stage level See below:
trigger:
- none
pool:
name: 'Azure Pipelines'
vmImage: windows-latest
# define the variable group under global level.
variables:
- group: my-group-dev
- group: my-group-test
stages:
- stage: DEV
jobs:
- template: sub-template.yml
parameters:
env: 'dev'
subscriptionName: '$(SubscriptionName)' # This reference from the variable group doesn't get passed in
subscriptionId: '$(SubscriptionId)'
- stage: TEST
jobs:
- template: sub-template.yml
parameters:
env: 'test'
subscriptionName: '$(SubscriptionName)' # This reference from the variable group doesn't get passed in
subscriptionId: '$(SubscriptionId)'
2, Link the variable groups on the UI page.
On the yaml pipeline edit page--> Click the 3dots-->Triggers-->Variables Tab-->Link Variable group
Please see below threads for more information.
Using a variable for the service connection
Azure subscription endpoint ID cannot be provided through a variable in build definition YAML file
You need to pass variable group name to template and on template level apply it on variables scope like this:
parameters:
env: 'DEFAULTVALUE'
subscriptionName: 'DEFAULTVALUE'
subscriptionId: 'DEFAULTVALUE'
variableGroupName: 'DEFAULTVALUE`
jobs:
- deployment: ResourceDeployment
displayName: Deploy Resources ${{ parameters.env }}
environment: ${{ parameters.env }}
variables:
- group: {{ paramaters.variableGroupName }}
strategy:
runOnce:
deploy:
steps:
- task: AzureFileCopy#4
displayName: 'Upload ARM Templates'
inputs:
sourcePath: '$(Pipeline.Workspace)/templates'
azureSubscription: '${{ parameters.subscriptionName }}'
destination: 'azureBlob'
storage: 'my-storage-account'
containerName: 'arm'
name: AzureFileCopy
- task: AzureResourceManagerTemplateDeployment#3
inputs:
deploymentScope: 'Resource Group'
azureResourceManagerConnection: '${{ parameters.subscriptionName }}'
subscriptionId: '${{ parameters.subscriptionId }}'
action: 'Create Or Update Resource Group'
resourceGroupName: 'my-resource-group'
location: 'eastus2'
templateLocation: 'URL of the file'
csmFileLink: '$(AzureFileCopy.StorageContainerUri)${{ parameters.env }}/templates/main.json$(AzureFileCopy.StorageContainerSasToken)'
This is similar to this topic:
For security reasons, we only allow you to pass information into templated code via explicit parameters.
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/templates?view=azure-devops
The means the author of the pipeline using your template needs to commit changes where they explicitly pass the needed info into your template code.
There are some exceptions to this, where the variable is statically defined in the same file or at pipeline compile time, but generally speaking, it’s probably better to use parameters for everything that does not involve system-defined read-only dynamic variable and custom-defined dynamic output variables.
I have to variables groups
QA
PROD
both has isProd variable
and then my template is
parameters:
variableGroupName: 'QA'
jobs:
- job: ResourceDeployment
variables:
- group: ${{ parameters.variableGroupName }}
steps:
- script: echo '$(isProd)'
and main file is
trigger: none
pr: none
stages:
- stage: QA
jobs:
- template: sub-template.yml
parameters:
variableGroupName: 'QA'
- stage: PROD
jobs:
- template: sub-template.yml
parameters:
variableGroupName: 'PROD'
and I got:
and