When deploying release using Azure Dev Ops, the variable value is preserved between two stages instead of being overwritten as expected.
I am using azure dev ops to deploy data factory with its resources to different environments
Test Environment
UAT Environment
I have 2 variable groups defined in "Library" respectively for each stage of deployment
Test-config and
Uat-config
Both of those have a variable by name varDataFactory - to keep the data factory name
When linking the variable group in the pipeline "Variables" tab, I specify the stage at which i am expecting it to execute against.
So, the variable varDataFactory is expected to have different value at each stage.
Basically the first stage executes, creating the Test Data Factory, but then when UAT stage deploys, it "sees" the "varDataFactory" with its old value - the one for the "test".
And I do not know why, and what to do about it. Especially, since I have another pair of variable groups for Key vaults (2 different vaults for 2 environments) and those get in just file. Please, help !!
As you have 2 different configuration groups for 2 different stages, then I don't understand what do you mean by overwriting.
I tried this with a simple command line task in release pipeline with below configurations:
Variable Group: test-config, Variable name: varDF, value: testdatafactory
Variable Group: uat-config, Variable name: varDF, value: uatdatafactory
Pipeline variable Configuration:
Below are the outputs for echo$(varDF) command in test and UAT stages:
Related
I am trying to write a pipeline to build an image and deploy to the test environment which is hosted on Azure. My codebase lies on GitHub. While trying to trigger the pipeline on a pull request from the source branch against the target branch, I am facing an issue where the pipeline doesn't trigger for the PR but runs fine for my other conditions, such as, push to develop or master.
The condition used for the PR trigger is as follows:
and(succeeded(), eq(variables['Build.Reason'], 'PullRequest'), startsWith(variables['System.PullRequest.SourceBranch'], 'release/'), eq(variables['System.PullRequest.TargetBranch'], 'master'))
The triggers in the yaml file can be seen below:
trigger:
branches:
include:
- develop
- master
paths:
exclude:
- k8s/*
- src/VERSION
- src/package.json
pr:
- master
Am I missing something here?
There are two scenarios:
Scenario 1: The pipeline was triggered when the pull request is created, but the stages/jobs/tasks with the condition you showed don't run.
Then the issue should be related to condition, not trigger.
I have tested and confirmed that your condition is right. So, it's probably not the condition notation but something else that's causing your task not to run.
Here is a troubleshooting advice:
Go to the build log, click on the stages/jobs/tasks that were skipped. You will find a comparison between the condition and the real value. From here, you can tell which part of the condition is keeping your tasks from running.
Scenario 2: The pipeline wasn't triggered when the pull request is created.
Then the issue should be related to trigger, not condition.
Please select documents below for detailed troubleshooting advice based on your case:
I just created a new YAML pipeline with CI/PR triggers, but the pipeline is not being triggered.
My CI or PR triggers have been working fine. But, they stopped working now.
I'm not sure if this will fix it but according to the documentation the System.PullRequest.TargetBranch variable has format refs/heads/main which would mean your condition needs updating to add refs/heads in front of the variables.
As such I would add a step to echo these variables just to confirm if they have the refs/head prefix and if so adjust your logic accordingly
I wrote a pipeline task with a variable passed like this
jobs:
- job: buildandpush
pool:
vmImage: 'ubuntu-latest'
steps:
- script: |
echo sanity check
echo $NOTING_SERVICE_ORIGIN
echo $NOTING_SERVICE_ORIGIN_2
env:
NOTING_SERVICE_ORIGIN: dummy-string-111
NOTING_SERVICE_ORIGIN_2: dummy-string-222
What I see printed is:
sanity check
https://some-url-we-used-in-the-past/
dummy-string-222
I did not ever add any variables through Azure DevOps UI. The value https://some-url-we-used-in-the-past/ is no longer anywhere in the codebase. I could not find anything interesting in Azure Pipelines docs.
Is Azure Pipelines caching NOTING_SERVICE_ORIGIN somewhere?
Ended up finding someone else already defined the variable for the pipeline. Surprising was that it took precedence over the same variable passed explicitly on the spot.
I'm currently building a deployment for Kubernetes through Helm. However, one of my values that I have to pass is an Endpoint that contain the following characters:
Endpoint=https://test.io;Id=001;Secret=test_test_test
The problem is that if I passed the following value:
test01:
- name: test
value: Endpoint=https://test.io;Id=001;Secret=test_test_test
The pod will never get created since is not getting and passing the value. If I add the following with single quotes is telling me that the pod is not ready.
test01:
- name: test
value: 'Endpoint=https://test.io;Id=001;Secret=test_test_test'
If I passed the value with single quotes it will tell me that the pod got created, but still the pod and namespace do not show up on in the AKS Cluster. However, if I run the same environmental variables on docker, I will see that all of these variables will be applicable to my app, and I will be able to see my app running as expected.
How can I set up the env variables inside of the Values and how can I run a command from the terminal to set & run multiple variables at the same time? & Does anyone has any other way to do this?
I am trying to use the AzureRmWebAppDeployment task to deploy WebJobs to an App Service (Azure App service deploy).
Here is the following YAML code (task: AzureRmWebAppDeployment#4):
- task: AzureRmWebAppDeployment#4
inputs:
ConnectionType: 'AzureRm'
azureSubscription: 'hidden'
appType: 'webApp'
WebAppName: 'hidden'
package: '$(Build.StagingDirectory)\WebJob'
When I try to run the build, it gives me this error:
Job DeployWebJobsCombo1: Step DeployWebJobsCombo1 task reference is invalid. The task name AzureRmWebAppDeployment is ambiguous. Specify one of the following identifiers to resolve the ambiguity: ae173df0-05de-4ac6-b5cc-853e9f951a33, 497d490f-eea7-4f2b-ab94-48d9c1acdcb1
I've tried replacing the task name with either of those IDs(task: ae173df0-05de-4ac6-b5cc-853e9f951a33#4 and task: ae173df0-05de-4ac6-b5cc-853e9f951a33 have both been tried), but it says it isn't a valid name.
I've tried setting the name property of the task (https://learn.microsoft.com/en-us/azure/devops/pipelines/yaml-schema?view=azure-devops&tabs=schema#task) but that doesn't accept GUIDs:
name: string # identifier for this step (A-Z, a-z, 0-9, and underscore)
I haven't been able to find any information online on how to actually specify the identifiers ae173df0-05de-4ac6-b5cc-853e9f951a33 or 497d490f-eea7-4f2b-ab94-48d9c1acdcb1. Can anyone point me to how to use these identifiers?
So 497d490f-eea7-4f2b-ab94-48d9c1acdcb1 appears to be the id of AzureRmWebAppDeployment#4
Github for the task here.
It looks like that error can happen when two tasks\extensions exist in your Azure DevOps organization with the same name. See support forums here and here.
I would expect that you should be able to use - task: 497d490f-eea7-4f2b-ab94-48d9c1acdcb1#4 not sure why that is not working. I just tried and it recognized it as the AzureRmWebAppDeployment task. Maybe give that another try.
Also if you would like to see the tasks that exist in your organization you can use the API and hit this url to get the YAML Schema.
https://dev.azure.com/{organization}/_apis/distributedtask/yamlschema?api-version=5.1 I would check here for possible duplicates.
Eric Smith mentions the ability to check for duplicates in his answer. You can actually graphically check for duplicates by going to https://dev.azure.com/{YourOrg}/_settings/extensions. There you can find the offending duplicate task and removing it. It was easier for me to remove the task that I no longer was using. In my case I had an AWS S3 upload task that was duplicated.
Removing the duplicated/unused task reference fixed my problem. I think this is a little cleaner than using the GUID of the DevOps task.
Attempting to trigger an Azure pipeline when another pipeline has been completed using a YAML. There's documentation indicating that you can add a pipeline resource with:
resources: # types: pipelines | builds | repositories | containers | packages
pipelines:
- pipeline: string # identifier for the pipeline resource
connection: string # service connection for pipelines from other Azure DevOps organizations
project: string # project for the source; optional for current project
source: string # source defintion of the pipeline
version: string # the pipeline run number to pick the artifact, defaults to Latest pipeline successful across all stages
branch: string # branch to pick the artiafct, optional; defaults to master branch
tags: string # picks the artifacts on from the pipeline with given tag, optional; defaults to no tags
However, I've been unable to figure out what the "source" means. For example, I have a pipeline called myproject.myprogram:
resources:
pipelines:
- pipeline: myproject.myprogram
source: XXXXXXXX
Moreover, it's unclear how you'd build based a trigger based on this.
I know that this can be done from the web-GUI, but it should be possible to do this from a YAML.
For trigger of one pipeline from another azure official docs suggest this below solution. i.e. use pipeline triggers
resources:
pipelines:
- pipeline: RELEASE_PIPELINE // any arbitrary name
source: PIPELINE_NAME. // name of the pipeline shown on azure UI portal
trigger:
branches:
include:
- dummy_branch // name of branch on which pipeline need to trigger
But actually what happens, is that it triggers two pipelines. Take an example, let suppose we have two pipelines A and B and we want to trigger B when A finishes. So in this scenario B runs 2 times, once when you do a commit (parallel with A) and second after A finishes.
To avoid this two times pipeline run problem follow the below solution
trigger: none // add this trigger value to none
resources:
pipelines:
- pipeline: RELEASE_PIPELINE // any arbitrary name
source: PIPELINE_NAME. // name of the pipeline shown on azure UI portal
trigger:
branches:
include:
- dummy_branch // name of branch on which pipeline need to trigger
By adding trigger:none second pipeline will not trigger at start commit and only trigger when first finish its job.
Hope it will help.
Microsoft documentation says that YAML is the preferred approach. So, instead of going for the build-trigger option let's understand the, little bit confusing, YAML trigger. The following tags will work from the original question and now with a bit easier documentation:
resources:
pipelines:
- pipeline: aUniqueNameHereForLocalReferenceCanBeAnything
project: projectNameNOTtheGUID
source: nameOfTheOtherPipelineNotTheDefinitionId
trigger:
branches:
include:
- master
- AnyOtherBranch
The documentation from Microsoft is confusing and the IDs are numerous. At times they want the Project GUID at times the project name. At times they want the pipeline name and at times the pipeline definition Id. But they use the same name for the variable (project and pipeline). And on top of that they write documentation that does not make it easy to guess which one to use the best way is to trial and error.
I think to avoid the confusion in other places I'm giving example of another place in the pipeline you refer to the same variables with different values. In the DownloadArtifact task, you need to use the project GUID and the pipeline definition Id as shown below:
- task: DownloadPipelineArtifact#2
inputs:
source: specific (a literal constant value not the pipeline name)
project: projectGUIDNOTtheProjectName
pipeline: numericDefinitionIdOfPipelineNotPipelineNameOrUniqueRef
runVersion: 'latest'
Just look at how they used the same variables in a different way, but both referring to a pipeline and in my case the same exact pipeline. That could create confusion and to avoid stumbling into the next issue I give it here for clarification.
The resources are not for the Build Completion trigger. according to the docs the build completion trigger not yet supported in YAML syntax.
After you create the YAML pipeline you can go to the classic editor (click on settings or variables) and there create the trigger.
Edit:
Now you need to click on the "Triggers":
And then:
Second Edit:
Microsoft added this feature also the YAML :) see here:
# this is being defined in app-ci pipeline
resources:
pipelines:
- pipeline: security-lib
source: security-lib-ci
trigger:
branches:
- releases/*
- master
In the above example, we have two pipelines - app-ci and security-lib-ci. We want the app-ci pipeline to run automatically every time a new version of the security library is built in master or a release branch.
If you're not publishing an artifact from the triggering pipeline, it won't trigger the triggered pipeline.
Also, if the defaultBranch for manual and scheduled builds in the triggered pipeline is not the same as your working branch, the triggered pipeline won't kick in at the end of the triggering pipeline execution.
I have created a minimum viable product for a pipeline trigger, and I explain better the two issues I just mentioned in this answer.