Cannot authorize variable group in Azure Pipelines - azure

I am constructing a multi-stage Azure Pipeline in Azure DevOps to build and release my product.
I want to use variable groups in this pipeline, so that I can substitute different configuration values for different deployment environments.
I am unable to authorize my variable groups to be used by the pipeline.
When I manually run a build, I see a message on the summary page telling me the variable group is not authorized:
The Azure DevOps documentation says this is to be expected:
When you make changes to the YAML file and add additional resources (assuming that these not authorized for use in all pipelines as explained above), then the build fails with a resource authorization error that is similar to the following: Could not find a {RESOURCE} with name {NAME}. The {RESOURCE} does not exist or has not been authorized for use.
In this case, on the Summary page, you will see an option to authorize the resources on the failed build. If you are a member of the User role for the resource, you can select this option. Once the resources are authorized, you can start a new build.
I am a member of the User role for the variable group, and I am seeing the message, but I am presented with no option to authorize. Is there something else I need to do? Or is there another way I can authorize a specific pipeline to use a variable group?

The provided solution proposed by #hey didn't work for me because i had to use deployment jobs. I've found a hacky way to resolve this error:
Go to your pipeline
Edit
Click on the tree dots > Triggers
Navigate to the variables tab
Variable groups
Add variable groups

Variable groups can only be accessed if they are imported at the "job" level.
Solution:
I have tested and tried to reproduce your issue. In order to solve it, you need to add the variable group under "job".
Explanation / Analysis:
This is how to reproduce and solve the issue:
First, I have tested with the below yaml script, by adding the variable group to the stage (literally at the same level as jobs):
stages:
- stage: build
variables:
- group: 789
jobs:
- job: run_build
pool:
vmImage: 'Ubuntu 16.04'
steps:
- script: echo Build
With this configuration, I was not able to use the variable group. I got the same issue as you:
I then moved the variable group into the job section of the yaml file:
stages:
- stage: build
jobs:
- job: run_build
pool:
vmImage: 'Ubuntu 16.04'
steps:
- script: echo Build
variables:
- group: 789
With the modified configuration, I was able to see and use the Authorize resources option in the error message:

I had this issue as well, but it was because when I created a variable group under Pipelines > Library, the name in Azure Portal did not match the name in my yml file.

Related

How can we define different variable group for different schedule trigger in Azure devops

Different variable group for different schedule trigger in Azure devops
Do you mean you would like to use variable in different variable groups in different schedule triggered pipelines? What about go to UI->library->choose the specific variable group then choose pipeline permission and add the corresponding schedule pipelines.
Then add the variable group name in the schedule pipelines yml and you could output the value of variables in the group, for example if there is a variable named 'SomeNumber' in the variable group.
variables:
- group: MyVarGroup
steps:
- task: PowerShell#2
inputs:
targetType: 'inline'
script: Write-Host "$(SomeNumber)"
But if you mean to add variable into the schedule trigger syntax, you can't use pipeline variables when specifying schedules in yml. You could see the official doc for more details.
If it is not what you want, please kindly specific your issue with the sample yaml, then I will try to do further investigation.

Access pipeline A´s variables from pipeline B... Azure Devops

After some internet search, I wasnt able to find a proper way or suggestion on how to access variables from different pipelines. Lets say, from Pipeline A access variables of Pipeline B.
What I did find, is the idea to use Key Vault, which I am not able to use right now. I was wondering if there is a workaround, lets say, with powershell.
All of this is happening in an Azure Devops environment, where I am trying to access/read variables from different pipelines.
Any ideas?
Kind regards
Leo.
You can make use of variable groups to call variables in multiple pipelines within a project.
You just need to reference that variable in the YAML script or release pipeline with the variable group and use it in any pipeline, Refer below :-
I went to my project > Pipelines > Library > Variable Group > And added a variable > You can add multiple variables here storing your secrets or values.
Using the variable group in a yaml pipeline :-
trigger:
- main
pool:
vmImage: ubuntu-latest
variables:
- group: SharedVariables
steps:
- script: |
echo $(databaseserverpassword)
Now, when you run the pipeline, It will ask you to permit the use of variable group for the pipeline.
This will enable access to all the variables in the SharedVariables group.
Output :-
We got our databaseservername value masked.
You can also enable this variable group for all the pipeline in the project by default.
You can use the same variable group in your Classic pipeline or release pipeline in release or specific stages like below :-
Reference :-
Variable groups for Azure Pipelines - Azure Pipelines | Microsoft Learn
For PowerShell - Azure DevOps: how to manage CI/CD variable groups using PowerShell – Radu Narita (ranari.com)

Azure devops service connection and central pipeline

I have a requirement of giving multiple teams access to a shared resource in azure. I therefore want to limit how people can publish changes to the shared resource.
The idea is to limit the use of a service connection to a specific pipeline, as per this documentation. However if the pipeline is stored in their own repo the developer could change it. This would not give me enough control. I therefore found that it was possible using a template from a central repo. Using a shared repo, would then allow me to have a service connection solely for the template?
So how I imagine doing the above is I need to grant project X a service connection for my BuildTemplates Repo. But this is basically just access to the repo and to be able to use the shared templates. Then in BuildTemplates repo I can have a service connection for my template A.
Now the developer in project X - creates her deployments and configurations for her pipeline with her own service connection scoped for her resources. Then she inherits a template from BuildTemplates Repo and passes relevant parameters for the template A.
She cannot alter the template pipeline A and only the template pipeline A can publish to the shared resource, because of the scoped service connection. I can therefore create relevant guards for the shared azure resource in the template pipeline A - so I restrict how developer X can publish to my shared azure resource.
does this make sense and is it viable?
The pipeline part in A cannot be edited by developer in X ?
The service connection in A will not propagate out so developer in X can use it in an inappropriate way?
Update
The above solution does not seem to be viable since the pipeline template is executed in the source branch scope.
Proposed Solution
The benefits I see with the above suggestion doe not seem possible, because of the issues. However one can utilise pipeline triggers, as a viable solution. This however results in a new issue. When a pipeline is triggered by Developer Y in Y's repository and it succeeds. Then a trigger is made in MAIN repository and the pipeline in MAIN fails e.g., because the artifacts from Y introduced an Issue. How does developer Y get notified about the issues in MAIN pipeline?
Here is my solution, in same Azure organization, we can create a Azure Project, then create a repo to save common pipeline template.
All the repos in other Azure project can access this pipeline template.
UserProject/UserRepo/azure-pipelines.yml
trigger:
branches:
include:
- master
paths:
exclude:
- nuget.config
- README.md
- azure-pipelines.yml
- .gitignore
resources:
repositories:
- repository: devops-tools
type: git
name: PipelineTemplateProject/CommonPipeline
ref: 'refs/heads/master'
jobs:
- template: template-pipeline.yml#devops-tools
PipelineTemplateProject/CommonPipeline/template-pipeline.yml
Since the inline script of pipeline has 5000 characters limitation,
you can put your script(not only powershell, but also other languages) in PipelineTemplateProject/CommonPipeline/scripts/test.ps1
# Common Pipeline Template
jobs:
- job: Test_Job
pool:
name: AgentPoolName
steps:
- script: |
echo "$(Build.RequestedForEmail)"
echo "$(Build.RequestedFor)"
git config user.email "$(Build.RequestedForEmail)"
git config user.name "$(Build.RequestedFor)"
git config --global http.sslbackend schannel
echo '------------------------------------'
git clone -c http.extraheader="AUTHORIZATION: bearer $(System.AccessToken)" -b $(ToolsRepoBranch) --single-branch --depth=1 "https://PipelineTemplateProject/_git/CommonPipeline" DevOps_Tools
echo '------------------------------------'
displayName: 'Clone DevOps_Tools'
- task: PowerShell#2
displayName: 'Pipeline Debug'
inputs:
targetType: 'inline'
script: 'Get-ChildItem -Path Env:\ | Format-List'
condition: always()
- task: PowerShell#2
displayName: 'Run Powershell Scripts'
inputs:
targetType: filePath
filePath: 'DevOps_Tools/scripts/test.ps1'
arguments: "$(System.AccessToken)"
Notes:
Organization Setting - Settings - Disable Limit job authorization scope to current project for release pipelines
Organization Setting - Settings - Limit job authorization scope to current project for non-release pipelines
Check some option in project setting as well.
So the normal user only access their own repo, cannot access DevOps project, and DevOps owner can edit template pipeline only.
For the notification issue, I use an Email extention "rvo.SendEmailTask.send-email-build-task.SendEmail#1"

Environments are not automatically created in Azure Devops when declared in yaml pipeline config

I'm new to Azure Devops. I would like to have devops pipeline environments to be created automatically during pipeline flow. So the 5th line below should create environment if it does not exist:
- deployment: Deploy
displayName: Deploy job
pool:
vmImage: $(vmImageName)
environment: 'production'
Instead I'm getting:
What am I missing?
To automate environment creation I could also use Terraform but this time I cannot find terraform resource config responsible for that.
I had a similar problem and found that the documentation lists some possible reasons for why this can happen:
Quote from learn.microsoft.com:
Q: Why am I getting error "Job XXXX: Environment XXXX could not be
found. The environment does not exist or has not been authorized for
use"?
A: These are some of the possible reasons of the failure:
When you author a YAML pipeline and refer to an environment that does not exist
in the YAML file, Azure Pipelines automatically creates the
environment in some cases:
You use the YAML pipeline creation wizard in the Azure Pipelines web
experience and refer to an environment that hasn't been created yet.
You update the YAML file using the Azure Pipelines web editor and save
the pipeline after adding a reference to an environment that does not
exist.
In the following flows, Azure Pipelines does not have
information about the user creating the environment: you update the
YAML file using another external code editor, add a reference to an
environment that does not exist, and then cause a manual or continuous
integration pipeline to be triggered. In this case, Azure Pipelines
does not know about the user. Previously, we handled this case by
adding all the project contributors to the administrator role of the
environment. Any member of the project could then change these
permissions and prevent others from accessing the environment.
If you are using runtime parameters for creating the environment, it
will fail as these parameters are expanded at run time. Environment
creation happens at compile time, so we have to use variables to
create the environment.
A user with stakeholder access level cannot create the environment as
stakeholders do not access to repository.
In our case, the problem was using runtime parameters for creating the environment.
You have the environment name 'production' hardcoded, so your problem might be related to one of the other cases.

Security hole in Azure Pipelines?

I've been researching Azure DevOps and I've come across what looks like a pretty obvious security hole in Azure pipelines.
So, I'm creating my pipeline as YAML and defining 2 stages: a build stage, and a deployment stage. The deployment stage looks like this:
- stage: deployApiProdStage
displayName: 'Deploy API to PROD'
dependsOn: buildTestApiStage
jobs:
- deployment: deployApiProdJob
displayName: 'Deploy API to PROD'
timeoutInMinutes: 10
condition: and(succeeded(), eq(variables.isRelease, true))
environment: PROD
strategy:
runOnce:
deploy:
steps:
- task: AzureWebApp#1
displayName: 'Deploy Azure web app'
inputs:
azureSubscription: '(service connection to production web app)'
appType: 'webAppLinux'
appName: 'my-web-app'
package: '$(Pipeline.Workspace)/$(artifactName)/**/*.zip'
runtimeStack: 'DOTNETCORE|3.1'
startUpCommand: 'dotnet My.Api.dll'
The Microsoft documentation talks about securing this by adding approvals and checks to an environment; in the above case, the PROD environment. This would be fine if the protected resource here that allows publishing to my PROD web app - the service connection in azureSubscription - were pulled from the PROD environment. Unfortunately, as far as I can tell, it's not. It's associated instead with the pipeline itself.
This means that when the pipeline is first run, the Azure DevOps UI prompts me to permit the pipeline access to the service connection, which is needed for any deployment to happen. Once access is permitted, that pipeline has access to that service connection for evermore. This means that from then on, that service connection can be used no matter which environment is specified for the job. Worse still, any environment name specified that is not recognized does not cause an error, but causes a blank environment to be created by default!
So even if I setup a manual approval for the PROD environment, if someone in the organization manages to slip a change through our code review (which is possible, with regular large code reviews) that changes the environment name to 'NewPROD' in the azure-pipelines.yml file, the CI/CD will create that new environment, and go ahead and deploy immediately to PROD because the new environment has no checks or approvals!
Surely it would make sense for the service connection to be associated with the environment instead. It would also make sense to have an option to ban the auto-creation of new environments - I don't really see how that's particularly useful anyway. Right now, as far as I can tell, this is a huge security hole that could allow deployments to critical environments by anyone who has commit access to the repo or manages to slip a change to the azure-pipelines.yml file through the approval process, introducing a major single point of failure/weakness. What happened to the much-acclaimed incremental approach to securing your pipelines? Am I missing something here, or is this security hole as bad as I think it is?
In your example, it seemed you created/used an empty environment, there is no deployment target. Currently, only the Kubernetes resource and virtual machine resource types are supported in an environment.
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/environments?view=azure-devops
The resource in your example is a service connection, so you need to go the service connection and define checks for this service connection.
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/approvals?view=azure-devops&tabs=check-pass

Resources