How do I use ARM 'outputs' values another release task? - azure

I have an ARM template that has and outputs section like the following:
"outputs": {
"sqlServerFqdn": {
"type": "string",
"value": "[reference(concat('Microsoft.Sql/servers/', variables('sqlserverName'))).fullyQualifiedDomainName]"
},
"primaryConnectionString": {
"type": "string",
"value": "[concat('Data Source=tcp:', reference(concat('Microsoft.Sql/servers/', variables('sqlserverName'))).fullyQualifiedDomainName, ',1433;Initial Catalog=', variables('databaseName'), ';User Id=', parameters('administratorLogin'), '#', variables('sqlserverName'), ';Password=', parameters('administratorLoginPassword'), ';')]"
},
"envResourceGroup": {
"type": "string",
"value": "[parameters('hostingPlanName')]"
}
}
I have a Azure Resource Group Deployment task that uses the template. I then want to use the variable $(sqlServerFqdn) in the next task for configuration. The variable doesn't seem to just populate and I cannot find anywhere that tells me how to use 'outputs' values on release.
What do I need to do to get the variable to populate for use in configuring tasks after this ARM template runs? An example would be in the parameters to a powershell script task or another ARM template.

VSTS Azure Resource Group Deployment task has outputs section now (since January 2018). So you can set variable name in Deployment outputs of Azure Resource Group Deployment task to, for example, ResourceGroupDeploymentOutputs and add PowerShell Script task with the following inline script:
# Make outputs from resource group deployment available to subsequent tasks
$outputs = ConvertFrom-Json $($env:ResourceGroupDeploymentOutputs)
foreach ($output in $outputs.PSObject.Properties) {
Write-Host "##vso[task.setvariable variable=RGDO_$($output.Name)]$($output.Value.value)"
}
And in subsequent tasks you can use your template variables. So, for example, if you have sqlServerFqdn variable in your template it will be available as $(RGDO_sqlServerFqdn) after PowerShell Script task is completed.

Capturing this answer because I always end up at this question when searching for the solution.
There is a marketplace task which makes ARM template output parameters available further down the pipeline. But in some cases you don't have permission to purchase marketplace items for your subscription, so the following PowerShell will do the same thing. To use it you add it as a powershell script step immediately following the ARM template resource group deployment step. It will look at the last deployment and pull the output variables into pipeline variables.
param(
[string] $resourceGroupName
)
$lastDeployment = Get-AzureRmResourceGroupDeployment -ResourceGroupName $resourceGroupName | Sort Timestamp -Descending | Select -First 1
if(!$lastDeployment) {
throw "Deployment could not be found for Resource Group '$resourceGroupName'."
}
if(!$lastDeployment.Outputs) {
throw "No output parameters could be found for the last deployment of Resource Group '$resourceGroupName'."
}
foreach ($key in $lastDeployment.Outputs.Keys){
$type = $lastDeployment.Outputs.Item($key).Type
$value = $lastDeployment.Outputs.Item($key).Value
if ($type -eq "SecureString") {
Write-Host "##vso[task.setvariable variable=$key;issecret=true]$value"
}
else {
Write-Host "##vso[task.setvariable variable=$key;]$value"
}
}
Note that the environmental variables won't be available in the context of this script, but will in subsequent tasks.

The output value shown on the UI for the Visual Studio Team Services task for Azure Resource Group Deployment only seems to work for the scenario described in Eddie's answer, which is for VMs. In fact, if your deployment doesn't include VMs, you will get an error something like:
No VMs found in resource group: 'MY-RESOURCE-GROUP-NAME'. Could not
register environment in the output variable: 'myVariableName'.
For non-VM examples, I created a powershell script that runs after the RG deployment. This script, as an example, takes input variables for resource group $resourceGroupName and the name of the output variable you need $rgDeploymentOutputParameterName. You could customize and use something similar:
#get the most recent deployment for the resource group
$lastRgDeployment = Get-AzureRmResourceGroupDeployment -ResourceGroupName $resourceGroupName |
Sort Timestamp -Descending |
Select -First 1
if(!$lastRgDeployment)
{
throw "Resource Group Deployment could not be found for '$resourceGroupName'."
}
$deploymentOutputParameters = $lastRgDeployment.Outputs
if(!$deploymentOutputParameters)
{
throw "No output parameters could be found for the last deployment of '$resourceGroupName'."
}
$outputParameter = $deploymentOutputParameters.Item($rgDeploymentOutputParameterName)
if(!$outputParameter)
{
throw "No output parameter could be found with the name of '$rgDeploymentOutputParameterName'."
}
$outputParameterValue = $outputParameter.Value
# From here, use $outputParameterValue, for example:
Write-Host "##vso[task.setvariable variable=$rgDeploymentOutputParameterName;]$outputParameterValue"

In Nov 2020, after this commit - https://github.com/microsoft/azure-pipelines-tasks/commit/1173324604c3f61ce52cdcc999f6d4d7ea9ab8f9 , the variables could directly be used in the subsequent tasks in the pipeline (No powershell scripts required!!)
This is what the steps look like -
In the ARM template deployment task, give any reference name to the Deployment Outputs section under Advanced drop down. In my case I have given armOutputVariable.
See image for visual description
Now to use the value of sqlServerFqdn in the subsequent tasks, simply use it in this manner $(armOutputVariable.sqlServerFqdn.value)
For example, let's say I want to use it to override a parameter in my test task which follows the deployment so I can use it in the following manner -
Example image
To summarize all the outputs in the ARM could be used in the further steps directly in this manner (make sure you assign a reference name in the ARM template deployment step) -
$(armOutputVariable.sqlServerFqdn.value)
$(armOutputVariable.sqlServerFqdn.type)
$(armOutputVariable.primaryConnectionString.value)
$(armOutputVariable.primaryConnectionString.type)
$(armOutputVariable.envResourceGroup.value)
$(armOutputVariable.envResourceGroup.type)

First you define the Azure Resource Deployment Task and in this context the Deployment outputs
In the next step you create a PowerShell Task that takes the Deployment outputs defined above as input arguments
The PowerShell script looks as follows and assigns for each output defined in the ARM template a separate VSTS environment variable with the same name as defined in the ARM template output section. These variables can then be used in subsequent tasks.
param (
[Parameter(Mandatory=$true)]
[string]
$armOutputString
)
Write-Host $armOutputString
$armOutputObj = $armOutputString | convertfrom-json
Write-Host $armOutputObj
$armOutputObj.PSObject.Properties | ForEach-Object {
$type = ($_.value.type).ToLower()
$key = $_.name
$value = $_.value.value
if ($type -eq "securestring") {
Write-Host "##vso[task.setvariable variable=$key;issecret=true]$value"
Write-Host "Create VSTS variable with key '$key' and value '$value' of type '$type'!"
} elseif ($type -eq "string") {
Write-Host "##vso[task.setvariable variable=$key]$value"
Write-Host "Create VSTS variable with key '$key' and value '$value' of type '$type'!"
} else {
Throw "Type '$type' not supported!"
}
}
In a subsequent task you can access the environment variables either by passing them as argument via '$(varName)' (this works for SecureString too) or e.g. in a PowerShell script via $env:varName (this does not work for SecureString)

VSTS allows setting variables in powershell scripts which you can use in other tasks.
The syntax is
Write-Host "##vso[task.setvariable variable=myvariable;]myvalue"
You can have an inline Powershell script which can set the required variable to consume in yet to be executed tasks.You can access it like $(myvariable).
You may need to system.debug variable to true to use this.
Read more details here.

You just need to add an output variable name for "Azure Resource Group Deployment" task like following:
And then use the variable in "PowerShell on Target Machines" task:
"PowerShell on Target Machines" task will use the resource configured in "Azure Resource Group Deployment" task:
Output variables:
Create/update action of the Azure Resource Group task now produces an
output variable during execution. The output variable can be used to
refer to the resource group object in the subsequent tasks. For
example "PowerShell on Target Machine" task can now refer to resource
group output variable as '$(variableName)' so that it can execute the
powershell script on the resource group VM targets.
Limitation: Output variable produced during execution will have
details about VM hostname(s) and (public) ports, if any. Credentials
to connect to the VM host(s) are to be provided explicitly in the
subsequent tasks.
Refer to this link for more details: Azure Resource Group Deployment Task

Related

using variables from variable group in release tasks

I have created a variable group and set a variable there CustomArtifactName and have given the scope as Release
The value of this variable is set in build pipeline so it generates a unique name for artifacts
Write-Host "##vso[task.setvariable variable=CustomArtifactName]$(Get-Date -Format yyyyMMddhhmmss)"
I can use this fine in build pipelline in yaml in PublishPipelineArtifact#1 tasks like this
'drop-pom-''$(CustomArtifactName)'
But when I use the below in release pipeline maven tasks in Maven POM File field
$(System.DefaultWorkingDirectory)/cucumber-java-junit5-webapp/drop-pom-$(CustomArtifactName)/s/target/pom.xml
I get
##[error]Unhandled: Not found mavenPOMFile:
I also tried to use this variable in Download Pipeline Artifact task but got similar result
How do I use variables from variable group in release tasks fields/properties? According to this msdoc we simply have to use $(variableName) but thats not working .
Based on your description, when you reference the variable group in the build pipeline, you set the variable with new value.
The new value only applies to the build pipeline, it does not update the value of the variable in the variable group, so it will still use the old variable in the release pipeline. This could be the root cause of this issue.
To solve this issue , you need to add a Powershell task to run Rest API(Variablegroups - Update) to update the variable in variable group.
Here is the Powershell Script example:
$token = "PAT"
$url="https://dev.azure.com/{organizationname}/{projectname}/_apis/distributedtask/variablegroups/{variablegroupid}?api-version=5.0-preview.1"
$token = [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes(":$($token)"))
$JSON = #'
{
"variables": {
"CustomArtifactName": {
"value": "$(CustomArtifactName)"
}
},
"type": "Vsts",
"name": "{variablegroupname}",
"description": "Updated variable group"
}
'#
$response = Invoke-RestMethod -Uri $url -Headers #{Authorization = "Basic $token"} -Method PUT -Body $JSON -ContentType application/json
Result: The variable in variable group will update based on each build.
Update:
variable group id:
You could open the variable group in website and check the URL.
When the Powershell script runs successfully, the value of the variable will be passed to the variable in Variable Group.

Azure YAML pipeline: find variable in group dynamically

I have a Powershell script that is executed in my YAML pipeline during the release stage.
I can reference group variables using $(myVar). However now I have a need to reference a variable dynamically given its name.
For instance, if I have the string 'myVar', I want to search the variable group for a variable named 'myVar' and get its value.
Is this possible to achieve ?
In the end I did it like this. First I installed the Devops module for powershell so I can get group variables through devops api :
Add-AzureDevOpsAccount -OrganisationName "myor" -ProjectName "myproj" -Token "mytoken"
Write-Host -ForegroundColor DarkGreen "Getting variables group based on name"
$group = Get-AzureDevOpsVariableGroup -SearchString "dev"
Then to get a variable based on its name :
$tokenValue = $group.variables | Select-Object -ExpandProperty $someName | Select-Object -ExpandProperty Value

How to generate reports for Azure Log Activity Data using python for any resource alongwith 'Tags'?

My colleague used the below powershell query to retrieve log data for past 4 days excluding today which matches operations of the resources and collects features such as EventTimeStamp, Caller, SubscriptionId etc.
Get-AzureRmLog -StartTime (Get-Date).AddDays(-4) -EndTime (Get-Date).AddDays(-1) | Where-Object {$_.OperationName.LocalizedValue -match "Start|Stop|Restart|Create|Update|Delete"} |
Select-Object EventTimeStamp, Caller, SubscriptionId, #{name="Operation"; Expression = {$_.operationname.LocalizedValue}},
I am a newbie to azure and want to generate a report where I can also fetch the 'Tags' name & value against a resource for past 90 days in this report. What will be the powershell query for this? Can I also use python to query this data? I tried searching the documentation and was unable to dig into it, so if anybody could redirect me to the right place it will be helpful.
First of all, you should know that not all azure resources can specify tags, so you should consider this in your code. Please refer to Tag support for Azure resources to check which azure resource supports tags.
For powershell query, I suggest using the new azure powershell az module instead of the old azureRM module.
Here is a simple powershell code with az module. And for testing purpose, I just introduce how to fetch and add tags to the output. Please feel free to change it as per your requirement.
#for testing purpose, I just get the azure activity logs from a specified resource group
$mylogs = Get-AzLog -ResourceGroupName "a resource group name"
foreach($log in $mylogs)
{
if(($log.Properties.Content.Values -ne $null))
{
#the tags is contains in the Properties of the log entry.
$s = $log.Properties.Content.Values -as [string]
if($s.startswith("{"))
{
$log | Select-Object EventTimeStamp, Caller, SubscriptionId,#{name="Operation"; Expression = {$_.operationname.LocalizedValue}}, #{name="tags"; Expression = {($s | ConvertFrom-Json).tags}}
}
#if it does not contains tags.
else
{
$log | Select-Object EventTimeStamp, Caller, SubscriptionId,#{name="Operation"; Expression = {$_.operationname.LocalizedValue}}, #{name="tags"; Expression = {""}}
}
}
#if it does not contains tags.
else
{
$log | Select-Object EventTimeStamp, Caller, SubscriptionId,#{name="Operation"; Expression = {$_.operationname.LocalizedValue}}, #{name="tags"; Expression = {""}}
}
Write-Output "************************"
}
The test result:
For python, you can take a look at this github issue which introduces how to fetch logs from azure activity logs, but you need do some research on how to add tags to the output.
Hope it helps.

Output variables in Octopus -Incorrectly Passing null value

I want to pass a value in Octopus from one step to another of a project via output variable, the value is "VM running" or "VM deallocated".
There are two server, one of the server is down, another one is running so values should be passed accordingly. Now when I use the exact syntax of Output variable, it is passing Null value to next step.
Octopus deploy Project Step 1:
$RG = $RGName
$VM = "#{StepTemplate_VMName}"
$WarningPreference = 'SilentlyContinue'
$VMStats = (Get-AzureRmVM -Name $VM -ResourceGroupName $RG -Status).Statuses
$stats = ($VMStats | Where Code -Like 'PowerState/*')[0].DisplayStatus
Set-OctopusVariable -name "RunStatus" -value $stats
write-host $stats #value can either be "VM running" or "VM deallocated"
Octopus deploy Project Step 2:
$VM = "#{StepTemplate_VMName}"
$Runstatus = $OctopusParameters["Octopus.Action[Step1].Output[$VM].RunStatus"]
write-host $Runstatus
If I do not use [$VM] in the code of step 2, it give only 1 value to both the machine as "VM running"
As per the syntax given in Octopus website, we should use the VM name to pass machine specific different values.
so I used [$VM] but it gives null values to both of the machine
Edit: Event If I hardcode the value of $VM to any one VMName, it still gives me null.
Based on what I do in Octopus it should be:
$Runstatus = $OctopusParameters["Octopus.Action[Step1].Output.RunStatus"]
Even states that format on the help documentation. https://octopus.com/docs/projects/variables/output-variables
Is the step that's creating the output variable actually being run on the server that's down? When the output variable is created in a step that's run on multiple machines, multiple machine-scoped values for that variable will be created. When the subsequent step is run on the same machines, Octopus will pick the correct value based on that scope. So maybe a little off topic, but if both steps are run on both targets, specifying the $VM scope is redundant, and is only needed if you want a different machine's value.
Since hardcoding the target name here returns null it seems like the output variable creation step wasn't run on that target. I'd suggest debugging this by creating a variable called OctopusPrintEvaluatedVariables with value True in your project. That'll increase the task log verbosity and log how all variables are evaluated through each step of the deployment.

Accessing pipeline activity status from an Azure Function

I have an Azure Function which triggers a Pipeline and I'm able to poll the pipeline status to check when it completes using: Pipeline.Properties.RuntimeInfo.PipelineState
My pipeline uses several parallel Copy activities and I'd like to be able to access the status of these activities incase they fail. The Azure documentation describes how to access the pipeline activities but you can only get at static properties like name and description but not dynamic properties like Status (like you can for the Pipeline via its RuntimeInfo property).
For completeness, I've accessed the activity list using:
IList<Microsoft.Azure.Management.DataFactories.Models.Activity> activityList = plHandle.Pipeline.Properties.Activities;
Is it possible to check individual activity statuses programmatically?
Its certainly possible.
I use the ADF PowerShell cmdlets in the Azure module to monitor our data factories.
Maybe do something like the below for what you need with Get-AzureRmDataFactoryActivityWindow command.
Eg:
$ActivityWindows = Get-AzureRmDataFactoryActivityWindow `
-DataFactoryName $ADFName.DataFactoryName `
-ResourceGroupName $ResourceGroup `
| ? {$_.WindowStart -ge $Now} `
| SELECT ActivityName, ActivityType, WindowState, RunStart, InputDatasets, OutputDatasets `
| Sort-Object ActivityName
This gives you the activity level details including the status. Being:
Ready
In Progress
Waiting
Failed
... I list them because they differ slightly from what you see in the portal blades.
The datasets are also arrays if you have multiple inputs and outputs for particular activities.
More ADF cmdlets available here: https://learn.microsoft.com/en-gb/powershell/module/azurerm.datafactories/?view=azurermps-3.8.0
Hope this helps
I've managed to resolve this by accessing the DataSliceRuns (i.e. activities) for the pipeline as follows:
var datasets = client.Datasets.ListAsync(<resourceGroupName>, <DataFactoryName>).Result;
foreach (var dataset in datasets.Datasets)
{
// Check the activity statuses for the pipelines activities.
var datasliceRunlistResponse = client.DataSliceRuns.List(<resourceGroupName>, <dataFactoryName>,<DataSetName>, new DataSliceRunListParameters()
{
DataSliceStartTime = PipelineStartTime.ConvertToISO8601DateTimeString()
});
foreach (DataSliceRun run in datasliceRunlistResponse.DataSliceRuns)
{
// Do stuff...
}
}

Resources