Azure Automation Purge CDN Endpoint - azure

I am trying to put in place a Azure Automation Runbook with the intent to purge all the cache when I change is made to a blob storage. So far, if I upload from azure portal 1 file that works just fine. But if I try to upload multiple file, some of them they just fail with the following error.
We can only accept 100 paths for purging concurrently. Please try again in a few minutes.
Here is the code I am using in the automation Runbook process:
param (
[Parameter (Mandatory = $false)]
[object] $WebHookData
)
## Authentication ##
# Runbook must authenticate to purge content
# Connect to Azure with RunAs account
$conn = Get-AutomationConnection -Name "AzureRunAsConnection"
# Connect to Azure Automation
$null = Add-AzAccount `
-ServicePrincipal `
-TenantId $conn.TenantId `
-ApplicationId $conn.ApplicationId `
-CertificateThumbprint $conn.CertificateThumbprint
## declarations ##
# Update parameters below
# CDN Profile name
$profileName = "<CDNProfileName>"
# CND Resource Group
$resourceGroup = "<Resource-Group>"
# CDN Endpoint Name
$endPointName = "<EndPointName>"
# Set Error Action Default
$errorDefault = $ErrorActionPreference
## Execution ##
# Convert Webhook Body to json
try {
$requestBody = $WebHookData.requestBody | ConvertFrom-json -ErrorAction 'stop'
}
catch {
$ErrorMessage = $_.Exception.message
write-error ('Error converting Webhook body to json ' + $ErrorMessage)
Break
}
# Convert requestbody to file path
try {
$ErrorActionPreference = 'stop'
$filePath = $requestBody.data.url -replace "https://<storageaccountname>.blob.core.windows.net",""
}
catch {
$ErrorMessage = $_.Exception.message
write-error ('Error converting file path ' + $ErrorMessage)
Break
}
finally {
$ErrorActionPreference = $errorDefault
}
# Run the purge command against the file
try {
Unpublish-AzCdnEndpointContent -ErrorAction 'Stop' -ProfileName $profileName -ResourceGroupName $resourceGroup `
-EndpointName $endPointName -PurgeContent '/*'
}
catch {
$ErrorMessage = $_.Exception.message
write-error ('Error purging content from CDN ' + $ErrorMessage)
Break
}
Anyone can help with this or clarify to me what could be the reason why the Purge is failing with that error ("BadRequest")
Thank you so much for your help

From the bottom of the article about CDN purge enpoint:
Purge requests take approximately 10 minutes to process with Azure CDN
from Microsoft, approximately 2 minutes with Azure CDN from Verizon
(standard and premium), and approximately 10 seconds with Azure CDN
from Akamai. Azure CDN has a limit of 100 concurrent purge requests
at any given time at the profile level.
There is a limit of 100 concurrent purge requests at any given time at the profile level.

Related

Not able to access Azure FileShare Storage container from Azure Automation Runbook

I've the following Azure Automation Runbook script which goal is to take an dump/export from a REST API call which must run from a target device which is able to reach the REST API device. So Azure Automation runbook is targeting a "proxy server" then from this we're taking the REST API backup.
The approach has been working exception the fact we're able not to copy this backup file from the target server once 'cm.vm.run_command' presents output size limitation and is truncating the backup. The workaround we found for this was copying the backup file from the 'target/proxy server' directly into a Storage Account Fileshare which is mounted on the target/proxy server. My problem now is when running from Azure Automation it's not able to access the drive mounted by other user and/or is not able to mount the device or access it directly like below errors messages. Does anybody have any alternative for this ? I was able to check the runbook is having connectivity on the storage account ports 443/445 from t. That was one of the possible reasons described here https://learn.microsoft.com/en-us/azure/storage/files/storage-troubleshoot-windows-file-connection-problems
Below the commands and errors I'm receiving and the whole script used.
Copy-item -Path C:\Devicebackup.txt -Destination \\storage_account_name.file.core.windows.net\configdatafileshare\Orchestration
net use w: \\storage_account_name.file.core.windows.net\configdatafileshare\Orchestration `'/yBapkthow==`' /user:Azure\storage_account_name
Copy-item : The network path was not found
At C:\Packages\Plugins\Microsoft.CPlat.Core.RunCommandWindows\1.1.5\Downloads\s
cript9.ps1:15 char:1
+ Copy-item -Path C:\Devicebackup.txt -Destination \\storage_account_name. ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Copy-Item], IOException
+ FullyQualifiedErrorId : System.IO.IOException,Microsoft.PowerShell.Comma
nds.CopyItemCommand
The option /DL2D2QKD1OU2ZKEOJVRK4LGPIRTJKAJBZ+EDKNHWVYYEJDDYSL9CPB5T8F/9VWQBMBWC37B1NJS4YBAPKTHOW== is unknown.
The syntax of this command is:
NET USE
[devicename | *] [\\computername\sharename[\volume] [password | *]]
[/USER:[domainname\]username]
[/USER:[dotted domain name\]username]
[/USER:[username#dotted domain name]
[/SMARTCARD]
[/SAVECRED]
[[/DELETE] | [/PERSISTENT:{YES | NO}]]
NET USE {devicename | *} [password | *] /HOME
NET USE [/PERSISTENT:{YES | NO}]
Param (
[Parameter(Mandatory=$false)][string] $rgName
,[Parameter(Mandatory=$false)][string] $ProxyServerName
)
function CreatePSCommandFile {
Param(
[parameter(Mandatory=$true)][String[]]$DeviceName,
[parameter(Mandatory=$true)][String[]]$DeviceIP,
[parameter(Mandatory=$true)][String[]]$ApiToken
)
$remoteCommand =
#"
add-type #`"
using System.Net;
using System.Security.Cryptography.X509Certificates;
public class TrustAllCertsPolicy : ICertificatePolicy {
public bool CheckValidationResult(
ServicePoint srvPoint, X509Certificate certificate,
WebRequest request, int certificateProblem) {
return true;
}
}
`"#
[System.Net.ServicePointManager]::CertificatePolicy = New-Object TrustAllCertsPolicy
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
Invoke-WebRequest -Uri 'www.mydownload.com' -UseBasicParsing -Headers #{ Authorization="Bearer $($ApiToken)" } | Out-file C:\Devicebackup.txt
net use w: \\storage_account_name.file.core.windows.net\configdatafileshare\Orchestration `'/STORAGE_KEY+EDknHWvyyeJDDYsL9cPB5T8F/9VwqBmbwc37B1NJS4yBapkthow==`' /user:Azure\storage_account_name
Copy-item -Path C:\Devicebackup.txt -Destination \\storage_account_name.file.core.windows.net\configdatafileshare\Orchestration
"#
Set-Content -Path .\InvokeCommand.ps1 -Value $remoteCommand
}
$connectionName = "AzureRunAsConnection"
try {
# Get the connection "AzureRunAsConnection "
$servicePrincipalConnection = Get-AutomationConnection -Name $connectionName
Write-Host "Logging in to Azure..."
$connectionResult = Connect-AzAccount `
-ServicePrincipal `
-Tenant $servicePrincipalConnection.TenantID `
-ApplicationId $servicePrincipalConnection.ApplicationID `
-CertificateThumbprint $servicePrincipalConnection.CertificateThumbprint
}
catch {
if (!$servicePrincipalConnection)
{
$ErrorMessage = "Connection $connectionName not found."
throw $ErrorMessage
} else{
Write-Error -Message $_.Exception
throw $_.Exception
}
}
function Backup-Device {
Param (
[Parameter(Mandatory=$false)][string] $DeviceName
,[Parameter(Mandatory=$false)][string] $DeviceIP
,[Parameter(Mandatory=$false)][string] $ApiToken
)
# Execute Backup on Fortigate Rest API
CreatePSCommandFile -DeviceName $DeviceName -DeviceIP $DeviceIP -ApiToken $ApiToken
$Output = Invoke-AzVMRunCommand -ResourceGroupName $rgName -VMName $ProxyServerName -CommandId 'RunPowerShellScript' -Scriptpath ".\InvokeCommand.ps1" -Parameter #{'api_url' = "10.29.255.212"; 'api_token' = "0p6h1rmspjf37kp80bc6ny88jw"}
($Output).Value.Message
}
Backup-Device -DeviceName "DeviceName" -DeviceIP '10.29.255.212' -ApiToken 'Api_Token'
Sharing the solution which was presented by a blessed colleague :)
Using New-SmbMapping we were able to mount the Storage Account File Share from Azure Automation PS script successfully.
if (!(Test-Path `$MapDrive)) {
New-SmbMapping -LocalPath `$MapDrive -RemotePath `$RemotePath -UserName `$UserName -Password `$Key
}
Copy-Item .\Devicebackup.txt `$MapDrive

Azure Data Factory webhook execution times out instead of relaying errors

I've attempted to set up a simple webhook execution in Azure Data Factory (v2), calling a simple (parameter-less) webhook for an Azure Automation Runbook I set up.
From the Azure Portal, I can see that the webhook is being executed and my runbook is being run, so far so good. The runbook is (currently) returning an error within 1 minute of execution - but that's fine, I also want to test failure scenarios.
Problem:
Data Factory doesn't seem to be 'seeing' the error result and spins until the timeout (10 minutes) elapses. When I kick off a debug run of the pipeline, I get the same - a timeout and no error result.
Update: I've fixed the runbook and it's now completing successfully, but Data Factory is still timing out and is not seeing the success response either.
Here is a screenshot of the setup:
And here is the portal confirming that the webhook is being run by azure data factory, and is completing in under a minute:
WEBHOOKDATA JSON is:
{"WebhookName":"Start CAMS VM","RequestBody":"{\r\n \"callBackUri\": \"https://dpeastus.svc.datafactory.azure.com/dataplane/workflow/callback/f7c...df2?callbackUrl=AAEAFF...0927&shouldReportToMonitoring=True&activityType=WebHook\"\r\n}","RequestHeader":{"Connection":"Keep-Alive","Expect":"100-continue","Host":"eab...ddc.webhook.eus2.azure-automation.net","x-ms-request-id":"7b4...2eb"}}
So as far as I can tell, things should be in place to pick up on the result (success of failure). Hopefully someone who's done this before knows what I'm missing.
Thanks!
I had assumed that Azure would automatically notify the ADF "callBackUri" with the result once the runbook completed or errored out (since they take care of 99% of the scaffolding without requiring a line of code).
It turns out that is not the case, and anyone wishing to execute a runbook from ADF will have to manually extract the callBackUri from the Webhookdata input parameter, and POST the result to it when done.
I haven't nailed this down yet, since the Microsoft tutorial sites I've found have a bad habit of taking screenshots of the code that does this rather than providing the code itself:
I guess I'll come back and edit this once I have it figured out.
EDIT I ended up implementing this by leaving my original Webhook untouched, and creating a "wrapper"/helper/utility Runbook that will execute an arbitrary webhook, and relay its status to ADF once it's complete.
Here's the full code I ended up with, in case it helps someone else. It's meant to be generic:
Setup / Helper Functions
param
(
[Parameter (Mandatory = $false)]
[object] $WebhookData
)
Import-Module -Name AzureRM.resources
Import-Module -Name AzureRM.automation
# Helper function for getting the current running Automation Account Job
# Inspired heavily by: https://github.com/azureautomation/runbooks/blob/master/Utility/ARM/Find-WhoAmI
<#
Queries the automation accounts in the subscription to find the automation account, runbook and resource group that the job is running in.
AUTHOR: Azure/OMS Automation Team
#>
Function Find-WhoAmI {
[CmdletBinding()]
Param()
Begin { Write-Verbose ("Entering {0}." -f $MyInvocation.MyCommand) }
Process {
# Authenticate
$ServicePrincipalConnection = Get-AutomationConnection -Name "AzureRunAsConnection"
Add-AzureRmAccount `
-ServicePrincipal `
-TenantId $ServicePrincipalConnection.TenantId `
-ApplicationId $ServicePrincipalConnection.ApplicationId `
-CertificateThumbprint $ServicePrincipalConnection.CertificateThumbprint | Write-Verbose
Select-AzureRmSubscription -SubscriptionId $ServicePrincipalConnection.SubscriptionID | Write-Verbose
# Search all accessible automation accounts for the current job
$AutomationResource = Get-AzureRmResource -ResourceType Microsoft.Automation/AutomationAccounts
$SelfId = $PSPrivateMetadata.JobId.Guid
foreach ($Automation in $AutomationResource) {
$Job = Get-AzureRmAutomationJob -ResourceGroupName $Automation.ResourceGroupName -AutomationAccountName $Automation.Name -Id $SelfId -ErrorAction SilentlyContinue
if (!([string]::IsNullOrEmpty($Job))) {
return $Job
}
Write-Error "Could not find the current running job with id $SelfId"
}
}
End { Write-Verbose ("Exiting {0}." -f $MyInvocation.MyCommand) }
}
Function Get-TimeStamp {
return "[{0:yyyy-MM-dd} {0:HH:mm:ss}]" -f (Get-Date)
}
My Code
### EXPECTED USAGE ###
# 1. Set up a webhook invocation in Azure data factory with a link to this Runbook's webhook
# 2. In ADF - ensure the body contains { "WrappedWebhook": "<your url here>" }
# This should be the URL for another webhook.
# LIMITATIONS:
# - Currently, relaying parameters and authentication credentials is not supported,
# so the wrapped webhook should require no additional authentication or parameters.
# - Currently, the callback to Azure data factory does not support authentication,
# so ensure ADF is configured to require no authentication for its callback URL (the default behaviour)
# If ADF executed this runbook via Webhook, it should have provided a WebhookData with a request body.
if (-Not $WebhookData) {
Write-Error "Runbook was not invoked with WebhookData. Args were: $args"
exit 0
}
if (-Not $WebhookData.RequestBody) {
Write-Error "WebhookData did not contain a ""RequestBody"" property. Data was: $WebhookData"
exit 0
}
$parameters = (ConvertFrom-Json -InputObject $WebhookData.RequestBody)
# And this data should contain a JSON body containing a 'callBackUri' property.
if (-Not $parameters.callBackUri) {
Write-Error 'WebhookData was missing the expected "callBackUri" property (which Azure Data Factory should provide automatically)'
exit 0
}
$callbackuri = $parameters.callBackUri
# Check for the "WRAPPEDWEBHOOK" parameter (which should be set up by the user in ADF)
$WrappedWebhook = $parameters.WRAPPEDWEBHOOK
if (-Not $WrappedWebhook) {
$ErrorMessage = 'WebhookData was missing the expected "WRAPPEDWEBHOOK" peoperty (which the user should have added to the body via ADF)'
Write-Error $ErrorMessage
}
else
{
# Now invoke the actual runbook desired
Write-Output "$(Get-TimeStamp) Invoking Webhook Request at: $WrappedWebhook"
try {
$OutputMessage = Invoke-WebRequest -Uri $WrappedWebhook -UseBasicParsing -Method POST
} catch {
$ErrorMessage = ("An error occurred while executing the wrapped webhook $WrappedWebhook - " + $_.Exception.Message)
Write-Error -Exception $_.Exception
}
# Output should be something like: {"JobIds":["<JobId>"]}
Write-Output "$(Get-TimeStamp) Response: $OutputMessage"
$JobList = (ConvertFrom-Json -InputObject $OutputMessage).JobIds
$JobId = $JobList[0]
$OutputMessage = "JobId: $JobId"
# Get details about the currently running job, and assume the webhook job is being run in the same resourcegroup/account
$Self = Find-WhoAmI
Write-Output "Current Job '$($Self.JobId)' is running in Group '$($Self.ResourceGroupName)' and Automation Account '$($Self.AutomationAccountName)'"
Write-Output "Checking for Job '$($JobId)' in same Group and Automation Account..."
# Monitor the job status, wait for completion.
# Check against a list of statuses that likely indicate an in-progress job
$InProgressStatuses = ('New', 'Queued', 'Activating', 'Starting', 'Running', 'Stopping')
# (from https://learn.microsoft.com/en-us/powershell/module/az.automation/get-azautomationjob?view=azps-4.1.0&viewFallbackFrom=azps-3.7.0)
do {
# 1 second between polling attempts so we don't get throttled
Start-Sleep -Seconds 1
try {
$Job = Get-AzureRmAutomationJob -Id $JobId -ResourceGroupName $Self.ResourceGroupName -AutomationAccountName $Self.AutomationAccountName
} catch {
$ErrorMessage = ("An error occurred polling the job $JobId for completion - " + $_.Exception.Message)
Write-Error -Exception $_.Exception
}
Write-Output "$(Get-TimeStamp) Polled job $JobId - current status: $($Job.Status)"
} while ($InProgressStatuses.Contains($Job.Status))
# Get the job outputs to relay to Azure Data Factory
$Outputs = Get-AzureRmAutomationJobOutput -Id $JobId -Stream "Any" -ResourceGroupName $Self.ResourceGroupName -AutomationAccountName $Self.AutomationAccountName
Write-Output "$(Get-TimeStamp) Outputs from job: $($Outputs | ConvertTo-Json -Compress)"
$OutputMessage = $Outputs.Summary
Write-Output "Summary ouput message: $($OutputMessage)"
}
# Now for the entire purpose of this runbook - relay the response to the callback uri.
# Prepare the success or error response as per specifications at https://learn.microsoft.com/en-us/azure/data-factory/control-flow-webhook-activity#additional-notes
if ($ErrorMessage) {
$OutputJson = #"
{
"output": { "message": "$ErrorMessage" },
"statusCode": 500,
"error": {
"ErrorCode": "Error",
"Message": "$ErrorMessage"
}
}
"#
} else {
$OutputJson = #"
{
"output": { "message": "$OutputMessage" },
"statusCode": 200
}
"#
}
Write-Output "Prepared ADF callback body: $OutputJson"
# Post the response to the callback URL provided
$callbackResponse = Invoke-WebRequest -Uri $callbackuri -UseBasicParsing -Method POST -ContentType "application/json" -Body $OutputJson
Write-Output "Response was relayed to $callbackuri"
Write-Output ("ADF replied with the response: " + ($callbackResponse | ConvertTo-Json -Compress))
At a high-level, steps I've taken are to:
Execute the "main" Webhook - get back a "Job Id"
Get the current running job's "context" (resource group and automation account info) so that I can poll the remote job.
Poll the job until it is complete
Put together either a "success" or "error" response message in the format that Azure Data Factory expects.
Invoke the ADF callback.
For those looking, I created a secondary approach to the above solution - one that executes a Runbook (with parameters) from a Webhook, rather than a invoking a nested Webhook. This has a couple of benefits:
Parameters can be passed through to the Runbook (rather than requiring parameters to be baked-into a new Webhook.
A Runbook from another Azure Automation Account / Resource Group can be invoked.
There's no need to poll the status of the job, since the Start-AzureRmAutomationRunbook commandlet has a -Wait parameter.
Here's the code:
param
(
# Note: While "WebhookData" is the only root-level parameter (set by Azure Data Factory when it invokes this webhook)
# The user should ensure they provide (via the ADF request body) these additional properties required to invoke the runbook:
# - RunbookName
# - ResourceGroupName (TODO: Can fill this in by default if not provided)
# - AutomationAccountName (TODO: Can fill this in by default if not provided)
# - Parameters (A nested dict containing parameters to forward along)
[Parameter (Mandatory = $false)]
[object] $WebhookData
)
Import-Module -Name AzureRM.resources
Import-Module -Name AzureRM.automation
Function Get-TimeStamp {
return "[{0:yyyy-MM-dd} {0:HH:mm:ss}]" -f (Get-Date)
}
# If ADF executed this runbook via Webhook, it should have provided a WebhookData with a request body.
if (-Not $WebhookData) {
Write-Error "Runbook was not invoked with WebhookData. Args were: $args"
exit 0
}
if (-Not $WebhookData.RequestBody) {
Write-Error "WebhookData did not contain a ""RequestBody"" property. Data was: $WebhookData"
exit 0
}
$parameters = (ConvertFrom-Json -InputObject $WebhookData.RequestBody)
# And this data should contain a JSON body containing a 'callBackUri' property.
if (-Not $parameters.callBackUri) {
Write-Error 'WebhookData was missing the expected "callBackUri" property (which Azure Data Factory should provide automatically)'
exit 0
}
$callbackuri = $parameters.callBackUri
# Check for required parameters, and output any errors.
$ErrorMessage = ''
$RunbookName = $parameters.RunbookName
$ResourceGroupName = $parameters.ResourceGroupName
$AutomationAccountName = $parameters.AutomationAccountName
if (-Not $RunbookName) {
$ErrorMessage += 'WebhookData was missing the expected "RunbookName" property (which the user should have added to the body via ADF)`n'
} if (-Not $ResourceGroupName) {
$ErrorMessage += 'WebhookData was missing the expected "ResourceGroupName" property (which the user should have added to the body via ADF)`n'
} if (-Not $AutomationAccountName) {
$ErrorMessage += 'WebhookData was missing the expected "AutomationAccountName" property (which the user should have added to the body via ADF)`n'
} if ($ErrorMessage) {
Write-Error $ErrorMessage
} else {
# Set the current automation connection's authenticated account to use for future Azure Resource Manager cmdlet requests.
# TODO: Provide the user with a way to override this if the target runbook doesn't support the AzureRunAsConnection
$ServicePrincipalConnection = Get-AutomationConnection -Name "AzureRunAsConnection"
Add-AzureRmAccount -ServicePrincipal `
-TenantId $ServicePrincipalConnection.TenantId `
-ApplicationId $ServicePrincipalConnection.ApplicationId `
-CertificateThumbprint $ServicePrincipalConnection.CertificateThumbprint | Write-Verbose
Select-AzureRmSubscription -SubscriptionId $ServicePrincipalConnection.SubscriptionID | Write-Verbose
# Prepare the properties to pass on to the next runbook - all provided properties exept the ones specific to the ADF passthrough invocation
$RunbookParams = #{ }
if($parameters.parameters) {
$parameters.parameters.PSObject.Properties | Foreach { $RunbookParams[$_.Name] = $_.Value }
Write-Output "The following parameters will be forwarded to the runbook: $($RunbookParams | ConvertTo-Json -Compress)"
}
# Now invoke the actual runbook desired, and wait for it to complete
Write-Output "$(Get-TimeStamp) Invoking Runbook '$($RunbookName)' from Group '$($ResourceGroupName)' and Automation Account '$($AutomationAccountName)'"
try {
# Runbooks have this nice flag that let you wait on their completion (unlike webhook-invoked)
$Result = Start-AzureRmAutomationRunbook -Wait -Name $RunbookName -AutomationAccountName $AutomationAccountName -ResourceGroupName $ResourceGroupName –Parameters $RunbookParams
} catch {
$ErrorMessage = ("An error occurred while invoking Start-AzAutomationRunbook - " + $_.Exception.Message)
Write-Error -Exception $_.Exception
}
# Digest the result to be relayed to ADF
if($Result) {
Write-Output "$(Get-TimeStamp) Response: $($Result | ConvertTo-Json -Compress)"
$OutputMessage = $Result.ToString()
} elseif(-Not $ErrorMessage) {
$OutputMessage = "The runbook completed without errors, but the result was null."
}
}
# Now for the entire purpose of this runbook - relay the response to the callback uri.
# Prepare the success or error response as per specifications at https://learn.microsoft.com/en-us/azure/data-factory/control-flow-webhook-activity#additional-notes
if ($ErrorMessage) {
$OutputJson = #{
output = #{ message = $ErrorMessage }
statusCode = 500
error = #{
ErrorCode = "Error"
Message = $ErrorMessage
}
} | ConvertTo-Json -depth 2
} else {
$OutputJson = #{
output = #{ message = $OutputMessage }
statusCode = 200
} | ConvertTo-Json -depth 2
}
Write-Output "Prepared ADF callback body: $OutputJson"
# Post the response to the callback URL provided
$callbackResponse = Invoke-WebRequest -Uri $callbackuri -UseBasicParsing -Method POST -ContentType "application/json" -Body $OutputJson
Write-Output "Response was relayed to $callbackuri"
Write-Output ("ADF replied with the response: " + ($callbackResponse | ConvertTo-Json -Compress))

Automation Runbook not able to set default subscription on run

Cloned a sample non classic runbook and attempted to set the subscription as default. This gives me following error:
Select-AzureSubscription : The subscription name {nameofsubscription} doesn't exist.
Parameter name: name
At step1_validate:18 char:18
+
+ CategoryInfo : CloseError: (:) [Select-AzureSubscription], ArgumentException
+ FullyQualifiedErrorId : Microsoft.WindowsAzure.Commands.Profile.SelectAzureSubscriptionCommand
How would it be possible to select or set a particular subscription as default? Below approaches throw above mentioned error:
Approach 1
Select-AzureSubscription -SubscriptionName $defaultSubscriptionname –Default
Get-AzureSubscription -SubscriptionId 123XXXXXXXXXXXXXXXXXX96eXX58 | Select-AzureSubscription
Approach 2
$subscriptionId = (Get-AzureRmSubscription | Out-GridView -Title 'Select Azure Subscription:' -PassThru).Id
Select-AzureRmSubscription -SubscriptionId $subscriptionId
Trying : To get particular resource name and make configurational changes, or to use Get-Azurewebsite cmdlet, setting a subscription appears mandatory.
Note: Hoping adding Service Principal used in the automation account to default subscription might help, I also tried to add the ApplicationId to Access control of Subscription as contributor. Though my Runbook being mentioned here and WebApps are in same subscription. Reference: https://blogs.technet.microsoft.com/knightly/2017/05/26/using-azure-automation-with-multiple-subscriptions/#comment-1555 ( Scenario for me is both are in same subscription )
EDIT: Pasting Code
<#
This PowerShell script was automatically converted to PowerShell Workflow so it can be run as a runbook.
Specific changes that have been made are marked with a comment starting with “Converter:”
#>
<#
.DESCRIPTION
To watch php version old in app, and if found turn it off and apply 5.6
.NOTES
AUTHOR: HBala
LASTEDIT: Jan 04, 2018
#>
workflow step1_validate {
# Converter: Wrapping initial script in an InlineScript activity, and passing any parameters for use within the InlineScript
# Converter: If you want this InlineScript to execute on another host rather than the Automation worker, simply add some combination of -PSComputerName, -PSCredential, -PSConnectionURI, or other workflow common parameters (http://technet.microsoft.com/en-us/library/jj129719.aspx) as parameters of the InlineScript
inlineScript {
$connectionName = "AzureRunAsConnection"
$myResourceGroupName = "DevstorageRG"
$defaultSubscriptionname = "StandardDevStaging"
$newPhpVersion = "5.6"
try
{
# Get the connection "AzureRunAsConnection "
$servicePrincipalConnection=Get-AutomationConnection -Name $connectionName
"Logging in to Azure..."
Add-AzureRmAccount `
-ServicePrincipal `
-TenantId $servicePrincipalConnection.TenantId `
-ApplicationId $servicePrincipalConnection.ApplicationId `
-CertificateThumbprint $servicePrincipalConnection.CertificateThumbprint
}
catch {
if (!$servicePrincipalConnection)
{
$ErrorMessage = "Connection $connectionName not found."
throw $ErrorMessage
} else{
Write-Error -Message $_.Exception
throw $_.Exception
}
}
#Get all ARM resources from all resource groups
$ResourceGroups = Get-AzureRmResourceGroup
foreach ($ResourceGroup in $ResourceGroups)
{
Write-Output ("Showing resources in resource group " + $ResourceGroup.ResourceGroupName)
if( $ResourceGroup.ResourceGroupName -eq $myResourceGroupName ){
$Resources = Find-AzureRmResource -ResourceGroupNameContains $ResourceGroup.ResourceGroupName | Select ResourceName, ResourceType
ForEach ($Resource in $Resources)
{
Write-Output ($Resource.ResourceName + " of type " + $Resource.ResourceType)
Write-Output ($Resource.ResourceName + " of type " + $Resource.ResourceType)
#Switch-AzureMode AzureServiceManagement
# Local powershell connects and works perfect with what I wanted to achieve.
# On Automation account, Runbook migration, it threw Azure subcription not set error for Set-AzureWebsite.
# so remove rest of the code and focused on few options to get the default as below
# which led me to post the thread.
#
# Approach 1
Select-AzureSubscription -SubscriptionName $defaultSubscriptionname –Default
Get-AzureSubscription -SubscriptionId 1238XXXXXXXXXXXe5XXXX8 | Select-AzureSubscription
# Approach 2
$subscriptionId = (Get-AzureRmSubscription | Out-GridView -Title 'Select Azure Subscription:' -PassThru).Id
Select-AzureRmSubscription -SubscriptionId $subscriptionId
Write-Output ( "==============Subscription ID :===========")
Write-Output( $subscriptionId)
# Wants to update config / php version parameters.
Set-AzureWebsite -Name $Resource.ResourceName -HttpLoggingEnabled 1 -PhpVersion 5.6
# Had tried this as well which was suggested by Jason. But appears not working..
# looks like I have messed it up.
Get-AzureRmWebApp -ResourceGroupName $myResourceGroupName -Name $Resource.ResourceName
Set-AzureRmWebApp -ResourceGroupName $myResourceGroupName -Name $Resource.ResourceName -HttpLoggingEnabled 1 -PhpVersion 5.6
}
}
Write-Output ("Completed!#Line83")
}
}
}
Hoping adding the serviceprincipal type addition for the
automationaccount to default subscription might help
You are right, we can use Azure automation account connections to select subscription.
For example, we can new a connection and the type is Azure service principal.
like this:
Then use this powershell script to select the connection:
$connectionName = "jason"
try
{
# Get the connection "jason "
$servicePrincipalConnection=Get-AutomationConnection -Name $connectionName
"Logging in to Azure..."
Add-AzureRmAccount `
-ServicePrincipal `
-TenantId $servicePrincipalConnection.TenantId `
-ApplicationId $servicePrincipalConnection.ApplicationId `
-CertificateThumbprint $servicePrincipalConnection.CertificateThumbprint
}
catch {
if (!$servicePrincipalConnection)
{
$ErrorMessage = "Connection $connectionName not found."
throw $ErrorMessage
} else{
Write-Error -Message $_.Exception
throw $_.Exception
}
}
In this way, we can use connection to select subscription.
By the way, we can follow this article to create service principal.

How to give a 200 response from elasticsearch without authentication to an app gateway?

I have an issue setting up an ES cluster on Azure. I'd like my cluster to be behind an application gateway and also use shield authentication.
The problem is that the azure application gateway needs to send a health ping to the cluster and get back a 200 response, otherwise it returns a 502 "bad gateway". If I create an anonymous use then I can get the cluster to return a 200 but I'd rather not enable an anonymous user and use basic authentication instead.
Is there some endpoint on the cluster that will return a 200 even if the user is not authenticated and anonymous users are turned off?
Thanks!
There is no such endpoint in Elasticsearch. There is status.allowAnonymous in Kibana for the stats api endpoint, but nothing similar in Elasticsearch.
You'd have to define your own user that has access to a specific healthcheck url and use that or anonymous access enabled.
The healthchecks story can have variations: you check the health of a specific node (/_cluster/health?local=true), or the health of the cluster. You can also get a 200 if you send a _search request (with preference=_local) to a specific node even if that cluster doesn't have an elected master node, for example, because by default a _search operation is permitted on a node even in such situation.
In addition to #Andrei's answer, I'd recommend taking a look at Elastic's Azure ARM template which can deploy a cluster with Application Gateway configured for load balancing and SSL offload to the cluster.
This also works with X-Pack Security by setting up anonymous access for the Application Gateway ping health check that is assigned a role with access only to
cluster:
- cluster:monitor/main
It would be great if the ping check supported supplying credentials in the future, in which case anonymous access would not be required, locking things down further.
To deploy a cluster with Application Gateway using Azure PowerShell
function PromptCustom($title, $optionValues, $optionDescriptions)
{
Write-Host $title
Write-Host
$a = #()
for($i = 0; $i -lt $optionValues.Length; $i++){
Write-Host "$($i+1))" $optionDescriptions[$i]
}
Write-Host
while($true)
{
Write-Host "Choose an option: "
$option = Read-Host
$option = $option -as [int]
if($option -ge 1 -and $option -le $optionValues.Length)
{
return $optionValues[$option-1]
}
}
}
function Prompt-Subscription() {
# Choose subscription. If there's only one we will choose automatically
$subs = Get-AzureRmSubscription
$subscriptionId = ""
if($subs.Length -eq 0) {
Write-Error "No subscriptions bound to this account."
return
}
if($subs.Length -eq 1) {
$subscriptionId = $subs[0].SubscriptionId
}
else {
$subscriptionChoices = #()
$subscriptionValues = #()
foreach($subscription in $subs) {
$subscriptionChoices += "$($subscription.SubscriptionName) ($($subscription.SubscriptionId))";
$subscriptionValues += ($subscription.SubscriptionId);
}
$subscriptionId = PromptCustom "Choose a subscription" $subscriptionValues $subscriptionChoices
}
return $subscriptionId
}
$subscriptionId = "{YOUR SUBSCRIPTION ID}"
try {
Select-AzureRmSubscription -SubscriptionId $subscriptionId -ErrorAction Stop
}
catch {
Write-Host "Please Login"
Login-AzureRmAccount
$subscriptionId = Prompt-Subscription
Select-AzureRmSubscription -SubscriptionId $subscriptionId
}
# Specify the template version to use. This can be a branch name, commit hash, tag, etc.
# NOTE: different template versions may require different parameters to be passed, so be sure to check
# the parameters/password.parameters.json file in the respective tag branch
$templateVersion = "master"
$templateSrc = "https://raw.githubusercontent.com/elastic/azure-marketplace/$templateVersion/src"
$elasticTemplate = "$templateSrc/mainTemplate.json"
$location = "Australia Southeast"
$resourceGroup = "app-gateway-cluster"
$name = $resourceGroup
$cert = [System.Convert]::ToBase64String([System.IO.File]::ReadAllBytes("{PATH TO cert.pfx}"))
$clusterParameters = #{
"artifactsBaseUrl"= $templateSrc
"esVersion" = "5.1.2"
"esClusterName" = $name
# Deploy to same location as Resource Group location
"location" = "ResourceGroup"
# Install X-Pack plugins.
# Will install trial license
"xpackPlugins" = "Yes"
# Use Application Gateway
"loadBalancerType" = "gateway"
"vmDataDiskCount" = 2
"dataNodesAreMasterEligible" = "Yes"
"adminUsername" = "russ"
"authenticationType" = "password"
"adminPassword" = "{Super Secret Password}"
"securityAdminPassword" = "{Super Secret Admin User Password}"
"securityReadPassword" = "{Super Secret Read User Password}"
"securityKibanaPassword" = "{Super Secret Kibana User Password}"
"appGatewayCertBlob" = $cert
"appGatewayCertPassword" = "{Password for cert.pfx (if it has one)}"
}
Write-Host "[$(Get-Date -format 'u')] Deploying cluster"
New-AzureRmResourceGroup -Name $resourceGroup -Location $location
New-AzureRmResourceGroupDeployment -Name $name -ResourceGroupName $resourceGroup -TemplateUri $elasticTemplate -TemplateParameterObject $clusterParameters
Write-Host "[$(Get-Date -format 'u')] Deployed cluster"
Take a look at the other parameters available for the template which can be used to control other elements such as size and number of disks to attach to each data node, setting up Azure Cloud plugin for snapshot/restore, etc.

Problems Running Azure Automation Powershell to Scale Database Back After Restore Operation

I am trying to scale back a database after the restore operation has completed and am running into some problems. I am getting this exception and wonder if there is something in this script not supported by Azure Automation Workflows?
Parameter set cannot be resolved using the specified named parameters.
workflow insertflowname
{
<#
.SYNOPSIS
The purpose of this runbook is to demonstrate how to restore a particular database to a new database using an Azure Automation workflow. Then it is scaled back to Basic.
.NOTES
#>
# Specify Azure Subscription Name
$subName = 'insertsubscription name'
# Connect to Azure Subscription
Connect-Azure -AzureConnectionName $subName
Select-AzureSubscription -SubscriptionName $subName
# Define source databasename
$SourceDatabaseName = 'insert database name'
# Define source server
$SourceServerName = 'insert source server'
# Define destination server
$TargetServerName = 'insert destination server'
Write-Output "`$SourceServerName [$SourceServerName]"
Write-Output "`$TargetServerName [$TargetServerName]"
Write-Output "`$SourceDatabaseName [$SourceDatabaseName]"
Write-Output "Retrieving recoverable database details for database [$SourceDatabaseName] on server [$SourceServerName]."
$RecoverableDatabase = Get-AzureSqlRecoverableDatabase –ServerName $SourceServerName -DatabaseName $SourceDatabaseName
$TargetDatabaseName = "$SourceDatabaseName-$($RecoverableDatabase.LastAvailableBackupDate.ToString('O'))"
Write-Output "`$TargetDatabaseName [$TargetDatabaseName]"
Write-Output "Starting recovery of database [$SourceDatabaseName] to server [$TargetServerName] as database [$TargetDatabaseName]."
Start-AzureSqlDatabaseRecovery -SourceDatabase $RecoverableDatabase -TargetServerName $TargetServerName –TargetDatabaseName $TargetDatabaseName
$PollingInterval = 10
Write-Output "Monitoring status of recovery operation, polling every [$PollingInterval] second(s)."
$KeepGoing = $true
while ($KeepGoing) {
$operation = Get-AzureSqlDatabaseOperation -ServerName $TargetServerName -DatabaseName $TargetDatabaseName | Where-Object {$_.Name -eq "DATABASE RECOVERY"} | Sort-Object StartTime -Descending
if ($operation) {
$operation[0]
if ($operation[0].State -eq "COMPLETED") { $KeepGoing = $false }
if ($operation[0].State -eq "FAILED") {
# Throw error
$KeepGoing = $false
}
} else {
# Throw error since something went wrong and object was not created
# May want to have this retry a few times before giving up or at least notify somebody
# since at this point the recovery has been kicked off and you don't want the database
# restore to remain at the elevated service level.
$KeepGoing = $false
}
if ($KeepGoing) { Start-Sleep -Seconds $PollingInterval }
}
if ($operation[0].State -eq "COMPLETED") {
Write-Output "Setting service level for database [$TargetDatabaseName] on server [$TargetServerName] to Basic."
$ServiceObjective = Get-AzureSQLDatabaseServiceObjective –ServerName $TargetServerName –ServiceObjectiveName "Basic"
$ServiceObjective
Set-AzureSqlDatabase –ServerName $TargetServerName –DatabaseName $TargetDatabaseName –Edition "Basic" –ServiceObjective $ServiceObjective -MaxSizeGB 2 –Force
}
}
You are probably hitting the issue described here: https://social.msdn.microsoft.com/Forums/en-US/ce6412b8-5cce-4573-befb-6017924ce0d0/whereobject-fails-with-parameter-set-cannot-be-resolved-using-the-specified-named-parameters?forum=azureautomation
Summary:
Use parameter names, don't rely on positional parameters, in PowerShell Workflow. In this case, you need to add the -FilterScript parameter name to Where-Object.

Resources