Is there option to auto terminate Azure SQL DW - azure

I am using Azure SQL DW which costs more per an hour. So I want to know is there option for auto terminate SQL DW after an hour or so?

You can pause the Azure Data warehouse and then you only pay for the storage used.
You can automate pausing your DWH by using an Azure automation account and a runbook.
This blog explains the process:
https://blogs.msdn.microsoft.com/allanmiller/2017/09/20/pausing-azure-sql-data-warehouse-using-an-automation-runbook/
Markus

Yes, it is possible.
To save costs, you can pause and resume compute resources on-demand. For example, if you won't be using the database during the night and on weekends, you can pause it during those times, and resume it during the day. You won't be charged for DWUs while the database is paused.
When you pause a database:
Compute and memory resources are returned to the pool of available resources in the data center
DWU costs are zero for the duration of the pause.
Data storage is not affected and your data stays intact.
SQL Data Warehouse cancels all running or queued operations.
To pause a database, use the Suspend-AzureRmSqlDatabase cmdlet.
Suspend-AzureRmSqlDatabase –ResourceGroupName "ResourceGroup1" `
–ServerName "Server01" –DatabaseName "Database02"
To start a database, use the Resume-AzureRmSqlDatabase cmdlet.
Resume-AzureRmSqlDatabase –ResourceGroupName "ResourceGroup1" `
–ServerName "Server01" -DatabaseName "Database02"
More information please refer to this official document.
As Markus Bohse said, you also could use Automation to do this.
Note: You also could write a runbook to start your database.
Update: If you want to use java to do this, please refer to this API document.
public void pauseDataWarehouse()
Update:
You also could Rest API to do this. See this link
POST https://management.azure.com/subscriptions/{subscription-id}/resourceGroups/{resource-group-name}/providers/Microsoft.Sql/servers/{server-name}/databases/{database-name}/pause?api-version=2014-04-01-preview HTTP/1.1

You can use runbooks or powershell script to pause dn resume SQL DWH.
Powershell script is below. If you would like "runbook" let me know
[CmdletBinding(DefaultParametersetName='None')]
Param
(
[Parameter(Mandatory=$true)][ValidateNotNullOrEmpty()]
[String]
$AzureSubscriptionId,
[Parameter(Mandatory=$true)][ValidateNotNullOrEmpty()]
[String]
$AzureDataWareHouseList="All",
[Parameter(Mandatory=$true)][ValidateSet("Suspend","Resume")]
[String]
$Action
)
function PauseAzureDWH
{
Param
(
[Parameter(Mandatory=$true)][ValidateNotNullOrEmpty()]
[String]
$AzureSubscriptionId,
[Parameter(Mandatory=$true)][ValidateNotNullOrEmpty()]
[String]
$AzureDataWareHouseList="All",
[Parameter(Mandatory=$true)][ValidateSet("Suspend","Resume")]
[String]
$Action
)
try
{
Login-AzureRmAccount
Select-AzureRmSubscription -SubscriptionId $AzureSubscriptionId
if($AzureDataWareHouseList -ne "All")
{
$AzureDWHList = #()
$AzureDWHTotalList = $AzureDataWareHouseList.Split(",")
foreach($DWHitem in $AzureDWHTotalList)
{
$DWH = "*$DWHitem*"
$DWH = Get-AzureRmResource | Where-Object ResourceName -like $DWH
if($DWH -ne $Null)
{
$dwc = $DWH.ResourceName.split("/")
# splat reused parameter lists
$ThisDW = #{
'ResourceGroupName' = $DWH.ResourceGroupName
'ServerName' = $dwc[0]
'DatabaseName' = $dwc[1]
}
$AzureDWHList += $ThisDW
}
else
{
Write-Warning "Given DataWarehouse '$DWHitem' is not found in given subscription"
}
}
}
else
{
[array]$TotalDataWareHouseList = Get-AzureRmResource | Where-Object ResourceType -EQ "Microsoft.Sql/servers/databases" | Where-Object Kind -Like "*datawarehouse*"
$AzureDWHList = #()
foreach($DWH in $TotalDataWareHouseList)
{
$dwc = $DWH.ResourceName.split("/")
$ThisDW = #{
'ResourceGroupName' = $DWH.ResourceGroupName
'ServerName' = $dwc[0]
'DatabaseName' = $dwc[1]
}
$AzureDWHList += $ThisDW
}
}
<# foreach($AzureDWHItem in $AzureDWHList)
{
if(!(Get-AzureRmResource | ? {$_.Name -eq $AzureDWHItem.ServerName}) )
{
throw " AzureDWH : [$AzureDWHItem] - Does not exist! - please Check your inputs "
}
} #>
if($Action -eq "Suspend")
{
Write-Output "Suspending Azure DataWareHouses";
foreach ($AzureDWH in $AzureDWHList)
{
$status = Get-AzureRmSqlDatabase #AzureDWH | Select Status
if($status.Status -eq "Online")
{
Suspend-AzureRmSqlDatabase #AzureDWH
}
}
}
else
{
Write-Output "Resuming Azure DataWareHouses";
foreach ($AzureDWH in $AzureDWHList)
{
$status = Get-AzureRmSqlDatabase #AzureDWH | Select Status
if($status.Status -eq "Paused")
{
Resume-AzureRmSqlDatabase #AzureDWH
}
}
}
}
catch
{
Write-Error " Exception while getting resource details and writing back to CSV"
Write-Error $_.Exception.message
Write-Error " ErrorStack: $Error[0] "
exit 1
}
}
PauseAzureDWH -AzureSubscriptionId $AzureSubscriptionId -AzureDataWareHouseList $AzureDataWareHouseList -Action $Action

Related

Azure Data Factory webhook execution times out instead of relaying errors

I've attempted to set up a simple webhook execution in Azure Data Factory (v2), calling a simple (parameter-less) webhook for an Azure Automation Runbook I set up.
From the Azure Portal, I can see that the webhook is being executed and my runbook is being run, so far so good. The runbook is (currently) returning an error within 1 minute of execution - but that's fine, I also want to test failure scenarios.
Problem:
Data Factory doesn't seem to be 'seeing' the error result and spins until the timeout (10 minutes) elapses. When I kick off a debug run of the pipeline, I get the same - a timeout and no error result.
Update: I've fixed the runbook and it's now completing successfully, but Data Factory is still timing out and is not seeing the success response either.
Here is a screenshot of the setup:
And here is the portal confirming that the webhook is being run by azure data factory, and is completing in under a minute:
WEBHOOKDATA JSON is:
{"WebhookName":"Start CAMS VM","RequestBody":"{\r\n \"callBackUri\": \"https://dpeastus.svc.datafactory.azure.com/dataplane/workflow/callback/f7c...df2?callbackUrl=AAEAFF...0927&shouldReportToMonitoring=True&activityType=WebHook\"\r\n}","RequestHeader":{"Connection":"Keep-Alive","Expect":"100-continue","Host":"eab...ddc.webhook.eus2.azure-automation.net","x-ms-request-id":"7b4...2eb"}}
So as far as I can tell, things should be in place to pick up on the result (success of failure). Hopefully someone who's done this before knows what I'm missing.
Thanks!
I had assumed that Azure would automatically notify the ADF "callBackUri" with the result once the runbook completed or errored out (since they take care of 99% of the scaffolding without requiring a line of code).
It turns out that is not the case, and anyone wishing to execute a runbook from ADF will have to manually extract the callBackUri from the Webhookdata input parameter, and POST the result to it when done.
I haven't nailed this down yet, since the Microsoft tutorial sites I've found have a bad habit of taking screenshots of the code that does this rather than providing the code itself:
I guess I'll come back and edit this once I have it figured out.
EDIT I ended up implementing this by leaving my original Webhook untouched, and creating a "wrapper"/helper/utility Runbook that will execute an arbitrary webhook, and relay its status to ADF once it's complete.
Here's the full code I ended up with, in case it helps someone else. It's meant to be generic:
Setup / Helper Functions
param
(
[Parameter (Mandatory = $false)]
[object] $WebhookData
)
Import-Module -Name AzureRM.resources
Import-Module -Name AzureRM.automation
# Helper function for getting the current running Automation Account Job
# Inspired heavily by: https://github.com/azureautomation/runbooks/blob/master/Utility/ARM/Find-WhoAmI
<#
Queries the automation accounts in the subscription to find the automation account, runbook and resource group that the job is running in.
AUTHOR: Azure/OMS Automation Team
#>
Function Find-WhoAmI {
[CmdletBinding()]
Param()
Begin { Write-Verbose ("Entering {0}." -f $MyInvocation.MyCommand) }
Process {
# Authenticate
$ServicePrincipalConnection = Get-AutomationConnection -Name "AzureRunAsConnection"
Add-AzureRmAccount `
-ServicePrincipal `
-TenantId $ServicePrincipalConnection.TenantId `
-ApplicationId $ServicePrincipalConnection.ApplicationId `
-CertificateThumbprint $ServicePrincipalConnection.CertificateThumbprint | Write-Verbose
Select-AzureRmSubscription -SubscriptionId $ServicePrincipalConnection.SubscriptionID | Write-Verbose
# Search all accessible automation accounts for the current job
$AutomationResource = Get-AzureRmResource -ResourceType Microsoft.Automation/AutomationAccounts
$SelfId = $PSPrivateMetadata.JobId.Guid
foreach ($Automation in $AutomationResource) {
$Job = Get-AzureRmAutomationJob -ResourceGroupName $Automation.ResourceGroupName -AutomationAccountName $Automation.Name -Id $SelfId -ErrorAction SilentlyContinue
if (!([string]::IsNullOrEmpty($Job))) {
return $Job
}
Write-Error "Could not find the current running job with id $SelfId"
}
}
End { Write-Verbose ("Exiting {0}." -f $MyInvocation.MyCommand) }
}
Function Get-TimeStamp {
return "[{0:yyyy-MM-dd} {0:HH:mm:ss}]" -f (Get-Date)
}
My Code
### EXPECTED USAGE ###
# 1. Set up a webhook invocation in Azure data factory with a link to this Runbook's webhook
# 2. In ADF - ensure the body contains { "WrappedWebhook": "<your url here>" }
# This should be the URL for another webhook.
# LIMITATIONS:
# - Currently, relaying parameters and authentication credentials is not supported,
# so the wrapped webhook should require no additional authentication or parameters.
# - Currently, the callback to Azure data factory does not support authentication,
# so ensure ADF is configured to require no authentication for its callback URL (the default behaviour)
# If ADF executed this runbook via Webhook, it should have provided a WebhookData with a request body.
if (-Not $WebhookData) {
Write-Error "Runbook was not invoked with WebhookData. Args were: $args"
exit 0
}
if (-Not $WebhookData.RequestBody) {
Write-Error "WebhookData did not contain a ""RequestBody"" property. Data was: $WebhookData"
exit 0
}
$parameters = (ConvertFrom-Json -InputObject $WebhookData.RequestBody)
# And this data should contain a JSON body containing a 'callBackUri' property.
if (-Not $parameters.callBackUri) {
Write-Error 'WebhookData was missing the expected "callBackUri" property (which Azure Data Factory should provide automatically)'
exit 0
}
$callbackuri = $parameters.callBackUri
# Check for the "WRAPPEDWEBHOOK" parameter (which should be set up by the user in ADF)
$WrappedWebhook = $parameters.WRAPPEDWEBHOOK
if (-Not $WrappedWebhook) {
$ErrorMessage = 'WebhookData was missing the expected "WRAPPEDWEBHOOK" peoperty (which the user should have added to the body via ADF)'
Write-Error $ErrorMessage
}
else
{
# Now invoke the actual runbook desired
Write-Output "$(Get-TimeStamp) Invoking Webhook Request at: $WrappedWebhook"
try {
$OutputMessage = Invoke-WebRequest -Uri $WrappedWebhook -UseBasicParsing -Method POST
} catch {
$ErrorMessage = ("An error occurred while executing the wrapped webhook $WrappedWebhook - " + $_.Exception.Message)
Write-Error -Exception $_.Exception
}
# Output should be something like: {"JobIds":["<JobId>"]}
Write-Output "$(Get-TimeStamp) Response: $OutputMessage"
$JobList = (ConvertFrom-Json -InputObject $OutputMessage).JobIds
$JobId = $JobList[0]
$OutputMessage = "JobId: $JobId"
# Get details about the currently running job, and assume the webhook job is being run in the same resourcegroup/account
$Self = Find-WhoAmI
Write-Output "Current Job '$($Self.JobId)' is running in Group '$($Self.ResourceGroupName)' and Automation Account '$($Self.AutomationAccountName)'"
Write-Output "Checking for Job '$($JobId)' in same Group and Automation Account..."
# Monitor the job status, wait for completion.
# Check against a list of statuses that likely indicate an in-progress job
$InProgressStatuses = ('New', 'Queued', 'Activating', 'Starting', 'Running', 'Stopping')
# (from https://learn.microsoft.com/en-us/powershell/module/az.automation/get-azautomationjob?view=azps-4.1.0&viewFallbackFrom=azps-3.7.0)
do {
# 1 second between polling attempts so we don't get throttled
Start-Sleep -Seconds 1
try {
$Job = Get-AzureRmAutomationJob -Id $JobId -ResourceGroupName $Self.ResourceGroupName -AutomationAccountName $Self.AutomationAccountName
} catch {
$ErrorMessage = ("An error occurred polling the job $JobId for completion - " + $_.Exception.Message)
Write-Error -Exception $_.Exception
}
Write-Output "$(Get-TimeStamp) Polled job $JobId - current status: $($Job.Status)"
} while ($InProgressStatuses.Contains($Job.Status))
# Get the job outputs to relay to Azure Data Factory
$Outputs = Get-AzureRmAutomationJobOutput -Id $JobId -Stream "Any" -ResourceGroupName $Self.ResourceGroupName -AutomationAccountName $Self.AutomationAccountName
Write-Output "$(Get-TimeStamp) Outputs from job: $($Outputs | ConvertTo-Json -Compress)"
$OutputMessage = $Outputs.Summary
Write-Output "Summary ouput message: $($OutputMessage)"
}
# Now for the entire purpose of this runbook - relay the response to the callback uri.
# Prepare the success or error response as per specifications at https://learn.microsoft.com/en-us/azure/data-factory/control-flow-webhook-activity#additional-notes
if ($ErrorMessage) {
$OutputJson = #"
{
"output": { "message": "$ErrorMessage" },
"statusCode": 500,
"error": {
"ErrorCode": "Error",
"Message": "$ErrorMessage"
}
}
"#
} else {
$OutputJson = #"
{
"output": { "message": "$OutputMessage" },
"statusCode": 200
}
"#
}
Write-Output "Prepared ADF callback body: $OutputJson"
# Post the response to the callback URL provided
$callbackResponse = Invoke-WebRequest -Uri $callbackuri -UseBasicParsing -Method POST -ContentType "application/json" -Body $OutputJson
Write-Output "Response was relayed to $callbackuri"
Write-Output ("ADF replied with the response: " + ($callbackResponse | ConvertTo-Json -Compress))
At a high-level, steps I've taken are to:
Execute the "main" Webhook - get back a "Job Id"
Get the current running job's "context" (resource group and automation account info) so that I can poll the remote job.
Poll the job until it is complete
Put together either a "success" or "error" response message in the format that Azure Data Factory expects.
Invoke the ADF callback.
For those looking, I created a secondary approach to the above solution - one that executes a Runbook (with parameters) from a Webhook, rather than a invoking a nested Webhook. This has a couple of benefits:
Parameters can be passed through to the Runbook (rather than requiring parameters to be baked-into a new Webhook.
A Runbook from another Azure Automation Account / Resource Group can be invoked.
There's no need to poll the status of the job, since the Start-AzureRmAutomationRunbook commandlet has a -Wait parameter.
Here's the code:
param
(
# Note: While "WebhookData" is the only root-level parameter (set by Azure Data Factory when it invokes this webhook)
# The user should ensure they provide (via the ADF request body) these additional properties required to invoke the runbook:
# - RunbookName
# - ResourceGroupName (TODO: Can fill this in by default if not provided)
# - AutomationAccountName (TODO: Can fill this in by default if not provided)
# - Parameters (A nested dict containing parameters to forward along)
[Parameter (Mandatory = $false)]
[object] $WebhookData
)
Import-Module -Name AzureRM.resources
Import-Module -Name AzureRM.automation
Function Get-TimeStamp {
return "[{0:yyyy-MM-dd} {0:HH:mm:ss}]" -f (Get-Date)
}
# If ADF executed this runbook via Webhook, it should have provided a WebhookData with a request body.
if (-Not $WebhookData) {
Write-Error "Runbook was not invoked with WebhookData. Args were: $args"
exit 0
}
if (-Not $WebhookData.RequestBody) {
Write-Error "WebhookData did not contain a ""RequestBody"" property. Data was: $WebhookData"
exit 0
}
$parameters = (ConvertFrom-Json -InputObject $WebhookData.RequestBody)
# And this data should contain a JSON body containing a 'callBackUri' property.
if (-Not $parameters.callBackUri) {
Write-Error 'WebhookData was missing the expected "callBackUri" property (which Azure Data Factory should provide automatically)'
exit 0
}
$callbackuri = $parameters.callBackUri
# Check for required parameters, and output any errors.
$ErrorMessage = ''
$RunbookName = $parameters.RunbookName
$ResourceGroupName = $parameters.ResourceGroupName
$AutomationAccountName = $parameters.AutomationAccountName
if (-Not $RunbookName) {
$ErrorMessage += 'WebhookData was missing the expected "RunbookName" property (which the user should have added to the body via ADF)`n'
} if (-Not $ResourceGroupName) {
$ErrorMessage += 'WebhookData was missing the expected "ResourceGroupName" property (which the user should have added to the body via ADF)`n'
} if (-Not $AutomationAccountName) {
$ErrorMessage += 'WebhookData was missing the expected "AutomationAccountName" property (which the user should have added to the body via ADF)`n'
} if ($ErrorMessage) {
Write-Error $ErrorMessage
} else {
# Set the current automation connection's authenticated account to use for future Azure Resource Manager cmdlet requests.
# TODO: Provide the user with a way to override this if the target runbook doesn't support the AzureRunAsConnection
$ServicePrincipalConnection = Get-AutomationConnection -Name "AzureRunAsConnection"
Add-AzureRmAccount -ServicePrincipal `
-TenantId $ServicePrincipalConnection.TenantId `
-ApplicationId $ServicePrincipalConnection.ApplicationId `
-CertificateThumbprint $ServicePrincipalConnection.CertificateThumbprint | Write-Verbose
Select-AzureRmSubscription -SubscriptionId $ServicePrincipalConnection.SubscriptionID | Write-Verbose
# Prepare the properties to pass on to the next runbook - all provided properties exept the ones specific to the ADF passthrough invocation
$RunbookParams = #{ }
if($parameters.parameters) {
$parameters.parameters.PSObject.Properties | Foreach { $RunbookParams[$_.Name] = $_.Value }
Write-Output "The following parameters will be forwarded to the runbook: $($RunbookParams | ConvertTo-Json -Compress)"
}
# Now invoke the actual runbook desired, and wait for it to complete
Write-Output "$(Get-TimeStamp) Invoking Runbook '$($RunbookName)' from Group '$($ResourceGroupName)' and Automation Account '$($AutomationAccountName)'"
try {
# Runbooks have this nice flag that let you wait on their completion (unlike webhook-invoked)
$Result = Start-AzureRmAutomationRunbook -Wait -Name $RunbookName -AutomationAccountName $AutomationAccountName -ResourceGroupName $ResourceGroupName –Parameters $RunbookParams
} catch {
$ErrorMessage = ("An error occurred while invoking Start-AzAutomationRunbook - " + $_.Exception.Message)
Write-Error -Exception $_.Exception
}
# Digest the result to be relayed to ADF
if($Result) {
Write-Output "$(Get-TimeStamp) Response: $($Result | ConvertTo-Json -Compress)"
$OutputMessage = $Result.ToString()
} elseif(-Not $ErrorMessage) {
$OutputMessage = "The runbook completed without errors, but the result was null."
}
}
# Now for the entire purpose of this runbook - relay the response to the callback uri.
# Prepare the success or error response as per specifications at https://learn.microsoft.com/en-us/azure/data-factory/control-flow-webhook-activity#additional-notes
if ($ErrorMessage) {
$OutputJson = #{
output = #{ message = $ErrorMessage }
statusCode = 500
error = #{
ErrorCode = "Error"
Message = $ErrorMessage
}
} | ConvertTo-Json -depth 2
} else {
$OutputJson = #{
output = #{ message = $OutputMessage }
statusCode = 200
} | ConvertTo-Json -depth 2
}
Write-Output "Prepared ADF callback body: $OutputJson"
# Post the response to the callback URL provided
$callbackResponse = Invoke-WebRequest -Uri $callbackuri -UseBasicParsing -Method POST -ContentType "application/json" -Body $OutputJson
Write-Output "Response was relayed to $callbackuri"
Write-Output ("ADF replied with the response: " + ($callbackResponse | ConvertTo-Json -Compress))

How to delete the ADFPipeline which is having the references Forcefully

I'm actually some automation for my ADF. As a part of that, I'm trying to delete all the ADF V2 pipelines. The problem is my pipelines having many references with different pipelines itself.
$ADFPipeline = Get-AzDataFactoryV2Pipeline -DataFactoryName $(datafactory-name) -ResourceGroupName $(rg)
$ADFPipeline | ForEach-Object { Remove-AzDataFactoryV2Pipeline -ResourceGroupName $(rg) -DataFactoryName $(datafactory-name) -Name $_.name -Force }
And most of the time I get the error like
The document cannot be deleted since it is referenced by "blabla"
I understand the error that it saying some references and cannot be deleted. However, when I tried the same deletion in the azure portal, irrespective of the reference I can able to delete. So I want to find a way that whether it possible to tell that Powershell even though it's having a reference delete it forcefully
Any other inputs much appreciated!
I run into the same issue, found out that it's rather complicated to build the whole dependency graph out of the pipeline's Activities property.
As a working solution (powershell):
function Remove-Pipelines {
param (
[Parameter(Mandatory=$true)]
[AllowEmptyCollection()]
[AllowNull()]
[System.Collections.ArrayList]$pipelines
)
if($pipelines.Count -gt 0) {
[System.Collections.ArrayList]$plsToProcess = New-Object System.Collections.ArrayList($null)
foreach ($pipeline in $pipelines) {
try {
$removeAzDFCommand = "Remove-AzDataFactoryV2Pipeline -dataFactoryName '$DataFactoryName' -resourceGroupName '$ResourceGroupName' -Name '$($pipeline.Name)' -Force -ErrorAction Stop"
Write-Host $removeAzDFCommand
Invoke-Expression $removeAzDFCommand
}
catch {
if ($_ -match '.*The document cannot be deleted since it is referenced by.*') {
Write-Host $_
$plsToProcess.Add($pipeline)
} else {
throw $_
}
}
}
Remove-Pipelines $plsToProcess
}
}
Here is the complete solution for clearing the whole DF: "trigger","pipeline","dataflow","dataset","linkedService"
Param(
[Parameter(Mandatory=$true)][string] $ResourceGroupName,
[Parameter(Mandatory=$true)][string] $DataFactoryName
)
$artfTypes = "trigger","pipeline","dataflow","dataset","linkedService"
function Remove-Artifacts {
param (
[Parameter(Mandatory=$true)][AllowEmptyCollection()][AllowNull()][System.Collections.ArrayList]$artifacts,
[Parameter(Mandatory=$true)][string]$artfType
)
if($artifacts.Count -gt 0) {
[System.Collections.ArrayList]$artToProcess = New-Object System.Collections.ArrayList($null)
foreach ($artifact in $artifacts) {
try {
$removeAzDFCommand = "Remove-AzDataFactoryV2$($artfType) -dataFactoryName '$DataFactoryName' -resourceGroupName '$ResourceGroupName' -Name '$($artifact.Name)' -Force -ErrorAction Stop"
Write-Host $removeAzDFCommand
Invoke-Expression $removeAzDFCommand
}
catch {
if ($_ -match '.*The document cannot be deleted since it is referenced by.*') {
Write-Host $_
$artToProcess.Add($artifact)
} else {
throw $_
}
}
}
Remove-Artifacts $artToProcess $artfType
}
}
foreach ($artfType in $artfTypes) {
$getAzDFCommand = "Get-AzDataFactoryV2$($artfType) -dataFactoryName '$DataFactoryName' -resourceGroupName '$ResourceGroupName'"
Write-Output $getAzDFCommand
$artifacts = Invoke-Expression $getAzDFCommand
Write-Output $artifacts.Name
Remove-Artifacts $artifacts $artfType
}
The same approach can be adapted for "Set-AzDataFactoryV2Pipeline" command as well.
It worth to mention that along with dependencies tracking, Remove/Set artifact's sequence should be right (because of cross artifacts' dependencies).
For Set - "linkedService","dataset","dataflow","pipeline","trigger"
For Remove - "trigger","pipeline","dataflow","dataset","linkedService"
Hello and thank you for the question. According to the Remove-AzDataFactoryV2Pipeline doc, the -Force flag simply skips the confirmation prompt. It does not actually 'Force' the deletion in spite of errors.
Since you are already doing automation, might I suggest leveraging the error message to recursively attempt to delete the referencing pipeline. $error[0] gets the most recent error.
(Pseudocode)
try_recurse_delete( pipeline_name )
do_delete(pipeline_name)
if not $error[0].contains("referenced by " + pipeline_name)
then return true
else
try_recurse_delete( get_refrencer_name($error[0]) )
Given that pipeline dependencies can be a many-to-many relationship, subsequent pipelines in your for-each loop might already be deleted by the recursion. You will have to adapt your code to react to 'pipeline not found' type errors.

Azure Powershell Tagging VM's from CSV file

I'm quite new to Powershell scripting. I'm trying to generate tags for azure vm's from a CSV file.
Inside the CSV I have the following column headings:
VMName
Application
SubCat
Environment
AppOwner
Location
I've got a test CSV which literally has the following data in it:
MattTestVM, TestApp, TestSubapp, Dev, Matt, UK South
I'm not sure what i've put wrong in my code to get it to add the tags.
Code
#Set Credentials
$cred = Get-credential
# Sign-in with Azure account credentials
add-azurermaccount -credential $cred
# Select Azure Subscription
$subscriptionId = (Get-AzureRmSubscription | Out-GridView -Title "Select an Azure Subscription ..." -PassThru).SubscriptionId
#Select specified subscription ID
Select-AzureRmSubscription -SubscriptionId $subscriptionId
$InputCSVFilePath = "C:\test\Tagging1.csv"
#Start loop
foreach ($eachRecord in $InputCSVFilePath)
{
$VMName = $eachrecord.VMName
$Application = $eachrecord.Application
$SubCat = $eachrecord.SubCat
$Environment = $eachrecord.Environment
$AppOwner = $eachrecord.AppOwner
$Location = $eachrecord.Location
$r = Get-AzureRmResource -ResourceId $ResourceId -ErrorAction Continue
if($r -ne $null)
{
if($r.tags)
{
# Tag - Application
if($r.Tags.ContainsKey("Application"))
{
$r.Tags["Application"] = $Application
}
else
{
$r.Tags.Add("Application", $Application)
}
# Tag - SubCat
if($r.Tags.ContainsKey("subCat"))
{
$r.Tags["subCat"] = $subCat
}
else
{
$r.Tags.Add("subCat", $subCat)
}
# Tag - Environment
if($r.Tags.ContainsKey("Environment"))
{
$r.Tags["Environment"] = $Environment
}
else
{
$r.Tags.Add("Environment", $Environment)
}
# Tag - AppOwner
if($r.Tags.ContainsKey("AppOwner"))
{
$r.Tags["AppOwner"] = $AppOwner
}
else
{
$r.Tags.Add("AppOwner", $AppOwner)
}
# Tag - Location
if($r.Tags.ContainsKey("Location"))
{
$r.Tags["Location"] = $Location
}
else
{
$r.Tags.Add("Location", $Location)
}
#Setting the tags on the resource
Set-AzureRmResource -Tag $r.Tags -ResourceId $r.ResourceId -Force
}
else
{
#Setting the tags on a resource which doesn't have tags
Set-AzureRmResource -Tag #{ Application=$Application; subCat=$subCat; Environment=$Environment; AppOwner=$AppOwner; Location=$Location } -ResourceId $r.ResourceId -Force
}
}
else
{
Write-Host "Resource Not Found with Resource Id: " + $ResourceId
}
}
Error message
Get-AzureRmResource : Cannot validate argument on parameter 'ResourceId'. The argument is null or empty. Provide an argument that is not null or empty, and then try the command again.
At line:10 char:43
+ $r = Get-AzureRmResource -ResourceId $ResourceId -ErrorAction Co ...
+ ~~~~~~~~~~~
+ CategoryInfo : InvalidData: (:) [Get-AzureRmResource], ParameterBindingValidationException
+ FullyQualifiedErrorId : ParameterArgumentValidationError,Microsoft.Azure.Commands.ResourceManager.Cmdlets.Implementation.GetAzureResourceCmdlet
OK, first thing's first: You can't do this without the ResourceGroupName, and the best way to add it in your case is as a column in your csv.
Secondly, everybody starts somewhere so never apologize. :) I commented about most of the issues I fixed in the code below. Your biggest problems were a) you didn't actually load the data from the CSV (you only created a variable with the file path in it), and b) you were attempting to use $ResourceID without ever populating it.
I cleaned things up a bit and it works now, and will throw an error if it can't find the VM. Here are the contents of my test.csv, with one good VM and one that doesn't exist.
VMName,ResourceGroup,Application,SubCat,Environment,AppOwner,Location
test-rm-01,test-rg,Thing,Subs,Dev,Nick,Europe
clambake,mygroup,,,,,
And here's the re-worked script.
# Set Credentials (Fixed this up)
$context = Get-AzureRmContext
if ($context.Account -eq $null) {
Login-AzureRmAccount
}
# Select Azure Subscription
$subscriptionId = (Get-AzureRmSubscription | Out-GridView -Title "Select an Azure Subscription ..." -PassThru).SubscriptionId
#Select specified subscription ID
Select-AzureRmSubscription -SubscriptionId $subscriptionId
$InputCSVFilePath = "C:\Users\Nick\Desktop\test.csv"
################
#
# Here you were missing loading the actual data. Also renamed the items in the loop, nitpicking. :)
#
################
$csvItems = Import-Csv $InputCSVFilePath
################
#Start loop
foreach ($item in $csvItems)
{
################
#
# Change here, you need ResourceGroupName
# Also cleared r because if you don't, it will still have a value if no matching VM is found, which would re-tag the previous item from the loop, instead of throwing an error that the VM was not found
#
################
Clear-Variable r
$r = Get-AzureRmResource -ResourceGroupName $item.ResourceGroup -Name $item.VMName -ErrorAction Continue
################
if ($r -ne $null)
{
if ($r.Tags)
{
# Tag - Application
if ($r.Tags.ContainsKey("Application"))
{
$r.Tags["Application"] = $item.Application
}
else
{
$r.Tags.Add("Application", $item.Application)
}
# Tag - SubCat
if ($r.Tags.ContainsKey("subCat"))
{
$r.Tags["subCat"] = $item.subCat
}
else
{
$r.Tags.Add("subCat", $item.subCat)
}
# Tag - Environment
if ($r.Tags.ContainsKey("Environment"))
{
$r.Tags["Environment"] = $item.Environment
}
else
{
$r.Tags.Add("Environment", $item.Environment)
}
# Tag - AppOwner
if ($r.Tags.ContainsKey("AppOwner"))
{
$r.Tags["AppOwner"] = $item.AppOwner
}
else
{
$r.Tags.Add("AppOwner", $item.AppOwner)
}
# Tag - Location
if ($r.Tags.ContainsKey("Location"))
{
$r.Tags["Location"] = $item.Location
}
else
{
$r.Tags.Add("Location", $item.Location)
}
#Setting the tags on the resource
Set-AzureRmResource -Tag $r.Tags -ResourceId $r.ResourceId -Force
}
else
{
#Setting the tags on a resource which doesn't have tags
Set-AzureRmResource -Tag #{ Application = $Application; subCat = $subCat; Environment = $Environment; AppOwner = $AppOwner; Location = $Location } -ResourceId $r.ResourceId -Force
}
}
else
{
Write-Host "No VM found named $($item.VMName) in ResourceGroup $($item.ResourceGroup)!"
}
}

Azure Automation RunBook from Another RunBook

I have a runbook name "RB_ConnectSQL"
workflow RB_ConnectSQL
{
[OutputType([string])]
param
(
[Parameter(Mandatory=$true)] [string] $SqlServer,
[Parameter(Mandatory=$false)] [int] $SqlServerPort = 1433,
[Parameter(Mandatory=$true)] [string] $Database,
[Parameter(Mandatory=$true)] [string] $Procedure,
[Parameter(Mandatory=$true)] [string] $SqlCredentialName
)
$SqlCredential = Get-AutomationPSCredential -Name $SqlCredentialName
$SqlUsername = $SqlCredential.UserName
$SqlPass = $SqlCredential.GetNetworkCredential().Password
inlinescript
{
$haveError = 0
$DatabaseConnection = New-Object System.Data.SqlClient.SqlConnection("Server=tcp:$using:SqlServer,$using:SqlServerPort; Database=$using:Database; User ID=$using:SqlUsername;Password=$using:SqlPass; Trusted_Connection=False; Encrypt=True; Connect Timeout=7200;")
$outputDataTable = New-Object System.Data.DataTable
[string[]] $ColumnNames
try
{
$DatabaseConnection.Open()
$Cmd=new-object system.Data.SqlClient.SqlCommand
$Cmd.Connection = $DatabaseConnection
$Cmd.CommandText = 'EXEC ' + $using:Procedure + ';'
$Cmd.CommandTimeout = 7200
$sqlDataAdapter = New-Object System.Data.SqlClient.SqlDataAdapter $Cmd
$dataSet = New-Object System.Data.DataSet
Write-Output($Cmd.CommandText)
$sqlDataAdapter.Fill($dataSet) | out-null
if ($dataSet.Tables[0].Rows.Count -gt 0)
{
$outputDataTable = $dataSet.Tables[0]
}
else
{
$outputDataTable = "SQL Stroc Proc Executed”
}
}
catch
{
#write your own error handling code here.
#if required send error message in email.
}
finally
{
if ($Cmd -ne $null)
{
$Cmd.Dispose
}
$DatabaseConnection.Close()
$DatabaseConnection.Dispose()
}
}
}
And another runnbook name "RB_Daily_Transaction_Summary_Record"
workflow RB_Daily_Transaction_Summary_Record
{
$dataTable = RB_ConnectSQL -SqlServer 'blahblah.database.windows.net' -Database 'blahDev' -Procedure 'sp_Daily_Transaction_Summary_Record' -SqlCredentialName 'blahCredential'
Write-Output($dataTable)
}
The runbook "RB_Daily_Transaction_Summary_Record" suppose to call "RB_ConnectSQL" and pass in the required parameter so that will execute the Store Procedure in Azure SQL Server.
However I get the error
At line:78 char:17
+ -SqlServer 'blahblah.database.windows.net'
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Cannot find the '-SqlServer' command. If this command is defined as a workflow, ensure it is defined before the workflow that calls it. If it is a command intended to run directly within Windows PowerShell (or is not available on this system), place it in an InlineScript: 'InlineScript { -SqlServer }'
May I know is there any mistake I make on the runbook?
When you want to separate a command into many lines, please add white-space and backquote at the end of each line, except for the last line.
In your case, it should work using the following format:
workflow RB_Daily_Transaction_Summary_Record
{
$dataTable = RB_ConnectSQL
-SqlServer 'blahblah.database.windows.net' `
-Database 'blahDev' `
-Procedure 'sp_Daily_Transaction_Summary_Record' `
-SqlCredentialName 'blahCredential'
Write-Output($dataTable)
}
I found out that the issue was I separate my code for readability become
workflow RB_Daily_Transaction_Summary_Record
{
$dataTable = RB_ConnectSQL
-SqlServer 'blahblah.database.windows.net'
-Database 'blahDev'
-Procedure 'sp_Daily_Transaction_Summary_Record'
-SqlCredentialName 'blahCredential'
Write-Output($dataTable)
}
But the new line seem like break the code.
I rewrite it as:
workflow RB_Daily_Transaction_Summary_Record
{
$dataTable = RB_ConnectSQL -SqlServer 'blahblah.database.windows.net' -Database 'blahDev' -Procedure 'sp_Daily_Transaction_Summary_Record' -SqlCredentialName 'blahCredential'
Write-Output($dataTable)
}
Then the error gone and it work well!

Problems Running Azure Automation Powershell to Scale Database Back After Restore Operation

I am trying to scale back a database after the restore operation has completed and am running into some problems. I am getting this exception and wonder if there is something in this script not supported by Azure Automation Workflows?
Parameter set cannot be resolved using the specified named parameters.
workflow insertflowname
{
<#
.SYNOPSIS
The purpose of this runbook is to demonstrate how to restore a particular database to a new database using an Azure Automation workflow. Then it is scaled back to Basic.
.NOTES
#>
# Specify Azure Subscription Name
$subName = 'insertsubscription name'
# Connect to Azure Subscription
Connect-Azure -AzureConnectionName $subName
Select-AzureSubscription -SubscriptionName $subName
# Define source databasename
$SourceDatabaseName = 'insert database name'
# Define source server
$SourceServerName = 'insert source server'
# Define destination server
$TargetServerName = 'insert destination server'
Write-Output "`$SourceServerName [$SourceServerName]"
Write-Output "`$TargetServerName [$TargetServerName]"
Write-Output "`$SourceDatabaseName [$SourceDatabaseName]"
Write-Output "Retrieving recoverable database details for database [$SourceDatabaseName] on server [$SourceServerName]."
$RecoverableDatabase = Get-AzureSqlRecoverableDatabase –ServerName $SourceServerName -DatabaseName $SourceDatabaseName
$TargetDatabaseName = "$SourceDatabaseName-$($RecoverableDatabase.LastAvailableBackupDate.ToString('O'))"
Write-Output "`$TargetDatabaseName [$TargetDatabaseName]"
Write-Output "Starting recovery of database [$SourceDatabaseName] to server [$TargetServerName] as database [$TargetDatabaseName]."
Start-AzureSqlDatabaseRecovery -SourceDatabase $RecoverableDatabase -TargetServerName $TargetServerName –TargetDatabaseName $TargetDatabaseName
$PollingInterval = 10
Write-Output "Monitoring status of recovery operation, polling every [$PollingInterval] second(s)."
$KeepGoing = $true
while ($KeepGoing) {
$operation = Get-AzureSqlDatabaseOperation -ServerName $TargetServerName -DatabaseName $TargetDatabaseName | Where-Object {$_.Name -eq "DATABASE RECOVERY"} | Sort-Object StartTime -Descending
if ($operation) {
$operation[0]
if ($operation[0].State -eq "COMPLETED") { $KeepGoing = $false }
if ($operation[0].State -eq "FAILED") {
# Throw error
$KeepGoing = $false
}
} else {
# Throw error since something went wrong and object was not created
# May want to have this retry a few times before giving up or at least notify somebody
# since at this point the recovery has been kicked off and you don't want the database
# restore to remain at the elevated service level.
$KeepGoing = $false
}
if ($KeepGoing) { Start-Sleep -Seconds $PollingInterval }
}
if ($operation[0].State -eq "COMPLETED") {
Write-Output "Setting service level for database [$TargetDatabaseName] on server [$TargetServerName] to Basic."
$ServiceObjective = Get-AzureSQLDatabaseServiceObjective –ServerName $TargetServerName –ServiceObjectiveName "Basic"
$ServiceObjective
Set-AzureSqlDatabase –ServerName $TargetServerName –DatabaseName $TargetDatabaseName –Edition "Basic" –ServiceObjective $ServiceObjective -MaxSizeGB 2 –Force
}
}
You are probably hitting the issue described here: https://social.msdn.microsoft.com/Forums/en-US/ce6412b8-5cce-4573-befb-6017924ce0d0/whereobject-fails-with-parameter-set-cannot-be-resolved-using-the-specified-named-parameters?forum=azureautomation
Summary:
Use parameter names, don't rely on positional parameters, in PowerShell Workflow. In this case, you need to add the -FilterScript parameter name to Where-Object.

Resources