How to delete the ADFPipeline which is having the references Forcefully - azure

Question

I'm actually some automation for my ADF. As a part of that, I'm trying to delete all the ADF V2 pipelines. The problem is my pipelines having many references with different pipelines itself.
$ADFPipeline = Get-AzDataFactoryV2Pipeline -DataFactoryName $(datafactory-name) -ResourceGroupName $(rg)
$ADFPipeline | ForEach-Object { Remove-AzDataFactoryV2Pipeline -ResourceGroupName $(rg) -DataFactoryName $(datafactory-name) -Name $_.name -Force }
And most of the time I get the error like
The document cannot be deleted since it is referenced by "blabla"
I understand the error that it saying some references and cannot be deleted. However, when I tried the same deletion in the azure portal, irrespective of the reference I can able to delete. So I want to find a way that whether it possible to tell that Powershell even though it's having a reference delete it forcefully
Any other inputs much appreciated!

Answer 1

I run into the same issue, found out that it's rather complicated to build the whole dependency graph out of the pipeline's Activities property.
As a working solution (powershell):
function Remove-Pipelines {
param (
[Parameter(Mandatory=$true)]
[AllowEmptyCollection()]
[AllowNull()]
[System.Collections.ArrayList]$pipelines
)
if($pipelines.Count -gt 0) {
[System.Collections.ArrayList]$plsToProcess = New-Object System.Collections.ArrayList($null)
foreach ($pipeline in $pipelines) {
try {
$removeAzDFCommand = "Remove-AzDataFactoryV2Pipeline -dataFactoryName '$DataFactoryName' -resourceGroupName '$ResourceGroupName' -Name '$($pipeline.Name)' -Force -ErrorAction Stop"
Write-Host $removeAzDFCommand
Invoke-Expression $removeAzDFCommand
}
catch {
if ($_ -match '.*The document cannot be deleted since it is referenced by.*') {
Write-Host $_
$plsToProcess.Add($pipeline)
} else {
throw $_
}
}
}
Remove-Pipelines $plsToProcess
}
}
Here is the complete solution for clearing the whole DF: "trigger","pipeline","dataflow","dataset","linkedService"
Param(
[Parameter(Mandatory=$true)][string] $ResourceGroupName,
[Parameter(Mandatory=$true)][string] $DataFactoryName
)
$artfTypes = "trigger","pipeline","dataflow","dataset","linkedService"
function Remove-Artifacts {
param (
[Parameter(Mandatory=$true)][AllowEmptyCollection()][AllowNull()][System.Collections.ArrayList]$artifacts,
[Parameter(Mandatory=$true)][string]$artfType
)
if($artifacts.Count -gt 0) {
[System.Collections.ArrayList]$artToProcess = New-Object System.Collections.ArrayList($null)
foreach ($artifact in $artifacts) {
try {
$removeAzDFCommand = "Remove-AzDataFactoryV2$($artfType) -dataFactoryName '$DataFactoryName' -resourceGroupName '$ResourceGroupName' -Name '$($artifact.Name)' -Force -ErrorAction Stop"
Write-Host $removeAzDFCommand
Invoke-Expression $removeAzDFCommand
}
catch {
if ($_ -match '.*The document cannot be deleted since it is referenced by.*') {
Write-Host $_
$artToProcess.Add($artifact)
} else {
throw $_
}
}
}
Remove-Artifacts $artToProcess $artfType
}
}
foreach ($artfType in $artfTypes) {
$getAzDFCommand = "Get-AzDataFactoryV2$($artfType) -dataFactoryName '$DataFactoryName' -resourceGroupName '$ResourceGroupName'"
Write-Output $getAzDFCommand
$artifacts = Invoke-Expression $getAzDFCommand
Write-Output $artifacts.Name
Remove-Artifacts $artifacts $artfType
}
The same approach can be adapted for "Set-AzDataFactoryV2Pipeline" command as well.
It worth to mention that along with dependencies tracking, Remove/Set artifact's sequence should be right (because of cross artifacts' dependencies).
For Set - "linkedService","dataset","dataflow","pipeline","trigger"
For Remove - "trigger","pipeline","dataflow","dataset","linkedService"

Answer 2

Hello and thank you for the question. According to the Remove-AzDataFactoryV2Pipeline doc, the -Force flag simply skips the confirmation prompt. It does not actually 'Force' the deletion in spite of errors.
Since you are already doing automation, might I suggest leveraging the error message to recursively attempt to delete the referencing pipeline. $error[0] gets the most recent error.
(Pseudocode)
try_recurse_delete( pipeline_name )
do_delete(pipeline_name)
if not $error[0].contains("referenced by " + pipeline_name)
then return true
else
try_recurse_delete( get_refrencer_name($error[0]) )
Given that pipeline dependencies can be a many-to-many relationship, subsequent pipelines in your for-each loop might already be deleted by the recursion. You will have to adapt your code to react to 'pipeline not found' type errors.