I am testing the Azure data factory deployment using ARM Templates and deleting the ADF instance (Data factory pipeline, linked services, data sets, data flow, Trigger etc.) using Azure Data Factory Delete item in-built task in DevOps Pipeline from azure Devops before deploying to UAT and production. All items deleted as per task outcome but there is one linked service which didn't delete.
Giving error= deleting LS_1 Linked Service: the document cannot be deleted since it is referenced by LS_2. basically the LS_2 deleted and it is not showing in the UAT ADF environment, only LS_1 is showing.
please find attached screenshot. please share your valuable suggestion on this how to resolve it.
Thanks
Related
Whenever I create a Source in an activity in a Synapse Pipeline, in the Linked Service tab, I get an option to either create a new Linked Service or to select from the dropdown (as shown below). One selection of that dropdown includes a default Linked Service (shown below) that shows as MySynapseWorkspaceName-WorkspaceDefaultStorage (where MySynapseWorkspaceName is name of a Synapse workspace that you create).
It seems that MySynapseWorkspaceName-WorkspaceDefaultStorage is the linked service that gets created when you specify an Azure Data Lake Storage Gen2 (ADLSGen2) account for your Synapse workspace.
Question: If a Dataset for the source or destination (Sink) of an activity in Synapse Pipeline is a ADLSGen2 storage, can we just select the above default linked service MySynapseWorkspaceName-WorkspaceDefaultStorage for that dataset; or choosing this linked service (created for Synapse workspace) for other datasets may cause an issue - and hence we should avoid using this linked service for other datasets inside our Synapse workspace?
From your comment, I understood that You want to know Whether same Linked Service can be used in both Source and Sink datasets ?
Unfortunately, you can not use same Linked Service in both Source and Sink. It may cause an issue and hence you should avoid using same linked service.
I made an ARM template which runs through an azure devops pipeline to create a new cosmos instance and put two collections inside it. I'd like to put some data inside the collections (fixed values, same every time). Everything is created in the standard way, e.g. the collections are using
"type": "Microsoft.DocumentDb/databaseAccounts/apis/databases/containers"
I think these are the relevant docs.
I haven't found mentions of automatically adding data much, but it's such an obviously useful thing I'm sure it will have been added. If I need to add another step to my pipeline to add data, that's an option too.
ARM templates are not able to insert data into Cosmos DB or any service with a data plane for many of the reasons listed in the comments and more.
If you need to both provision a Cosmos resource and then insert data into it you may want to consider creating another ARM template to deploy an Azure Data Factory resource and then invoke the pipeline using PowerShell to copy the data from Blob Storage into the Cosmos DB collection. Based upon the ARM doc you referenced above it sounds as though you are creating a MongoDB collection resource. ADF supports MongoDB so this should work very well.
You can find the ADF ARM template docs here and the ADF PowerShell docs can be found here. If you're new to using ARM to create ADF resources, I recommend first creating it using the Azure Portal, then export it and examine the properties you will need to drive with parameters or variables during deployment.
PS: I'm not sure why but this container resource path (below) you pointed to in your question should not used as it breaks a few things in ARM, namely, you cannot put a resource lock on it or use Azure Policy. Please use the latest api-version which as of this writing is 2021-04-15.
"type": "Microsoft.DocumentDb/databaseAccounts/apis/databases/containers"
I have a Data Factory pipeline that contains a bunch of Linked Services. Some of them are not used anymore and I would like to delete them. However, when I try to delete some of them, Data Factory complains that it is still used with the following error:
Error: Not able to delete linked service, message returned from
service - The document cannot be deleted since it is referenced by
staging_zone_dim.
Question
I made sure that no Datasets or Pipelines references it and still, I do get the error message when I try to delete the linked services. What am I missing?
You should have a look in the published view of the data factory and delete the pipelines there too before deleting the linked service.
You can switch here by choosing data factory instead of Azure DevOps:
I have been able to work around this issue by directly going to the Git repository that contained the json files and deleting the unused configuration files.
I have a very simple requirement to setup an Azure dev resource on the same subscription as the prod environment. This dev resource would be a 1 to 1 clone of a single Azure Datafactory resource.
Is there a way to simply clone the Azure DataFactory resource on the same subscription and rename it to a "...-dev" version?
(At present, I do not see a simple clone function...)
There is no clone option, but you can export the automation script and re-run that in order to replicate the resource. If you want to automate many similar deployments in Azure, look in to ARM Templates.
Keep in mind data factory has two key aspects:
The actual Data Factory service instance
Your Data Factory pipelines (i.e. the data workflows you write in Data Factory)
Both can be automated and deployed using ARM Templates. (2) above can also be linked to source control where you can then clone and re-use the definition files.
The client 'abc#abc.com' with object id 'abcabcabcabcabc' does not
have authorization to perform action
'Microsoft.Resources/deployments/write' over scope
'/subscriptions/abcabcabc/resourcegroups/abc-01-east/providers/Microsoft.Resources/deployments/publishing-123123123123'
I was trying to create a pipeline using azure data factory to pull data from sql-server to azure blob, but i am facing the above issue while i was trying to use my integration runtime which already exsist in my azure portal.
At present I have data factory contributor role assigned to me, what other roles should I have to avoid this issue?
I had a similar issue being a contributor for an ADF. With this role, you seem to be able to open the ADF UI, but the moment you try to publish anything, you get the above error. Making me a data factory contributor for that ADF didn't help.
What did help was making me a data factory contributor on the resource group level. So go to the resource group that contains the ADF, go to IAM and add you as a data factory contributor.
I also noticed, you need to close the data factory ui before IAM changes take effect.
Azure's roles are a bit of a mystery to me so it would be useful if someone could provide an explanation of how and why.
Steps
1 - Register an Enterprise APP in your Azure Active Directory
2 - Create a key in the Enterprise APP and save the value somewhere
3 - Go to your Azure SQL Database through Management Console and
CREATE USER [your application name] FROM EXTERNAL PROVIDER;
4 - Change the authentication method for Principal and use the application id and key on the form
For more information:
https://learn.microsoft.com/en-us/azure/data-factory/connector-azure-sql-database