How do I grant the framework orchestrator access to itself allowing the 'Pipeline Already Running' to occur?
In the documentation found here: https://mrpaulandrew.github.io/procfwk/pipelinealreadyrunning
It states:
Data Factory must explicitly be granted access to itself in order to query its own pipeline runs.
The easiest way to do this is via the Azure portal control/management plane UI.
If the framework pipelines are running from Azure Data Factory
In the Azure portal, navigate to the Access Control (IAM) panel for the Data Factory instance.
Add a new Role Assignment using the Service Principal name that matches the Data Factory instance (itself).
Grant permissions using the Microsoft predefined Owner role. Shown below.
If the framework pipelines are running from Azure Synapse Analytics
The 'Pipeline Already Running' utility is not currently supported.
Related
Our organization having an Azure Synapse Dedicated Pool instance. I am trying to register the Azure Synapse Dedicated Pool with Azure Purview and want to scan the Synapse DB. However, I am getting the following Error every time:
“Failed to access the Azure Dedicated SQL pool with the given credentials”
Following are the process I followed to Register the Data Source:
I opened “Purview Studio”
There I have created a “Collection”
Then I go to “Register Sources”
Then I search for “Azure Synapse Dedicated Pool”
Then I select the subscription where my Azure Synapse Dedicated Pool is present
Then I Registered my Data Source
Now I am trying to create a New Scan for my Synapse Dedicated Pool
The problem starts from here, First of all I selected the subscription, then I selected the resource group and then I selected the Synapse DB name. I tried two authentication methods to authenticate my Synapse Instance. First one is Purview MSI account and second one is SQL Authentication. I have added my Purview MSI account as a user in Synapse Dedicated pool using following command.
CREATE USER [PurviewAccountName] FROM EXTERNAL PROVIDER
GO
EXEC sp_addrolemember 'db_datareader', [PurviewAccountName]
GO
Now I tried to test the connection but it is not working and giving me following Error:
“Failed to access the Azure Dedicated SQL pool with Purview MSI account”
My Azure Synapse Dedicated Pool instance in not publically accessible, we have put it behind the private link. I can connect my Azure Synapse Instance using VPN connectivity on my machine and login through SSMS and Azure Data Studio.
I also tried with SQL authentication by using SQL username and Password which is kept under the keyvault. I have checked it multiple times and I am confident I have configured it correctly. But still when I try to test the connection. It is showing following error:
“Failed to access the Azure Dedicated SQL pool with the given credentials”
Some where I have read I need self-hosted-integration runtime if the Azure Synapse instance is behind private link.
So I installed integration runtime on my machine, configure it and tested for the Synapse connection with SQL Authentication by connecting to VPN. Self-Hosted IR configured successfully. I tested with both the IR. Azure IR and Self-hosted-IR. But no luck, I am getting the same error.
I have also added Purview MSI account to Access Policy in keyVault and provided GET, List permission on keys and Secrets.
However, I am not getting what I am missing here and why it is giving me the same error.
Any help on this is really means a lot me..
CREATE USER [PurviewAccountName] FROM EXTERNAL PROVIDER
GO
EXEC sp_addrolemember 'db_datareader', [PurviewAccountName]
GO
According to Microsoft official documentation, to execute the above command one must be Azure Synapse Administrator in the workspace. It is alsi required that your purview account name must have reader role set which can be done from Access Control (IAM) under the Azure Synapse Workspace resource.
To create SQL Pools, Apache Spark Pools and Integration Runtimes, users must have at least Azure Contributor role in the workspace. The contributor role also allows these users to manage the resources, including pausing and scaling. If you're- using Azure Portal or Synapse Studio to create SQL Pools, Apache Spark Pools and INtegration Runtimes, then you need Azure Contributor role at the resource group level.
To GRANT access to a Dedicated SQL Pool database, the scripts can be run by the workspace creator or any member of the workspace1_SynapseAdministrators group.
Follow the below steps in the Azure Synapse SQL script editor:
Create the USER in the database by running the following command on the target database, selected using the Connect to dropdown:
CREATE USER [<alias#domain.com>] FROM EXTERNAL PROVIDER;
Grant a user a role to access the database
EXEC sp_addrolemember 'db_owner', '<alias#domain.com>'
I have connected my azure account in Data Studio and I am using Azure SQL migration extension (v0.1.12) to migrate on-prem SQL to Azure Managed Instance.
However my subscription details are not getting fetched.
Screen Shot Attached Here
When I manually add Azure Subscription details I am getting following error
Manually Entered Details
And the error message Error
The issue seems to be more of access level issues.
Below are the type of access levels that you need to have for creating Azure Migrate Appliance project
Contributor or Owner permissions in the Azure subscription.
Permissions to register Azure Active Directory (Azure AD) apps.
Owner or Contributor and User Access Administrator permissions in the Azure subscription to create an instance of Azure Key Vault, which is used during agentless server migration.
Below are the steps to set contributor or Owner permissions
From Azure Subscriptions panel select the subscription
Move to Access Control IAM and select Add role Assignment
Assign the following roles.
For complete information check the Microsoft Document on providing access.
We are implementing row level security in Azure Synapse Analytics and we want to check if user is member of specific Azure AAD group, user can access data. As per [documentation][1]
[1]: https://learn.microsoft.com/en-us/sql/t-sql/functions/is-member-transact-sql?view=sql-server-ver15 it says this function only check windows group. Is there any work around or ETA when this features will be available?
We tried using below query but it always returns NULL
SELECT IS_MEMBER('AAD_Group_Name')
The document you have shared clearly mentioned that IS_MEMBER function is not supported for Azure Active Directory Groups.
You can raise the feature request here.
Alternatively, you can check this official document about How to set up access control for your Azure Synapse workspace. This will help you to understand and implement control access to a Microsoft Azure Synapse workspace using Azure roles, Azure Synapse roles, SQL permissions, and Git permissions.
I created a Web task in Azure Data Factory to start/stop SSIS Integration Runtime. However, when I try to run the pipeline, I keep getting this error - Activity Start My IR failed:
{"error":{"code":"AuthorizationFailed","message":"The client
'2b36c922-34cf-43d8-be05-5d757017e2ed' with object id
'2b36c922-34cf-43d8-be05-5d757017e2ed' does not have authorization to
perform action
'Microsoft.DataFactory/factories/integrationruntimes/start/action'
over scope
'/subscriptions/e8e76bdc-4591-4936-9899-31bffef9c5cd/resourcegroups/XXXXXBI-RG/providers/Microsoft.DataFactory/factories/XXXXXBI-DF/integrationruntimes/XXXXXSSIS-IR'."}}
I followed this instruction - https://learn.microsoft.com/en-us/azure/data-factory/how-to-schedule-azure-ssis-integration-runtime
For Authentication, select MSI to use the managed identity for your ADF, see Managed identity for Data Factory article for more info.
What is the issue?
According to the error, you may not add the service principal to the ADF as a role.
You need to assign the managed identity for your ADF a Contributor role to itself, so Web activities in its pipelines can call REST API to start/stop Azure-SSIS IRs provisioned in it.
Follow the step 5 in this link:
I'm looking for my VSTS deployment agent service principal to get the Object ID of a managed service principal (created by Data Factory V2).
It needs this for assigning ACL's in data lake store.
However as far as I can tell, it requires Read Permission on Azure AD.
I wish to avoid granting it read permission if possible, to follow the 'least privilege' mantra.
For non-managed service principals, I allow the deployment agent to manage service principals it has created (thus not needing full read access). However I suspect the fact that the service principals are managed by Azure it is unlikely I'll be able to give the deployment agent any ownership over the managed service principal (however I'm working with AD admins to see if there is a way).
I've tried everything I can think of mixing and matching Azure RM data factory v2 powershell modules and Azure AD modules.
Is it possible to get the id without Azure AD read access? Or a novel workaround (I'm considering a constrained Web API wrapper)?