I am trying to list azure pipelines run for a given data factory pipeline resource. The account I am using has Contributor role to the data factory resource.
But I get a permission error when I try to query running pipelines
await client.PipelineRuns.QueryByFactoryAsync(dataFactoryResourceName, factoryRG ...
So question I have is why Contributor role is not sufficient to be able to list pipelines. From role description, it sounds like Contributor role should have all required access. ContributorRole permissions Is this a bug in implementation?
According to my test, Contributor role is enough. You need to set your AAD application as Contributor.
My test code:
var context = new AuthenticationContext("https://login.windows.net/" + tenantID);
ClientCredential cc = new ClientCredential(applicationId, authenticationKey);
AuthenticationResult result = context.AcquireTokenAsync("https://management.azure.com/", cc).Result;
ServiceClientCredentials cred = new TokenCredentials(result.AccessToken);
client = new DataFactoryManagementClient(cred)
{
SubscriptionId = subscriptionId
};
PipelineRunsQueryResponse response = await client.PipelineRuns.QueryByFactoryAsync(dataFactoryResourceName, factoryName,new RunFilterParameters());
Console.WriteLine(response.Value.Count);
Related
What permissions are needed to run the azure data factory management apis. For instance I am trying to execute the Pipeline runs, Query by factory api in a web activity.
Error:
User configuration issue
{"error":{"code":"AuthorizationFailed","message":"The client with object id does not have authorization to perform action 'Microsoft.DataFactory/factories/pipelineruns/read' over scope '/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.DataFactory/factories/{factoryName}/pipelineruns/{runId}' or the scope is invalid. If access was recently granted, please refresh your credentials."}}
Could you please guide me on how to pass the credentials and get the token for GET and POST methods.
You need to create an Azure Active Directory Application that can access your Data factory.
Create an Azure Active Directory application, For the sign-on URL, you can provide a dummy URL (https://contoso.org/exampleapp).
Get values for signing in, get the application ID and tenant ID, and note down these values that you use later.
Certificates and secrets, get the authentication key, and note down this value that you use later in this tutorial.
Assign the application to a role, assign the application to the Contributor role at the subscription level so that the application can create data factories in the subscription.
After you do the above steps, you need to create the DataFactoryManagementClient and authenticate your application using the below code snippet:
var context = new AuthenticationContext("https://login.microsoftonline.com/" + tenantID);
ClientCredential cc = new ClientCredential(applicationId, authenticationKey);
AuthenticationResult result = context.AcquireTokenAsync("https://management.azure.com/", cc).Result;
ServiceClientCredentials cred = new TokenCredentials(result.AccessToken);
var client = new DataFactoryManagementClient(cred) {
SubscriptionId = subscriptionId };
Once you've authenticated your application, you can start the Pipeline run using the below code snippet:
// Create a pipeline run
Console.WriteLine("Creating pipeline run...");
CreateRunResponse runResponse = client.Pipelines.CreateRunWithHttpMessagesAsync(
resourceGroup, dataFactoryName, pipelineName, parameters: parameters
).Result.Body;
Console.WriteLine("Pipeline run ID: " + runResponse.RunId);
Below is the complete code of a console application to run an Azure Data Factory Pipeline:
using Microsoft.Azure.Management.DataFactory;
using Microsoft.Azure.Management.DataFactory.Models;
using Microsoft.IdentityModel.Clients.ActiveDirectory;
using Microsoft.Rest;
using System;
namespace ADF
{
class Program
{
static void Main(string[] args)
{
// Set variables
string tenantID = "<your tenant ID>";
string applicationId = "<your application ID>";
string authenticationKey = "<your authentication key for the application>";
string subscriptionId = "<your subscription ID where the data factory resides>";
string resourceGroup = "<your resource group where the data factory resides>";
string dataFactoryName ="<specify the name of data factory to create. It must be globally unique.>";
string pipelineName = "<specify the name of pipeline to run. It must be globally unique.>";
// Authenticate and create a data factory management client
var context = new AuthenticationContext("https://login.microsoftonline.com/" + tenantID);
ClientCredential cc = new ClientCredential(applicationId, authenticationKey);
AuthenticationResult result = context.AcquireTokenAsync("https://management.azure.com/", cc).Result;
ServiceClientCredentials cred = new TokenCredentials(result.AccessToken);
var client = new DataFactoryManagementClient(cred)
{
SubscriptionId = subscriptionId
};
// Create a pipeline run
Console.WriteLine("Creating pipeline run...");
CreateRunResponse runResponse = client.Pipelines.CreateRunWithHttpMessagesAsync(
resourceGroup, dataFactoryName, pipelineName
).Result.Body;
Console.WriteLine("Pipeline run ID: " + runResponse.RunId);
}
}
}
Don't forget to add the Nuget Packages:
Install-Package Microsoft.Azure.Management.DataFactory
Install-Package Microsoft.Azure.Management.ResourceManager -IncludePrerelease
Install-Package Microsoft.IdentityModel.Clients.ActiveDirectory
I'm trying to access Azure Data Fabric V2 programmatically.
First, I had created an App Registration in Azure portal and a Client secret. Then I gave Contributor permission to this App registration on the entire suscription, and also in the resource group where my data factory lives.
Using this credentials I'm able to login to the portal and create an DataFactoryManagementClient
private void CreateAdfClient()
{
var authenticationContext = new AuthenticationContext($"https://login.windows.net/{tenantId}");
var credential = new ClientCredential(clientId: appRegistrationClientId, clientSecret: appRegistrationClientkey);
var result = authenticationContext.AcquireTokenAsync(resource: "https://management.core.windows.net/", clientCredential: credential).ConfigureAwait(false).GetAwaiter().GetResult();
if (result == null)
{
throw new InvalidOperationException("Failed to obtain the JWT token");
}
var token = result.AccessToken;
var tokenCloudCredentials = new TokenCloudCredentials(subscriptionId, token);
datafactoryClient = new DataFactoryManagementClient(tokenCloudCredentials);
}
However, when I try to get my pipeline with
var pipeline = datafactoryClient.Pipelines.Get(resourceGroup, dataFactory, pipelineName);
it throws an error:
System.Private.CoreLib: Exception while executing function:
StartRawMeasuresSync. Microsoft.Azure.Management.DataFactories:
ResourceNotFound: The Resource
'Microsoft.DataFactory/dataFactories/MyPipeline' under resource group
'MyResGroup' was not found.
I had verified that the resource group, the data factory name and the pipeline name are correct, but it keeps throwing this error.
I had the same issue, and it was due to referencing the Nuget package for Azure Data Factory v1 instead of v2.
Version 1: Microsoft.Azure.Management.DataFactories
Version 2: Microsoft.Azure.Management.DataFactory
I have followed the Azure documentation steps to create a simple Copy Data Factory from Blob to SQL.
Now I want to run the pipeline through VS code.
I have checked the authentication keys and Roles assigned are correct.
Below is the code -
var context = new AuthenticationContext("https://login.windows.net/" + tenantID);
ClientCredential cc = new ClientCredential(applicationId, authenticationKey);
AuthenticationResult result = context.AcquireTokenAsync("https://management.azure.com/", cc).Result;
ServiceClientCredentials cred = new TokenCredentials(result.AccessToken);
var client = new DataFactoryManagementClient(cred) { SubscriptionId = subscriptionId };
Console.WriteLine("Creating pipeline run...");
var st = client.Pipelines.Get(resourceGroup, dataFactoryName, pipelineName);
CreateRunResponse runResponse = client.Pipelines.CreateRunWithHttpMessagesAsync(resourceGroup, dataFactoryName, pipelineName).Result.Body;
Console.WriteLine("Pipeline run ID: " + runResponse.RunId);
However, I get Forbidden error.
The client 'xxxx' with object id 'xxxx' does not have authorization to perform action 'Microsoft.DataFactory/factories/pipelines/read' over scope '/subscriptions/xxxxx/resourceGroups/'
How can I fix this?
How can I fix this?
According to the exception message that it indicates that you don't assign the corresponding role to application to access the data factory.
I test your code with Azure Datafactory(V2) on my side , it works correctly. The following is my details steps.
Registry an Azure AD WebApp application.
Get the clientId and clientscret from created Application.
Assign the role the application to access the datafactory.
Test code on my side.
I have a requirement to create user or group in azure active directory programmatically. I searched on google and I found multiple solutions like using Graph APIs, C# Code etc.. but I am bit confused with the approach.
Can any one help me out with the difference between these approaches and suggest me the best approach? Please let me know if there are any code samples available.
Thanks in advance !!
Azure ad support multiple protocols. To acquire the token for the Azure AD Graph we need to choose the suitable flow in OAuth 2.0/OpenId connect to interact with Azure AD.
For example, if you were developing a web app the OAuth code grant flow maybe is a good choice. And if the app is daemon app or service application, the client credentials flow the better one. More about the scenarios you can refer this document.
And to acquire the token for Azure AD Graph in a web app, you can refer this code sample. At the line of 104 of this code sample, it acquire the access token for Azure AD Graph. And then in the controller, you can use the code below to acquire the token from cache and create the user using Azure AD Graph:
string graphResourceId = "https://graph.windows.net";
string tenantId = "xxx.onmicrosoft.com";
AuthenticationContext authContext = new AuthenticationContext("https://login.microsoftonline.com/xxx.onmicrosoft.com");
ClientCredential credential = new ClientCredential("{clientId}", "{secret}");
string userObjectID = ClaimsPrincipal.Current.FindFirst("http://schemas.microsoft.com/identity/claims/objectidentifier").Value;
AuthenticationResult result = await authContext.AcquireTokenSilentAsync(graphResourceId, credential, new UserIdentifier(userObjectID, UserIdentifierType.UniqueId));
var accessToken = result.AccessToken;
Uri servicePointUri = new Uri(graphResourceId);
Uri serviceRoot = new Uri(servicePointUri, tenantId);
ActiveDirectoryClient graphClient = new ActiveDirectoryClient(serviceRoot, async () => await Task.FromResult(accessToken));
var user = new User();
user.AccountEnabled = true;
user.DisplayName = "testName";
user.UserPrincipalName = "testName#xxx.onmicrosoft.com";
user.MailNickname = "testName";
user.UsageLocation = "US";
user.PasswordProfile = new PasswordProfile
{
Password = "xxxxxx",
ForceChangePasswordNextLogin = true
};
await graphClient.Users.AddUserAsync(user);
And the application requires Directory.ReadWrite.All to create user and group. More detail about the permission you can refer here.
I've been trying (and failing) to create an Azure Stream Analytics job programatically. I was following this example originally:
https://azure.microsoft.com/en-gb/documentation/articles/stream-analytics-dotnet-management-sdk/
But it pops up a dialog for you to log in. I want to be able to do this server side. It looks like I need to use Azure AD to use the Resource Manager APIs. I've been working my way through this:
https://msdn.microsoft.com/en-us/library/azure/dn790557.aspx#bk_portal
And the code looks like this:
var authContext = new AuthenticationContext("https://login.microsoftonline.com/{tenant id}/oauth2/token");
var clientId = "{app client id}";
var appKey = "{app key}";
var subscriptionId = "{subscription id}";
var clientCredential = new ClientCredential(clientId, appKey);
var result = authContext.AcquireToken("https://management.core.windows.net/", clientCredential);
var creds = new TokenCloudCredentials(subscriptionId, result.AccessToken);
var client = new StreamAnalyticsManagementClient(creds);
var jobCreateParameters = new JobCreateOrUpdateParameters
{
Job = new Job
{
Name = streamAnalyticsJobName,
Location = "North Europe",
Properties = new JobProperties
{
EventsOutOfOrderPolicy = EventsOutOfOrderPolicy.Adjust,
Sku = new Sku
{
Name = "Standard"
}
}
}
};
var jobCreateResponse = client.StreamingJobs.CreateOrUpdate(resourceGroupName, jobCreateParameters);
I can successfully acquire a token, but creating the job fails:
AuthorizationFailed: The client 'REDACTED' with object id 'REDACTED' does not have authorization to perform action 'Microsoft.StreamAnalytics/streamingjobs/write' over scope '/subscriptions/REDACTED/resourcegroups/REDACTED/providers/Microsoft.StreamAnalytics/streamingjobs/REDACTED'
Am I doing something wrong? The app has the delegated permissions set.
UPDATE - 08-Dec-2015
There's an easy way to assign roles to Service Principals now. Please see this link for more details: https://azure.microsoft.com/en-in/documentation/articles/resource-group-create-service-principal-portal/.
Original Response
When you grant access to an application to your Azure subscription, behind the scenes a user is created with Service Principal user type in your Azure AD. The code you're using below assumes that you're using this Service Principal user when getting access token.
var clientCredential = new ClientCredential(clientId, appKey);
var result = authContext.AcquireToken("https://management.core.windows.net/", clientCredential);
var creds = new TokenCloudCredentials(subscriptionId, result.AccessToken);
However by default this user is not granted any permissions (RBAC) on your subscription and that's why you're getting the authorization error.
To solve this problem, what you would need to do is grant appropriate role to this user in your subscription. Now you can use PowerShell to do so or you can do it via code using ADAL library + making some web requests.
What I did was I made use of ADAL library to get access tokens and then used Google Postman (or Fiddler) to do other stuff. In my case, it was a web application. Here's what I did:
I logged in into the application as Global Administrator (Subscription Owner) and got a code. Using that code and ADAL library, I got the access token (let's call it token1).
var authContext = new AuthenticationContext(string.Format("{0}/common", signinEndpoint));//signinEndpoint = https://login.windows.net
var result = await authContext.AcquireTokenByAuthorizationCodeAsync(code, redirectUri, credential);
I copied the tenant id and access token returned to me in result above.
Next thing I did was I found out the object id of the Service Principal user using POSTMAN. This is the GET URL I executed there. For Authorization header, you would need to use Bearer {token1}.
https://graph.windows.net/{subscription-id}/servicePrincipals?api-version=1.5&$filter=appId eq '{app-client-id}'
After that I acquired another access token (let's call it token2) for Service Management API operation using the code below:
authContext = new AuthenticationContext(string.Format("{0}/{1}", signinEndpoint, result.TenantId));
result = await authContext.AcquireTokenSilentAsync(serviceManagementApiEndpoint, credential, new UserIdentifier(request.UserInfo.UniqueId, UserIdentifierType.UniqueId));//serviceManagementApiEndpoint = https://management.core.windows.net/
After that I listed the roles in my subscription and picked the role I wanted to assign to my Service Principal user. In my case, I wanted to assign a Reader role so I noted down the role's id. For Authorization header, you would need to use Bearer {token2}.
https://management.azure.com/subscriptions/{subscription-id}/providers/Microsoft.Authorization/roleDefinitions?api-version=2015-06-01
Next is assignment of this role to the user. For this I created a guid as role assignment id and used the following URL:
https://management.azure.com/subscriptions/{subscription-id}/providers/microsoft.authorization/roleassignments/{role-assignment-id}?api-version=2015-06-01
It's going to be a PUT request and this was the request body would be something like:
{
"properties":
{
"roleDefinitionId": "{role id of the selected role from step 5}",
"principalId": "{id of the service principal from step 3"
}
}
Please ensure that the content-type of the request is set as application/json;odata=verbose and not application/json.
That's pretty much it! After that your code should work just fine :)
Give it a try and see what happens.