I'm trying to access Azure Data Fabric V2 programmatically.
First, I had created an App Registration in Azure portal and a Client secret. Then I gave Contributor permission to this App registration on the entire suscription, and also in the resource group where my data factory lives.
Using this credentials I'm able to login to the portal and create an DataFactoryManagementClient
private void CreateAdfClient()
{
var authenticationContext = new AuthenticationContext($"https://login.windows.net/{tenantId}");
var credential = new ClientCredential(clientId: appRegistrationClientId, clientSecret: appRegistrationClientkey);
var result = authenticationContext.AcquireTokenAsync(resource: "https://management.core.windows.net/", clientCredential: credential).ConfigureAwait(false).GetAwaiter().GetResult();
if (result == null)
{
throw new InvalidOperationException("Failed to obtain the JWT token");
}
var token = result.AccessToken;
var tokenCloudCredentials = new TokenCloudCredentials(subscriptionId, token);
datafactoryClient = new DataFactoryManagementClient(tokenCloudCredentials);
}
However, when I try to get my pipeline with
var pipeline = datafactoryClient.Pipelines.Get(resourceGroup, dataFactory, pipelineName);
it throws an error:
System.Private.CoreLib: Exception while executing function:
StartRawMeasuresSync. Microsoft.Azure.Management.DataFactories:
ResourceNotFound: The Resource
'Microsoft.DataFactory/dataFactories/MyPipeline' under resource group
'MyResGroup' was not found.
I had verified that the resource group, the data factory name and the pipeline name are correct, but it keeps throwing this error.
I had the same issue, and it was due to referencing the Nuget package for Azure Data Factory v1 instead of v2.
Version 1: Microsoft.Azure.Management.DataFactories
Version 2: Microsoft.Azure.Management.DataFactory
Related
What permissions are needed to run the azure data factory management apis. For instance I am trying to execute the Pipeline runs, Query by factory api in a web activity.
Error:
User configuration issue
{"error":{"code":"AuthorizationFailed","message":"The client with object id does not have authorization to perform action 'Microsoft.DataFactory/factories/pipelineruns/read' over scope '/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.DataFactory/factories/{factoryName}/pipelineruns/{runId}' or the scope is invalid. If access was recently granted, please refresh your credentials."}}
Could you please guide me on how to pass the credentials and get the token for GET and POST methods.
You need to create an Azure Active Directory Application that can access your Data factory.
Create an Azure Active Directory application, For the sign-on URL, you can provide a dummy URL (https://contoso.org/exampleapp).
Get values for signing in, get the application ID and tenant ID, and note down these values that you use later.
Certificates and secrets, get the authentication key, and note down this value that you use later in this tutorial.
Assign the application to a role, assign the application to the Contributor role at the subscription level so that the application can create data factories in the subscription.
After you do the above steps, you need to create the DataFactoryManagementClient and authenticate your application using the below code snippet:
var context = new AuthenticationContext("https://login.microsoftonline.com/" + tenantID);
ClientCredential cc = new ClientCredential(applicationId, authenticationKey);
AuthenticationResult result = context.AcquireTokenAsync("https://management.azure.com/", cc).Result;
ServiceClientCredentials cred = new TokenCredentials(result.AccessToken);
var client = new DataFactoryManagementClient(cred) {
SubscriptionId = subscriptionId };
Once you've authenticated your application, you can start the Pipeline run using the below code snippet:
// Create a pipeline run
Console.WriteLine("Creating pipeline run...");
CreateRunResponse runResponse = client.Pipelines.CreateRunWithHttpMessagesAsync(
resourceGroup, dataFactoryName, pipelineName, parameters: parameters
).Result.Body;
Console.WriteLine("Pipeline run ID: " + runResponse.RunId);
Below is the complete code of a console application to run an Azure Data Factory Pipeline:
using Microsoft.Azure.Management.DataFactory;
using Microsoft.Azure.Management.DataFactory.Models;
using Microsoft.IdentityModel.Clients.ActiveDirectory;
using Microsoft.Rest;
using System;
namespace ADF
{
class Program
{
static void Main(string[] args)
{
// Set variables
string tenantID = "<your tenant ID>";
string applicationId = "<your application ID>";
string authenticationKey = "<your authentication key for the application>";
string subscriptionId = "<your subscription ID where the data factory resides>";
string resourceGroup = "<your resource group where the data factory resides>";
string dataFactoryName ="<specify the name of data factory to create. It must be globally unique.>";
string pipelineName = "<specify the name of pipeline to run. It must be globally unique.>";
// Authenticate and create a data factory management client
var context = new AuthenticationContext("https://login.microsoftonline.com/" + tenantID);
ClientCredential cc = new ClientCredential(applicationId, authenticationKey);
AuthenticationResult result = context.AcquireTokenAsync("https://management.azure.com/", cc).Result;
ServiceClientCredentials cred = new TokenCredentials(result.AccessToken);
var client = new DataFactoryManagementClient(cred)
{
SubscriptionId = subscriptionId
};
// Create a pipeline run
Console.WriteLine("Creating pipeline run...");
CreateRunResponse runResponse = client.Pipelines.CreateRunWithHttpMessagesAsync(
resourceGroup, dataFactoryName, pipelineName
).Result.Body;
Console.WriteLine("Pipeline run ID: " + runResponse.RunId);
}
}
}
Don't forget to add the Nuget Packages:
Install-Package Microsoft.Azure.Management.DataFactory
Install-Package Microsoft.Azure.Management.ResourceManager -IncludePrerelease
Install-Package Microsoft.IdentityModel.Clients.ActiveDirectory
I am trying to list azure pipelines run for a given data factory pipeline resource. The account I am using has Contributor role to the data factory resource.
But I get a permission error when I try to query running pipelines
await client.PipelineRuns.QueryByFactoryAsync(dataFactoryResourceName, factoryRG ...
So question I have is why Contributor role is not sufficient to be able to list pipelines. From role description, it sounds like Contributor role should have all required access. ContributorRole permissions Is this a bug in implementation?
According to my test, Contributor role is enough. You need to set your AAD application as Contributor.
My test code:
var context = new AuthenticationContext("https://login.windows.net/" + tenantID);
ClientCredential cc = new ClientCredential(applicationId, authenticationKey);
AuthenticationResult result = context.AcquireTokenAsync("https://management.azure.com/", cc).Result;
ServiceClientCredentials cred = new TokenCredentials(result.AccessToken);
client = new DataFactoryManagementClient(cred)
{
SubscriptionId = subscriptionId
};
PipelineRunsQueryResponse response = await client.PipelineRuns.QueryByFactoryAsync(dataFactoryResourceName, factoryName,new RunFilterParameters());
Console.WriteLine(response.Value.Count);
I need to get access to Key Vault and Service Bus from code, using a Service Principle for authentication.
I can use the following code to access Service Bus, which works as expected - when I enable to Service Principle in the Access Policies I can pull the list of topics:
var credentials = SdkContext.AzureCredentialsFactory.FromServicePrincipal(APPID, APPSECRET, TENANTID, AzureEnvironment.AzureGlobalCloud);
var serviceBusManager = ServiceBusManager.Authenticate(credentials, SUBSCRIPTIONID);
var serviceBusNamespace = serviceBusManager.Namespaces.List().SingleOrDefault(n => n.Name == "SERVICEBUSNAMESPACE");
var topics = serviceBusNamespace.Topics.ListAsync().GetAwaiter().GetResult();
However, I also need to get some information from Key Vault and I was trying to establish a common way to authenticate.
METHOD 1
Similar to the above, I tried this code to access KeyVault:
var credentials = SdkContext.AzureCredentialsFactory.FromServicePrincipal(APPID, APPSECRET, TENANTID, AzureEnvironment.AzureGlobalCloud);
var kvManager = new KeyVaultClient(credentials);
var secret = kvManager.GetSecretAsync("https://VAULTNAMESPACE.vault.azure.net", "SECRETNAME").GetAwaiter().GetResult().Value;
I get the the following error:
Microsoft.Azure.KeyVault.Models.KeyVaultErrorException: 'Operation
returned an invalid status code 'Unauthorized''
METHOD 2
This code does work for Key Vault however (showing I have correct permissions):
string GetSecret()
{
var client = new KeyVaultClient(GetAccessToken);
var secret = client.GetSecretAsync("https://VAULTNAMESPACE.vault.azure.net", "SECRETNAME").GetAwaiter().GetResult();
return secret;
}
private static async Task<string> GetAccessToken(string authority, string resource, string scope)
{
var context = new AuthenticationContext("https://login.windows.net/" + tenantId);
var credential = new ClientCredential(appId, appSecret);
var tokenResult = await context.AcquireTokenAsync("https://vault.azure.net", credential);
return tokenResult.AccessToken;
}
But, again, it's a very KeyVault specific way to Authenticate and I was hoping to establish a common mechanism using SdkContext.AzureCredentialsFactory. Any reason why I'd be getting an Unauthorized exception with the code above connecting to Key Vault? (all is set up correctly in Azure).
Thanks for any tips!
When you use SdkContext.AzureCredentialsFactory.FromServicePrincipal to authenticate, it will use https://management.azure.com/ as its Resource Uri.
While Azure Key Vault has its own authorization system and its Resource Uri is https://vault.azure.net, so you may get the Unauthorized error message.
So, you could use Method2 to get access to Azure Key Vault with right Resource Uri.
For more details, you could refer to this article.
I have followed the Azure documentation steps to create a simple Copy Data Factory from Blob to SQL.
Now I want to run the pipeline through VS code.
I have checked the authentication keys and Roles assigned are correct.
Below is the code -
var context = new AuthenticationContext("https://login.windows.net/" + tenantID);
ClientCredential cc = new ClientCredential(applicationId, authenticationKey);
AuthenticationResult result = context.AcquireTokenAsync("https://management.azure.com/", cc).Result;
ServiceClientCredentials cred = new TokenCredentials(result.AccessToken);
var client = new DataFactoryManagementClient(cred) { SubscriptionId = subscriptionId };
Console.WriteLine("Creating pipeline run...");
var st = client.Pipelines.Get(resourceGroup, dataFactoryName, pipelineName);
CreateRunResponse runResponse = client.Pipelines.CreateRunWithHttpMessagesAsync(resourceGroup, dataFactoryName, pipelineName).Result.Body;
Console.WriteLine("Pipeline run ID: " + runResponse.RunId);
However, I get Forbidden error.
The client 'xxxx' with object id 'xxxx' does not have authorization to perform action 'Microsoft.DataFactory/factories/pipelines/read' over scope '/subscriptions/xxxxx/resourceGroups/'
How can I fix this?
How can I fix this?
According to the exception message that it indicates that you don't assign the corresponding role to application to access the data factory.
I test your code with Azure Datafactory(V2) on my side , it works correctly. The following is my details steps.
Registry an Azure AD WebApp application.
Get the clientId and clientscret from created Application.
Assign the role the application to access the datafactory.
Test code on my side.
I've been trying (and failing) to create an Azure Stream Analytics job programatically. I was following this example originally:
https://azure.microsoft.com/en-gb/documentation/articles/stream-analytics-dotnet-management-sdk/
But it pops up a dialog for you to log in. I want to be able to do this server side. It looks like I need to use Azure AD to use the Resource Manager APIs. I've been working my way through this:
https://msdn.microsoft.com/en-us/library/azure/dn790557.aspx#bk_portal
And the code looks like this:
var authContext = new AuthenticationContext("https://login.microsoftonline.com/{tenant id}/oauth2/token");
var clientId = "{app client id}";
var appKey = "{app key}";
var subscriptionId = "{subscription id}";
var clientCredential = new ClientCredential(clientId, appKey);
var result = authContext.AcquireToken("https://management.core.windows.net/", clientCredential);
var creds = new TokenCloudCredentials(subscriptionId, result.AccessToken);
var client = new StreamAnalyticsManagementClient(creds);
var jobCreateParameters = new JobCreateOrUpdateParameters
{
Job = new Job
{
Name = streamAnalyticsJobName,
Location = "North Europe",
Properties = new JobProperties
{
EventsOutOfOrderPolicy = EventsOutOfOrderPolicy.Adjust,
Sku = new Sku
{
Name = "Standard"
}
}
}
};
var jobCreateResponse = client.StreamingJobs.CreateOrUpdate(resourceGroupName, jobCreateParameters);
I can successfully acquire a token, but creating the job fails:
AuthorizationFailed: The client 'REDACTED' with object id 'REDACTED' does not have authorization to perform action 'Microsoft.StreamAnalytics/streamingjobs/write' over scope '/subscriptions/REDACTED/resourcegroups/REDACTED/providers/Microsoft.StreamAnalytics/streamingjobs/REDACTED'
Am I doing something wrong? The app has the delegated permissions set.
UPDATE - 08-Dec-2015
There's an easy way to assign roles to Service Principals now. Please see this link for more details: https://azure.microsoft.com/en-in/documentation/articles/resource-group-create-service-principal-portal/.
Original Response
When you grant access to an application to your Azure subscription, behind the scenes a user is created with Service Principal user type in your Azure AD. The code you're using below assumes that you're using this Service Principal user when getting access token.
var clientCredential = new ClientCredential(clientId, appKey);
var result = authContext.AcquireToken("https://management.core.windows.net/", clientCredential);
var creds = new TokenCloudCredentials(subscriptionId, result.AccessToken);
However by default this user is not granted any permissions (RBAC) on your subscription and that's why you're getting the authorization error.
To solve this problem, what you would need to do is grant appropriate role to this user in your subscription. Now you can use PowerShell to do so or you can do it via code using ADAL library + making some web requests.
What I did was I made use of ADAL library to get access tokens and then used Google Postman (or Fiddler) to do other stuff. In my case, it was a web application. Here's what I did:
I logged in into the application as Global Administrator (Subscription Owner) and got a code. Using that code and ADAL library, I got the access token (let's call it token1).
var authContext = new AuthenticationContext(string.Format("{0}/common", signinEndpoint));//signinEndpoint = https://login.windows.net
var result = await authContext.AcquireTokenByAuthorizationCodeAsync(code, redirectUri, credential);
I copied the tenant id and access token returned to me in result above.
Next thing I did was I found out the object id of the Service Principal user using POSTMAN. This is the GET URL I executed there. For Authorization header, you would need to use Bearer {token1}.
https://graph.windows.net/{subscription-id}/servicePrincipals?api-version=1.5&$filter=appId eq '{app-client-id}'
After that I acquired another access token (let's call it token2) for Service Management API operation using the code below:
authContext = new AuthenticationContext(string.Format("{0}/{1}", signinEndpoint, result.TenantId));
result = await authContext.AcquireTokenSilentAsync(serviceManagementApiEndpoint, credential, new UserIdentifier(request.UserInfo.UniqueId, UserIdentifierType.UniqueId));//serviceManagementApiEndpoint = https://management.core.windows.net/
After that I listed the roles in my subscription and picked the role I wanted to assign to my Service Principal user. In my case, I wanted to assign a Reader role so I noted down the role's id. For Authorization header, you would need to use Bearer {token2}.
https://management.azure.com/subscriptions/{subscription-id}/providers/Microsoft.Authorization/roleDefinitions?api-version=2015-06-01
Next is assignment of this role to the user. For this I created a guid as role assignment id and used the following URL:
https://management.azure.com/subscriptions/{subscription-id}/providers/microsoft.authorization/roleassignments/{role-assignment-id}?api-version=2015-06-01
It's going to be a PUT request and this was the request body would be something like:
{
"properties":
{
"roleDefinitionId": "{role id of the selected role from step 5}",
"principalId": "{id of the service principal from step 3"
}
}
Please ensure that the content-type of the request is set as application/json;odata=verbose and not application/json.
That's pretty much it! After that your code should work just fine :)
Give it a try and see what happens.