What permissions are needed to run the azure data factory management apis. For instance I am trying to execute the Pipeline runs, Query by factory api in a web activity.
Error:
User configuration issue
{"error":{"code":"AuthorizationFailed","message":"The client with object id does not have authorization to perform action 'Microsoft.DataFactory/factories/pipelineruns/read' over scope '/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.DataFactory/factories/{factoryName}/pipelineruns/{runId}' or the scope is invalid. If access was recently granted, please refresh your credentials."}}
Could you please guide me on how to pass the credentials and get the token for GET and POST methods.
You need to create an Azure Active Directory Application that can access your Data factory.
Create an Azure Active Directory application, For the sign-on URL, you can provide a dummy URL (https://contoso.org/exampleapp).
Get values for signing in, get the application ID and tenant ID, and note down these values that you use later.
Certificates and secrets, get the authentication key, and note down this value that you use later in this tutorial.
Assign the application to a role, assign the application to the Contributor role at the subscription level so that the application can create data factories in the subscription.
After you do the above steps, you need to create the DataFactoryManagementClient and authenticate your application using the below code snippet:
var context = new AuthenticationContext("https://login.microsoftonline.com/" + tenantID);
ClientCredential cc = new ClientCredential(applicationId, authenticationKey);
AuthenticationResult result = context.AcquireTokenAsync("https://management.azure.com/", cc).Result;
ServiceClientCredentials cred = new TokenCredentials(result.AccessToken);
var client = new DataFactoryManagementClient(cred) {
SubscriptionId = subscriptionId };
Once you've authenticated your application, you can start the Pipeline run using the below code snippet:
// Create a pipeline run
Console.WriteLine("Creating pipeline run...");
CreateRunResponse runResponse = client.Pipelines.CreateRunWithHttpMessagesAsync(
resourceGroup, dataFactoryName, pipelineName, parameters: parameters
).Result.Body;
Console.WriteLine("Pipeline run ID: " + runResponse.RunId);
Below is the complete code of a console application to run an Azure Data Factory Pipeline:
using Microsoft.Azure.Management.DataFactory;
using Microsoft.Azure.Management.DataFactory.Models;
using Microsoft.IdentityModel.Clients.ActiveDirectory;
using Microsoft.Rest;
using System;
namespace ADF
{
class Program
{
static void Main(string[] args)
{
// Set variables
string tenantID = "<your tenant ID>";
string applicationId = "<your application ID>";
string authenticationKey = "<your authentication key for the application>";
string subscriptionId = "<your subscription ID where the data factory resides>";
string resourceGroup = "<your resource group where the data factory resides>";
string dataFactoryName ="<specify the name of data factory to create. It must be globally unique.>";
string pipelineName = "<specify the name of pipeline to run. It must be globally unique.>";
// Authenticate and create a data factory management client
var context = new AuthenticationContext("https://login.microsoftonline.com/" + tenantID);
ClientCredential cc = new ClientCredential(applicationId, authenticationKey);
AuthenticationResult result = context.AcquireTokenAsync("https://management.azure.com/", cc).Result;
ServiceClientCredentials cred = new TokenCredentials(result.AccessToken);
var client = new DataFactoryManagementClient(cred)
{
SubscriptionId = subscriptionId
};
// Create a pipeline run
Console.WriteLine("Creating pipeline run...");
CreateRunResponse runResponse = client.Pipelines.CreateRunWithHttpMessagesAsync(
resourceGroup, dataFactoryName, pipelineName
).Result.Body;
Console.WriteLine("Pipeline run ID: " + runResponse.RunId);
}
}
}
Don't forget to add the Nuget Packages:
Install-Package Microsoft.Azure.Management.DataFactory
Install-Package Microsoft.Azure.Management.ResourceManager -IncludePrerelease
Install-Package Microsoft.IdentityModel.Clients.ActiveDirectory
Related
I wondering if it's possible to update Azure Web App Configuration settings using the LinqPad application? For reference of where in the web app that needs to be updated, please see image below.
I know it's possible to update the Networking Settings using LinqPad (a colleague created the script). I have the login credentials (Microsoft.Rest.Azure.Authentication.ApplicationTokenProvider) and subscription id (Microsoft.Azure.Management.WebSites.WebSiteManagementClient). Yet, when try to use WebApps (from the 'Microsoft.Azure.Management.WebSites' assembly/Namespace), I'm not sure what I'm suppose to use or if it's even possible to do this.
Yes, it's possible. You can create a Service Principle(use azure cli or azure portal) first for authentication, then install this package Microsoft.Azure.Management.Fluent.
Then use the code below to Add or Update Application Settings in azure portal:
public static void UpdateSetting(string key, string value)
{
string tenantId = "xx";
string clientSecret = "xx";
string clientId = "xxx";
string subscriptionId = "xx";
//resource group name
string rg_name = "xx";
//azure web app name
string app_name = "xxx";
var azureCredentials = new AzureCredentials(new
ServicePrincipalLoginInformation
{
ClientId = clientId,
ClientSecret = clientSecret
}, tenantId, AzureEnvironment.AzureGlobalCloud);
var myazure = Azure
.Configure()
.Authenticate(azureCredentials)
.WithSubscription(subscriptionId);
var webapp = myazure.WebApps.GetByResourceGroup(rg_name, app_name);
webapp.Update()
.WithAppSetting(key, value)
.Apply();
}
I'm trying to access Azure Data Fabric V2 programmatically.
First, I had created an App Registration in Azure portal and a Client secret. Then I gave Contributor permission to this App registration on the entire suscription, and also in the resource group where my data factory lives.
Using this credentials I'm able to login to the portal and create an DataFactoryManagementClient
private void CreateAdfClient()
{
var authenticationContext = new AuthenticationContext($"https://login.windows.net/{tenantId}");
var credential = new ClientCredential(clientId: appRegistrationClientId, clientSecret: appRegistrationClientkey);
var result = authenticationContext.AcquireTokenAsync(resource: "https://management.core.windows.net/", clientCredential: credential).ConfigureAwait(false).GetAwaiter().GetResult();
if (result == null)
{
throw new InvalidOperationException("Failed to obtain the JWT token");
}
var token = result.AccessToken;
var tokenCloudCredentials = new TokenCloudCredentials(subscriptionId, token);
datafactoryClient = new DataFactoryManagementClient(tokenCloudCredentials);
}
However, when I try to get my pipeline with
var pipeline = datafactoryClient.Pipelines.Get(resourceGroup, dataFactory, pipelineName);
it throws an error:
System.Private.CoreLib: Exception while executing function:
StartRawMeasuresSync. Microsoft.Azure.Management.DataFactories:
ResourceNotFound: The Resource
'Microsoft.DataFactory/dataFactories/MyPipeline' under resource group
'MyResGroup' was not found.
I had verified that the resource group, the data factory name and the pipeline name are correct, but it keeps throwing this error.
I had the same issue, and it was due to referencing the Nuget package for Azure Data Factory v1 instead of v2.
Version 1: Microsoft.Azure.Management.DataFactories
Version 2: Microsoft.Azure.Management.DataFactory
I have followed the Azure documentation steps to create a simple Copy Data Factory from Blob to SQL.
Now I want to run the pipeline through VS code.
I have checked the authentication keys and Roles assigned are correct.
Below is the code -
var context = new AuthenticationContext("https://login.windows.net/" + tenantID);
ClientCredential cc = new ClientCredential(applicationId, authenticationKey);
AuthenticationResult result = context.AcquireTokenAsync("https://management.azure.com/", cc).Result;
ServiceClientCredentials cred = new TokenCredentials(result.AccessToken);
var client = new DataFactoryManagementClient(cred) { SubscriptionId = subscriptionId };
Console.WriteLine("Creating pipeline run...");
var st = client.Pipelines.Get(resourceGroup, dataFactoryName, pipelineName);
CreateRunResponse runResponse = client.Pipelines.CreateRunWithHttpMessagesAsync(resourceGroup, dataFactoryName, pipelineName).Result.Body;
Console.WriteLine("Pipeline run ID: " + runResponse.RunId);
However, I get Forbidden error.
The client 'xxxx' with object id 'xxxx' does not have authorization to perform action 'Microsoft.DataFactory/factories/pipelines/read' over scope '/subscriptions/xxxxx/resourceGroups/'
How can I fix this?
How can I fix this?
According to the exception message that it indicates that you don't assign the corresponding role to application to access the data factory.
I test your code with Azure Datafactory(V2) on my side , it works correctly. The following is my details steps.
Registry an Azure AD WebApp application.
Get the clientId and clientscret from created Application.
Assign the role the application to access the datafactory.
Test code on my side.
How to enumerate Azure subscriptions and tenants programmatically? This is related to my previous question Login-AzureRmAccount (and related) equivalent(s) in .NET Azure SDK.
Basically I try to replicate the behavior of Login-AzureRmAccount and Get-AzureRmSubscription in desktop or a console application. Thus far I've figured out MSAL seems to always require client ID and tenant ID, so there needs to be some other library to acquire those from. After this I would like to go about creating a service principal programmatically using the most current library, but I suppose that is a subject for further investigation (and questions if needed).
Actually, the Login-AzureRmAccount and Get-AzureRmSubscription use the Microsoft Azure PowerShell app to operate the Azure resource through Resource Manager REST APIs.
To simulate the same operations using REST as PowersShell commands, we can also use this app. However since this app is register on Azure portal(not the v2.0 app) so we are not able to acquire the token using this app via MSAL. We need to use Adal instead of MSAL.
Here is a code sample to list the subscriptions using admin account via Microsoft.WindowsAzure.Management using this app for your reference:
public static void ListSubscriptions()
{
string authority = "https://login.microsoftonline.com/common";
string resource = "https://management.core.windows.net/";
string clientId = "1950a258-227b-4e31-a9cf-717495945fc2";
Uri redirectUri = new Uri("urn:ietf:wg:oauth:2.0:oob");
AuthenticationContext authContext = new AuthenticationContext(authority);
var access_token = authContext.AcquireTokenAsync(resource, clientId, redirectUri, new PlatformParameters (PromptBehavior.Auto)).Result.AccessToken;
var tokenCred = new Microsoft.Azure.TokenCloudCredentials(access_token);
var subscriptionClient = new SubscriptionClient(tokenCred);
foreach (var subscription in subscriptionClient.Subscriptions.List())
{
Console.WriteLine(subscription.SubscriptionName);
}
}
Update:
string resource = "https://management.core.windows.net/";
string clientId = "1950a258-227b-4e31-a9cf-717495945fc2";
string userName = "";
string password = "";
HttpClient client = new HttpClient();
string tokenEndpoint = "https://login.microsoftonline.com/common/oauth2/token";
var body = $"resource={resource}&client_id={clientId}&grant_type=password&username={userName}&password={password}";
var stringContent = new StringContent(body, Encoding.UTF8, "application/x-www-form-urlencoded");
var result = client.PostAsync(tokenEndpoint, stringContent).ContinueWith<string>((response) =>
{
return response.Result.Content.ReadAsStringAsync().Result;
}).Result;
JObject jobject = JObject.Parse(result);
var token = jobject["access_token"].Value<string>();
client.DefaultRequestHeaders.Add("Authorization", $"bearer {token}");
var subcriptions = client.GetStringAsync("https://management.azure.com/subscriptions?api-version=2014-04-01-preview").Result;
Console.WriteLine(subcriptions);
I have found how to upload/manage Azure Batch job Application Packages through the UI:
https://learn.microsoft.com/en-us/azure/batch/batch-application-packages
And how to upload and manage Resource Packages programmatically:
https://github.com/Azure/azure-batch-samples/tree/master/CSharp/GettingStarted/02_PoolsAndResourceFiles
But I can't quite seem to put 2 and 2 together on how to manage Application Packages programmatically. Is there an API endpoint we can call to upload/manage an Application Package when setting up a batch job?
Since this is not quite straightforward, I'll write down my findings.
These are the steps to programmatically upload Application Packages via an application that is unattended - no user input (e.g. Azure credentials) is needed.
In Azure Portal:
Create the Azure Batch application
Create a new Azure AD application (as Application Type use Web app / API)
Follow these steps to create the secret key and assign the role to the Azure Batch account
Note down the following credentials/ids:
Azure AD application id
Azure AD application secret key
Azure AD tenant id
Subscription id
Batch account name
Batch account resource group name
In your code:
Install NuGet packages Microsoft.Azure.Management.Batch, WindowsAzure.Storage and Microsoft.IdentityModel.Clients.ActiveDirectory
Get the access token and create the BatchManagementClient
Call the ApplicationPackageOperationsExtensions.CreateAsync method, which should return an ApplicationPackage
ApplicationPackage contains the StorageUrl which can now be used to upload the Application Package via the storage API
After you have uploaded the ApplicationPackage you have to activate it via ApplicationPackageOperationsExtensions.ActivateAsync
Put together the whole code looks something like this:
private const string ResourceUri = "https://management.core.windows.net/";
private const string AuthUri = "https://login.microsoftonline.com/" + "{TenantId}";
private const string ApplicationId = "{ApplicationId}";
private const string ApplicationSecretKey = "{ApplicationSecretKey}";
private const string SubscriptionId = "{SubscriptionId}";
private const string ResourceGroupName = "{ResourceGroupName}";
private const string BatchAccountName = "{BatchAccountName}";
private async Task UploadApplicationPackageAsync() {
// get the access token
var authContext = new AuthenticationContext(AuthUri);
var authResult = await authContext.AcquireTokenAsync(ResourceUri, new ClientCredential(ApplicationId, ApplicationSecretKey)).ConfigureAwait(false);
// create the BatchManagementClient and set the subscription id
var bmc = new BatchManagementClient(new TokenCredentials(authResult.AccessToken)) {
SubscriptionId = SubscriptionId
};
// create the application package
var createResult = await bmc.ApplicationPackage.CreateWithHttpMessagesAsync(ResourceGroupName, BatchAccountName, "MyPackage", "1.0").ConfigureAwait(false);
// upload the package to the blob storage
var cloudBlockBlob = new CloudBlockBlob(new Uri(createResult.Body.StorageUrl));
cloudBlockBlob.Properties.ContentType = "application/x-zip-compressed";
await cloudBlockBlob.UploadFromFileAsync("myZip.zip").ConfigureAwait(false);
// create the application package
var activateResult = await bmc.ApplicationPackage.ActivateWithHttpMessagesAsync(ResourceGroupName, BatchAccountName, "MyPackage", "1.0", "zip").ConfigureAwait(false);
}
Azure Batch Application Packages management operations occur on the management plane. The MSDN docs for this namespace are here:
https://learn.microsoft.com/en-us/dotnet/api/microsoft.azure.management.batch
The nuget package for Microsoft.Azure.Management.Batch is here:
https://www.nuget.org/packages/Microsoft.Azure.Management.Batch/
And the following sample shows management plane operations in C#, although it is for non-application package operations:
https://github.com/Azure/azure-batch-samples/tree/master/CSharp/AccountManagement