Upload Azure Batch Job Application Package programmatically - azure

I have found how to upload/manage Azure Batch job Application Packages through the UI:
https://learn.microsoft.com/en-us/azure/batch/batch-application-packages
And how to upload and manage Resource Packages programmatically:
https://github.com/Azure/azure-batch-samples/tree/master/CSharp/GettingStarted/02_PoolsAndResourceFiles
But I can't quite seem to put 2 and 2 together on how to manage Application Packages programmatically. Is there an API endpoint we can call to upload/manage an Application Package when setting up a batch job?

Since this is not quite straightforward, I'll write down my findings.
These are the steps to programmatically upload Application Packages via an application that is unattended - no user input (e.g. Azure credentials) is needed.
In Azure Portal:
Create the Azure Batch application
Create a new Azure AD application (as Application Type use Web app / API)
Follow these steps to create the secret key and assign the role to the Azure Batch account
Note down the following credentials/ids:
Azure AD application id
Azure AD application secret key
Azure AD tenant id
Subscription id
Batch account name
Batch account resource group name
In your code:
Install NuGet packages Microsoft.Azure.Management.Batch, WindowsAzure.Storage and Microsoft.IdentityModel.Clients.ActiveDirectory
Get the access token and create the BatchManagementClient
Call the ApplicationPackageOperationsExtensions.CreateAsync method, which should return an ApplicationPackage
ApplicationPackage contains the StorageUrl which can now be used to upload the Application Package via the storage API
After you have uploaded the ApplicationPackage you have to activate it via ApplicationPackageOperationsExtensions.ActivateAsync
Put together the whole code looks something like this:
private const string ResourceUri = "https://management.core.windows.net/";
private const string AuthUri = "https://login.microsoftonline.com/" + "{TenantId}";
private const string ApplicationId = "{ApplicationId}";
private const string ApplicationSecretKey = "{ApplicationSecretKey}";
private const string SubscriptionId = "{SubscriptionId}";
private const string ResourceGroupName = "{ResourceGroupName}";
private const string BatchAccountName = "{BatchAccountName}";
private async Task UploadApplicationPackageAsync() {
// get the access token
var authContext = new AuthenticationContext(AuthUri);
var authResult = await authContext.AcquireTokenAsync(ResourceUri, new ClientCredential(ApplicationId, ApplicationSecretKey)).ConfigureAwait(false);
// create the BatchManagementClient and set the subscription id
var bmc = new BatchManagementClient(new TokenCredentials(authResult.AccessToken)) {
SubscriptionId = SubscriptionId
};
// create the application package
var createResult = await bmc.ApplicationPackage.CreateWithHttpMessagesAsync(ResourceGroupName, BatchAccountName, "MyPackage", "1.0").ConfigureAwait(false);
// upload the package to the blob storage
var cloudBlockBlob = new CloudBlockBlob(new Uri(createResult.Body.StorageUrl));
cloudBlockBlob.Properties.ContentType = "application/x-zip-compressed";
await cloudBlockBlob.UploadFromFileAsync("myZip.zip").ConfigureAwait(false);
// create the application package
var activateResult = await bmc.ApplicationPackage.ActivateWithHttpMessagesAsync(ResourceGroupName, BatchAccountName, "MyPackage", "1.0", "zip").ConfigureAwait(false);
}

Azure Batch Application Packages management operations occur on the management plane. The MSDN docs for this namespace are here:
https://learn.microsoft.com/en-us/dotnet/api/microsoft.azure.management.batch
The nuget package for Microsoft.Azure.Management.Batch is here:
https://www.nuget.org/packages/Microsoft.Azure.Management.Batch/
And the following sample shows management plane operations in C#, although it is for non-application package operations:
https://github.com/Azure/azure-batch-samples/tree/master/CSharp/AccountManagement

Related

How to Get Azure Event Hub Connection String in C#?

Given a Event Hub Name, how can I get connection string in C#?
I googled a bit, but nothing useful found so far.
Thanks
Using AAD authentication for an EventHub
var credential = new DefaultAzureCredential();
// or use
// var credential = new Azure.Identity.ClientSecretCredential("tenantId", "clientId", "clientSecret");
EventHubProducerClient producerClient = new EventHubProducerClient(txtNamespace.Text, txtEventHub.Text, credential
var consumerClient = new EventHubConsumerClient(EventHubConsumerClient.DefaultConsumerGroupName, txtNamespace.Text, txtEventHub.Text, credential)
Full example and docs
Acquiring the Connection Strings of configured Access Policies
You can use these two Nuget packages:
Azure.ResourceManager.EventHubs
Azure.Identity
Then you can use the resource group name and the eventhub name to retrieve the connection string. You will need to iterate the subscriptions and resource groups if you don't have this information.
using Azure.Identity;
using Azure.ResourceManager;
using Azure.ResourceManager.EventHubs;
ArmClient client = new ArmClient(new DefaultAzureCredential());
// Or use
// ArmClient client = new ArmClient(new Azure.Identity.ClientSecretCredential("tenantId", "clientId", "clientSecret"));
var subscription = await client.GetDefaultSubscriptionAsync();
var resourceGroup = await subscription.GetResourceGroupAsync("myresourcegroup");
var eventhubNamespace = await resourceGroup.Value.GetEventHubsNamespaceAsync("namespacename");
var rules = eventhubNamespace.Value.GetEventHubsNamespaceAuthorizationRules();
foreach (var rule in rules)
{
var keys = await rule.GetKeysAsync();
Console.WriteLine(keys.Value.PrimaryConnectionString);
Console.WriteLine(keys.Value.SecondaryConnectionString);
}
Not sure if this is what you mean, but if you want to access an Event Hub through C# you need to provide the EH connection string into your code. This can be retrieved by adding a Shared access policy for the Event hub that you are trying to access.
Edit: If you are trying to actually create the connection string yourself you could follow this sample where you create the SAS-token yourself. But you would still need to provide the Primary key that is set on the policy from Azure.

What permissions are needed to run the adf pipeline api

What permissions are needed to run the azure data factory management apis. For instance I am trying to execute the Pipeline runs, Query by factory api in a web activity.
Error:
User configuration issue
{"error":{"code":"AuthorizationFailed","message":"The client with object id does not have authorization to perform action 'Microsoft.DataFactory/factories/pipelineruns/read' over scope '/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.DataFactory/factories/{factoryName}/pipelineruns/{runId}' or the scope is invalid. If access was recently granted, please refresh your credentials."}}
Could you please guide me on how to pass the credentials and get the token for GET and POST methods.
You need to create an Azure Active Directory Application that can access your Data factory.
Create an Azure Active Directory application, For the sign-on URL, you can provide a dummy URL (https://contoso.org/exampleapp).
Get values for signing in, get the application ID and tenant ID, and note down these values that you use later.
Certificates and secrets, get the authentication key, and note down this value that you use later in this tutorial.
Assign the application to a role, assign the application to the Contributor role at the subscription level so that the application can create data factories in the subscription.
After you do the above steps, you need to create the DataFactoryManagementClient and authenticate your application using the below code snippet:
var context = new AuthenticationContext("https://login.microsoftonline.com/" + tenantID);
ClientCredential cc = new ClientCredential(applicationId, authenticationKey);
AuthenticationResult result = context.AcquireTokenAsync("https://management.azure.com/", cc).Result;
ServiceClientCredentials cred = new TokenCredentials(result.AccessToken);
var client = new DataFactoryManagementClient(cred) {
SubscriptionId = subscriptionId };
Once you've authenticated your application, you can start the Pipeline run using the below code snippet:
// Create a pipeline run
Console.WriteLine("Creating pipeline run...");
CreateRunResponse runResponse = client.Pipelines.CreateRunWithHttpMessagesAsync(
resourceGroup, dataFactoryName, pipelineName, parameters: parameters
).Result.Body;
Console.WriteLine("Pipeline run ID: " + runResponse.RunId);
Below is the complete code of a console application to run an Azure Data Factory Pipeline:
using Microsoft.Azure.Management.DataFactory;
using Microsoft.Azure.Management.DataFactory.Models;
using Microsoft.IdentityModel.Clients.ActiveDirectory;
using Microsoft.Rest;
using System;
namespace ADF
{
class Program
{
static void Main(string[] args)
{
// Set variables
string tenantID = "<your tenant ID>";
string applicationId = "<your application ID>";
string authenticationKey = "<your authentication key for the application>";
string subscriptionId = "<your subscription ID where the data factory resides>";
string resourceGroup = "<your resource group where the data factory resides>";
string dataFactoryName ="<specify the name of data factory to create. It must be globally unique.>";
string pipelineName = "<specify the name of pipeline to run. It must be globally unique.>";
// Authenticate and create a data factory management client
var context = new AuthenticationContext("https://login.microsoftonline.com/" + tenantID);
ClientCredential cc = new ClientCredential(applicationId, authenticationKey);
AuthenticationResult result = context.AcquireTokenAsync("https://management.azure.com/", cc).Result;
ServiceClientCredentials cred = new TokenCredentials(result.AccessToken);
var client = new DataFactoryManagementClient(cred)
{
SubscriptionId = subscriptionId
};
// Create a pipeline run
Console.WriteLine("Creating pipeline run...");
CreateRunResponse runResponse = client.Pipelines.CreateRunWithHttpMessagesAsync(
resourceGroup, dataFactoryName, pipelineName
).Result.Body;
Console.WriteLine("Pipeline run ID: " + runResponse.RunId);
}
}
}
Don't forget to add the Nuget Packages:
Install-Package Microsoft.Azure.Management.DataFactory
Install-Package Microsoft.Azure.Management.ResourceManager -IncludePrerelease
Install-Package Microsoft.IdentityModel.Clients.ActiveDirectory

Possible to update Azure Web App Setting with LinqPad?

I wondering if it's possible to update Azure Web App Configuration settings using the LinqPad application? For reference of where in the web app that needs to be updated, please see image below.
I know it's possible to update the Networking Settings using LinqPad (a colleague created the script). I have the login credentials (Microsoft.Rest.Azure.Authentication.ApplicationTokenProvider) and subscription id (Microsoft.Azure.Management.WebSites.WebSiteManagementClient). Yet, when try to use WebApps (from the 'Microsoft.Azure.Management.WebSites' assembly/Namespace), I'm not sure what I'm suppose to use or if it's even possible to do this.
Yes, it's possible. You can create a Service Principle(use azure cli or azure portal) first for authentication, then install this package Microsoft.Azure.Management.Fluent.
Then use the code below to Add or Update Application Settings in azure portal:
public static void UpdateSetting(string key, string value)
{
string tenantId = "xx";
string clientSecret = "xx";
string clientId = "xxx";
string subscriptionId = "xx";
//resource group name
string rg_name = "xx";
//azure web app name
string app_name = "xxx";
var azureCredentials = new AzureCredentials(new
ServicePrincipalLoginInformation
{
ClientId = clientId,
ClientSecret = clientSecret
}, tenantId, AzureEnvironment.AzureGlobalCloud);
var myazure = Azure
.Configure()
.Authenticate(azureCredentials)
.WithSubscription(subscriptionId);
var webapp = myazure.WebApps.GetByResourceGroup(rg_name, app_name);
webapp.Update()
.WithAppSetting(key, value)
.Apply();
}

Reference to type 'IResourceGroups' claims it is defined in 'Microsoft.Azure.Management.ResourceManager.Fluent', but it could not be found

I am getting this error when trying to call azure.ResourceGroups.List() where azure is an instance of Iazure.
project.json
Please have a try to following code to get list of resources. I test it on my side, it works correctly. We also could use the Resources - List By Resource Group Rest API to do that.
I install Microsoft.Azure.Management.ResourceManager.Fluent 1.0.0 as you and use console app to test. Here is my testing code:
var credentials = SdkContext.AzureCredentialsFactory.FromServicePrincipal(clientId, clientSecret, tenantId, AzureEnvironment.AzureGlobalCloud);
var subscriptionId = "your_subscriptionId";
var azure = new ResourceManagementClient(credentials) { SubscriptionId = subscriptionId };
var resource = azure.ResourceGroups.ListAsync().Result;

Not able to run Azure Data Factory Pipeline using Visual Studio 2015

I have followed the Azure documentation steps to create a simple Copy Data Factory from Blob to SQL.
Now I want to run the pipeline through VS code.
I have checked the authentication keys and Roles assigned are correct.
Below is the code -
var context = new AuthenticationContext("https://login.windows.net/" + tenantID);
ClientCredential cc = new ClientCredential(applicationId, authenticationKey);
AuthenticationResult result = context.AcquireTokenAsync("https://management.azure.com/", cc).Result;
ServiceClientCredentials cred = new TokenCredentials(result.AccessToken);
var client = new DataFactoryManagementClient(cred) { SubscriptionId = subscriptionId };
Console.WriteLine("Creating pipeline run...");
var st = client.Pipelines.Get(resourceGroup, dataFactoryName, pipelineName);
CreateRunResponse runResponse = client.Pipelines.CreateRunWithHttpMessagesAsync(resourceGroup, dataFactoryName, pipelineName).Result.Body;
Console.WriteLine("Pipeline run ID: " + runResponse.RunId);
However, I get Forbidden error.
The client 'xxxx' with object id 'xxxx' does not have authorization to perform action 'Microsoft.DataFactory/factories/pipelines/read' over scope '/subscriptions/xxxxx/resourceGroups/'
How can I fix this?
How can I fix this?
According to the exception message that it indicates that you don't assign the corresponding role to application to access the data factory.
I test your code with Azure Datafactory(V2) on my side , it works correctly. The following is my details steps.
Registry an Azure AD WebApp application.
Get the clientId and clientscret from created Application.
Assign the role the application to access the datafactory.
Test code on my side.

Resources