We need to set up continuous data export from Kusto to an external table for data older than 60 days into ADLS. Based on documentation it looks like we should use AAD token to Productionize, however, the documentation does not clearly specify the process to generate AAD token.
We also followed the Microsoft Documentation to acquire the access token and registered an application, generated client secret key.
Need some help/suggestion regarding the process to generate the token
Append ;token=AadToken to the URI, with AadToken being a base-64 encoded AAD access token (make sure the token is for the resource https://storage.azure.com/).
.create external table logs (ing_dt:date,record:string)
kind=adl
partition by bin(ing_dt, 1d)
dataformat=json
(
h#'abfss://filesystem#<storageaccountname>.dfs.core.windows.net/input;token=*****)
with
(
docstring = "External Table",
folder = "External",
nameprefix= "ext"
)
.create-or-alter continuous-export exp_logs
to table logs
with
(intervalBetweenRuns=1h)
<| source
| extend ing_dt=format_datetime(ingestion_time(),'yyyy-MM-dd')
| project todatetime(ing_dt), record
| where todatetime(ing_dt) < ago(60d)
The documentation you linked to references AAD client libraries to generate access token (see this section). For example, AcquireToken method if you're using .NET. However, AAD access tokens have a very short expiry (can be extended up to 1 day, I think) so it's not recommended to use those for continuous export, as ADX cannot renew the token and the export will start failing once the token has expired. The recommended authentication method for continuous export is either using an account key, or a SAS key (with a very long expiry). Both are documented here.
Related
Upon successful login with Azure AD (via MSAL 2), I get the following object:
environment:"login.windows.net"
homeAccountId:"ID_1.ID_2"
idTokenClaims:Object
aio:"xxx"
aud:"xxx"
exp:1661778482
iat:1661774582
idp:"https://sts.windows.net/ID_2"
iss:"https://login.microsoftonline.com/TENANT_ID"
name:"xxx"
nbf:1661774582
nonce:"xxx"
oid:"ID_3"
preferred_username:"xxx"
rh:"xxx"
sub:"xxx"
tid:"xxx"
uti:"xxx"
ver:"xxx"
localAccountId:"ID_3"
name:"xxx"
nativeAccountId:undefined
tenantId:"TENANT_ID"
It seems that according to the documentation, if I understand correctly, you need to take idTokenClaims.oid which is equal to localAccountId.
But I thought that if you want real permanent UserId to store in database, then you should look at the properties of the user through the Azure portal, and takes its Object ID field. I am a bit confused that Azure user's Object ID field also existed in the returned object (upon Azure AD login via MSAL). Object ID is in the homeAccountId field, namely ID_1, but for some reason it is followed by a dot and some other incomprehensible ID_2, which is present in idTokenClaims.idp.
Please tell me what is correct to store in the database as UserId in the CreatedBy field for an Azure AD user when authorizing through MSAL 2:
idTokenClaims.oid, aka localAccountId?
or ID_1 from homeAccountId (which equals to the user's Object ID) ?
Object ID i.e. oid is the primary claim that you should use as part of a system of records (plus tenant ID i.e. tid if you are serving multiple tenants).
homeAccountId and localAccountId are MSAL SDK specific terms, but they map to claims in the ID token eventually -homeAccountId is "oid.tid", while localAccountId is simply "oid" as it's only used with ADFS which doesn't have tenancy.
The idTokenClaims in the response payload is the actual decoded ID token from Azure AD, and then MSAL adds further useful metadata to the response.
See also:
Microsoft identity platform ID tokens.
Data access sample
In existing pipeline the REST API key values (client ID and client secret) are expired. So, I have generated new keys values. I am able to generate token and extract API data in Postman using new client id and client secret.
In ADF pipeline, I am giving the new key values to generate token. But when calling API the newly generated token is not working. It is giving me the below error.
Do I need to change anything else?enter image description here
Details Failure happened on ' Source side . ErrorCode RestSourceCallFailed , Type Microsoft.Data
Transfer.Common.Shared.Hybrid DeliveryException , Message The
HttpStatusCode 401 indicates failure .
From the error message it seems to be a permission issue. please make sure that the service principal has enough permissions.
More information about RestAPI Calling in Azure Data Factory use below Reference in which we have detailed information about
• REST linked service using UI
• Connector configuration details
• AAD service principal authentication
• API key authentication
• Copy data from REST connector into Azure Data Lake Storage in JSON format using OAuth.
Reference:
https://learn.microsoft.com/en-us/azure/data-factory/connector-rest?tabs=data-factory#how-to-use-this-solution-template
I have a pipeline in Azure Data Factory that starts by going to a REST API to obtain an authorization token. In order to obtain this token, the initial POST request needs to contain a username, password, and private key in the request body. It looks like this:
{
"Username": "<myusername>",
"Password": "<mypassword>",
"PrivateKey":"<privatekey>"
}
Currently I just have this stored as plain text in the Web activity in ADF
To me this doesn't seem very secure and I'm wondering if there is a better way to store this JSON string. I've looked into Azure Key Vault, but that seems to be for storing "data store" credentials.... What is the best practice for storing credentials like this to be used by ADF?
You can save the individual values as Secrets in Key vault and fetch them individually via Web activity from KeyVault with masked output thereby making your ADF secure.
Below GITHUb location contains the Pipeline JSON :
https://github.com/NandanHegde15/Azure-DataFactory-Generic-Pipelines/blob/main/Get%20Secret%20From%20KeyVault/Pipeline/GetSecretFromKeyVault.json
Other way would be to use SecureString Parameter
But would say to avoid using the parameter and leverage the Key Vault
the credentials can be saved in the key vault secret
The secret can be called for authentication in the linked service that connects to the required base url
Refer https://learn.microsoft.com/en-us/azure/data-factory/connector-http?tabs=data-factory#create-a-linked-service-to-an-http-source-using-ui
I encoded a bunch of videos in Azure Media Services and created streaming endpoints. Everything works as expected. Now, I'd like to protect the endpoint URL for each video. For example, I'd like to make each URL available conditionally (for example up to an hour per user). Is it possible to define SAS tokens (similar to Azure Storage SAS tokens) for streaming endpoints? Or something similar that could do the job?
You can read offical doc first.
Tutorial: Use DRM dynamic encryption and license delivery service
After consulting the information, there should be no similar use of SAS to protect media resources. Generally, SAS is used to access resources in Storage.
In this tutorial, we specify for the content key policy to have a token restriction. The token-restricted policy must be accompanied by a token issued by a security token service (STS). Media Services supports tokens in the JWT formats and that's what we configure in the sample.
JwtSecurityToken token = new JwtSecurityToken(
issuer: issuer,
audience: audience,
claims: claims,
notBefore: DateTime.Now.AddMinutes(-5),
expires: DateTime.Now.AddMinutes(60),
signingCredentials: cred);
I think expires can meet your needs.
Build a streaming URL
// Look for just the DASH path and generate a URL for the Azure Media Player to playback the encrypted DASH content.
// Note that the JWT token is set to expire in 1 hour.
if (path.StreamingProtocol == StreamingPolicyStreamingProtocol.Dash)
{
uriBuilder.Path = path.Paths[0];
dashPath = uriBuilder.ToString();
}
I have created an AAD app as per https://learn.microsoft.com/en-us/azure/storage/common/storage-auth-aad-app.
The access is given to the azure storage account for the AAD app created.
Got the client id and client secret.
To create a user delegation key and user delegation sas, I am using the approach and code as defined in
https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blob-user-delegation-sas-create-dotnet.
(set environment variables as mentioned in article).
I am able to generate the user delegation key using method GetUserDelegationSasBlob.
The container and blob file is existing one.
Now I am using the method ReadBlobWithSasAsync to read the contents of the blob using the SAS uri as generated above.
But, I get error as below.
This request is not authorized to perform this operation using this
permission. RequestId:5d127eaf-101e-00be-6666-6a3875000000
Time:2019-09-13T19:04:15.4109144Z
Status: 403 (This request is not authorized to perform this operation
using this permission.)
ErrorCode: AuthorizationPermissionMismatch
In another approach, I am generating the user delegation key using rest api.
https://learn.microsoft.com/en-us/rest/api/storageservices/get-user-delegation-key
I am able to get user delegation key in xml format.
I am creating SAS from it as per steps in
https://learn.microsoft.com/en-us/rest/api/storageservices/create-user-delegation-sas
For signature, I am using this code, using StringToSign and secret value as received from delegation key.
var encoding = new System.Text.ASCIIEncoding();
byte[] keyByte = encoding.GetBytes(secret);
byte[] messageBytes = encoding.GetBytes(ToSign);
using (var hmacsha256 = new HMACSHA256(keyByte))
{
byte[] hashmessage = hmacsha256.ComputeHash(messageBytes);
String sig= Convert.ToBase64String(hashmessage);
}
I am doing the GET request.
I have tried various set of parameter values, like,
sr: b and c
sks: b and c
sp: racwd and r and rw and few more
skv and sv is 2018-11-09 because this version is required for creating user delegation key.
But the GET api returns the error.
AuthenticationFailed
Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the
signature. RequestId:e4bc8f0f-d01e-0046-7367-6af368000000
Time:2019-09-13T19:12:27.7780695Z
Signature fields not well formed.
Try to assign the Storage Blob Data Contributor role to the storage account.
The Reader role is an Azure Resource Manager role that permits users to view storage account resources, but not modify them. It does not provide read permissions to data in Azure Storage, but only to account management resources.
Refer to this article.