I have a pipeline in Azure Data Factory that starts by going to a REST API to obtain an authorization token. In order to obtain this token, the initial POST request needs to contain a username, password, and private key in the request body. It looks like this:
{
"Username": "<myusername>",
"Password": "<mypassword>",
"PrivateKey":"<privatekey>"
}
Currently I just have this stored as plain text in the Web activity in ADF
To me this doesn't seem very secure and I'm wondering if there is a better way to store this JSON string. I've looked into Azure Key Vault, but that seems to be for storing "data store" credentials.... What is the best practice for storing credentials like this to be used by ADF?
You can save the individual values as Secrets in Key vault and fetch them individually via Web activity from KeyVault with masked output thereby making your ADF secure.
Below GITHUb location contains the Pipeline JSON :
https://github.com/NandanHegde15/Azure-DataFactory-Generic-Pipelines/blob/main/Get%20Secret%20From%20KeyVault/Pipeline/GetSecretFromKeyVault.json
Other way would be to use SecureString Parameter
But would say to avoid using the parameter and leverage the Key Vault
the credentials can be saved in the key vault secret
The secret can be called for authentication in the linked service that connects to the required base url
Refer https://learn.microsoft.com/en-us/azure/data-factory/connector-http?tabs=data-factory#create-a-linked-service-to-an-http-source-using-ui
Related
We need to set up continuous data export from Kusto to an external table for data older than 60 days into ADLS. Based on documentation it looks like we should use AAD token to Productionize, however, the documentation does not clearly specify the process to generate AAD token.
We also followed the Microsoft Documentation to acquire the access token and registered an application, generated client secret key.
Need some help/suggestion regarding the process to generate the token
Append ;token=AadToken to the URI, with AadToken being a base-64 encoded AAD access token (make sure the token is for the resource https://storage.azure.com/).
.create external table logs (ing_dt:date,record:string)
kind=adl
partition by bin(ing_dt, 1d)
dataformat=json
(
h#'abfss://filesystem#<storageaccountname>.dfs.core.windows.net/input;token=*****)
with
(
docstring = "External Table",
folder = "External",
nameprefix= "ext"
)
.create-or-alter continuous-export exp_logs
to table logs
with
(intervalBetweenRuns=1h)
<| source
| extend ing_dt=format_datetime(ingestion_time(),'yyyy-MM-dd')
| project todatetime(ing_dt), record
| where todatetime(ing_dt) < ago(60d)
The documentation you linked to references AAD client libraries to generate access token (see this section). For example, AcquireToken method if you're using .NET. However, AAD access tokens have a very short expiry (can be extended up to 1 day, I think) so it's not recommended to use those for continuous export, as ADX cannot renew the token and the export will start failing once the token has expired. The recommended authentication method for continuous export is either using an account key, or a SAS key (with a very long expiry). Both are documented here.
I can understand that we can do JWT validation easily by using symmetric keys(HS256) and open id for the RS256.
In my case I want to validate the JWT using existing private key that I already have. Payload will be encoded by public key, and at the API Management side I want to validate that token with my own predefined private key. Is there a way to do that?
It should be possible to decrypt it manually if needed. But perhaps using output-token-variable-name attribute of validate-jwt policy will be a simpler way out.
Well I can't decrypt the payload but we managed to make it pass di validate-jwt set open id url to a mock openid-configuration file, and convert our *.pem key into jwt key, and then refer the jwtk-url in the mock openid-configuration to that jwtkey json.
I have created an AAD app as per https://learn.microsoft.com/en-us/azure/storage/common/storage-auth-aad-app.
The access is given to the azure storage account for the AAD app created.
Got the client id and client secret.
To create a user delegation key and user delegation sas, I am using the approach and code as defined in
https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blob-user-delegation-sas-create-dotnet.
(set environment variables as mentioned in article).
I am able to generate the user delegation key using method GetUserDelegationSasBlob.
The container and blob file is existing one.
Now I am using the method ReadBlobWithSasAsync to read the contents of the blob using the SAS uri as generated above.
But, I get error as below.
This request is not authorized to perform this operation using this
permission. RequestId:5d127eaf-101e-00be-6666-6a3875000000
Time:2019-09-13T19:04:15.4109144Z
Status: 403 (This request is not authorized to perform this operation
using this permission.)
ErrorCode: AuthorizationPermissionMismatch
In another approach, I am generating the user delegation key using rest api.
https://learn.microsoft.com/en-us/rest/api/storageservices/get-user-delegation-key
I am able to get user delegation key in xml format.
I am creating SAS from it as per steps in
https://learn.microsoft.com/en-us/rest/api/storageservices/create-user-delegation-sas
For signature, I am using this code, using StringToSign and secret value as received from delegation key.
var encoding = new System.Text.ASCIIEncoding();
byte[] keyByte = encoding.GetBytes(secret);
byte[] messageBytes = encoding.GetBytes(ToSign);
using (var hmacsha256 = new HMACSHA256(keyByte))
{
byte[] hashmessage = hmacsha256.ComputeHash(messageBytes);
String sig= Convert.ToBase64String(hashmessage);
}
I am doing the GET request.
I have tried various set of parameter values, like,
sr: b and c
sks: b and c
sp: racwd and r and rw and few more
skv and sv is 2018-11-09 because this version is required for creating user delegation key.
But the GET api returns the error.
AuthenticationFailed
Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the
signature. RequestId:e4bc8f0f-d01e-0046-7367-6af368000000
Time:2019-09-13T19:12:27.7780695Z
Signature fields not well formed.
Try to assign the Storage Blob Data Contributor role to the storage account.
The Reader role is an Azure Resource Manager role that permits users to view storage account resources, but not modify them. It does not provide read permissions to data in Azure Storage, but only to account management resources.
Refer to this article.
In Azure Portal > Key vaults > Secrets, I have secrets with json values (I did not create it). Something like:
...
"SubscriptionId": "XXXXXXX",
"BaseAuthUri": "https://login.microsoftonline.com/XXXXX/oauth/authorize?client_id="&api-version=
...
I would like to add another url value to it. How can I edit the
secrets with Azure portal?
How the value of api-version set?
Thanks
You can only change secret attributes such as expiration date, activation date. You cannot change secret's value programatically or via Azure Portal. If you want to update your secret without creating a new vault (meaning the secret identifier still remains intact) you can create a new version of the existing secret.
If the secret value contains the variables to get authorization code, you don't need api version because the URI you call is the authorization endpoint.
I have uploaded the SSl certificate to keyvault. Now I wanted to upload this certificate to my Cloud service
With this API I am able to access the Cert from keyvault
GET https://{vaultBaseUrl}/secrets/{secret-name}/{secret-version}?api-version={api-version}
But to upload certifcate, I need the password too.
"path": "subscriptions/%sub_Id%/resourceGroups/%rg_Name%/providers/Microsoft.ClassicCompute/domainNames/%cloudService_Name%/servicecertificates/SHA1-%THUMBPRINT%",
"body": {
"thumbprintAlgorithm": "SHA1",
"thumbprint": "%THUMBPRINT%",
"data": "%base64encodedcert%",
"certificateFormat": "pfx",
"password": "password" << this is in plain text I believe
}
How to get the password from keyvault or upload service certificate without password or with secret URI ?
When you need to pass a secure value (like a password) as a parameter during deployment, you can retrieve the value from an Azure Key Vault. You retrieve the value by referencing the key vault and secret in your parameter file. The value is never exposed because you only reference its key vault ID. You do not need to manually enter the value for the secret each time you deploy the resources.
For more details, refer to this documentation. Use Key Vault to pass secure parameter.
Also, check the Azure Key Vault REST API reference for examples.