I have a pipeline in Azure Data Factory that starts by going to a REST API to obtain an authorization token. In order to obtain this token, the initial POST request needs to contain a username, password, and private key in the request body. It looks like this:
{
"Username": "<myusername>",
"Password": "<mypassword>",
"PrivateKey":"<privatekey>"
}
Currently I just have this stored as plain text in the Web activity in ADF
To me this doesn't seem very secure and I'm wondering if there is a better way to store this JSON string. I've looked into Azure Key Vault, but that seems to be for storing "data store" credentials.... What is the best practice for storing credentials like this to be used by ADF?
You can save the individual values as Secrets in Key vault and fetch them individually via Web activity from KeyVault with masked output thereby making your ADF secure.
Below GITHUb location contains the Pipeline JSON :
https://github.com/NandanHegde15/Azure-DataFactory-Generic-Pipelines/blob/main/Get%20Secret%20From%20KeyVault/Pipeline/GetSecretFromKeyVault.json
Other way would be to use SecureString Parameter
But would say to avoid using the parameter and leverage the Key Vault
the credentials can be saved in the key vault secret
The secret can be called for authentication in the linked service that connects to the required base url
Refer https://learn.microsoft.com/en-us/azure/data-factory/connector-http?tabs=data-factory#create-a-linked-service-to-an-http-source-using-ui
I have generated a wildcard certificate using Azure's App Service Certificate.
After this Azure asks you to configure the certificate in a KeyVault, and this is where I am getting stuck.
I have a Keyvault which is in the same region, same subscription, same resource group.
I have the necessary permissions to the key vault. I am a User Admin, Cert officer, Secrets officer, contributor to the key vault .
This step is failing with an error
Failed to link certificate with the selected Key Vault. Check below errors for more detail.
The Activity Log under App Service Certificate shows :
Operation Name : Add or Update Certificate
Status : Failed
Summary :
Operation name : Add or Update Certificate
Time stamp : Tue Jan 19 2021 07:10:46 GMT+1100 (Australian Eastern Daylight Time)
Event initiated by: xxxx
Error code: BadRequest
Message : The parameter keyVaultCsmId has an invalid value.
The JSON has the following important information :
"authorization": {
"action": "Microsoft.CertificateRegistration/certificateOrders/certificates/write",
"scope": "/subscriptions/xxx/resourceGroups/yyyyyyy/providers/Microsoft.CertificateRegistration/certificateOrders/InternalWildCard/certificates/InternalWildCard"
}
"properties": {
"statusCode": "BadRequest",
"serviceRequestId": null,
"statusMessage": "{\"Code\":\"BadRequest\",\"Message\":\"The parameter keyVaultCsmId has an invalid value.\",\"Target\":null,\"Details\":[{\"Message\":\"The parameter keyVaultCsmId has an invalid value.\"},{\"Code\":\"BadRequest\"},{\"ErrorEntity\":{\"ExtendedCode\":\"51008\",\"MessageTemplate\":\"The parameter {0} has an invalid value.\",\"Parameters\":[\"keyVaultCsmId\"],\"Code\":\"BadRequest\",\"Message\":\"The parameter keyVaultCsmId has an invalid value.\"}}],\"Innererror\":null}",
"eventCategory": "Administrative",
"entity": "/subscriptions/xxx/resourceGroups/xxx/providers/Microsoft.CertificateRegistration/certificateOrders/InternalWildCard/certificates/InternalWildCard",
"message": "Microsoft.CertificateRegistration/certificateOrders/certificates/write",
"hierarchy": "xxxx"
}
Any help on this or direction you can provide will be really appreciated :)
There should be a service principal in the Azure AD, if not you can create it.
Get-AzADServicePrincipal -DisplayName microsoft.azure.certificateregistration
You need to assign that permission to keyvault via either access policies OR RBAC.
In Azure Key Vault, supported certificate formats are PFX and PEM.
• .pem file format contains one or more X509 certificate files.
• .pfx file format is an archive file format for storing several cryptographic objects in a single file i.e. server certificate (issued for your domain), a matching private key, and may optionally include an intermediate CA.
Certificates used by App Service first needs to be converted to (and marked as) application/x-pkcs12. Re-importing the cert from a pfx file with the --password parameter (az keyvault certificate import), and after that import it from the key vault to the webapp might help. You may refer to this Blog might be helpful.
Also, look if Cert and the Key Vault are in their original resource group.
Additional details: https://learn.microsoft.com/en-us/azure/key-vault/certificates/tutorial-import-certificate
https://azure.github.io/AppService/2016/05/24/Deploying-Azure-Web-App-Certificate-through-Key-Vault.html
Stuff which I didn't mention in the original question..
The existing key vault Permission Model is using RBAC for policies which is in preview [ Azure role-based access control (preview) ].
It has "Selected Networks" open and allows Trusted MS Networks..
It already has another wildcard cert in it as a secret ( different name )
I temporarily switched the Key Vault Permission Model to Vault access policy, and tried linking - and it worked.
Looks like a bug with the Permission Model using Azure RBAC.
To fix the issue:
https://github.com/Azure/azure-quickstart-templates/tree/master/quickstarts/microsoft.web/app-service-certificate-standard
By default, 'Microsoft.CertificateRegistration' and 'Microsoft.Web' RPs don't have access to the Key Vault specified in the template hence you need to authorize these RPs by executing the following PowerShell commands before deploying the template:
Login-AzureRmAccount
Set-AzureRmContext -SubscriptionId AZURE_SUBSCRIPTION_ID
Set-AzureRmKeyVaultAccessPolicy -VaultName KEY_VAULT_NAME -ServicePrincipalName f3c21649-0979-4721-ac85-b0216b2cf413 -PermissionsToSecrets get,set,delete
Set-AzureRmKeyVaultAccessPolicy -VaultName KEY_VAULT_NAME -ServicePrincipalName abfa0a7c-a6b6-4736-8310-5855508787cd -PermissionsToSecrets get
ServicePrincipalName parameter represents these RPs in user tenant and will remain same for all Azure subscriptions. This is a onetime operation. Once you have a configured a Key Vault properly, you can use it to store as many App Service Certificates as you want without executing these PowerShell commands again.
I have a Function App Hosted in Azure. I access the functions via a Key in the Host Keys that i created , MyKey. This is linked to a secret in the KeyVault via the following format :
#Microsoft.KeyVault(SecretUri=secret_uri_with_version)
Now if the Key inside the function App is renewed, I lose the edited value as above and it is replaced with a random key value .
How can i make it so that if someone renews the key in the function app then the link to the Key Vault is not lost ?
In Azure Portal > Key vaults > Secrets, I have secrets with json values (I did not create it). Something like:
...
"SubscriptionId": "XXXXXXX",
"BaseAuthUri": "https://login.microsoftonline.com/XXXXX/oauth/authorize?client_id="&api-version=
...
I would like to add another url value to it. How can I edit the
secrets with Azure portal?
How the value of api-version set?
Thanks
You can only change secret attributes such as expiration date, activation date. You cannot change secret's value programatically or via Azure Portal. If you want to update your secret without creating a new vault (meaning the secret identifier still remains intact) you can create a new version of the existing secret.
If the secret value contains the variables to get authorization code, you don't need api version because the URI you call is the authorization endpoint.
I've uploaded a pfx certificate as a secret to my Azure Portal and I now want to use this to sign the credentials in IdentityServer4.
I've got reference to the vault in the startup of the api using:
builder.AddAzureKeyVault(
$"https://{config["Vault"]}.vault.azure.net/",
config["ClientId"],
config["ClientSecret"]);
But not too sure how to get the certificate out to pass to:
services.AddIdentityServer()
.AddSigningCredential(...)
Is it possibly to be able to reference the certificate directly from the key vault, or do I need to deploy the cert to the web app the api is running on?
Thanks
You are already building the IConfiguration object, so you can reference the pfx key just like you would reference any other object from the configuration.
If the key is in the keyvault you can reference it something like:
// Name of your pfx in the keyvault.
var key = _configuration["pfx"];
var pfxBytes = Convert.FromBase64String(key);
// Create the certificate.
var cert = new X509Certificate2(pfxBytes);
services.AddIdentityServer()
.AddSigningCredential(cert);
I would recommend using the keyvault for this but you could decide to upload the pfx to the certificate store of the web app. You can do this in Azure by going to your web app -> SSL Certificates -> Upload certificate and enter the password. Go to the Application Settings and add the app setting WEBSITE_LOAD_CERTIFICATES : <thumbprint>. The last thing you would do is retrieve the certificate from the store and add it again like AddSigningCredential(cert);.