We are trying to setup CI/deployment with DevOps using the documentation provided here: https://learn.microsoft.com/en-us/azure/data-factory/continuous-integration-deployment. We are using a shared IR that has been set up in the target environment prior to deployment.
The release succeeds if the deployment mode setting is set to validation only, but fails when incremental or complete is selected. We get the following error when using override template parameters:
2018-09-21T17:07:43.2936188Z ##[error]BadRequest: {
"error": {
"code": "BadRequest",
"message": "Failed to encrypt sub-resource payload
Please make sure your shared IR is online when doing the deployment, otherwise you may meet this problem because self-hosted IR will be used to encrypt your payload.
If you confirm the above action is done and you still have this error, please share the request activity ID to us and we can do some further investigation.
Make sure that you've entered the right connection string into your parameters JSON for any linked services you are using. This fixed the error for me although I don't have a full CI/CD environment with IR established.
I can solve it using the Azure Key Vault.
I added the connection string as a Secret.
In the connection string I also included the authentication data (username and password).
The limitation of this approach is that the possibility of passing the parameters is lost.
For example: dynamic values such as the name of the database or the user.
I would request you to look into the connection string for the respective Linked Service for which you have attached IR. For my ASQL based Linked service I had to use something like this , simple server name would not suffice and you will get "message": "Failed to encrypt sub-resource payload
"typeProperties": {
"connectionString": "Integrated Security=False;Encrypt=True;Connection Timeout=30;Data Source=axxx-xxx-xx-xxxx.database.windows.net;Initial Catalog=\"#{split(linkedService().LS_ASQL_SERVERDB,';')[1]}\""
}
I override parameter because of the connection string was secure. Use dummy value of(username, password, connection string) if You don't have original ones and then deploy.
The IR already being running doesn't make sense when doing a full deployment of an ADF instance. The IR key is generated within the instance of ADF you deploy, meaning you've created circular logic: you cannot deploy IR until the deployment of ADF is complete, but you can't complete the deployment of ADF until the IR is deployed.
So far our answer has been to let the arm template fail at this point, which is after the IR registration in the template so the IR key is then generated. We use that to deploy the IR, then re-run the template and it succeeds... it's stupid and hacky and there has to be a more sane way to do this than intentional failure/retry.
Related
So I've got an Azure Machine Learning pipeline here that consists of a number of PythonScriptStep tasks - pretty basic really.
Some of these script steps fail intermittently due to network issues or somesuch - really nothing unexpected. The solution here is always to simply trigger a rerun of the failed experiment in the browser interface of Azure Machine Learning studio.
Despite my best efforts I haven't been able to figure out how to set a retry parameter either on the script step objects, the pipeline object, or any other AZ ML-related object.
This is a common pattern in pipelines of any sort: Task fails once - retry a couple of times before deciding it actually fails.
Does anyone have pointers for me please?
Edit: One helpful user suggested an external solution to this which requires an Azure Logic App that listens to ML pipeline events and re-triggers failed pipelines via an HTTP request. While this solution may work for some it just takes you down another rabbit hole of setting up, debugging, and maintaining another external component. I'm looking for a simple "retry upon task failure" option that (IMO) must be baked into the Azure ML pipeline framework and is hopefully just poorly documented.
I assume that if a script fails, you want to rerun the entire pipeline. In that case, it is pretty simple with Logic Apps. What you need is the following:
You need to make a PipelineEndpoint for your pipeline so it can be triggered by something outside Azure ML.
You need to set up a Logic App to listen for failed runs. See the following: https://medium.com/geekculture/notifications-on-azure-machine-learning-pipelines-with-logic-apps-5d5df11d3126. Instead of printing a message to Microsoft Teams as in that example, you instead invoke your pipeline through its endpoint.
(this would ideally be a comment but it exceeded the word limit)
#user787267's answer above help me set up the re-try pipeline. So I thought I'd add a few more details that might help someone else set this up.
How to set up the HTTP action
Method: POST
URI: The pipeline endpoint that you configured
Headers: `Key`: Content-Type -- `Value`: application/json
Body:
{
"ExperimentName": "my_experiment_name",
"ParameterAssignments": {
"param1": "value1",
"param2": "value2" },
"RunSource": "SDK"
}
Authentication Type: Managed Identity
Managed Identity: System-assigned managed identity
You can set up the managed identity by going to the logic app's page and then clicking on the Identity tab as shown below. After that just follow the steps. You'll need to give the managed identity permissions over the space in which your ML instance lives.
I've a WebAPI project and I'm using Azure CI/CD pipeline to deploy it in azure. The project contains unit testing and integration testing. And for integration testing it needs to access database. But since I don't want to check in my connectionstring to source control, the build pipeline will always fail.
So, the question is what solutions/features or workaround exist that can help me accomplish this scenario?
You can use token replace task to feed your config file with connection string. For that you need to install and add token replace task
and for this configuration:
you need to have such appsettings.json
{
"ConnectionStrings": {
"BloggingDatabase": "#{ConnectionString}#"
},
}
and in your pipeline please variable ConnectionString:
You can also use variable groups with Azure KeyVault. For that appeoach please check this blog post.
Azure Key Vault is a good place to securely store secrets such as db server credentials; this keeps them out of source control.
The general approach is:
in advance, save the db server password as a keyvault secret
in the pipeline, get the db server password using the Azure Key Vault task; it is now available as a secret variable in the pipeline
use the db server password in subsequent tasks; either directly, or by substituting into app settings as described in Krzysztof Madej's answer
I am developing an ETL solution using .NET and Azure Data Factory to move data from an on-premises SQL Server to Azure. Self-hosted IR is set up correctly and is running on Azure Portal, but when I run the code I get this exception:
Failed to encrypt sub-resource payload and error is: Failed to encrypted linked service credentials on self-hosted IR reason is: InternalServerError, error message is: Internal Server Error..
ConnectionString for the on-premises SQL Server server is in plain text and not encrypted.
How can I fix this problem?
I found a solution myself, and it works. I hope it helps someone who also encounters the same problem.
You need to set the value for EncryptedCredential and don't need to set the Password property.
And set Connection String like this:
I have installed self hosted IR on my PC and am trying to use it in my ADF (SQL Server to Azure SQL DB) pipeline. When i run the pipeline it fails with the below error.
InvalidParameter,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The value of the property 'msiAuthenticator' is invalid: 'The required property is not specified. Parameter name:
I think you can try the copy tool UI and set up it again.
did you use the encrypted credential for your linked service, what authentication type did you use ? to know your scenario, I need more information.
I want my Data Factory's Web Activity to authenticate to my Azure Function using MSI (Managed Service Identity), so I don't have to include the API key in the Web-URL.
What type of resource string do I need to enter here in the configuration section of the Web Activity:
I tried:
https://myfunctionapp.azurewebsites.net/
https://myfunctionapp.azurewebsites.net/api/myfunction
but none worked. The pipeline activity times out after a few minutes with the error "Error calling the endpoint".
I had the same issue, but using https://management.azure.com/ as the resource string worked for me in terms of making MSI function, but after a bunch of testing it seems to me as though the HTTPTrigger still requires a key in either the URI or the header.
Both of these websites are good resources:
https://learn.microsoft.com/en-us/azure/app-service/app-service-managed-service-identity
https://learn.microsoft.com/en-us/azure/active-directory/managed-service-identity/services-support-msi