I have installed self hosted IR on my PC and am trying to use it in my ADF (SQL Server to Azure SQL DB) pipeline. When i run the pipeline it fails with the below error.
InvalidParameter,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The value of the property 'msiAuthenticator' is invalid: 'The required property is not specified. Parameter name:
I think you can try the copy tool UI and set up it again.
did you use the encrypted credential for your linked service, what authentication type did you use ? to know your scenario, I need more information.
Related
I am trying to deploy a DACPAC using Azure Release pipeline.
Following are the methods which I used
Using Azure SQL DACPAC task - While using this approach, as part of additional properties following set of params are passed for deployment
AzureKeyVaultAuthMethod:ClientIdSecret /ClientId:'$(SERVICEPRINCIPALID)' /Secret:'$(SERVICEPRINCIPALKEY)'. On enabling diagnostics log, getting following error:
SqlPackage build version - 16.0.6161.0
Using a power-shell script - While using the power-shell script in order to do the DACPAC deployment, I am getting the following error.
Failed to decrypt a column encryption key. Invalid key store provider name: 'AZURE_KEY_VAULT'. A key store provider name must denote either a system key store provider or a registered custom key store provider. Valid system key store provider names are: 'MSSQL_CERTIFICATE_STORE', 'MSSQL_CNG_STORE', 'MSSQL_CSP_PROVIDER'. Valid (currently registered) custom key store provider names are: . Please verify key store provider information in column master key definitions in the database, and verify all custom key store providers used in your application are registered properly.
Logs attached here
SqlPackage build version - 15.0.5472.2
As a pre-requisite before deploying the DACPAC, using a power-shell script I am creating the keys and then inserting them to DB. Contents of the script below.
All the deployments are happening through a service principal and it has admin level access on all the resources in the Azure AD.
Am I missing any steps here before the deployment of DACPAC through release pipeline.
Thanks,
Nandan
I have created a simple SSIS project and in this project, I have a package that will delete a particular file in Downloads folder.
I deployed this project to Azure. And when I am trying to execute this package using Azure Data Factory then the pipeline fails with an empty error (I am attaching the screenshot here).
enter image description here
What I have done to fix this error is:
I have added self-hosted IR to Azure-SSIS IR as the proxy to access the data on-premise.
Set the ConnectByProxy as True.
Converted the project to Project Deployment Model.
Please help me out to fix this error and if you need more details then just leave a comment.
Windows Authentication :
To access data stores such as SQL servers/file shares on-premises or Azure Files, check the Windows authentication check box.
If this check box is selected, fill in the Domain, Username, and Password fields with the values for your package execution credentials. The domain is Azure, the username is storage account name>, and the password is storage account key> to access Azure Files, for example.
Using the secrets stored in your Azure Key Vault
As a substitute, you can leverage secrets from your Azure Key Vault as values. Select the AZURE KEY VAULT check box next to them to do so. Create a new key vault connected service or choose or update an existing one. Then choose your value's secret name and version. You can pick or update an existing key vault or create a new one when creating or editing your key vault connected service. If you haven't previously done so, allow Data Factory managed identity access to your key vault. You may also directly input your secret in the format key vault linked service name>/secret name>/secret version>.
Note : If you are using Windows Authentication, there are four methods to
access data stores with Windows authentication from SSIS packages
running on your Azure-SSIS IR: Access data stores and file shares with
Windows authentication from SSIS packages in Azure | Docs
Make Sure it Falls under one of such methods, else it could potentially fail at the Run Time.
I have created the app service in existing internal app service environment. And then I’m trying to deploy the source code into above app service using Azure DevOps CI & CD pipelines. But, in the release pipeline I’m getting the below error message in Azure DevOps:
Error: Error Code: ERROR_DESTINATION_INVALID
More Information: Could not connect to the remote computer ("xx-xxxx-v1-api-dev.scm.xxx-int-nonp.xxxcloudapps.com"). Make sure that the remote computer name is correct and that you are able to connect to that computer. Learn more at: http://go.microsoft.com/fwlink/?LinkId=221672#ERROR_DESTINATION_INVALID.
Error: The remote name could not be resolved: 'xx-xxxx-v1-api-dev.scm.xxx-int-nonp.xxxxcloudapps.com'
Error count: 1.
Note: I’m using self-hosted agent in the release pipeline. I have tried to access Kudu environment for the above service in the private virtual machine. But, the kudu not opening.
So, please suggest me how to fix the above issue.
Kindly set/add the following to the Application setting and see if it helps.
Name: WEBSITE_WEBDEPLOY_USE_SCM
Value: false
If I have understood your issue correctly (using private link).
For accessing the Kudu console, or Kudu REST API (deployment with Azure DevOps self-hosted agents for example), you must create two records in your Azure DNS private zone or your custom DNS server. Kindly check this document for more details. If you haven't added the records, kindly see if that works.
Name Type Value
mywebapp.privatelink.azurewebsites.net A PrivateEndpointIP
mywebapp.scm.privatelink.azurewebsites.net A PrivateEndpointIP
https://learn.microsoft.com/azure/app-service/networking/private-endpoint#dns
Ensure that the release pipeline is running in the right host. Because the endpoint is private for the SCM too, the DevOps Agent must be in a network with access with the endpoint, so mandatory to use self-hosted Devops Agent.
Kindly see this doc for more details:
https://learn.microsoft.com/azure/app-service/networking/private-endpoint#dns
If the issue still persists, kindly capture a network trace/HAR or client-side HTTP log/review the logs.
P.S. Kindly do not share any PII data on the public forum for your privacy.
I've a WebAPI project and I'm using Azure CI/CD pipeline to deploy it in azure. The project contains unit testing and integration testing. And for integration testing it needs to access database. But since I don't want to check in my connectionstring to source control, the build pipeline will always fail.
So, the question is what solutions/features or workaround exist that can help me accomplish this scenario?
You can use token replace task to feed your config file with connection string. For that you need to install and add token replace task
and for this configuration:
you need to have such appsettings.json
{
"ConnectionStrings": {
"BloggingDatabase": "#{ConnectionString}#"
},
}
and in your pipeline please variable ConnectionString:
You can also use variable groups with Azure KeyVault. For that appeoach please check this blog post.
Azure Key Vault is a good place to securely store secrets such as db server credentials; this keeps them out of source control.
The general approach is:
in advance, save the db server password as a keyvault secret
in the pipeline, get the db server password using the Azure Key Vault task; it is now available as a secret variable in the pipeline
use the db server password in subsequent tasks; either directly, or by substituting into app settings as described in Krzysztof Madej's answer
We are trying to setup CI/deployment with DevOps using the documentation provided here: https://learn.microsoft.com/en-us/azure/data-factory/continuous-integration-deployment. We are using a shared IR that has been set up in the target environment prior to deployment.
The release succeeds if the deployment mode setting is set to validation only, but fails when incremental or complete is selected. We get the following error when using override template parameters:
2018-09-21T17:07:43.2936188Z ##[error]BadRequest: {
"error": {
"code": "BadRequest",
"message": "Failed to encrypt sub-resource payload
Please make sure your shared IR is online when doing the deployment, otherwise you may meet this problem because self-hosted IR will be used to encrypt your payload.
If you confirm the above action is done and you still have this error, please share the request activity ID to us and we can do some further investigation.
Make sure that you've entered the right connection string into your parameters JSON for any linked services you are using. This fixed the error for me although I don't have a full CI/CD environment with IR established.
I can solve it using the Azure Key Vault.
I added the connection string as a Secret.
In the connection string I also included the authentication data (username and password).
The limitation of this approach is that the possibility of passing the parameters is lost.
For example: dynamic values such as the name of the database or the user.
I would request you to look into the connection string for the respective Linked Service for which you have attached IR. For my ASQL based Linked service I had to use something like this , simple server name would not suffice and you will get "message": "Failed to encrypt sub-resource payload
"typeProperties": {
"connectionString": "Integrated Security=False;Encrypt=True;Connection Timeout=30;Data Source=axxx-xxx-xx-xxxx.database.windows.net;Initial Catalog=\"#{split(linkedService().LS_ASQL_SERVERDB,';')[1]}\""
}
I override parameter because of the connection string was secure. Use dummy value of(username, password, connection string) if You don't have original ones and then deploy.
The IR already being running doesn't make sense when doing a full deployment of an ADF instance. The IR key is generated within the instance of ADF you deploy, meaning you've created circular logic: you cannot deploy IR until the deployment of ADF is complete, but you can't complete the deployment of ADF until the IR is deployed.
So far our answer has been to let the arm template fail at this point, which is after the IR registration in the template so the IR key is then generated. We use that to deploy the IR, then re-run the template and it succeeds... it's stupid and hacky and there has to be a more sane way to do this than intentional failure/retry.