I’m trying to implement a pipeline in ADF where I copy data from a Function App to an on-prem SQL Server. I have installed the Self-Hosted Integration Runtime to access the on-prem database and set my copy activity to use the self-hosted IR.
First I was a getting a firewall error, so I added a rule to allow the node where IR is installed to call the function app but now I am getting a timeout error.
Any ideas why the timeout?
Please check the General parameters of the copy activity. Try to Increase Timeout of your copy activity. By default it is 7 days.
Also, try to increase the retry count in the copy activity. The default is zero (no retry). Increasing the count and retry interval should allow it to attempt to regain connection.
Please refer this Microsoft Documentation: Troubleshoot copy activity on Self-hosted IR
Related
I have a problem in self hosted IR which is running slow.
Although i had a check on CPU and memory usage which is fine in ADF.
Please let me know to overcome the issue.
To overcome the issue you need to first identify the issue.
You can gather the self-hosted IR logs. For a self-hosted IR, you can upload logs that are related to the failed activity or all logs on the self-hosted IR node.
On the Monitor page for the service UI, select Pipeline runs.
Under Activity runs, in the Error column, select the highlighted button to display the activity logs, as shown in the following screenshot:
Select Send logs.
Later you can go through the Copy activity performance and scalability guide to improve the performance of the activity.
I'm trying to copy data, using the copy activity in a synapse-pipeline, from a self hosted integration runtime rest api call to a azure data lake gen2. Using preview I can see the data from the rest api call but when I try to do the copy activity it is queued endlessly. Any idea why this happens? The Source is working with a self hosted integration Runtime and the Sink with azure integration runtime. Could this be the problem? Otherwise both connections are tested and working...
Edit: When trying the the web call, it tells me it's processing for a long time but I know I can connect to the rest api source since when using the preview feature in the copy activity it shows me the response....
Running the diagnostic tool, I receive the following error:
It seems to be a problem with the certificate. Any ideas?
Integration runtime
If you use a Self-hosted Integration Runtime (IR) and copy activity waits long in the queue until the IR has available resource to execute, suggest scaling out/up your IR.
If you use an Azure Integration Runtime that is in a not optimal region resulting in slow read/write, suggest configuring to use an IR in another region.
You can also try performance tuning tips
how do i scale up the IR?
Scale Considerations
Good day,
ADF v2 data loads that were working for months are suddenly failing on the Sink (to Azure SQL database) activity.
The error message points to a firewall entry for the target database:
Check the linked service configuration is correct, and make sure the SQL Database firewall allows the integration runtime to access.
I have confirmed that the IP of my on premise integration runtime server is specified in the firewall rules of the Azure database.
When testing the linked service connectivity with the Integration Runtime parameter as 'AutoResolveIntegrationRuntime' the test is successful but when setting it to my self hosted IR then I get the error message above.
Also, when the data factory job executes it, processes 'Lookup' or 'Stored Proc' activities to my Azure database without any problems (and I can see it uses both IRs at this point) but for 'Sink' activities it fails and I can see it uses my self hosted IR at these points.
I have one IR node only and have confirmed it is active and running. It is also on the latest version.
Ok - so the firewall engineer fixed it. Apparently the Azure database traffic was subject to general internet traffic firewall rule. The connection failed on MSSQL traffic so he created a firewall rule specifically for that and now it is sorted.
I have been using Azure Data Factory V2 for a while to execute SSIS packages from the SSISDB catalouges.
Today (16-11-2018) I have encountered "Unexpected Termination" Failure message without any Warning and Error message.
Things than I have done:
Executing the SSIS package manually from the SSISDB catalogue in SQL Server Management Services (SSMS). What i have noticed is that it took an exceptionally long time to assign the task to a machine. When the package is assigned to a machine, within 1 or two minutes it throws back the Failure message.
There are 3 SSIS packages that is excecuted "sequentially" with the Azure Data Factory Pipeline. Often the 1st package is executed successfully, however the 2nd and 3rd package never succeded.
Another error message that I got is "Failed pull task from SSISDB, please check if SSISDB has exceeded its limit".
I hope anyone can help me with this issue. I have been searching the web and could not find anything on this subject.
What tier of Azure sql server have you provisioned for the SSISDB to run on? If its too small, it may be taking too much time starting and throw a timeout.
Personally, I've had no problems provisioning an S3 Azure Sql Server.
Hope this helped!
Martin
We have SSIS packages running on a server with sql server agent. However, we want to move this job to a cloud solution. One solution is to use a powershell script, but we also tried to replace SSIS with Azure Data Factory.
However, as stated above, the gateway requires my computer to be online and can't be installed on a domain controller (server). Does this mean that data factory cannot be used to fill our database at night (when the pc's are shutdown) and is therefore not a good replacement for SSIS?
Are there any other solutions for this problem?
The Data Gateway can be installed on any computer in your network that has access to the SQL Server. Obviously both the gateway and the SQL server need to be up at the time the activity runs.