is possible to configure a linked service between 2 or more datafactory?
I red documentation but i didn't found it
Thanks
Per my experience, we can't do that and never heard such configuration.
Just as I know, we only could share the Integration runtime between 2 or more Data Factory.
But we still need to create the linked service to connect to the same on-premise data source through shared shared self-hosted integration runtime
In one word, it's impossible to configure a linked service between 2 or more Data Factory.
Related
I'm trying to copy data, using the copy activity in a synapse-pipeline, from a self hosted integration runtime rest api call to a azure data lake gen2. Using preview I can see the data from the rest api call but when I try to do the copy activity it is queued endlessly. Any idea why this happens? The Source is working with a self hosted integration Runtime and the Sink with azure integration runtime. Could this be the problem? Otherwise both connections are tested and working...
Edit: When trying the the web call, it tells me it's processing for a long time but I know I can connect to the rest api source since when using the preview feature in the copy activity it shows me the response....
Running the diagnostic tool, I receive the following error:
It seems to be a problem with the certificate. Any ideas?
Integration runtime
If you use a Self-hosted Integration Runtime (IR) and copy activity waits long in the queue until the IR has available resource to execute, suggest scaling out/up your IR.
If you use an Azure Integration Runtime that is in a not optimal region resulting in slow read/write, suggest configuring to use an IR in another region.
You can also try performance tuning tips
how do i scale up the IR?
Scale Considerations
I have an Azure Data Factory V2 with an Integration Runtime installed on the our internal cloud server and connected to our Java web platform API. This passes data one way into ADF on a scheduled trigger via a request to the IR API.
The Java web platform also has a DR solution at another site, which is a mirror build of the same servers and platforms. If I was to install another IR on this DR platform and link to ADF as a secondary IR. Is there a way for ADF to detect if the primary is down and auto failover to the secondary IR?
Thanks
For you question "Is there a way for ADF to detect if the primary is down and auto failover to the secondary IR?", the answer is no, Data Factory doesn't have the failover feature. The shared integration runtime nodes don't affect each other.
For another question in the comment, the IR can't be stop/pause automatically, we must set it manually on the machine:
I'm wondering if anyone has any experience in calling datasets dynamically in Azure Data Factory. The situation we have is that we dynamically sweep all tables in from IaaS (on-premise SQL Server installations on an Azure VM) application systems to a data lake. We want to have one pipeline that can pass server name, database name, user name and password to the pipeline's activities. The pipelines will then sweep whatever source they've been told to read from the parameters. The source systems are currently within a separate subscription and domain within our Enterprise Agreement.
We have looked into using the AutoResolveIntegrationRuntime on a generic SQL Server dataset but, as it is Azure and the runtimes on the VMs are self-hosted, it can't resolve and we get 'cannot connect' errors. So,
i) I don't know if this problem goes away if they are in the same subscription and domain?
That leaves whether anyone can assist with:
ii) A way of getting a dynamic runtime to resolve which SQL Server runtime it should use (we have one per VM for resilience purposes, but they can all see each other's instances). We don't want to parameterise a linked service on a particular VM as it places reliance for other VMs on that single VM.
iii) Ability to parameterise a dataset to call a runtime (doesn't look possible in the UI).
iv) Ability to parameterise the source and sink connections with pipeline activities to call a dataset parameter.
Servers, database, tableNames are possible to be dynamic by using parameters. The key problem here is that all the reference in ADF can’t be parameterized, like linked services reference in dataset, integrationRuntime reference in linked service. If you don’t have too many selfhosted integrationRuntime, maybe you can try setup different pipelines for different network?
I am trying to understand what is the right architecture to use to access data from servers hosted on a private network (still running on Azure but not publicly accessible) and the Azure Data Factory service.
On some documentation Microsoft mentions the Integration Runtime as the solution:
https://learn.microsoft.com/en-us/azure/data-factory/concepts-integration-runtime
While on other documentation it refers to a Data Gateway:
https://learn.microsoft.com/en-us/azure/analysis-services/analysis-services-gateway
Both articles seem fairly recent. The two applications have different recommended requirements (one mentions 8 CPU cores! Which is an overkill for my requirements to ship a few hundred megabytes per night)
Given that the data sources are running on Azure, just not publicly accessible, is there a way to connect Azure Data Factory directly?
The Self Hosted Integration Runtime in ADF should meet your requirement, this link gives a complete example to access data under Azure VNet or private network.
My question is slightly similar to this question however adequately different to merit a new thread.
I have a requirement to extract data from a number of different on-premises SQL Server instances over the internet. I am using Azure Data-Factory 2 and the Integration Runtime to access data from these servers.
The problem is that i will have many pipelines to manage and update. I want to have a single Data Factory process which uses parameters for linked service names.
Is it possible to have 1 pipeline which uses a parameter to reference a linked service name which is updated before re-executing the pipeline?
I am struggling to find a useful article on how this can be achieved.
Reference name can’t be parametied. But making linked Service support parameters is in the plan as the post you mentioned said.