What are possible solutions to do per-request or on schedule 1-way sync of one SQL Server database to the other in Azure?
Both DBs are configured to allow access only via private endpoints.
I've just started exploring options, appreciate expert's opinion on the question.
1-way replication, incrementally data sync and on schedule configuration -- Azure Data Factory is the most suitable service to achieve the following requirements.
Using ADF, you can incrementally load data from multiple tables in a SQL Server Database to one or more databases in another or same SQL Server by creating pipeline using Copy activity. You can also schedule the pipeline trigger based on your requirement.
This official tutorial from Microsoft on Incrementally load data from multiple tables in SQL Server to a database in Azure SQL Database using the Azure portal will help you to create the ADF environment using Linked Service, Datasets, and copy Activity to accomplish your requirement (skip setting Self hosted Integration Runtime as it is required when one of your database is in on-premises).
Once your pipeline has been created, you can schedule it by creating New Trigger. Follow Create a trigger that runs a pipeline on a schedule to create new trigger.
Related
I am working on a migration project where I have a few SQL Server Integration Service projects that will be moved to Azure Data Factory. While I go through this we have a few jobs scheduled via SQL Server Agent which has multiple steps. If were to replicate the same using Azure Data Factory triggers is there a way to group multiple pipelines together and sequence the execution accordingly like we have multiple job steps in SQL Server Agents.
For instance:
Load all of the lookup tables
Load all of the staging tables
Load all of the dimension tables
Load Fact table
Please guide in the right direction.
You can use the Execute Pipeline Activity to build a master pipeline that runs your other pipelines. eg
I am trying to test out Azure Purview and connect it to an Azure SQL Server. Since the SQL server is hosted in the cloud I want to use the default AutoResolve Integrated Runtime to get connected but there is not one setup or an option to setup a new one. Has anyone else using Purview been able to setup (or needed to setup) an AutoResolve IR?
To connect to Azure SQL DB/MI you can directly go to the Azure Purview portal and register new data sources and select Azure SQL DB/MI.
In this article - Manage data sources in Azure Purview (Preview), you learn how to register new data sources, manage collections of data sources, and view sources in Azure Purview (Preview).
Only to connect on-premise SQL server you need to Set up a
self-hosted integration runtime to scan the data source.
If the data source is located on Azure, you don't need any integration runtime to scan the data source.
Reference: Register and scan an Azure SQL Database.
CHEEKATLAPRADEEP-MSFT is absolutely correct, to go a step further, since you know what an auto resolve integration runtime is, you probably are utilizing Azure Data Factory so in addition to registering your SQL Server, you can also link your Azure Data Factory for data lineage purposes. Based on the pipelines that are executed, it will autonomously create the data lineage.
Navigation to Link Data Factory
Data lineage created by linking Data Factory
Keep in mind, you will have to execute pipelines after linkage for it to pick up the data lineage. Also, for sources or destinations not supported yet, it will not get the data lineage.
My requirements are as below :
Move 3 SAP local databases to 3 Azure SQL DB.
Then Sync daily transactions or data to azure every night. If transactions of local DB are already exists in azure, update process will do on these transactions if not insert process will do.
Local systems will not stop after moving to azure. They will still goes about 6 months.
Note :
We are not compatible with Azure Data Sync process because of it's
limitations - only support 500 tables, can't sync no primary keys
table, no views and no procedure. It also increase database size on
both(local and azure).
Azure Data Factory Pipeline can fulfill my requirements but I have
to create pipeline and procedure manually for each table. (SAP has
over 2000 tables, not good for me)
We don't use azure VM and Manage Instance
Can you guide me the best solution to move and sync? I am new to azure.
Thanks all.
Since you mentioned that ADF basically meets your needs, I will try to start from ADF. Actually,you don't need to manually create each table one by one.The creation could be done in the ADF sdk or powershell script or REST api. Please refer to the official document:https://learn.microsoft.com/en-us/azure/data-factory/
So,if you could get the list of SAP table names(i found this thread:https://answers.sap.com/questions/7375575/how-to-get-all-the-table-names.html) ,you could loop the list and execute the codes to create pipelines in the batch.Only table name property need to be set.
I have some T-sql scripts which generate some data and we manually update them into the excel spreedsheet, we need a way to push this into azure sql database, from a job so that we can access them from there and remove the manual process of uploading the information to the azure sql database every time. What is the best way to do this?
I assume you are trying to move data from an on prem server to Azure. The simplest method may be Azure Data Sync.
You could load your data from your queries into an on prem table which syncs to Azure.
On all your SQL Server instances, you can create a Linked Server to one Azure SQL Database. Once the linked server is created you can directly insert on Azure SQL Database from your on-premises SQL Server instances.
Here is how you create the Linked Server.
Below image shows how you insert data on Azure SQL Database using the linked server.
For detailed steps, you can visit this tutorial.
I think you can think about Azure Data Factory.
Azure Data Factory Copy Active can help you use T-sql scripts to move data to another Azure SQL database.
For more details, please the Azure tutorial:Copy multiple tables in bulk by using Azure Data Factory.
When the pipeline created, you can trigger and monitor the pipeline runs.
Trigger the pipeline on a schedule:
You can create a scheduler trigger to schedule the pipeline to run periodically (hourly, daily, and so on). In this procedure, you create a trigger to run every minute until the end date and time that you specify.
Please see: Trigger the pipeline on a schedule.
This can help you push the data to Azure SQL Database automatically.
Hope this helps.
you can try the SSIS package? which automates the process of data upload data into azure sql database.... i have not used ssis for Azure but to sink data from csv/xls/xlsx into ms sql server database,,I refered this article which can be helpful in anyway
Currently, we are mirroring our local SQL server Database with Azure SQL Server Database. For this, we are using Azure Data Management gateway but the problem is we are not able to handle the update or delete scenario. Update and Delete are not reflecting in Azure SQL Server database(Mirror).
Thanks.
What type of activity are you using in your pipeline? I'm assuming a simple copy activity?
My suggestion would be to have a copy activity that lands a clone of the on premises data in the Azure SQL DB first, maybe in a staging schema set of table or something. Then have a second downstream activity that performs a stored procedure execution activity. You can code a MERGE statement or whatever in the procedure to output the data into a separate table.
It sounds like you almost want a secondary node for your SQL Server in Azure. Maybe just use an availability group?! SQL 2014 or higher required on prem though.
You can also take a look at Transactional Replication to Azure SQL database, with transactional replication, Azure SQL database acts as a subscriber, update/delete changes you made in on-premises SQL Server database will be reflected in Azure SQL database.