I am thinking about using Snowflake as data warehouse. My databases are in Azure SQl Database and I would like to know what tools I need for etl my data from Azure SQL Database to Snowflake.
I think Snowpark could work for data transformations, but I wonder what other code tool could I use.
Also, I wonder if I use azure blob storage as staging area or snowflake has its own.
Thanks
You can use HEVO data a third-party tool where you can directly migrate data from Microsoft SQL Server to Snowflake.
STEPS TO BE FOLLOWED
Make a connection to your Microsoft SQL Server database.
Choose a replication mode.
Create a Snowflake Data Warehouse configuration.
Alternatively, You can use SnowSQL to Connect Microsoft SQL Server to Snowflake where you export data from SQL Server to SSMS, upload the same to either Azure storage or S3, and move the data from Storage to Snowflake.
REFERENCES:
Microsoft SQL Server to Snowflake
How to move the data from Azure Blob Storage to Snowflake
Related
A client of mine needs to join tables from his Azure SQL financial data mart with external tables built upon a Data Lakehouse (Parquet files) in Azure Synapse Analytics.
I was wondering if it's possible to create a database link within a Azure SQL database accessing a Azure Synapse Analytics Serverless (on-demand) SQL Pool.
Yes, it’s possible open the Integrate hub, and select and add a link connection.
Select test connection and make sure to check whether SQL firewall rules are properly configured or not.
Reference:
Get started with Azure Synapse Link for Azure SQL Database (Preview) - Azure Synapse Analytics | Microsoft Docs
We are looking at options for moving our on premise SQL Server(s) to Azure and trying to understand whether we will be able to run cross database queries should we have data residing across multiple database technologies both in Azure ( specifically Azure Managed Instance, Azure Synapse Analytics, Azure SQL Database), and in an on-premise SQL Server instance.
We cannot find much information anywhere on whether these are supported and would appreciate if any of ye could help in filling out the table below:
TO->
Azure SQL DB
Azure Managed Instance
Azure Synapse Analytics
On Premise SQL Server
Azure SQL DB
Supported through Elastic Search Query (Ref: https://azure.microsoft.com/en-us/blog/querying-remote-databases-in-azure-sql-db/)
?
Azure Data Share supports sharing of both tables and views from Azure SQL Database and Azure Synapse Analytics (formerly Azure SQL DW), and sharing of tables from Azure Synapse Analytics (workspace) dedicated SQL pool. Sharing from Azure Synapse Analytics (workspace) serverless SQL pool is not currently supported. (Ref: https://learn.microsoft.com/en-us/azure/data-share/how-to-share-from-sql)
Azure SQL database doesn't support the linked server property so you wont be able to access on prem tables in Azure SQL database and the elastic query in Azure SQL database is to query tables between 2 Azure SQL databases and not On prem. (Ref: https://learn.microsoft.com/en-us/answers/questions/289105/how-can-i-query-on-premise-sql-server-database-fro.html)
Azure Managed Instance
?
?
?
Available through the use of Linked Servers (Ref: http://thewindowsupdate.com/2019/03/22/lesson-learned-81-how-to-create-a-linked-server-from-azure-sql-managed-instance-to-sql-server-onpremise-or-azure-vm/)
Azure Synapse Analytics
?
?
?
?
On Premise SQL Server
?
?
?
Using a linked server you can query data in an Azure SQL database from an on premised SQL Server (Ref: https://learn.microsoft.com/en-us/answers/questions/289105/how-can-i-query-on-premise-sql-server-database-fro.html)
Azure SQL database need elastic query to achieve cross database query. It doesn't support create linked server.
Azure Managed instance has almost same feature with on-premise SQL server, you could use USE statement to execute cross database query. It's same with local SQL Server.
Azure Synapse Analytics also doesn't support cross database query.
Per my knowledge and experiense, I will put ✔ or X in table to express support or not supported. please ref:
TO->
Azure SQL DB
Azure Managed Instance
Azure Synapse Analytics
On Premise SQL Server
Azure SQL DB
Supported through Elastic Search Query (Ref: https://azure.microsoft.com/en-us/blog/querying-remote-databases-in-azure-sql-db/)
X, not supported
Azure Data Share supports sharing of both tables and views from Azure SQL Database and Azure Synapse Analytics (formerly Azure SQL DW), and sharing of tables from Azure Synapse Analytics (workspace) dedicated SQL pool. Sharing from Azure Synapse Analytics (workspace) serverless SQL pool is not currently supported. (Ref: https://learn.microsoft.com/en-us/azure/data-share/how-to-share-from-sql)
Azure SQL database doesn't support the linked server property so you wont be able to access on prem tables in Azure SQL database and the elastic query in Azure SQL database is to query tables between 2 Azure SQL databases and not On prem. (Ref: https://learn.microsoft.com/en-us/answers/questions/289105/how-can-i-query-on-premise-sql-server-database-fro.html)
Azure Managed Instance
X, not supported
✔,same with local SQL Server
X, not supported
Available through the use of Linked Servers (Ref: http://thewindowsupdate.com/2019/03/22/lesson-learned-81-how-to-create-a-linked-server-from-azure-sql-managed-instance-to-sql-server-onpremise-or-azure-vm/)
Azure Synapse Analytics
Azure Data Share
X, not supported
X, not supported
X, not supported
On Premise SQL Server
✔,linked server
✔,linked server
X, not supported
Using a linked server you can query data in an Azure SQL database from an on premised SQL Server (Ref: https://learn.microsoft.com/en-us/answers/questions/289105/how-can-i-query-on-premise-sql-server-database-fro.html)
Please choose the suitable Azure database version according your request.
HTH.
AFAIK there is no cross-DB facade that provides a single interface to talk to multiple Databases at the same time. Be it on-prem/in-cloud or SQL-Server/Synapse/MySQL/...
There are individual ways and means by which you can access a single Database from somewhere/anywhere. E.g. accessing an on-prem DB from code in cloud or access a cloud DB from code running on on-prem "servers". List of interfaces available is specific to each "source" and "target" combination.
Azure newbie here.
I have an architecture requirement to move the data from on-premise to cloud database. This is done in two steps due to security restrictions and timelines.
Move the file to azure blob storage
Read from the blob and Import to the sql database .
Azure blob is suggested for unstructured data. However, the data we want to export to cloud is a simple export of data from sql tables to csv files.
For such requirements what is recommended? Azure blob or Azure file share? When to use blob versus azure file share ?
Actually, if you want to migrate database from local/on-premise SQL Server to Azure SQL, there are many ways can help do that directly without Blob or File Storage.
Such as:
Using Data Migration Assistant(DMA) to help you migrate the data/database to Azure.
Ref: Migrate on-premises SQL Server or SQL Server on Azure VMs to Azure SQL Database using the Data Migration Assistant
SQL Server Management Studio(SSMS) task: Deploy Database to Microsoft Azure SQL Database.
Ref: Using the Deploy Database to SQL Azure Wizard in SQL Server Management Studio to move to the Cloud
Of course you could export the sql server data as CSV files to Blob or File storage, then import the csv file to Azure SQL. In usually, we often use Blob Storage work with Azure SQL database. It's up to you, reference the document #Gaurav Mantri-AIS mentioned.
Hope this helps.
I'm doing some tests with Azure Data Lake Analytics and I can’t add a new SQL Server database as a Data Source. When I click on "Add data source", the only two available options are: "Azure Data Lake Storage Gen1" and "Azure Storage".
What I want is to add one SQL Server database so that I can run U-SQL queries against it.
Our SQL Server firewall is correctly configured to allow access to Azure Services, but I am not allowed to add it as a data source.
How can this be done? Is it a matter of other configuration issues?
Any help would be greatly appreciated.
Per my research ,there is no other configuration issues for sql server data source in DLA. Based on this official doc, DLA only supports two data sources:Data Lake Store and Azure Storage.
As workaround , I suggest you using Azure Data Factory to transfer data from sql server database to azure storage so that you could run U-SQL script against data source.
Any concern,please let me know.
I am trying to perform a replication using Azure data sync between an Azure sql database and an Azure SQL Data Warehouse. I have a table with a primary key, identity (1,1). When i try to perform the sync, I receive an error saying something related to a syntax error around "PRIMARY". Is this issue related to how pk's are managed in azure data warehouses or, simply said, this kind of replication is not supported?
SQL Data Sync synchronizes data between SQL Azure Databases and SQL Server instances only. SQL Azure Data Warehouse is not supported.
https://learn.microsoft.com/en-us/azure/sql-database/sql-database-sync-data
Hope this helps.
Regards,
Alberto Morillo