A client of mine needs to join tables from his Azure SQL financial data mart with external tables built upon a Data Lakehouse (Parquet files) in Azure Synapse Analytics.
I was wondering if it's possible to create a database link within a Azure SQL database accessing a Azure Synapse Analytics Serverless (on-demand) SQL Pool.
Yes, it’s possible open the Integrate hub, and select and add a link connection.
Select test connection and make sure to check whether SQL firewall rules are properly configured or not.
Reference:
Get started with Azure Synapse Link for Azure SQL Database (Preview) - Azure Synapse Analytics | Microsoft Docs
Related
I am thinking about using Snowflake as data warehouse. My databases are in Azure SQl Database and I would like to know what tools I need for etl my data from Azure SQL Database to Snowflake.
I think Snowpark could work for data transformations, but I wonder what other code tool could I use.
Also, I wonder if I use azure blob storage as staging area or snowflake has its own.
Thanks
You can use HEVO data a third-party tool where you can directly migrate data from Microsoft SQL Server to Snowflake.
STEPS TO BE FOLLOWED
Make a connection to your Microsoft SQL Server database.
Choose a replication mode.
Create a Snowflake Data Warehouse configuration.
Alternatively, You can use SnowSQL to Connect Microsoft SQL Server to Snowflake where you export data from SQL Server to SSMS, upload the same to either Azure storage or S3, and move the data from Storage to Snowflake.
REFERENCES:
Microsoft SQL Server to Snowflake
How to move the data from Azure Blob Storage to Snowflake
Difference between the dedicated sql pool and dedicated sql pool inside the azure synapse analytics?
While provision the azure synapse analytics we will use the Azure storage layer gen2 ,as per the msdn the data will be stored in the azure storage gen2 but azure gen2 will use the hdfs features.so how the dfs feature will use the syanpse analytics?
They both are the same thing. Either you can first create a Dedicated SQL pool and link it with Synapse Workspace, or you can first create the Synapse Workspace and then dedicated pool inside it.
A dedicated SQL pool offers T-SQL based compute and storage capabilities. After creating a dedicated SQL pool in your Synapse workspace, data can be loaded, modeled, processed, and delivered for faster analytic insight.
Apart from Dedicated SQL pool, Azure Synapse provide Serverless SQL and Apache Spark pools. Based on your requirement you can choose the appropriate.
Serverless SQL pool is a query service over the data in your data lake. It enables you to access your data through the following functionalities:
A familiar T-SQL syntax to query data in place without the need to copy or load data into a specialized store.
Integrated connectivity via the T-SQL interface that offers a wide range of business intelligence and ad-hoc querying tools, including the most popular drivers.
You will be directly passing the file path stored in Data Lake Gen2 in T-SQL statement. Refer example below:
select top 10 *
from openrowset(
bulk 'https://pandemicdatalake.blob.core.windows.net/public/curated/covid-19/ecdc_cases/latest/ecdc_cases.csv',
format = 'csv',
parser_version = '2.0',
firstrow = 2 ) as rows
For more related information, I recommend you to go through this document.
We are looking at options for moving our on premise SQL Server(s) to Azure and trying to understand whether we will be able to run cross database queries should we have data residing across multiple database technologies both in Azure ( specifically Azure Managed Instance, Azure Synapse Analytics, Azure SQL Database), and in an on-premise SQL Server instance.
We cannot find much information anywhere on whether these are supported and would appreciate if any of ye could help in filling out the table below:
TO->
Azure SQL DB
Azure Managed Instance
Azure Synapse Analytics
On Premise SQL Server
Azure SQL DB
Supported through Elastic Search Query (Ref: https://azure.microsoft.com/en-us/blog/querying-remote-databases-in-azure-sql-db/)
?
Azure Data Share supports sharing of both tables and views from Azure SQL Database and Azure Synapse Analytics (formerly Azure SQL DW), and sharing of tables from Azure Synapse Analytics (workspace) dedicated SQL pool. Sharing from Azure Synapse Analytics (workspace) serverless SQL pool is not currently supported. (Ref: https://learn.microsoft.com/en-us/azure/data-share/how-to-share-from-sql)
Azure SQL database doesn't support the linked server property so you wont be able to access on prem tables in Azure SQL database and the elastic query in Azure SQL database is to query tables between 2 Azure SQL databases and not On prem. (Ref: https://learn.microsoft.com/en-us/answers/questions/289105/how-can-i-query-on-premise-sql-server-database-fro.html)
Azure Managed Instance
?
?
?
Available through the use of Linked Servers (Ref: http://thewindowsupdate.com/2019/03/22/lesson-learned-81-how-to-create-a-linked-server-from-azure-sql-managed-instance-to-sql-server-onpremise-or-azure-vm/)
Azure Synapse Analytics
?
?
?
?
On Premise SQL Server
?
?
?
Using a linked server you can query data in an Azure SQL database from an on premised SQL Server (Ref: https://learn.microsoft.com/en-us/answers/questions/289105/how-can-i-query-on-premise-sql-server-database-fro.html)
Azure SQL database need elastic query to achieve cross database query. It doesn't support create linked server.
Azure Managed instance has almost same feature with on-premise SQL server, you could use USE statement to execute cross database query. It's same with local SQL Server.
Azure Synapse Analytics also doesn't support cross database query.
Per my knowledge and experiense, I will put ✔ or X in table to express support or not supported. please ref:
TO->
Azure SQL DB
Azure Managed Instance
Azure Synapse Analytics
On Premise SQL Server
Azure SQL DB
Supported through Elastic Search Query (Ref: https://azure.microsoft.com/en-us/blog/querying-remote-databases-in-azure-sql-db/)
X, not supported
Azure Data Share supports sharing of both tables and views from Azure SQL Database and Azure Synapse Analytics (formerly Azure SQL DW), and sharing of tables from Azure Synapse Analytics (workspace) dedicated SQL pool. Sharing from Azure Synapse Analytics (workspace) serverless SQL pool is not currently supported. (Ref: https://learn.microsoft.com/en-us/azure/data-share/how-to-share-from-sql)
Azure SQL database doesn't support the linked server property so you wont be able to access on prem tables in Azure SQL database and the elastic query in Azure SQL database is to query tables between 2 Azure SQL databases and not On prem. (Ref: https://learn.microsoft.com/en-us/answers/questions/289105/how-can-i-query-on-premise-sql-server-database-fro.html)
Azure Managed Instance
X, not supported
✔,same with local SQL Server
X, not supported
Available through the use of Linked Servers (Ref: http://thewindowsupdate.com/2019/03/22/lesson-learned-81-how-to-create-a-linked-server-from-azure-sql-managed-instance-to-sql-server-onpremise-or-azure-vm/)
Azure Synapse Analytics
Azure Data Share
X, not supported
X, not supported
X, not supported
On Premise SQL Server
✔,linked server
✔,linked server
X, not supported
Using a linked server you can query data in an Azure SQL database from an on premised SQL Server (Ref: https://learn.microsoft.com/en-us/answers/questions/289105/how-can-i-query-on-premise-sql-server-database-fro.html)
Please choose the suitable Azure database version according your request.
HTH.
AFAIK there is no cross-DB facade that provides a single interface to talk to multiple Databases at the same time. Be it on-prem/in-cloud or SQL-Server/Synapse/MySQL/...
There are individual ways and means by which you can access a single Database from somewhere/anywhere. E.g. accessing an on-prem DB from code in cloud or access a cloud DB from code running on on-prem "servers". List of interfaces available is specific to each "source" and "target" combination.
I'm doing some tests with Azure Data Lake Analytics and I can’t add a new SQL Server database as a Data Source. When I click on "Add data source", the only two available options are: "Azure Data Lake Storage Gen1" and "Azure Storage".
What I want is to add one SQL Server database so that I can run U-SQL queries against it.
Our SQL Server firewall is correctly configured to allow access to Azure Services, but I am not allowed to add it as a data source.
How can this be done? Is it a matter of other configuration issues?
Any help would be greatly appreciated.
Per my research ,there is no other configuration issues for sql server data source in DLA. Based on this official doc, DLA only supports two data sources:Data Lake Store and Azure Storage.
As workaround , I suggest you using Azure Data Factory to transfer data from sql server database to azure storage so that you could run U-SQL script against data source.
Any concern,please let me know.
I have an existing Azure SQL DB (Paas solution) in resource group 1. I created a new OMS work space in Log Analytics resource group 2. I want to connect SQL DB in resource group 1 to OMS work space for SQL assessment in resource group 2.
How can enable the connectivity ?
I followed the article from https://learn.microsoft.com/en-us/azure/log-analytics/log-analytics-sql-assessment
If I understand it correctly , I don't need to install any agent. How can I add SQL DB to SQL Assessment .
Currently I see no DB connected in SQL assessment
You can't, SQL Assessment solution is designed for on premise SQL Server or SQL Database (IaaS). For Azure SQL Database(PaaS), you should use the Azure SQL Analytics solution instead of SQL Assessment.
The Azure SQL Analytics solution doesn't use agents to connect to the Log Analytics service.