I am looking at a Data Lake csv file and want to create an external table in the serverless SQL Pool of Microsoft Synapse. The goal is to query this file with Row Level Security constraints in place.
When the external table is created on a dedicated Server, I am able to query the file with Row Level Security constraints in place.
How can I make the Row Level security for external tables on a serverless SQL Pool?
Unfortunately, row level-security is not supported in serverless SQL pool at the moment.
Can you please vote for this on our User Voice?
https://feedback.azure.com/forums/307516-sql-data-warehouse?category_id=171048
You can't use the feature as it is. T-SQL support on Serverless is limited.
E.g. CREATE FUNCTION isn't supported.
This syntax is not supported by serverless SQL pool in Azure Synapse Analytics.
You could of course try to DIY using Views which are supported in Serverless.
In the figure below Entitlements would become another CSV and EXTERNAL TABLE that you would create.
You'll have to either find the right function to get current user and/or role for View's SELECT query, or provide it via some wrapper code from some other place where you maintain your own Context.
Disclaimer: I've not done this in Serverless so can't say for sure.
Related
I am trying to load the data (tabular data in tables, in a schema named 'x' from a spark pool in Azure Synapse. I can't seem to find how to do that. Until now i have only linked synapse and my pool to the ML studio. How can I do that?
The Lake Database contents are stored as Parquet files and exposed via your Serverless SQL endpoint as External Tables, so you can technically just query them via the endpoint. This is true for any tool or service that can connect to SQL, like Power BI, SSMS, Azure Machine Learning, etc.
WARNING, HERE THERE BE DRAGONS: Due to the manner in which the serverless engine allocates memory for text queries, using this approach may result in significant performance issues, up to and including service interruption. Speaking from personal experience, this approach is NOT recommended. I recommend that you limit use of the Lake Database for Spark workloads or very limited investigation in the SQL pool. Fortunately there are a couple ways to sidestep these problems.
Approach 1: Read directly from your Lake Database's storage location. This will be in your workspace's root container (declared at creation time) under the following path structure:
synapse/workspaces/{workspacename}/warehouse/{databasename}.db/{tablename}/
These are just Parquet files, so there are no special rules about accessing them directly.
Approach 2: You can also create Views over your Lake Database (External Table) in a serverless database and use the WITH clause to explicitly assign properly sized schemas. Similarly, you can ignore the External Table altogether and use OPENROWSET over the same storage mentioned above. I recommend this approach if you need to access your Lake Database via the SQL Endpoint.
Is it possible for me to insert some data from one database to another in Azure sql?
Let's say I have a trigger in db1 that updates some values in db2.
I read about elastic queries but it seems like they are read-only so they don't solve my problem.
You can't use cross-database in Azure Sql Server because databases can't see eachother physically , you could use elastic pools but they are Read Only.
A solution is to use SQL Managed Instance to upload your instance . This supports cross-database queries but it was expensive.
There was some previous discussion here about doing similar:
C# Azure Function trigger when SQL Database has a new row added without polling
There is also the Azure SQL Bindings for Azure Functions but they are input bindings and not triggers and they're still in preview and limited to C#, JavaScript and Python.
Azure SQL bindings for Azure Functions overview (preview)
There was a new announcement last week after MS Build however for Azure SQL Database External REST Endpoints Integration (hopefully they don't refer to it as ASDEREI) but this is currently in preview under Early Adoption Program (EAP).
Announcing the “Azure SQL Database External REST Endpoints Integration” Early Adoption Program
I have a view in an on-demand (or "serverless") sql pool. My goal is to over data from the serverless views and materialize them as tables in the dedicated pool. Is this possible?
There are a couple of options here:
create a Synapse Pipeline with Copy activity. Use the serverless and the source and the dedicated sql pool as the sink. Make sure the 'Auto create table' option is set on the sink
create a Synapse notebook that connects via jdbc to the serverless sql pool (it's just a sql endpoint right), and writes into dedicated sql pool via the synapsesql.write method. I did an example of that technique here.
As per the official Microsoft documentation:
Limitations
Views in Synapse SQL are only stored as metadata. Consequently, the following options aren't available:
There isn't a schema binding option
Base tables can't be updated through the view
Views can't be created over temporary tables
There's no support for the EXPAND / NOEXPAND hints
There are no indexed views in Synapse SQL
But, as an alternative, if your table is in dedicated SQL pool you can use CREATE TABLE AS SELECT (CTAS) that creates a new table based on the output of a SELECT statement. CTAS is the simplest and fastest way to create a copy of a table.
To know more, please refer CREATE TABLE AS SELECT (Azure Synapse Analytics).
I am trying to see whether it is possible to access some data stored within a table in a dedicated SQL in Azure Synapse using REST API but I have not been able to figure much out. I checked the official docs at Microsoft and at most I have been able to query for a specific column within a table, not much more beyond that. I am wondering whether it is even possible to get data through the Azure Synapse REST API. Would appreciate any help.
Docs for reference: https://learn.microsoft.com/en-us/rest/api/synapse/
It is not possible to access the data in Synapse Dedicated SQL pools using REST APIs, today it is possible only to manage compute using REST API.
We are using Azure SQL as our database across multiple regions, with one primary and multiple secondaries.
The scenario is, we want to find if there are any active replication problems through some sort of API call / so we can integrate this into our overall environment.
Is there an Azure Management API / or SQL query we can run and build an API on top of - so we can some result like replica DB X is having problems and the data is outdated?
Azure doesn't have an API for this yet. But you should be able to build APIs on top of the views.
There are out of box views 'sys.dm_database_replica_states' and 'sys.dm_db_resource_stats' provided on Azure SQL Database that could be used for your requirement. However, I would recommend you to go through the below link and choose them appropriately:
https://learn.microsoft.com/en-us/azure/azure-sql/database/read-scale-out#monitoring-and-troubleshooting-read-only-replicas
https://learn.microsoft.com/en-us/azure/azure-sql/database/read-scale-out#data-consistency