I am working on Tutorial 4 of this doc from Azure Team where this section in a Dedicated SQL pool (that actually is also a database - as item 2 of Tutorial states) creates a database named nyctaxi with a table nyctaxi.trip as follows:
%%pyspark
spark.sql("CREATE DATABASE IF NOT EXISTS nyctaxi")
df.write.mode("overwrite").saveAsTable("nyctaxi.trip")
Then Tutorial 5 creates another table NYCTaxiTripSmall. After completing these tutorials, I can see (on the Data Hub of the Synapse studio) my Dedicated SQL Pool as shown below. But when I click on any database object folder it does not show any db object (tables, external tables etc.) and instead, it shows a red cross sign (as shown below).
Question: Why I am not seeing the db objects (described above) in the Dedicated SQL pool below.
Remarks: Please note that I also created DataExplorationDB db and an external source in tutorial 2 using serverless SQL pool - and, as shown in yellow below, I can see that db and its objects. So why same is not true for Dedicated SQL Pool db? I have also restarted the Dedicated SQL pool and it's online. But still no db objects are showing.
When you create a Spark database the tables aren't automatically added to your Dedicated SQL Pool. You can add them as External Tables if you want, but there's no automatic metadata sync between Spark and Dedicated SQL Pool.
Synapse does create a serverless "Lake Database" corresponding to your Spark database, which you can use from SQL Scripts or access with SQL Server reporting tools.
Related
Following the official documentation from Microsoft, it is possible to create a stored procedure in Azure Synapse Serverless SQL pool. I manage to execute the following script
Create proc Test
as
Select 1 as X
However, I Don't see the Test stored procedure in the Serverless SQL pool.
I don't also see the programmability folder in the Serverless SQL pool
The only way I've found to view the Stored Procedure in Synapse Studio or Azure Data Studio is to query the sys schema tables. However SPs are visible if you use SSMS via steps here https://learn.microsoft.com/en-us/azure/synapse-analytics/sql/get-started-ssms
Query in Synapse Studio
SELECT definition, uses_ansi_nulls, uses_quoted_identifier, is_schema_bound
FROM sys.sql_modules
WHERE object_id = OBJECT_ID('dbo.Test');
GO
SSMS
I have a view in an on-demand (or "serverless") sql pool. My goal is to over data from the serverless views and materialize them as tables in the dedicated pool. Is this possible?
There are a couple of options here:
create a Synapse Pipeline with Copy activity. Use the serverless and the source and the dedicated sql pool as the sink. Make sure the 'Auto create table' option is set on the sink
create a Synapse notebook that connects via jdbc to the serverless sql pool (it's just a sql endpoint right), and writes into dedicated sql pool via the synapsesql.write method. I did an example of that technique here.
As per the official Microsoft documentation:
Limitations
Views in Synapse SQL are only stored as metadata. Consequently, the following options aren't available:
There isn't a schema binding option
Base tables can't be updated through the view
Views can't be created over temporary tables
There's no support for the EXPAND / NOEXPAND hints
There are no indexed views in Synapse SQL
But, as an alternative, if your table is in dedicated SQL pool you can use CREATE TABLE AS SELECT (CTAS) that creates a new table based on the output of a SELECT statement. CTAS is the simplest and fastest way to create a copy of a table.
To know more, please refer CREATE TABLE AS SELECT (Azure Synapse Analytics).
I am trying to transform data from ADLS by using Azure Synapse's Dataflow and store it in a table in Dedicated SQL Pool.
I created a Dataset 'UserSinkDataset' pointing to this table in Dedicated SQL Pool.
This 'UserSinkDataset' is not visible in sink dataset of dataflow
There is no option to create a dataset pointing to Dedicated pool from dataflow
Could someone help me understand why is it not being shown in the dropdown?
There is no option to create a dataset referring to dedicated SQL pool instead it provides Azure Synapse Analytics. That is why it is not showing the UserSinkDataset (Azure Synapse Dedicated SQL pool) in the dropdown. So, you can use Azure Synapse Analytics option to point to the table in dedicated SQL pool and create your dataset.
You can follow the steps given below.
Once you reach the sink step, click on new.
Browse for Azure Synapse Analytics and continue.
Create a new linked service by clicking on new.
Specify your workspace, dedicated SQL pool (the one you want to point to) and authentication for the synapse workspace. Test the connection and create the linked service.
After creating the linked service, you can select dbo.SFUser from your SQL pool and click ok.
Now you can go ahead and set the rest of the properties for sink.
You can also create ‘UserSinkDataset’ by choosing azure synapse analytics instead of azure synapse dedicated SQL pool before creating dataflow. This way the dataset created will appear in the dropdown list on sink dataset property.
I created an SQL database using ASP.NET Core 1.1 Migrations.
After I created the database I added some data to the database.
What options do I have to upload this database to Azure?
I need to send the Scheme and the initial data.
Is it possible to run Entity Framework migrations on Azure?
This article describes the possibilities to migrate an existing database to SQL Azure.
https://learn.microsoft.com/en-us/azure/sql-database/sql-database-cloud-migrate
However, in your scenario, this might be overkill to go through the steps of realy doing a migration.
If your number of tables and data is rather small, why not create a SQL script to create the tables & insert the data?
Connect to your SQL Azure using SQL Server Management Studio and execute the script.
As for the Entity Framework, yes, you can run those on SQL Azure as well.
Are there any tutorials available on the subject of migrating from an existing BI-stack based on SQL Server 2008 to Azure SQL Data Warehouse? I'm specifically interested in best practices with regards to how to handle cross database joins on non-premium tiers (our existing procedures and UDFs are full of joins on multiple database objects) and how to migrate existing SSAS cubes and its related programmability and ETL.
What BI-stack are you using? This will determine your next steps for the actual BI tools.
Specifically for cross-database queries when moving to the cloud, the guidance is to move the databases into schemas and then update your scripts to use schema based (2 part names) vs. database (3 part names) when referencing objects. For example, if you have staging and production databases you can simply move your staging objects into a [staging] schema within a single database.
Azure SQL Data Warehouse is commonly used as a backing store for SSAS cubes (MOLAP/ROLAP/Tabular mode). In the Azure cloud, customers have created IaaS SQL Server VMs to host ETL process (SSIS) and cubes (SSAS) with direct connections to SQL Data Warehouse.