Does ODBC Tableau for Azure Data Explorer have support to connect to external tables. I have these 3 tables I am able to query:
But in Tableau, I am unable to search and see the tables:
While it seems the ADX external tables are indeed not listed, they can be referenced by a New Custom SQL.
Below is a demonstration on the publicly available cluster, Help.
Although in KQL external tables are referenced with a special syntax, e.g.:
external_table('TaxiRides')
| take 10
When using the SQL syntax, you refer to it as a regular table, e.g.:
select top 10 * from TaxiRides
Related
I have a view in an on-demand (or "serverless") sql pool. My goal is to over data from the serverless views and materialize them as tables in the dedicated pool. Is this possible?
There are a couple of options here:
create a Synapse Pipeline with Copy activity. Use the serverless and the source and the dedicated sql pool as the sink. Make sure the 'Auto create table' option is set on the sink
create a Synapse notebook that connects via jdbc to the serverless sql pool (it's just a sql endpoint right), and writes into dedicated sql pool via the synapsesql.write method. I did an example of that technique here.
As per the official Microsoft documentation:
Limitations
Views in Synapse SQL are only stored as metadata. Consequently, the following options aren't available:
There isn't a schema binding option
Base tables can't be updated through the view
Views can't be created over temporary tables
There's no support for the EXPAND / NOEXPAND hints
There are no indexed views in Synapse SQL
But, as an alternative, if your table is in dedicated SQL pool you can use CREATE TABLE AS SELECT (CTAS) that creates a new table based on the output of a SELECT statement. CTAS is the simplest and fastest way to create a copy of a table.
To know more, please refer CREATE TABLE AS SELECT (Azure Synapse Analytics).
I need to implement an incremental load for sap tables via the sap table connector in azure data factory within a copy activity.
Currently, I am trying to set the watermark of the sap table (e.g. MAX(ERDAT)) through a lookup activity. Unfortunately, I cannot create a custom query,
because the sap table connector is very limited and only supports basic filtering options (link).
Does someone know how to set a watermark via the sap table connector?
Thanks in advance!
https://learn.microsoft.com/en-us/azure/data-factory/load-sap-bw-data
This document providing the information regarding the watermark in SAP table connector. Check it out for the implementation.
Personally, I'd prefer loading the delta e.g."ERDAT = today" to an OpenHub, then loading that regularly to Azure.
If that option does not work, using the link you provided, you can create a custom function module on SAP, and filter out records based on date.
Use this link for more info
https://www.cdata.com/kb/entries/sap-custom-read-table-function.rst
I'm trying to get my head around Databricks.
I've found documentation stepping through importing data from S3 or Azure Datalake, and then outputting into Azure Synapse Analytics or another Data Warehouse solution.
After a quick play, I've recognised that you can simply save a table in Databricks, access it using SQL, and even pull it into PowerBI as a source.
So my question: for a small Datamart (10 dims, 5 facts), why would I choose to pay for an additional database solution like Azure SQL, Synapse, RDS or other when I could simply leave the data in a table in Databricks and then access it directly from my reporting tool from there?
Thank you in advance.
Andy
Yes this is very much possible . Just to let you know that SQL Azure and Synapse may be a Microsoft offering but they are for different purpose , Synapse supports MPP and so it more big data implementation . Also its not only how many dimension and fact table you have , how much data you have , what kind of aggregation it has etc becomes decisive .
I am using an Azure SQL Database for our team's reporting and the data size right now is too big to handle by a single data (at least I think so, it has 2 fact tables with around 100m rows in each table).
The Azure SQL Database is named "operation-db" and the Synapse is named "operation-synapse".
I want to make the transition for my team become as smooth as possible. So I'm planning to copy all the tables, views, stored procedure and user-defined function over to Synapse.
Once I'm done with that, is there a way to rename "operation-synapse" to "operation-db" so the team doesn't have to go to their code base to change the name of the db?
Thanks!
It is not possible to rename a SQL Pool via SQL Server Management Studio and you will receive the following error:
ALTER DATABASE NAME statement is not supported in a Synapse workspace.
To update the name of a SQL pool, use the Azure Synapse Portal or the
Synapse REST API. (Microsoft SQL Server, Error: 49978)
The REST API however does list a move method to change names:
POST https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Synapse/workspaces/{workspaceName}/sqlPools/{sqlPoolName}/move?api-version=2019-06-01-preview
I couldn't get it to work though. YMMV. Not renaming your db shouldn't be a big deal though. Your team should feel comfortable with changing connection strings etc and it will help them understand they are moving to a different product (Synapse) with different characteristics.
Before you move to Synapse however, have you look at Clustered Columnstore indexes in Azure SQL DB? They are default type of index in a SQL Pool database but are also available in SQL DB. They can compress your data 5-10x so it might end up not that big at all. Columnstore is great for aggregate queries but less so for point lookups so have a think about your workload before you migrate.
100 million rows is not big enough for synapse. Cci data in each shard will only have 1 row group (1mil rows).
Consider using partitioning or CCI in your sql db itself.
Also what's your usage pattern? If you are doing point lookups and updates clustered indexes will perform better.
You can rename a Synapse database easily using the SSMS GUI. (I've just tried this on v18.8).
Just click once on the database name in the Object Explorer to select it, then press the F2 key to rename it.
The Synapse service must be running (i.e. not paused) for the rename to work.
You can rename Synapse database using T-SQL. The command is as follows:
ALTER DATABASE [OldSynapseDBName]
MODIFY NAME = [NewSynapseDBName]
Note you need to be connected to/issue the command from the master database otherwise it will not work.
The command takes can 30 seconds on 100GB DB and there are some caveats such as DB must not be used during operation.
I have to querying two tables from two different databases in Azure SQL. Is there any option available in azure?
If your database are Azure single databases, maybe you can use OPENROWSET (Transact-SQL). It now support Azure SQL database.
Includes all connection information that is required to access remote data from an OLE DB data source. This method is an alternative to accessing tables in a linked server and is a one-time, ad hoc method of connecting and accessing remote data by using OLE DB. For more frequent references to OLE DB data sources, use linked servers instead. For more information, see Linked Servers (Database Engine). The OPENROWSET function can be referenced in the FROM clause of a query as if it were a table name. The OPENROWSET function can also be referenced as the target table of an INSERT, UPDATE, or DELETE statement, subject to the capabilities of the OLE DB provider. Although the query might return multiple result sets, OPENROWSET returns only the first one.
Hope this helps.