I have a cell with multiple SQL queries like MERGE, SELECT etc. When I run the notebook through a workflow TASK I can see the output of the last run SQL query for a particular cell only. How to see output for each queries?
Related
I need to create a dashboard inside Databricks that summarizes the number of rows in the current workspace right now.
Is there a way to create a SQL query to calculate the number of rows by table, schema, and catalog? The expected result would be:
Catalog
Schema
Table
Rows
example_catalog_1
Finance
table_example_1
1567000
example_catalog_1
Finance
table_example_2
67000
example_catalog_2
Procurement
table_example_1
45324888
example_catalog_2
Procurement
table_example_2
89765987
example_catalog_2
Procurement
table_example_3
145000
Currently, I am working on a pure SQL workflow. So I would like to understand if it's possible to execute such an action using SQL, because as much as I know, the dashboards in Databricks do not accept PySpark Codes.
I was looking for a way to do that. I know that it's possible to access the tables in the workspace by using system.information_schema.tables but how to use it to count to total rows for each table presented there?
I was checking that via SQL Server it's possible via sys schema, dynamic query, or BEGIN...END clause. I couldn't find a way in Databricks to do that.
I strongly doubt if you can run that kind of query in the databricks dashboard . The link shared by #Sharma is more as to how to get the record count using dataframe and not how to link that with the databricks dashboard .
This is my 1st time working with Azure synapse and It seems Select Insert is not working, is there any workaround for this one, where I will just use select statement and then dump it into a temporary table?
here are the error prompted
The query references an object that is not supported in distributed processing mode.
and this is my query
Select *
Into #Temp1
FROM [dbo].[TblSample]
This is the azure synapse we are currently using
ondemand-sql.azuresynapse.net
In Synapse On-Demand, the use of Temporary Tables is limited. In your case I am assuming that dbo.TblSample is an External Table which is possibly why you are facing this restriction.
Instead of using a Temp Table, can you either just JOIN the TblSample directly or use a CTE if you are SELECTing specific rows and columns?
I'm directly querying a local access database using power automate desktop. I'm performing an sql query directly into access by way of an open sql connection in power automate rather than pulling data from a stored results table. When I write the query results to an excel file in the same flow I get my results but not the result column headers. I've looked through most of the answers provided here and they all suggest MySQL, SQL server, or other solutions that I simply don't have.
My current query looks like:
SELECT table1.uid, table2.[job.title], table1.[start.date],
table1.[end.date], table2.company, table1.role, table2.active
FROM table1 INNER JOIN table.2 ON table1.uid=tables.uid
And so when I run the query my column headers are UID, job title, start date, end date, company, role, active.
I'd like those headers generated by the query to carry over to the Excel sheet so I don't have to manually insert them.
In short can I write an sql query that directly connects to access through an open sql connection in power automate and have it return the data and the data column headers?
I am trying to MERGE two tables using spark sql and getting error with the statement.
The tables are created as external tables pointing to the Azure ADLS storage. The sql is executing using Databricks.
Table 1:
Name,Age.Sex
abc,24,M
bca,25,F
Table 2:
Name,Age,Sex
abc,25,M
acb,25,F
The Table 1 is the target table and Table 2 is the source table.
In the table 2 I have one Insert and one update record which needs to be merged with source table 1.
Query:
MERGE INTO table1 using table2 ON (table1.name=table2.name)
WHEN MATCHED AND table1.age <> table2.age AND table1.sex<>table2.sex
THEN UPDATE SET table1.age=table2.age AND table1.sex=table2.sex
WHEN NOT MATCHED
THEN INSERT (name,age,sex) VALUES (table2.name,table2.age,table2.sex)
Is the spark SQL support merge or is there another way of achieving it ?
Thanks
Sat
To use MERGE you need the Delta Lake option (and associated jars). Then you can use MERGE.
Otherwise, SQL Merge is not supported by Spark. The Dataframe Writer APIs with own logic are then needed. There are a few different ways to do this. Even with ORC ACID, Spark will not work in this way.
I have set of SQL select queries to execute and share the consolidated results in excel sheet, I'm using sqlyog to do this.
Every time I execute results are in multiple tables. Can I get the results in a single table?
Select * from a.table;
Select * from b.table;
To get the result in a single table you need to use JOIN's in your query.
For Example, I have a area_table and I have a height_table.
to get the result in the consolidated table I would use JOIN and write the query as:
Select a.*,b.* from area_table a
Join height_table b;