I have just spun up my cluster and I was about to delete a table like I often do from with with Data workspace. However, it seems like Databricks have changed there interface and I'm now unsure how to delete a table without writing code.
Can someone show me how to delete a table from with Data Explorer please. For example I would like to delete the table trips from without Data Explorer
I would like to return to the option where I can delete a table from the Data Explorer tab as shown in the image
Data Explorer is a readonly tool. You can explore data, but for any action (DML, DDL) just use a cluster/warehouse and run the query. The only exception is that you can grant ownership/permissions with it.
Related
I am loading data via pipelines in ADLS gen2 container.
Now I want to create a table that has details that when the pipeline start running and then completed. like below fields
where
startts - start time of job
endts- end time of job
but extractts is the extraction time which is what i want to create.. is there any approch i can create table ?? help will be really appreciated.
It might not be exactly what you need, but you can add columns inside a copy activity with the property #pipeline().TriggerTime to get your "startts" field. Go to the bottom option on the source tab and select new, and on value choose "Add dynamic content".
Here is an example
For your "extractts" you could use the property of the activity called "executionDuration", which will give you the time it took to adf to run the activity in seconds. Again, use dynamic content and get this: #activity('ReadFile').output.executionDuration, then store this value wherever you need.
In this example I'm storing the value in a variable, but you could use it anywhere.
I dont understand your need for a field "endts", I would just simply do startts+extractts to get it.
I would like to find out if it's possible to replace the data in azure sql database with the latest one everytime when my pipeline is ingesting new data. I have a pipeline that takes data from azure blob storage to azure sql but the challenge that I am facing is that the pipeline runs every 5 minutes and it is concatenating new data on the existing table in the database and I want it to replace the data everytime the pipeline is running. I have tried everything on the internet and nothing worked for me and now the space of my azure sql is exhausted.
Anything that will help will be highly appreciated. thanks
How you update your data depends on the data lifecycle. It sounds like you want a truncate & replace, or a MERGE.
Create a landing table and a final table to report against.
The ADF copy component has a pre-script feature in the Sink. In the pre-script feature truncate the table you are bringing the data into, then bring in the data from ADL to a landing table.
Via stored procedure wrapped in a transaction, update the final table with a try/catch from the landing table. On a failed try, roll back the transaction and throw an error. The landing table allows you to wrap the truncate in a transaction with the insert preventing you from having an empty table on issues with the insert.
Truncate & Replace
Begin a transaction
Truncate the final table
BEGIN TRANSACTION
Insert into it from the landing table and commit transaction
BEGIN CATCH
Rollback transaction and throw error.
Merge
Begin a transaction
BEGIN TRY
When data found in source that is not in target THEN inert into target.
When data found in source that is in target but differs THEN update target.
When data found in target that is not in source THEN delete from target.
COMMIT TRANSACTION
BEGIN CATCH
Rollback transaction and throw error.
I would like to create a script in Excel to look at my IBM Personal Communications Screen or data source(preferred) and then to parse that screen text into cells.
Any ideas on where to even begin?
Example of Screen
I would handle this using SQL inside of Excel. This technique will not grab data from your display file. It will instead use your second option of getting it from the underlying database tables.
Create an .odc file with connection information to get to the IBM i DB2 database. On my PC, ODBC and OLE DB drivers were installed along with the emulator so that you can connect to the IBM i using Microsoft protocols.
In Excel, choose "Data, Connections". In that dialog box pick "Add" and choose your connection file. It may ask you for a specific table to select from, but you can pick whichever because we are going to change it in just a minute.
A new data connection is now on the list in the dialog box. Choose "Properties" to edit it.
Replace the autogenerated SQL on the "Definition" tab "Command Text" field with SQL that actually selects the data relevant to you from the tables you are interested in. You can get as fancy as you want with this SELECT statement: joins, CTE's, sub-queries, anything that is well-formed DB2 for i SQL.
Close these windows, saving changes. The connection is now independent of the original .odc source.
Choose "Exisiting Connections" to run the query and to have Excel populate a worksheet table with the data. If you have SQL errors, you will get them at this point when the query is actually sent to the IBM i.
Edit the SQL until you are happy with the data you are returning. You can also use "Refresh" to re-run it and get the latest data whenever you want.
I am facing issue, with Azure Power Apps. When I add users from Azure AD:
AzureAD.GetGroupMembers("Object ID").value
And then I get users to my table(my table name: TableCollection), but then I have created a refresh for my Table , but it does not refresh it when I click refresh.
My syntax for Refresh button is:
AzureAD.RefreshTokens("Object ID")
I also have tried to just simple refresh for my table(my table name: TableCollection):
Refresh(TableCollection)
Does anyone know on this situation how to give correct syntax for refresh button?
Refresh() method can only be used on a data source.
So you should add AzureAD.GetGroupMembers("Object ID").value into a data source (I name it as AADData).
Collect( AADData, AzureAD.GetGroupMembers("ed9a0284-3ce5-4ae4-9d0c-68eb8e460330").value)
And reference the data source from your table.
Then add a refresh button with the OnSelect = Clear(AADData); Collect( AADData, AzureAD.GetGroupMembers("Object ID").value).
In order to fill in the TableCollections you would have used the Collect function.
Something like this
collect(TableCollections,AzureAD.GetGroupMembers("Object ID").value)
In order to refresh this you can use the method ClearCollect
The ClearCollect function deletes all the records from a collection. And then adds a different set of records to the same collection.
Clearcollect(TableCollections,AzureAD.GetGroupMembers("Object ID").value)
So in the above line, the tablecollection is cleared and the records are freshly updated from the data source.
I am attempting a simple SELECT action on a Source JSON dataset in an Azure Data Factory data flow, but I am getting an error message that none of the columns from my source are valid. I use the exact configuration as the video, except instead of a CSV file, I use a JSON file.
In the video, at 1:12, you can see that after configuring the source dataset, the source projection shows all of the columns from the source schema. Below is a screen shot from the tutorial video:
image.png
And below is a screen shot from my attempt:
(I blurred the column names because they match column names from a vendor app)
Note in my projection, I am unable to modify the data types or the format. I'm not sure why not, but I don't need to modify either so I moved on. I did try with a CSV and I was able to modify the data types. I'm assuming this is a JSON thing, but I'm noting here just in case there is some configuration that I should take a look at.
At 6:48 in the video, you'll see the user add a select task, exactly as I have done. Below is a screen shot of the select task in the tutorial immediately following adding the task:
Notice the source columns all appear. Below is a screen shot of my select task:
I'm curious why the column names are missing? If I type them in manually, I get an error: "Column not found"
For reference, below are screen shots of my Data Source setup. I'm using a Data Lake Storage Gen2 Linked Service connected via Managed Identity and the AutoResolvingIntegrationRuntime.
Note that I tried to do this with a CSV as well. I was able to edit the datatype and format on a CSV, but I get the same column not found error on the next step.
Try doing this in a different browser or clear your browser cache. It may just be a formatting thing in the auto-generated JSON. This has happened to me before.