In the power query, I cannot use the + operator to add type date to type time natively. What are some ways to add date and time to create a datetime value?
Definitely not intuitive for an Excel user, but the Power Query method is:
date & time
let
Source = Table.FromRecords(
{[date=#date(2022,1,1), time = #time(1,15,0)]},
type table [date=date, time=time]),
#"Added Custom" = Table.AddColumn(Source, "datetime", each [date] & [time], type datetime)
in
#"Added Custom"
In the MS documentation for Power Query operators it shows x & y, where x=date and y=time, =>merged datetime
There are some code snippets that work. You can create a custom function as well if you use these often.
DateTime.FromText(Text.Combine({Text.From(DateValue), " ", Text.From(TimeValue)}))
Another is to convert them to durations and process them that way.
(DateValue - #date(1900,1,1))+(TimeValue - #time(0,0,0)) + #datetime(1900,1,1,0,0,0)
OR
List.Sum({DateValue - #date(1900,1,1), TimeValue - #time(0,0,0),#datetime(1900,1,1,0,0,0)})
Finally
#datetime(Date.Year(DateValue), Date.Month(DateValue), Date.Day(DateValue), Time.Hour(TimeValue), Time.Minute(TimeValue), Time.Second(TimeValue))
Related
I have a list of elemental impurities in power query which I wish to filter according to whether or not they exist on another list known as the prop65 list.
The screenshot below shows a simplified example of what I am trying to achieve.
I appreciate that using formulas However I don't know how to achieve this using a Power query solution. If anyone know how to achieve this it would be appreciated.
Data shown:
Aluminium 33.885
Antimony 0.6777
Arsenic 3.5064
Barium 2.259
Boron 1.3554
Bromoform 0.555
Cadmium 3.18895
Chromium 0.33885
Cobalt 1.1295
Copper 0.4518
Indium 0.4518
Simplified Prop65 List
Arsenic
Bromoform
Cadmium
Furan
Lead
Nafenopin
Here is one way to do that:
Read in the two tables
Do an Inner Join
let
//get original data
Source = Excel.CurrentWorkbook(){[Name="Data"]}[Content],
data = Table.TransformColumnTypes(Source,{{"Impurity", type text}, {"Amount (ppm)", type number}}),
//get Filter
Source2 = Excel.CurrentWorkbook(){[Name="Prop65"]}[Content],
filter = Table.TransformColumnTypes(Source2,{"Simplified Prop65 List", Text.Type}),
//Join them
filteredData = Table.Join(data,"Impurity", filter, "Simplified Prop65 List",JoinKind.Inner),
//Remove unneeded column
#"Removed Columns" = Table.RemoveColumns(filteredData,{"Simplified Prop65 List"})
in
#"Removed Columns"
Another method would be a filter (Table.SelectRows) method, but it may be slower with a large dataset. At least, in a single instance where I had an opportunity to compare, the Table.Join method was faster on a 100,000 row data set.
let
//get original data
Source = Excel.CurrentWorkbook(){[Name="Data"]}[Content],
data = Table.TransformColumnTypes(Source,{{"Impurity", type text}, {"Amount (ppm)", type number}}),
//get Filter
Source2 = Excel.CurrentWorkbook(){[Name="Prop65"]}[Content],
filter = Table.TransformColumnTypes(Source2,{"Simplified Prop65 List", Text.Type})[#"Simplified Prop65 List"],
//filter the rows
filteredData = Table.SelectRows(data, each List.Contains(filter,[Impurity]))
in
filteredData
I have a BigQuery table that contains a struct column called daySliderTimes in the following form:
daySliderTimes STRUCT<_field_1 STRUCT<_seconds INT, _nanoseconds INT>, _field_1 STRUCT<_seconds INT, _nanoseconds INT>.
_field_1 and _field_2 represent two different timestamps. _seconds and _nanoseconds represent time since the unix epoch.
I want to convert the data into a new STRUCT with the following form:
daySlidertimes STRUCT<startTime DATETIME, endTime DATETIME>
This is the table as seen in the BigQuery UI:
If you want to create a new table from the old one with the format daySlidertimes STRUCT<startTime DATETIME, endTime DATETIME>, you can cast the data in milliseconds and so then transform it to TIMESTAMP with the function "TIMESTAMP_MICROS", check this link to see the amount of functions to parse timestamp [1].
An example of the query should look something like this:
CREATE TABLE `project.dataset.new_table` AS
SELECT searchDocId,
STRUCT(TIMESTAMP_MICROS(CAST(
((daySliderTimes.field1.seconds * 1e+6) +
ROUND(daySliderTimes.field1.nanoseconds * 0.001)) AS INT64)) as
startTime,
TIMESTAMP_MICROS(CAST( ((daySliderTimes.field2.seconds * 1e+6) +
ROUND(daySliderTimes.field2.nanoseconds * 0.001)) AS INT64)) as endTime)
as daySliderTimes,
enabledDaySliders
FROM `project.dataset.old_table`
[1] https://cloud.google.com/bigquery/docs/reference/standard-sql/functions-and-operators#parse_timestamp
You can use TIMESTAMP_SECONDS() function. This function converts the seconds to DATETIME format.
Therefore, you are ale to transform daySliderTimes._field_1.seconds to a date using TIMESTAMP_SECONDS() function. As well as, for _field_2, then aggregate them in a new struct format.
During the creation of the view or table, in your select you can do the following:
WITH table_newStruct as(
SELECT
#Select all the desired fields
searchDocId,
STRUCT(TIMESTAMP_SECONDS(daySliderTimes._field_1.seconds) as startTime,
TIMESTAMP_SECONDS(daySliderTimes._field_.seconds) as endTime) as new_daySlidertimes
FROM 'table_source')
SELECT searchDocId, new_daySlidertimes
FROM 'table_newStruct'
In addition, the returned TIMESTAMP should be in the following format 1970-01-01 00:00:00 UTC. You can format it using the FORMAT_DATE() function.
I have a widget called 'filedate' where you can specify a date, were you enter the date in the format 'yyyy-mm-dd', my example will use '2019-10-01' .
I get the value from the widget with the following:
val fileloaddate = dbutils.widgets.get("filedate")
If I print the fileloaddate, it shows 2019-10-01, I need to use it in a query so if I do a 'select to_timestamp(${fileloaddate}, "yyyy-mm-dd")' it errors as it's seeing the variable as '((2019 - 8) -18). If I cast the string to a date, for example
select to_timestamp(to_date(${prundate}), "yyyy-mm-dd")
with the error of 'cannot resolve 'CAST(((2019 - 8) - 18) AS DATE)'
select to_timestamp(to_date('2019-10-01'), "yyyy-mm-dd")
works fine. I have googled around the answer but can't seem to see what I doing wrong.
thanks
Azure DataBrick you can use getArgument to convert date into desire output
dbutils.widgets.text("x","2018-09-12")
select to_timestamp(to_date(getArgument("x")), "yyyy-mm-dd")
hope this helps you
If Scala is disabled then you need to use the legacy method for retrieving sql parameters from widgets
.
select * from database where parameter='$file'
For sql I can pass the date as a text
%sql
create widget 'StartDate' DEFAULT 'YYYY-MM-DD' (this is simply text to remind the user to input the date in a format the sql query can use)
select * from my_table Between '$StartDate' and '$EndDate'
https://docs.databricks.com/notebooks/widgets.html#legacy-input
https://docs.databricks.com/notebooks/widgets.html
Trying to figure out how to perform the sum of two columns inside a Table.Group step.
This one throws:
Expression.Error: We cannot convert Type to List type.
M Code:
= Table.Group(PreviousStep, {"PF"}, {{"ColumnName", each List.Sum({[Column1], [Column2]})}, type number})
M Gurus: Is this even possible?
Objective: Simplify steps (actually I have 12 columns I want to reduce to 6 - adding pairs, grouping them, all in one simple step)
You can kind of do it if you create a custom column to sum within that step. Something like this:
= Table.Group(
PreviousStep,
{"PF"},
{{"ColumnName",
each List.Sum(
Table.AddColumn(
_,
"Custom", each [Column1] + [Column2]
)[Custom]
),
type number
}}
)
I've got an Excel 2016 spreadsheet set up with an Oracle db data source and I already have several queries set up in PowerQuery to get data from the tables in a specific schema and all is working well.
I now need to get some data from the data dictionary - I need to find the name of a trigger associated with a specific table in the schema - so I've set up a query to try to get data from user_triggers, but so far I've not been able to get it to work.
This is the query I have set up so far (SourceTableName is a reference to a named cell in the sheet to get the table name) :
let
STN = Excel.CurrentWorkbook(){[Name="SourceTableName"]}[Content],
#"Changed Type" = Table.TransformColumnTypes(STN,{{"Column1", type text}}),
table_name = #"Changed Type"{0}[Column1],
Source = Oracle.Database("MY_DB", [HierarchicalNavigation=true]),
Schema = Source{[Schema="MY_SCHEMA"]}[Data],
USER_TRIGGERS = Schema{[Name="USER_TRIGGERS"]}[Data]
in
USER_TRIGGERS
This works perfectly fine for the other queries I already have set up as long as the table name is one of the tables in the schema, but referring to a data dictionary view as in the above example doesn't seem to work.
The error I get when trying to run the this query is:
Expression.Error: The key didn't match any rows in the table.
Details:
Key=Record
Table=Table
Does anyone know if it's actually possible to get data from the data dictionary using powerquery and if it is what do I need to change to get it to work?
Thanks in advance for any help with this!
Cheers,
Dave
I've figured it out! Answering my own question in case it's useful for anyone else in the future
It's actually possible to specify an SQL query directly in the db connection line and you can include variable names from other parts of the query in the SQL, like so:
let
STN = Excel.CurrentWorkbook(){[Name="SourceTableName"]}[Content],
#"Changed Type" = Table.TransformColumnTypes(STN,{{"Column1", type text}}),
table_name = #"Changed Type"{0}[Column1],
upper_tn = Text.Upper(table_name),
Triggers = Oracle.Database("MY_DB", [HierarchicalNavigation=true, Query="select trigger_name from user_triggers where table_name = '" & upper_tn & "'"])
in
Triggers
Using the SQL query directly in this way seems to work fine for data dictionary views :)