Condition to Test Date
I am using Power Query to create a status column that checks the date against a specified date, like so:
However, this gives me the following error:
Expression.Error: We cannot convert the value null to type Logical.
Details:
Value=
Type=[Type]
The column does contain empty cells, which I want to report as "null" in the new column. I then tried the following logic, and it errors out as well:
Then I moved the null test to the top, and it finally works:
Why Does Order Matter?
Why does the third query produce the expected results but not the first one? This seems bizarre to me, so if there is something I am missing please let me know.
M is using lazy evaluation in the if statement. If the first statement is true, then it doesn't even bother evaluating the other conditions.
https://learn.microsoft.com/en-us/powerquery-m/m-spec-introduction
For computer language theorists: the formula language specified in
this document is a mostly pure, higher-order, dynamically typed,
partially lazy functional language.
Easy fix
On a step before your filter, choose "remove nulls" or "replace nulls with values"
using catch
If you want more flexibility, you can use a try + catch pair.
Step FirstTry is meant to; be your filter, then I added two ways to handle errors.
let
Source = Table.FromList(sample, Splitter.SplitByNothing(),
type table[Date = nullable date], null, ExtraValues.Error),
sample = {
#date(2020, 1, 1),
"text", null,
#date(2024, 1, 1)
},
filter = #date(2022, 1, 1),
FirstTry = Table.AddColumn(
Source , "Comparison", each filter > [Date], Logical.Type),
WithFallback = Table.AddColumn(FirstTry, "WithFallback",
each try
filter > [Date]
catch (e) => e[Message], type text),
WithPreservedDatatype = Table.AddColumn(WithFallback, "PreserveColumnType",
each try
filter > [Date]
catch (e) => null meta [ Reason = e[Message] ],
type logical)
in
WithPreservedDatatype
things to note
the query steps are "out of order", which is totally valid. ( above sample was referenced "before" its line )
Errors are propagated so an error on step4 could actually be step2. Just keep going up until you find it.
the schema says column [Date] is type date -- but it's actually type any.
What you need is to call Table.TransformColumnTypes to convert and assert datatypes
= Table.TransformColumnTypes( Source,{{"Date", type date}})
Now row 2 will correctly show an error, because text couldn't convert into a date
Better Understanding of NULLs
I was not understanding how Excel (or any other data tool) handles null values and how logical tests are performed on null values. This response on Reddit really helped clarify this in my mind:
https://www.reddit.com/r/excel/comments/xu37dr/comment/iqtmivn/?utm_source=share&utm_medium=web2x&context=3
In short, logical tests involving nulls do not behave like you would expect.
For a deeper dive into this, read this excellent post:
https://bengribaudo.com/blog/2018/09/13/4617/power-query-m-primer-part9-types-logical-null-binary
My Solution
Given this enlightened understanding of nulls and how they behave in logical tests, I now know that I must either:
Convert the null values to empty strings ("") before the query
Test first for nulls in the query
My choice is to test for nulls first within the query, like so:
Related
I'm very new in using Cognos report studio and trying to filter some of the values and replace them into others.
I currently have values that are coming out as blanks and want to replace them as string "Property Claims"
what i'm trying to use in my main query is
CASE WHEN [Portfolio] is null
then 'Property Claims'
ELSE [Portfolio]
which is giving me an error. Also have a different filter i want to put in to replace windscreen flags to a string value rather than a number. For example if the flag is 1 i want to place it as 'Windscreen Claims'.
if [Claim Windscreen Flag] = 1
then ('Windscreen')
Else [Claim Windscreen Flag]
None of this works with the same error....can someone give me a hand?
Your first CASE statement is missing the END. The error message should be pretty clear. But there is a simpler way to do that:
coalesce([Portfolio], 'Property Claims')
The second problem is similar: Your IF...THEN...ELSE statement is missing a bunch of parentheses. But after correcting that you may have problems with incompatible data types. You may need to cast the numbers to strings:
case
when [Claim Windscreen Flag] = 1 then ('Windscreen')
else cast([Claim Windscreen Flag], varchar(50))
end
In future, please include the error messages.
it might be syntax
IS NULL (instead of = null)
NULL is not blank. You might also want = ' '
case might need an else and END at the bottom
referring to a data type as something else can cause errors. For example a numeric like [Sales] = 'Jane Doe'
For example (assuming the result is a string and data item 2 is also a string),
case
when([data item 1] IS NULL)Then('X')
when([data item 1] = ' ')Then('X')
else([data item 2])
end
Also, if you want to show a data item as a different type, you can use CAST
An Excel table as data source may contain error values (#NA, #DIV/0), which could disturbe later some steps during the transformation process in Power Query.
Depending of the following steps, we may get no output but an error. So how to handle this cases?
I found two standard steps in Power Query to catch them:
Remove errors (UI: Home/Remove Rows/Remove Errors) -> all rows with an error will be removed
Replace error values (UI: Transform/Replace Errors) -> the columns have first to be selected for performing this operations.
The first possibility is not a solution for me, since I want to keep the rows and just replace the error values.
In my case, my data table will change over the time, means the column name may change (e.g. years), or new columns appear. So the second possibility is too static, since I do not want to change the script each time.
So I've tried to get a dynamic way to clean all columns, indepent from the column names (and number of columns). It replaces the errors by a null value.
let
Source = Excel.CurrentWorkbook(){[Name="Tabelle1"]}[Content],
//Remove errors of all columns of the data source. ColumnName doesn't play any role
Cols = Table.ColumnNames(Source),
ColumnListWithParameter = Table.FromColumns({Cols, List.Repeat({""}, List.Count(Cols))}, {"ColName" as text, "ErrorHandling" as text}),
ParameterList = Table.ToRows(ColumnListWithParameter ),
ReplaceErrorSource = Table.ReplaceErrorValues(Source, ParameterList)
in
ReplaceErrorSource
Here the different three queries messages, after I've added two new column (with errors) to the source:
If anybody has another solution to make this kind of data cleaning, please write your post here.
let
src = Excel.CurrentWorkbook(){[Name="Tabelle1"]}[Content],
cols = Table.ColumnNames(src),
replace = Table.ReplaceErrorValues(src, List.Transform(cols, each {_, "!"}))
in
replace
Just for novices like me in Power Query
"!" could be any string as substitute for error values. I initially thought it was a wild card.
List.Transform(cols, each {_, "!"}) generates the list of error handling by column for the main funcion:
Table.ReplaceErrorValues(table_with errors, {{col1,error_str1},{col2,error_str2},{},{}, ...,{coln,error_strn}})
Nice elegant solution, Sergei
I have a field named field, and I would like to see if it is null, but I get an error in the query, my code is this:
let
Condition= Excel.CurrentWorkbook(){[Name="test_table"]}[Content],
field= Condition{0}[fieldColumn],
query1="select * from students",
if field <> null then query1=query1 & " where id = '"& field &"',
exec= Oracle.Database("TESTING",[Query=query1])
in
exec
but I get an error in the condition, do you identify the mistake?
I got Expression.SyntaxError: Token Identifier expected.
You need to assign the if line to a variable. Each M line needs to start with an assignment:
let
Condition= Excel.CurrentWorkbook(){[Name="test_table"]}[Content],
field= Condition{0}[fieldColumn],
query1="select * from students",
query2 = if field <> null then query1 & " some stuff" else " some other stuff",
exec= Oracle.Database("TESTING",[Query=query2])
in
exec
In query2 you can build the select statement. I simplified it, because you also have conflicts with the double quotes.
I think you're looking for:
if Not IsNull(field) then ....
Some data types you may have to check using IsEmpty() or 'field is Not Nothing' too. Depending on the datatype and what you are using.
To troubleshoot, it's best to try to set a breakpoint and locate where the error is happening and watch the variable to prevent against that specific value.
To meet this requirement, I would build a fresh Query using the PQ UI to select the students table/view from Oracle, and then use the UI to Filter the [id] column on any value.
Then in the advanced editor I would edit the generated FilteredRows line using code from your Condition + field steps, e.g.
FilteredRows = Table.SelectRows(TESTING_students, each [id] = Excel.CurrentWorkbook(){[Name="test_table"]}{0}[fieldColumn])
This is a minor change from a generated script, rather than trying to write the whole thing from scratch.
I have a nested if statement is returning "False" rather than the expected outcome.
Scenario
Table "High VoltageCables" has data in it that default to numeric but may contain characters: kVa
Table "Master" checks "High VoltageCables" data as blank or not blank, and returns "Failed Check 1","Passed Check 1". This works fine.
Table "Meta" then checks the results of "Master" and then tests "High VoltageCables" data for length between 1 and 6, regardless of whether record is numeric or string.
Formula
=IF(MASTER!H2="Passed Check 1",IF(LEN('High VoltageCables'!O2)>=1,IF(LEN('High VoltageCables'!O2<6),"Passed Check 2","Failed Check 2")))
This is partially succesful, as it returns "Passed Check 2" for the following sample data in the source table "High VoltageCables".
1 numeric, or
1kVa str, or
50000 numeric
However if a field in "High VoltageCables"is blank, the formula returns "FALSE" rather than "Failed Check 1"
I inherited this task, (and would have preferred to do the whole thing in Access using relatively simple queries) - and unfortunately I am new to nested If statements, so I am probably missing something basic...
NB the data in High VoltageCables must default to numeric for a further check to work.
The first and second IF's seem to be missing the else part. They should be added at the end between the ))) like ), else ), else )
Every IF statement consists of IF( condition, truepart, falsepart) if you have two nested ifs it will be something like IF( condition, IF( condition2, truepart2, falsepart2), falsepart)
Hope that makes it a little clearer
You do have an unaccounted for FALSE in the middle IF. Try bring the latter two conditions together.
=IF(Master!H2="Passed Check 1",IF(OR(LEN('High VoltageCables'!O2)={1,2,3,4,5}),"Passed Check 2","Failed Check 2"))
It's still a bit unclear on what to show or not show if Master!H2 does not equal "Passed Check 1".
I failed to construct the formula with a concluding "else" - "Failed Check 1"
Using jeeped's and Tom's suggestion and adding the final "else" part I have solved the problem:
=IF(MASTER!H2="Passed Check 1",IF(OR(LEN('High VoltageCables'!O2)={1,2,3,4,5}),"Passed Check 2","Failed Check 2"),"Failed Check 1")
I have a query calculation that should throw me either a value (if conditions are met) or a blank/null value.
The code is in the following form:
if([attribute] > 3)
then ('value')
else ('')
At the moment the only way I could find to obtain the result is the use of '' (i.e. an empty character string), but this a value as well, so when I subsequently count the number of distinct values in another query I struggle to get the correct number (the empty string should be removed from the count, if found).
I can get the result with the following code:
if (attribute='') in ([first_query].[attribute]))
then (count(distinct(attribute)-1)
else (count(distinct(attribute))
How to avoid the double calculation in all later queries involving the count of attribute?
I use this Cognos function:
nullif(1, 1)
I found out that this can be managed using the case when function:
case
when ([attribute] > 3)
then ('value')
end
The difference is that case when doesn't need to have all the possible options for Handling data, and if it founds a case that is not in the list it just returns a blank cell.
Perfect for what I needed (and not as well documented on the web as the opposite case, i.e. dealing with null cases that should be zero).