There is a table 'Phones', which includes a column 'phone_no', declared as varchar(20). It can be NULL, too.
Some of the values stored in this column are:
'(310) 369-1000', '(415) 623-1000', '(310) 449-3000', '(323) 956-8398', and '(800) 864-8377'.
I would like to filter out all the records where the phone number ends with '0', so I use the expression phone_no LIKE '%0'. However, the resulting recordset is empty! The same happens when using any number (not just 0) at the end of the pattern. Why? Where is the problem?
Related
Looks like I have 672 mission values, according to statistics.
There are NULL value in QuotedPremium column.
I implemented Clean Missing Data module where it should substitute missing values with 0, but for some reason I'm still seeing NULL values as QuotedPremium, but...it says that missing values are = 0
Here you see it tells me that missing values = 0, but there are still NULLs
So what really happened after I ran Clean Missing Data module? Why it ran succesfully but there are still NULL values, even though it tells that number of missing values are 0.
NULL is indeed a value; entries containing NULLs are not missing, hence they are neither cleaned with the 'Clean Missing Data' operator nor reported as missing.
Since they are not really missing values, its a string NULL which is added to all these cells. So, in order to substitute these values with 0 you can use this below:
Use Execute R Script module, and add this code in it.
dataset1 <- maml.mapInputPort(1); # class: data.frame
dataset1[dataset1 == "NULL"] = 0; # Wherever cell's value is "NULL", replace it with 0
maml.mapOutputPort("dataset1"); # return the modified data.frame
Image for same:
An Excel table as data source may contain error values (#NA, #DIV/0), which could disturbe later some steps during the transformation process in Power Query.
Depending of the following steps, we may get no output but an error. So how to handle this cases?
I found two standard steps in Power Query to catch them:
Remove errors (UI: Home/Remove Rows/Remove Errors) -> all rows with an error will be removed
Replace error values (UI: Transform/Replace Errors) -> the columns have first to be selected for performing this operations.
The first possibility is not a solution for me, since I want to keep the rows and just replace the error values.
In my case, my data table will change over the time, means the column name may change (e.g. years), or new columns appear. So the second possibility is too static, since I do not want to change the script each time.
So I've tried to get a dynamic way to clean all columns, indepent from the column names (and number of columns). It replaces the errors by a null value.
let
Source = Excel.CurrentWorkbook(){[Name="Tabelle1"]}[Content],
//Remove errors of all columns of the data source. ColumnName doesn't play any role
Cols = Table.ColumnNames(Source),
ColumnListWithParameter = Table.FromColumns({Cols, List.Repeat({""}, List.Count(Cols))}, {"ColName" as text, "ErrorHandling" as text}),
ParameterList = Table.ToRows(ColumnListWithParameter ),
ReplaceErrorSource = Table.ReplaceErrorValues(Source, ParameterList)
in
ReplaceErrorSource
Here the different three queries messages, after I've added two new column (with errors) to the source:
If anybody has another solution to make this kind of data cleaning, please write your post here.
let
src = Excel.CurrentWorkbook(){[Name="Tabelle1"]}[Content],
cols = Table.ColumnNames(src),
replace = Table.ReplaceErrorValues(src, List.Transform(cols, each {_, "!"}))
in
replace
Just for novices like me in Power Query
"!" could be any string as substitute for error values. I initially thought it was a wild card.
List.Transform(cols, each {_, "!"}) generates the list of error handling by column for the main funcion:
Table.ReplaceErrorValues(table_with errors, {{col1,error_str1},{col2,error_str2},{},{}, ...,{coln,error_strn}})
Nice elegant solution, Sergei
I have a query calculation that should throw me either a value (if conditions are met) or a blank/null value.
The code is in the following form:
if([attribute] > 3)
then ('value')
else ('')
At the moment the only way I could find to obtain the result is the use of '' (i.e. an empty character string), but this a value as well, so when I subsequently count the number of distinct values in another query I struggle to get the correct number (the empty string should be removed from the count, if found).
I can get the result with the following code:
if (attribute='') in ([first_query].[attribute]))
then (count(distinct(attribute)-1)
else (count(distinct(attribute))
How to avoid the double calculation in all later queries involving the count of attribute?
I use this Cognos function:
nullif(1, 1)
I found out that this can be managed using the case when function:
case
when ([attribute] > 3)
then ('value')
end
The difference is that case when doesn't need to have all the possible options for Handling data, and if it founds a case that is not in the list it just returns a blank cell.
Perfect for what I needed (and not as well documented on the web as the opposite case, i.e. dealing with null cases that should be zero).
I have this strange problem. i have a table with 10 columns of type character varying.
I need to have a function that searches all records and returns the id of the record which has all strings. Lets say records:
1. a,b,c,d,e
2. a,k,l,h
3. f,t,r,e,w,q
if i call this function func(a,d) it should return 1, if i call func(e,w,q) its should return 3.
The function is
CREATE OR REPLACE FUNCTION func(ma1 character varying,ma2 character varying,ma3 character varying,ma4 character varying)
DECLARE name numeric;
BEGIN
SELECT Id INTO name from Table WHERE
ma1 IN (col1,col2,col3,col4) AND
ma2 IN (col1,col2,col3,col4) AND
ma3 IN (col1,col2,col3,col4) AND
ma4 IN (col1,col2,col3,col4);
RETURN name;
END;
It's working 90% of the time, the weird problem is that some rows are not found.
Its not uppercase or lowercase problem.
What can be wrong, its version 9.1 on 64 bit win 7. I feel its like encoding or string problem but i can't see where and what.
//Ok i found the problem, it has to do with all column, if all 24 columns are filled in then its not working ?? but why ? are there limitations becouse there are 24 columns that i must compare with//
Can someone help me plz.
thanks.
The problem is (probably) that some of your columns have nulls.
In SQL, any equality comparison with a null is always false. This extends to the list of values used with the IN (...) condition.
If any of the values in the list are null, the comparison will be false, even if the value being sought is in the list.
The work-around is to make sure no values are null. which unfortunately results in a verbose solution:
WHERE ma1 IN (COALESCE(col1, ''), COALESCE(col2, ''), ...)
I suspect Bohemian is correct that the problem is related to nulls in your IN clauses. An alternative approach is to use Postgres's array contained in operator to perform your test.
where ARRAY[ma1,ma2,ma3,ma4] <# ARRAY[col1,col2,...,colN]
I am rolling up a huge table by counts into a new table, where I want to change all the empty strings to NULL, and typecast some columns as well. I read through some of the posts and I could not find a query, which would let me do it across all the columns in a single query, without using multiple statements.
Let me know if it is possible for me to iterate across all columns and replace cells with empty strings with null.
Ref: How to convert empty spaces into null values, using SQL Server?
To my knowledge there is no built-in function to replace empty strings across all columns of a table. You can write a plpgsql function to take care of that.
The following function replaces empty strings in all basic character-type columns of a given table with NULL. You can then cast to integer if the remaining strings are valid number literals.
CREATE OR REPLACE FUNCTION f_empty_text_to_null(_tbl regclass, OUT updated_rows int)
LANGUAGE plpgsql AS
$func$
DECLARE
_typ CONSTANT regtype[] := '{text, bpchar, varchar}'; -- ARRAY of all basic character types
_sql text;
BEGIN
SELECT INTO _sql -- build SQL command
'UPDATE ' || _tbl
|| E'\nSET ' || string_agg(format('%1$s = NULLIF(%1$s, '''')', col), E'\n ,')
|| E'\nWHERE ' || string_agg(col || ' = ''''', ' OR ')
FROM (
SELECT quote_ident(attname) AS col
FROM pg_attribute
WHERE attrelid = _tbl -- valid, visible, legal table name
AND attnum >= 1 -- exclude tableoid & friends
AND NOT attisdropped -- exclude dropped columns
AND NOT attnotnull -- exclude columns defined NOT NULL!
AND atttypid = ANY(_typ) -- only character types
ORDER BY attnum
) sub;
-- RAISE NOTICE '%', _sql; -- test?
-- Execute
IF _sql IS NULL THEN
updated_rows := 0; -- nothing to update
ELSE
EXECUTE _sql;
GET DIAGNOSTICS updated_rows = ROW_COUNT; -- Report number of affected rows
END IF;
END
$func$;
Call:
SELECT f_empty2null('mytable');
SELECT f_empty2null('myschema.mytable');
To also get the column name updated_rows:
SELECT * FROM f_empty2null('mytable');
db<>fiddle here
Old sqlfiddle
Major points
Table name has to be valid and visible and the calling user must have all necessary privileges. If any of these conditions are not met, the function will do nothing - i.e. nothing can be destroyed, either. I cast to the object identifier type regclass to make sure of it.
The table name can be supplied as is ('mytable'), then the search_path decides. Or schema-qualified to pick a certain schema ('myschema.mytable').
Query the system catalog to get all (character-type) columns of the table. The provided function uses these basic character types: text, bpchar, varchar, "char". Only relevant columns are processed.
Use quote_ident() or format() to sanitize column names and safeguard against SQLi.
The updated version uses the basic SQL aggregate function string_agg() to build the command string without looping, which is simpler and faster. And more elegant. :)
Has to use dynamic SQL with EXECUTE.
The updated version excludes columns defined NOT NULL and only updates each row once in a single statement, which is much faster for tables with multiple character-type columns.
Should work with any modern version of PostgreSQL. Tested with Postgres 9.1, 9.3, 9.5 and 13.