I can not find any documentation on how to view and drop fully empty rows from an ADX source table.
Using where field == "" doesn't return what we are looking for, and the microsoft docs don't provide much insight. Does anyone know any ways to maybe filter these rows out of the ingestion source table in the first place, or run a cleaning function to automatically drop these rows?
Thanks in advance!
Prefer filter during ingestion over delete.
Demo
.create table t(Id:int, txt:string, val:real)
.ingest inline into table t <|
1,Hello,2.3
,,
,,
2,World,4.5
3,,null
,,
t
Id
txt
val
1
Hello
2.3
2
World
4.5
3
Naive checking
t
| where not
(
isnull(Id)
and isempty(txt) // string is never null
and isnull(val)
)
or
t
| where isnotnull(Id)
or isnotempty(txt)
or isnotnull(val)
Shortcut
t
| where tostring(pack_all(true)) != "{}"
Id
txt
val
1
Hello
2.3
2
World
4.5
3
Related
How do I query in cassandra for != null columns.
Select * from tableA where id != null;
Select * from tableA where name != null;
Then I wanted to store these values and insert these into different table.
I don't think this is possible with Cassandra. First of all, Cassandra CQL doesn't support the use of NOT or not equal to operators in the WHERE clause. Secondly, your WHERE clause can only contain primary key columns, and primary key columns will not allow null values to be inserted. I wasn't sure about secondary indexes though, so I ran this quick test:
create table nullTest (id text PRIMARY KEY, name text);
INSERT INTO nullTest (id,name) VALUES ('1','bob');
INSERT INTO nullTest (id,name) VALUES ('2',null);
I now have a table and two rows (one with null data):
SELECT * FROM nullTest;
id | name
----+------
2 | null
1 | bob
(2 rows)
I then try to create a secondary index on name, which I know contains null values.
CREATE INDEX nullTestIdx ON nullTest(name);
It lets me do it. Now, I'll run a query on that index.
SELECT * FROM nullTest WHERE name=null;
Bad Request: Unsupported null value for indexed column name
And again, this is done under the premise that you can't query for not null, if you can't even query for column values that may actually be null.
So, I'm thinking this can't be done. Also, if null values are a possibility in your primary key, then you may want to re-evaluate your data model. Again, I know the OP's question is about querying where data is not null. But as I mentioned before, Cassandra CQL doesn't have a NOT or != operator, so that's going to be a problem right there.
Another option, is to insert an empty string instead of a null. You would then be able to query on an empty string. But that still doesn't get you past the fundamental design flaw of having a null in a primary key field. Perhaps if you had a composite primary key, and only part of it (the clustering columns) had the possibility of being empty (certainly not part of the partitioning key). But you'd still be stuck with the problem of not being able to query for rows that are "not empty" (instead of not null).
NOTE: Inserting null values was done here for demonstration purposes only. It is something you should do your best to avoid, as inserting a null column value WILL create a tombstone. Likewise, inserting lots of null values will create lots of tombstones.
1) select * from test;
name | id | address
------------------+----+------------------
bangalore | 3 | ramyam_lab
bangalore | 4 | bangalore_ramyam
bangalore | 5 | jasgdjgkj
prasad | 11 | null
prasad | 12 | null
india | 6 | karnata
india | 7 | karnata
ramyam-bangalore | 3 | jasgdjgkj
ramyam-bangalore | 5 | jasgdjgkj
2)cassandra does't support null values selection.It is showing null for our understanding.
3) For handling null values use another strings like "not-available","null",then we can select data
I want to combine 2 result set into one.
Requirement: I am working on "Workbook" in azure and trying to add a drop-down as a parameter.I need to add values in the drop down using query. I retrieved the running apps
from below query. I need to add custom value to the result set.
Table 1: (Holds all the Function Apps running)
let AvailableApps = customEvents
| summarize by operation_Name
| order by operation_Name asc;
I need to combine one value to the above result set ie: "All Apps" .
How can we achieve it. I tried altering below code to suit mine,
let Range10 = view () { range MyColumn from 1 to 10 step 1 };
Can anyone shed some light on this.
Sorry if this is a duplicate question.
Edited: I figured out one way, Let me know if this is the best approach.
let AvailableApps = customEvents
| summarize by operation_Name
| order by operation_Name asc;
let DefaultValue = datatable (operation_Name:string)
["All Apps"];
union DefaultValue,AvailableApps;
When dealing with Drop Down parameters you can add an "All" choice from within the workbook UI that will be appended to the result set of the kusto query. For example:
See the docs for more info.
As the title states, I am trying to do a merge of 2 tables. I want a nested joint where the values from the first table are always there and rows matching the second table are added to the first. I believe this is known as the nested join.
Unfortunately, it only allows for 1 key to 1 key matching where as I need it for 1 key in table 1 to 2 keys in table 2
Here is an example
Table1:
Group
..
..
Time
Date
Table2:
Group 1
Group 2
..
..
..
Other Info
What I want is where "Group = Group 1 OR Group = Group 2" and display the matching row from table 2 nested into Table 1
I looked at the following example but I must be confused by the syntax because it doesn't seem to be working for me.
How to join two tables in PowerQuery with one of many columns matching?
So after further investigation of the answer post I linked earlier, I will add an explanation of it here:
Table.AddColumn(Source, "Name_of_Column",
(Q1) => Table.SelectRows(Query2,
each Q1[Col_from_q1] = [Col_from_q2] or Q1[Col_from_q1] = [2_Col_from_q2]
)
)
So this did work for me and it adds an extra column that needs to be expanded to get all the values from the table. What i would add is that I don't know / haven't tested if there are multiple matches and how it treats it, based on nestedjoin, I would assume that it will duplicate rows in the first table.
So I have a lot of rows taken up by excel. I have 10,000 rows or so taken up by data and I am working with 10,000 or different IDs. Is there a way to query off an oracle database just 1 time by capturing the entire ID column as a group and including the group in the WHERE query instead of looping the 10,000 assets and query the database 10,000 times?
Sorry for not providing code. I really have not attempted this because I dont know if a solution exists.
Something like what you are asking can be accomplished in a two step process. First, by creating SELECT-FROM-DUAL queries for the relevant IDs, and second, inputting those queries into your main query and joining against them to limit to only the returns you need.
For the first step, use Excel to create SELECT-FROM-DUAL subqueries.
If your ID column starts in cell A2, copy the following formula into an empty cell on the same row and drag it down the column until all rows with an ID also have the formula. Alter the references to cell A2 and A3 if your IDs don't start in cell A2.="SELECT "&A2&" AS id FROM DUAL"&IF(NOT(ISBLANK(A3)), " UNION ALL", "")
Ultimately, what we want is a block of SELECT-FROM-DUAL statements that look like the below. Note that the last statement will not end in "UNION ALL", but all other statements should.
| IDs | Formula |
|----- |------------------------------------ |
| 1 | SELECT 1 AS id FROM DUAL UNION ALL |
| 2 | SELECT 2 AS id FROM DUAL UNION ALL |
| 3 | SELECT 3 AS id FROM DUAL UNION ALL |
| 4 | SELECT 4 AS id FROM DUAL UNION ALL |
| 5 | SELECT 5 AS id FROM DUAL UNION ALL |
| 6 | SELECT 6 AS id FROM DUAL |
For the second step, add all the SELECT-FROM-DUAL statements to your main query and then add an appropriate JOIN condition.SELECT
*
FROM table_you_need tyn
INNER JOIN (
SELECT 1 AS id FROM DUAL UNION ALL
SELECT 2 AS id FROM DUAL UNION ALL
SELECT 3 AS id FROM DUAL UNION ALL
SELECT 4 AS id FROM DUAL UNION ALL
SELECT 5 AS id FROM DUAL UNION ALL
SELECT 6 AS id FROM DUAL
) your_ids yi
ON tyn.id = yi.id
;
If you had a shorter list of IDs you could use a similar strategy to create an ID list for a WHERE ids IN (<list_of_numbers>) clause, but the IN list is typically limited to 100 items, and consequently would not work for your current question.
You can import data from Excel using Toad or SQL Developer. You need to create a table first in the database.
You can read the data directly with external tables if you save the excel file as a CSV file to a folder on the database server that the database can access.
You can read files as Excel (xls or xlsx format) using a PL/SQL library.
There are probably a few other ways I haven't thought of as well. This is a very common question.
I have taken a good look at the Teradata Syntax reference to no avail
I have some rows with numbers:
ID
Mickey
Laura9
Larry59N
How do I take away the integers from my rows?
I understand that SUBSTR(id, 0, index(id, '%FORMAT%')) would work, but I don't know what could I enter in the %FORMAT% area to just find integers.
You can use oTranslate to remove numbers:
BTEQ -- Enter your SQL request or BTEQ command:
Select the_name, oTranslate( the_name, 'a0123456789','a')
from
( SELECT 'Larry59N' the_name FROM ( SELECT 'X' DUMMY ) a
UNION ALL
SELECT 'Laura9' FROM ( SELECT 'X' DUMMY ) b
UNION ALL
SELECT 'Mickey' the_name FROM ( SELECT 'X' DUMMY ) c
) d
;
*** Query completed. 3 rows found. 2 columns returned.
*** Total elapsed time was 1 second.
the_name oTranslate(the_name,'a0123456789','a')
-------- -----------------------------------------------------
Larry59N LarryN
Laura9 Laura
Mickey Mickey
HTH.
Cheers.
Unfortunately, I don't believe there is a function native to Teradata that will accomplish this. I would suggest looking at the UDF's posted on the Teradata Developer Exchange (link). The function eReplaceChar in particular looks like it may help you accomplish what you are looking to do with this data. The UDF's found at the link above were published under the Apache 2.0 license so you should not have any problems using them.