I have a source table and I want to use the update policy to copy the data from the source table to the target table but with an additional column that its value is obtained from an external table.
In the update policy I should copy the row from the origin table and look for the value I need to add in the external table through an Id.
I could easily use a view (join) but the problem is performance so I would like to have a table in ADX with the values I need.
The question is how the function should look like to copy the row from the source table to the target table by adding the column from the external table.
Update: it seems that an external table cannot be used in an Update Policy. I get an error when I try to define it
Exception: Request is invalid and cannot be processed: Semantic error: SEM0457: external_table(): usage is not allowed in this context.
your update policy command might look like this:
.alter table table2 policy update
#'[{"IsEnabled": true, "Source": "table1", "Query": "MyFunctionThatDoesTheLookup(table1, table2)", "IsTransactional": true, "PropagateIngestionProperties": false}]'
This means the table is getting data from table1. In between, you use a function. The function would look like this
.create-or-alter function
MyFunctionThatDoesTheLookup
(
raw_table:(id:string,col1:string,col2:int),
target_table:(id:string,col1:string,col2:int, col3:int)
)
{
let mylookuptable = lookuptablename;
raw_table
| join kind=left mylookuptable | project col3 on id
}
Related
Below is part of my table schema.
my table schema
I can use statement select factor.column(2) as code from factor to get the second column.
I wonder if I can SELECT the 2nd column, and the 4th to the last column.
Using metaprogramming is a good choice.
Get the column names of the table first using function columnNames.
Access column names by index and join them.
Create and execute a SQL statement using function sql and eval with the metacode generated by function sqlCol.
factor=table(2015.01.15 as date,`00000.SZ as code,-1.05 as factor_value,1.1 as factor01,1.2 as factor02)
colNames = factor.columnNames()
finalColNames = colNames[1] join colNames[3:]
sql(sqlCol(finalColNames), factor).eval()
code factor01 factor02
-------- -------- --------
00000.SZ 1.1 1.2
How do you select just one item for an object in an object via Select in Azure Data Factory
{
"CorrelationId": 123,
"ComponentInfo": {
"ComponentId": "1",
"ComponentName": "testC"
}
}
I have a join1 step in my ADF as such and Inspect and see results in that step:
But when I select just the two I need the Data Preview errors out:
Column source1#ComponentInfo not found. The stream is either not connected or column is unavailable
The Select is set as such:
source1#{source1#ComponentInfo}.ComponentName
What is wrong with my selecting ComponentName since it is an object - the selected method was selected from a drop down. I have tried to flatten the data but it is not an array and modify the schema but not sure if I am researching the right select object method.
I reproduced this with above sample data and used select transformation after join. I got the same error as above.
Here, the select may looking the source1#ComponentInfo as column which is an object in this case.
You can get the desired result using derived column transformation.
After join, use derived column and create two new columns for the required columns and give like below in the data flow expression from the input schema.
ComponentName column:
CorrelationId column:
You can see the result in the Data preview.
Then, you can filter the required columns using select transformation.
Result:
I'm working on a website in node js and use SQLite as a database for the first time.
I want to be able to use real for some form data and I noticed that every real in my database are converted to integer once the query is made.
To vizualize the database i am using DB Browser and i checked if the columns are defined as REAL which they are.
If i try to query a data set as 0.1 in my DB I get this :
sqlite> select step_variable
from variables
where id=38;
0.0
After trying as suggested the command TYPEOF(step_variable) it returned :
0.0|real
In the SQLite CREATE TABLE command, one defines a data type affinity, not a data type. SQLite supports the following five column affinities: TEXT, NUMERIC, INTEGER, REAL, NONE.
Thus the data type you specify when creating a table does not enforce a certain data type. You can supply any data type you want or even omit the data type.
CREATE TABLE table1(
column1 ABC,
column2 Others,
column3 WHATEVER);
CREATE TABLE table2(column1, column2, column3);
Populate tables:
INSERT INTO table1 VALUES( 1, 'my text', 123.45);
INSERT INTO table2 VALUES( 1, 'my text', 123.45);
Now let us check what SQLite made out of it:
SELECT column1, TYPEOF(column1) from table1
SELECT column2, TYPEOF(column1) from table1
SELECT column3, TYPEOF(column1) from table1
With:
column TYPEOF(column)
------------------------
1 INTEGER
my text TEXT
123.45 REAL
When you go through a query result e.g. by using sqlite2_step you can use the sqlite3_column_type statement to confirm the column type - unless you know the result anyway and simply cast the result to the data type expected.
Martin
I found the solution it was simply that i didn't save my file after modifying it.
So my original table is like this:
And I want to compress the table which only shows that factoryID and the month, and the max output like this:
How do I do this with spotfire in table visualization?
To go from the top table to the bottom table:
Do these steps:
Create a new calculated column off of your date for mmm-yy: Concatenate(Month([Date]),"-",right(String(Year([Date])),2))
Add a pivot transformation with the following parameters:
I created it as a new table but you do not have to.
Create two caculated columns.
c_month, c_max
for c_month, use the expression: Concatenate(Month([date]),"-",Year([date]))
for c_max, use the expression: Max([max]) OVER ([factory id])
Add third column called c_rank
expression:
If(Rank(RowId(),"asc",[factory id])=Min(Rank(RowId(),"asc",[factory id])),True)
//this will create a new calculated column which will give true only for every unique value of factory id.
Add a data table to your visualization
Add data limiter
Right click and under data --> Limit Data using expression, enter [c_rank] = true
Create Table with limted columns
Only add factory id, c_month and c_max to your column list.
I am trying to read in a table called operations that looks like so
"id";"name";
"1";"LASER CUTTING";
"2";"DEBURR";
"3";"MACHINING";
"4";"BENDING";
"5";"PEM";
"6";"WELDING";
"7";"PAINT PREPARATION";
"8";"PAINTING";
"9";"SILKSCREEN PREPARATION";
"10";"SILKSCREEN";
"11";"ASSEMBLY - PACKAGING";
"12";"LASER PREP";
I want to have a column in a worksheet that gets the appropriate name based on value of an operation_id column in another worksheet.
How do I lookup a particular cell in another worksheet dependent on the value of a cell?
Example
userid, operation_id, operation_name
bob, 3, MACHINING
You should look at the DGET(database,field,criteria) function, reference here.
Or you can use this worksheet function:
VLOOKUP(cellWithID, Sheet2!A1:B13, 2, FALSE)
where cellWithID is the cell with the ID value you want to use.
Maybe the Lookup() function would work better for you.
http://www.techonthenet.com/excel/formulas/lookup.php
Basing off this: "What I really want to do is just lookup the name for an operation without having to run a sql query every time or have an ugly huge if statement in every cell."
I guess what confuses me here is why you don't just use a join. You can always join that table as a lookup to whatever your sql statement is;
select operations.name, tableA.* from tableA
left outer join operations on operations.id = tableA.operationid
If you wanted to, you could functionize this; not recommended. Subqueries are generally speaking bad news. However,
create function dbo.LookupOperationName
(
#id int
)
returns varchar(100)
as
declare #returnvalue varchar(100)
select #returnvalue = name from Operations where id = #id
return #returnValue
would do the trick. Then you could:
select tablea.*,LookupOperationName(operationid) from tablea
Again, remember that your join example is much more performant. You could also create a view that had the join, and use the view in place of the table.... all kinds of things.