I need conditional output in stream analytics,like i have one lookup data table.while data streaming i need to check in lookup table with if data is exists or not. if data is exists then add data in eg.output1 or if data is not exists then add data in eg.output2.
This if statement is more then 4 or 5 times so multiple query is not possible
You should use CASE statements, sample code from doc and you can modify to meet your needs:
CASE xxx
WHEN xxx THEN yyy
ELSE
Related
I am currently using the Azure Data Factory to retrieve a fixed-length file from blob storage and trying to import the record into my database.
Fixed-length.txt
0202212161707
1Tom
1Kelvin
1Michael
23
The first row is the header record, which is start with '0' and comes up with the creation time.
The following row are the detail record, started with '1' and comes up with user name.
The last row is the end record, which started with '2' and comes up with the sum of the detail record.
However, I want to validate that the data of the file is correct before I insert those records. I would like to check if the checksum is correct first, and then only insert all those record started with 1.
Currently, I insert all those record line by line into SQL DB and run a stored procedures to perform the tasks. Is it possible to utlize Azure Data Factory to do it?? Thank you.
I reproduced your issue follow below steps.
First take one look up activity to view all the data from file and apply filter on that data.
Then take one set variable activity and get the last row's last element e.g. 23 as 3 with below dynamic expression.
#last(activity('Lookup1').output.value[sub(length(activity('Lookup1').output.value),1)].Prop_0)
Then take one Filter activity to filter rows with 1 prefix with below items value and condition
items : #activity('Lookup1').output.value
condition : #startswith(item().Prop_0,'1')
after filter take ForEach activity to Append those values in an array
Then inside for each activity take Append variable activity it will create an array with filtered values.
Now take If condition with expression which checking value of set variable 1 and length of Append array variable is same or not.
#equals(int(variables('sum')),length(variables('username')))
Then inside true condition, add your copy activity to copy data if condition is true
My Sample Output:
0202212161707
1Tom
1Kelvin
23
for above data control is going to false condition.
0202212161707
1Tom
1Kelvin
1Michael
23
for above data control is going to true condition.
With the "upsert option" , should I expect to see "0" as "Rows Written" in a copy activity result summary?
My situation is this: The source and sink table columns are not exactly the same but the Key columns to tell it how to know the write behavior are correct.
I have tested and made sure that it does actually do insert or update based on the data I give to it BUT what I don't understand is if I make ZERO changes and just keep running the pipeline , why does it not show "zero" in the Rows Written summary?
The main reason why rowsWritten is not shown as 0 even when the source and destination have same data is:
Upsert inserts data when a key column value is absent in target table and updates the values of other rows whenever the key column is found in target table.
Hence, it is modifying all records irrespective of the changes in data. As in SQL Merge, there is no way to tell copy activity that if an entire row already exists in target table, then ignore that case.
So, even when key_column matches, it is going to update the values for rest of the columns and hence counted as row written. The following is an example of 2 cases
The rows of source and sink are same:
The rows present:
id,gname
1,Ana
2,Ceb
3,Topias
4,Jerax
6,Miracle
When inserting completely new rows:
The rows present in source are (where sink data is as above):
id,gname
8,Sumail
9,ATF
I have a question regarding Power Query and Tables as parameters for excel.
Right now I can create a table and use it as a parameter for Power query via Drill down.
But I'm unsure how i would proceed with a Table that has multiple values. How can a table be recognized with multiple "values" as a parameter
For example:
I have the following rawdata and parameter tables
Rawdata+parametertables
Now if I wanted to filter after Value2 with a parameter tables I would do a drill down of the parameter tables and load them to excel.
After that I have two tables that I can filter Value2 with an OR Function by 1 and 2
Is it possible to somehow combine this into 1 Table and that it still uses an OR Function to search
Value2
Im asking because I want it to be potentially possible to just add more and more parameters into the table without creating a new table everytime. Basically just copy paste some parameters into the parameter table and be done with it
Thanks for any help in advance
Assuming, you use Parameters only for filtering. There are other ways, but this one looks the best from performance point of view.
You may create Parameters table, so you have such tables:
Note, it's handy to have the same names (Value2) for key column in both tables, otherwise Table.Join will create additional column(s) after merging tables.
Add similar step to filter RawData table:
join = Table.Join(RawData, "Value2", Parameters, "Value2")
I'm trying to do a logical test to compare two activity outputs.
The first one is giving back a file name (derived from GetMetaData)and the other one distinct filenames that are already in the database (derived from a lookup Activity).
So the first activity is giving X.csv (a file in a Blob0 while the second one is giving a list Y.csv; Z.csv (the result of the lookup Select distinct from table X)
Based on this outcome I would say that the logical test is true so ADF has to start a particular activity. I'm using the expresion below, but despite the fact there are no errors the outcome is always false. What am I doing wrong? I guess it has something to do with the lookup activity because the query will give a list of values I think.
please help thanks in advance!
#equals(activity('GetBlobName').output,activity('LookupBestandsnaam').output)
Output activity LookupBestandsnaam:
Output activity GetBlobName:
The output of Lookup and Get Metadata are different:
Lookup activity reads and returns the content of a configuration file
or table.
Get Metadata activity to retrieve the metadata of any data in Azure
Data Factory
We can't compare the output directly. You will always get false in the if condition expression.
Please try bellow expression:
#equals(activity('GetBlobName').output.value.name,activity('LookupBestandsnaam').output.value.bestandsnaam)
Update:
Congratulations that you use another way to solved it:
"I have now replaced the if condition with a stored procedure that uses an IF exists script running on the basis of look-up activity in ADF."
For a requirement I need to create a SQL which should check the data from Table A.
If data exists in table a, query should return that result.
If data doesn't exists on table A, query should check data in table B and return the result.
I tried to use the case statement for this requirement, but I guess case statement is allowed for a single record.
Any help for this is appreciated.
Thanks in Advance.