I'm trying to do a logical test to compare two activity outputs.
The first one is giving back a file name (derived from GetMetaData)and the other one distinct filenames that are already in the database (derived from a lookup Activity).
So the first activity is giving X.csv (a file in a Blob0 while the second one is giving a list Y.csv; Z.csv (the result of the lookup Select distinct from table X)
Based on this outcome I would say that the logical test is true so ADF has to start a particular activity. I'm using the expresion below, but despite the fact there are no errors the outcome is always false. What am I doing wrong? I guess it has something to do with the lookup activity because the query will give a list of values I think.
please help thanks in advance!
#equals(activity('GetBlobName').output,activity('LookupBestandsnaam').output)
Output activity LookupBestandsnaam:
Output activity GetBlobName:
The output of Lookup and Get Metadata are different:
Lookup activity reads and returns the content of a configuration file
or table.
Get Metadata activity to retrieve the metadata of any data in Azure
Data Factory
We can't compare the output directly. You will always get false in the if condition expression.
Please try bellow expression:
#equals(activity('GetBlobName').output.value.name,activity('LookupBestandsnaam').output.value.bestandsnaam)
Update:
Congratulations that you use another way to solved it:
"I have now replaced the if condition with a stored procedure that uses an IF exists script running on the basis of look-up activity in ADF."
Related
I am currently using the Azure Data Factory to retrieve a fixed-length file from blob storage and trying to import the record into my database.
Fixed-length.txt
0202212161707
1Tom
1Kelvin
1Michael
23
The first row is the header record, which is start with '0' and comes up with the creation time.
The following row are the detail record, started with '1' and comes up with user name.
The last row is the end record, which started with '2' and comes up with the sum of the detail record.
However, I want to validate that the data of the file is correct before I insert those records. I would like to check if the checksum is correct first, and then only insert all those record started with 1.
Currently, I insert all those record line by line into SQL DB and run a stored procedures to perform the tasks. Is it possible to utlize Azure Data Factory to do it?? Thank you.
I reproduced your issue follow below steps.
First take one look up activity to view all the data from file and apply filter on that data.
Then take one set variable activity and get the last row's last element e.g. 23 as 3 with below dynamic expression.
#last(activity('Lookup1').output.value[sub(length(activity('Lookup1').output.value),1)].Prop_0)
Then take one Filter activity to filter rows with 1 prefix with below items value and condition
items : #activity('Lookup1').output.value
condition : #startswith(item().Prop_0,'1')
after filter take ForEach activity to Append those values in an array
Then inside for each activity take Append variable activity it will create an array with filtered values.
Now take If condition with expression which checking value of set variable 1 and length of Append array variable is same or not.
#equals(int(variables('sum')),length(variables('username')))
Then inside true condition, add your copy activity to copy data if condition is true
My Sample Output:
0202212161707
1Tom
1Kelvin
23
for above data control is going to false condition.
0202212161707
1Tom
1Kelvin
1Michael
23
for above data control is going to true condition.
As someone with a background in Alteryx, it has been a slow process to get up to speed with the expressions and syntax within Azure Data Factory data flows. I am trying to filter out rows containing the following string in a similar manner to this Alteryx filter code below:
!Contains([Subtype], "News")
After scrolling through all the string expressions in Azure Data Factory, I am struggling to find anything similar to the logic above. Thanks in advance for any help you can provide me on this front!
You can use Filter transformation in ADF Data flow and give the condition
for any column like below:
My Sample Data:
Here I am filtering out the rows the which contains a string of "Rakesh" in the Name column with the Data flow expression instr(Name,"Rakesh")==0.
instr() returns number of common letters. Our condition satisfies if its result is 0.
Filter Transformation:
.
Output in Data preview of filter:
You can see the remaining rows only in the result.
I am trying to parameterize a mapping data flow in Azure DF UI. I am declaring a variable in the pipeline, which is taking the result from a lookup value - the current timestamp of a table. In the mapping dataflow though I am defining a parameter, which I would like to take the value of the variable declared in the pipeline. The variable is of type string, holding the current timestamp value.
this is the pipeline in debug
the dataflow parameter is taking the value of the variable
I am setting empty string as a default value - I tried also setting it to take variable value
that is a filter expression to filter based on this timestamp
So I guess I have to add a default value to the parameter value inside the Data Flow. How should this be done using the expression language? Thank you in advance!
When you are passing the Data flow parameter value from the Pipeline, the default value is not at all required. You can leave it as it is without giving any value.
When you debug the pipeline, the value is taken from the pipeline only and not from the default value.
For the Data preview of each transformation, you can provide a temporary hardcoded value by going through the click to see parameters below. It will help us to check whether the Data flow preview is giving correct results or not.
Please go through my repro for your reference:
This is my sample data
After creating the myparam parameter in the Data flow, I have passed my variable myname to it from pipeline. Here, I have given the value of the myname variable as #string('Rakesh') to filter this name.
Don't check that checkbox.
The condition in the Filter as Name==$myparam.
Result:
The best way to add a default value you can add a previous timestamp JUST LIKE THIS to hold and btw it will not be used as its an dynamic paramter so dont worry just place it there, but it will allow the debug to preview data and example is and it run in my case as i am also getting the same lookup max date and using in an dynamic Query
I'm having issues using the Exists Transformation within a Data Flow with a generic dataset.
I have two sources (one from staging table "sourceStg", one from DWH table "sourceDwh") and want to compare if the UniqueIdentifier-Column in the staging table is existing in the UniqueIdentifier-Column in the DWH table. For that I have a generic data set which I query with a SQL statement containing parameters.
When I open the "Exists settings" I cannot choose any Column from the source in the conditions since the source is generic and has no Projection until I run the data flow. However, I have a parameter which I get from the parent pipeline which provides me the name of the Column containing the UniqueIdentifier (both column names in staging / DWH are the same).
I tried to add following statement "byName($UniqueIdentifier)" in the left and right column field but the engine resolves them both as the sourceStg-Column since the prefix of the source-transformations is missing and it defaults to the first one. What I basically now try to achieve is having some statement as followed defining the correct source-transformation and the column containing the unique identifier with a parameter.
exists(sourceStg#$UniqueIdentifier == sourceDwh#$UniqueIdentifier)
But either the expression cannot be parsed or the result does not retrieve the actual UniqueIdentifier value from the column but writes the statement (e.g. sourceStg#$UniqueIdentifier) as column value.
The only workaround I found so far is having two derived columns which adds a suffix to the UniqueIdentifier-Column in one source and a new parameter $UniqueIdentiferDwh which is populate with the parameter $UniqueIdentifier and the same suffix as used in the derived column.
Any Azure Data Factory experts out there to help?
Thanks in advance!
In my logic app one step takes some filtered entities from Azure Table Storage. Filter consist of two conditions:
One field has to be equal to some constant value
Other field (datetime) has be to less or equal than current time minus 10min
It worked ok until last month when it started to return wrong results as seen in the screen below:
And the connector in Edit Mode:
I cannot work out what is happening. If I edit the row in the Azure Table (just click Update without changing anything) it starts to work properly. I thought that maybe the field was set with wrong type, but everything seems ok:
Maybe your error is caused by the wrong type of CreatedDate, you can refer to this post.
Simply put, you insert a time-formatted String into Azure Table data formatted as DateTime. This is shown in the portal as type DateTime, but it is actually a String.
Solution:
1. If you want to insert data of type DateTime, you can specify odata.type, please refer to the following example：
{
"Address":"Mountain View",
"Age":23,
"AmountDue":200.23,
"CustomerCode#odata.type":"Edm.Guid",
"CustomerCode":"c9da6455-213d-42c9-9a79-3e9149a57833",
"CustomerSince#odata.type":"Edm.DateTime",
"CustomerSince":"2008-07-10T00:00:00",
"IsActive":true,
"NumberOfOrders#odata.type":"Edm.Int64",
"NumberOfOrders":"255",
"PartitionKey":"mypartitionkey",
"RowKey":"myrowkey"
}
Reference:
https://learn.microsoft.com/en-us/rest/api/storageservices/understanding-the-table-service-data-model#property-types
https://learn.microsoft.com/en-us/rest/api/storageservices/inserting-and-updating-entities
2. Define CreatedDate as String type, but this is not a very good solution, it is better to insert the correct DateTime data.