Filter inside ForEach activity in Azure Datafactory - azure

I was trying to filter an array based on values from different array. As filter activity doesn't allow arrays inside conditions, so, I am trying to filter one array by iterating over each value in second array. However, while writing the condition in the "Filter Activity", I am unable to reference "item()" value of the ForEach loop (value from second array on which iteration is running). Is there a way to reference outer item() inside filter activity?
I saw a post which showed we can use -: items("ForEachActivity") to refer foreach activity's values however, it throws an error -: {"code":"BadRequest","message":"ErrorCode=InvalidTemplate, ErrorMessage=The template validation failed: 'The workflow action 'FilterFilter1' at line '1 and column '42236' references the action 'ForEach1' of type 'Http': only the actions of type 'foreach' are allowed to be referenced by 'repeatItems' or 'items' functions","target":"pipeline/TableIeratorPipeline/runid/f332271e-4628-4a3f-95a2-7794e3a4216f","details":null,"error":null} .

You can create a variable to save the current item from For Each activity.
My test:
1.create three variables in pipeline.
2.create a For Each activity and check Sequential option.
3.pass value of current item to variable.
4.determine whether two values are equal.

Related

How to send the output values of a Lookup activity in an email in Data Factory?

I'm trying to send a LookUp activity output values as part of a body parameter in a POST request using LogicApp, which uses three parameters: "to", "email_body", "subject".
The LookUp activity depends on a query, and it may return from 2 rows up to 10 rows.
According to Azure, the output of the activity should look like this:
{
"count": 2,
"value": [
{
"column1":value1,
"column2":value2,
"column3":value3
},
{
"column1":value4,
"column2":value5,
"column3":value6
}
]
}
In this case, the query returned 2 rows, but how can I attach every output value to the POST body without having to use #activity('lookup_act').output.value[0].column1 and so on for every value?
The POST body is the following:
{
"email_body": "Hi, the following tables have been updated:
#{activity('lookup_act').output.value[0].column1}
#{activity('lookup_act').output.value[1].column1}",
"subject": "Update on tables",
"to": "email#domain.com"
}
I've tried using #activity('lookup_act').output.value to bring every value but it won't work.
Is there a way to call every single output value? If so, how can it be done and paste into a table?
Thanks beforehand.
There are two ways to get all values in mail:
1. Get whole lookup output array in mail.
First get the results from Lookup activity and then pass the output of this activity by converting it into a string otherwise you will get error regarding deserialization.
{"message":"#string(activity('Lookup1').output.value)",
"dataFactoryName":"#{pipeline().DataFactory}",
"pipelineName":"#{pipeline().Pipeline}",
"receiver":"#{pipeline().parameters.receiver}"}
OUTPUT
2. Get all the respective values column wise.
First get the results from Lookup activity then take a foreach loop and create append variable for every column to store every column value in single array.
ForEach activity setting:
Took append variable activity and created Idarray variable. and gave item().id as value to store all id values in a single array.
Then in web activity passed below body for getting all arrays.
{"message":"#{string(variables('Idarray'))} as Id, #{string(variables('Namearray'))} as Name, #{string(variables('ProfessionArray'))} as Profession",
"dataFactoryName":"#{pipeline().DataFactory}",
"pipelineName":"#{pipeline().Pipeline}",
"receiver":"#{pipeline().parameters.receiver}"}
OUTPUT

ADF/Synapse all Objects iterate and remove the Underscore

Wanted to iterate the list of objects/Tables and exclusively for one object which is not getting picked up as there is Underscore between the words "Admin_process" Expectation is to get as "Adminprocess" in the adf/synapse by removing the underscore,such that all objects will be passed to the copy operation.
Objects/Tables list
AdminUser
Admin_process
TempUser
Currently it is above, However is not reading the object "Admin_Process" as there is underscore.
Could you someone please tell me how to handle this case.
Thank you,
You can use replace function in ADF dynamic content.
please follow the demonstration below.
Here I am using an array parameter with keys and the above list of tables as values.
[
{
"Objectname": "AdminUser"
},
{
"Objectname": "Admin_process"
},
{
"Objectname": "TempUser"
}
]
Parameter array to ForEach activity:
To use replace function, create a set variable activity and give the below expression.
#replace(item().Objectname, '_','' )
Output with required result(Underscore removed):
Now you can pass this value to a copy activity inside the same ForEach activity.

Merge or Union a list of dynamically created querysets - Django

I have a list of querysets as follows. Is there a way to merge all these query sets of a list into one query set ? The queryset elements of this list are dynamically determined. As in the example below it is three querysets in the list. In next hypothetical iteration it could be 40 querysets.
capture_list =[<QuerySet [<Fields_Model: Fields_Model object (11)>]>,<QuerySet [<Fields_Model: Fields_Model object (12)>]>,<QuerySet [<Fields_Model: Fields_Model object (13)>]>]
You can union these with:
Fields_Model.objects.none().union(*capture_list)
Here the asterisk will thus "unpack" the querysets as individual parameters for the .union(…) call [Django-doc].
A call with .union(…) will only select distinct objects, so no duplicates. If you want to obtain the duplicates, you can the all=True parameter:
Fields_Model.objects.none().union(*capture_list, all=True)

Why is my Watir::Table not found?

I'm using Watir-5.0.0, selenium-webdriver-2.40 and testing on IE-8. When I execute the following code:
puts "#browser.tables.length=#{#browser.tables.length}"
#browser.tables.each { |t| puts t.to_s }
t=#browser.table(:class => "jrPage")
puts "jrPage=#{t}"
t.rows.each do |row|
# do something
end
I get the following results:
#browser.tables.length=5
#<Watir::Table:0x3921cc0>
#<Watir::Table:0x3921c48>
#<Watir::Table:0x3921c00>
#<Watir::Table:0x3921bd0>
#<Watir::Table:0x3921b88>
jrPage=#<Watir::Table:0x39219d8>
Selenium::WebDriver::Error::StaleElementReferenceError: Element is no longer valid
Any thoughts on why is table I explicitly locate, Watir::Table:0x39219d8, is not in the #browser.tables.each collection?
I can understand why I get the StaleElementReferenceError (table not found) but not why the table I explicit locate isn't in the tables list.
I can find this table in the HTML.
The code is calling to_s for the table object. Watir-webdriver does not specifically define this method for elements. Therefore, it calls Ruby's default Object#to_s method:
Returns a string representing obj. The default to_s prints the
object’s class and an encoding of the object id. As a special case,
the top-level object that is the initial execution context of Ruby
programs returns “main.”
As you can see, this is what Watir is doing - Watir::Table is the object's class and 0x39219d8 is an encoding of the object id.
When iterating through the tables or getting a table, you are retrieving the table from scratch. In other words, a new table object is created for each of these commands. Even if you run the collection again, you will see that you get 4 different table object ids each time.
Note that while the table objects you are seeing are unique, the jsPage is referring to one of the elements in the collection. You can use the == method to check if two objects are referencing the same html element.
For example, you can see this with:
# Get the specific jrPage
jrPage = browser.table(:class => "jrPage")
# Check which element in the collection is the jrPage
browser.tables.each { |t| puts t == jrPage }
#=> false
#=> false
#=> true
#=> false
#=> false
The above tells you that the jrPage is the 3rd table on the page.

Mongo DB C# code using 2 level , 3 level and 4 level of array

I have to update in C# code using MongoDB. Here I had implement 2nd level array of update in below (subBranchindex is taken in a generic list object):-
for (var index = 0; index < subBranchindex.Count; index++)
{
if (subBranchindex[index]._id == new ObjectId(subBranchid))
{
IMongoQuery queryEdit = Query.EQ("BranchOffice.SubBranchlist._id", new ObjectId(subBranchid));
UpdateBuilder update = Update.Set("BranchOffice.$.SubBranchlist."+ index +".Name",subBranch.SubName).
SafeModeResult s = dc.Collection.Update(queryEdit, update,
UpdateFlags.None, SafeMode.True);
}
}
Here 2nd level array, I was using (for loop Statement) to taken Index value for array. Next I can use 3rd, 4th and 5th level of array means more than (for loop statement) will be assign. So don't need [for loop Statement] and also don't need to assign hardcore number in index.
For example: ("BranchOffice.$.SubBranchlist.0.Name",subBranch.SubName). Here Don't Hardcore number[index] 0 or 1 or 2. "2nd" level array more than 100 record is there.
Is there any way I can use to array index value? Please explain how to solve this probelm. Please explain me with Example.
Based on your example above, my understanding of your schema is the following:
The top-level document has a BranchOffice field
BranchOffice is an array of objects
Each object within BranchOffice has an _id, SubName and SubBranchlist field
SubBranchlist is an array of objects
Each object within SubBranchlist has a Name field
Your update statement appears to be copying the SubName field to each Name field among objects within SubBranchlist (a sibling field of SubName).
Using the property path syntax to select fields through arrays (e.g. SubBranchlist.0.Name), there is no "wildcard" index that will allow you to modify Name fields among all objects in the array.
On a somewhat related note, the $ positional operator only applies to the first-matched array element, so you cannot use that to update multiple array elements. In your case, it would not be an option anyway, since you're using the positional operator for the BranchOffice array field.
You can either issue a series of update queries (for each element in SubBranchlist), or consider using $set to modify the entire SubBranchlist array in one query. The downside with using $set is that you'll need to read and write back the entire array, which may be a problem if other, concurrent operations are also issuing updates to the array.

Resources