I am trying this new functionality, and when I try to use set variable activity inside foreach loop I cannot select a variable that I declared in a pipeline.
Also inside IF activity.
Is it supposed to behave like this? That you cant set variable inside some inner activities, only at the root level of the pipeline?
This is a known bug, where set variables and append variables activities are not correctly detecting changes when they're nested in another activity. Actively working on a fix for, hopefully will resolve this problem soon :)
Related
I'm working with ADF and trying to leverage parameters to make life easier and reduce the number of objects being created in the ADF itself. What I am trying to do, would appear on the surface to be extremely simple, bu in reality its driving me slowly crazy. Would greatly appreciate any assistance!
I am trying to set up a parameterised dataset to be used as a sink target. Inside that dataset I have added a param named "filenames" of type string. In the connection tab I have added that param to the file part of the path. The folder part point to my Azure Data Lake folder and the file part is set to: #dataset().filename which is the result of choosing 'dynamic content' then selecting the param.
So far so good.. my sink target is, as far as I am aware, ready to receive "filenames" to write out to.
This is where it all goes wrong.
I now create a new pipeline. I want to use a list or array of values inside that pipeline which represent the names of the files I want to process. I have been told that I'll need a Foreach to send each of the values one at a time to the COPY DATA task behind the Foreach. I am no stranger to Foreach type loops and behaviors.. but for the life of me I CANNOT see where to set up the list of filenames. I can create a param as a type "array" but how the heck do you populate it?
I have another use case which this problem is preventing me from completing. This use case is, I think, the same problem but perhaps serves to explain the situation more clearly. It goes like this:
I have a linked service to a remote database. I need to copy data from that database (around 12 tables) into the data lake. At the moment I have about 12 "COPY DATA" actions linked together - which is ridiculous. I want to use a Foreach loop to copy the data from source to data lake one after the other. Again, I can set up the sink dataset to be parameterised, just fine... but how the heck do I create the array/list of table names in the pipeline to pass to the sink dataset?
I add the Foreach and inside the foreach a "COPY DATA" but where do I add all the table names?
Would be very grateful for any assistance. THANK YOU.
If you want to manually populate values of an array as a pipeline parameter, you create the parameter with Array type and set the value with syntax like: ["File1","File2","File3"]
You then iterate that array using a ForEach activity.
Inside the ForEach, you reference #item() to get the current file name value the loop is on.
You can also use a Lookup activity to get data from elsewhere and iterate over that using the ForEach.
I have a folder in ADLS that has few files. For the purpose of understanding, I will keep it simple. I have the following three files. When I loop through this folder, I want to get the "file name" and "source" as separate parameters so that I can pass it subsequent activities/pipelines.
employee_crm.txt
contractor_ps.txt
manager_director_sap.txt
I want to put this in an array so that it can be passed accordingly to the subsequent activities.
(employee, contractor, manager_director)
(crm, ps, sap)
I want to pass two parameters to my subsequent activity (may be a stored procedure) as usp_foo (employee, crm) and it will execute the process based on the parameters. Similary, usp_foo (contractor, ps) and usp_foo (manager_director, sap).
How do I get the child items as two separate parameters so that it can be passed to SP?
To rephrase the question, you would like to 1) get a list of blob names and 2) parse those names into 2 variables. This pattern occurs frequently, so the following steps will guide you through how to accomplish these tasks.
Define an ADLS DataSet that specifies the folder. You do not need a schema, and you can optionally parameterize the FileSystem and Directory names:
To get a list of the objects within, use the GetMetadata activity. Expand the "Field list" section and select "Child Items" in the drop down:
Add a Filter activity to make sure you are only dealing with .txt files. Note it targets the "childItems" property:
You may obviously alter these expressions to meet the specific needs of your project.
Use ForEach activity to loop through each element in the Filter sequentially:
Inside the ForEach, add activities to parse the filename. To access the fileName, use "item().name":
In my example, I am storing these values as pipeline variables, which are global [hence the need to perform this operation sequentially]. Storing them in an Array for further use gets complicated and tricky in a hurry because of the limited Array and Object support in the Pipeline Expression Language. The inability to have nested foreach activities may also be a factor.
To overcome these, at this point I would pass these values to another pipeline directly inside the ForEach loop.
This pattern has the added benefit of allowing individual file execution apart from the folder processing.
I have a pipeline built that reads metadata from a blob container subfolder raw/subfolder. I then execute a foreach loop with another get metadata task to get data for each subfolder, it returns the following type of data. /raw/subfolder1/folder1, /raw/subfolder2/folder1, /raw/subfolder2/folder1 and so on. I need another foreach loop to access the files inside of each folder. The problem is that you cannot run a foreach loop inside of another foreach loop so I cannot iterate further on the files.
I have an execute datapipline that calls the above pipeline and then uses a foreach. My issue with this is that I'm not finding a way to pass the item().name from the above iteration to my new pipeline. It doesn't appear you can pass in objects form the previous pipeline? How would I be able to accomplish this nested foreach metat data gathering so I can iterate further on my files?
Have you tried using parameters? Here is how it would look like:
In your parent pipeline, click on the "Execute Pipeline" activity which triggers the inner (your new pipeline) go to Settings and specify item name as a parameter "name".
In your inner pipeline, click anywhere on empty space and add new parameter "name".
Now you can refer to that parameter like this: pipeline().parameters.name
Using Parameters works in this scenario as #Andrii mentioned.
For more on passing parameters between activities refer to this link.
https://azure.microsoft.com/en-in/resources/azure-data-factory-passing-parameters/
I’ve created new Segmented Keys in ACUMATICA for use in a specific module. I would like to assign the Dimension name dynamically but I noticed it works only with hard code or name like [PXDimension(“VENDOR”)]
Also, I have some limitation to create an IF Conditional inside the customized field… it does not recognize the IF clause (see the image).
I would appreciate any suggestion how to solve this issue.
I haven't seen how your original attempt at PXDimension looked, but I'm going to take a guess and assume you tried to reference a new custom field contained in a setup table, something like:
[PXDimension(typeof(XXMySetup.usrMyCustomField))]
If that's indeed what you tried to do, one very important thing to do is to ensure that you have a view for your table in your graph, otherwise the attribute will not find the table and record in your cache. For instance:
public PXSetup<XXMySetup> XXMySetup;
Without this view declared in the graph, the dimension attribute will not work as expected. It would be nice if a clear exception was thrown in this case - I made the same mistake recently and it would have been helpful.
I am working on this project https://github.com/tanema/express-helpers that I forked fixed up and added new functionality to.
What I want to do it, instead of having to use form_tag_end() or even end_tag('tagname') I would just like to use a end() method. For this to work I need some sort of stack implementation for when a start tag is issued push onto the stack ect. I cannot just make a variable in the module called tagStack and just use that because it would create a race condition where the module is being used at the same time by two requests and the stack order gets messed up.
I was thinking if there was some way to have access to the request variable I would just store it in that and delete the variable if empty but I can't figure out how to do that.
Any suggestions?
Create your variable within a closure; it will be available within the scope of the instance, but not outside the instantiation of the functions, and will be garbage collected when the specific instances go out of scope.