I have a pipeline that accepts an array as parameters.
Currently, an array has been hardcoded as the default value.
Is it possible to make this dynamic? there is a file called Array.txt in azure blob which is updated frequently, how can I extract the content of Array.txt and pass it as parameter values to the Pipeline.
I tried using Lookup but receive error 'Object cannot be passed, pipeline is expecting an Array'
Please make sure the data in Array.txt is array format-compatible, then use a lookup activity for data extracting, pass #array(activity('Lookup1').output.value) to the subsequent activity. Remember to use #array() function to convert data into an Array.
Related
I'm writing a pipeline, where I fetch SQL queries from a metadata database in a lookup with the hope to execute those later on in the pipeline. Imagine a string stored in a database:
"SELECT * FROM #{pipeline().parameters.SchemaName}.#{pipeline().parameters.TableName}"
My hope was when passing this string to another Lookup activity, it would pick up the necessary parameters. However it's being passed to the activity as-is, without parameter substitution and I'm getting errors as a result. Is there any clean fix for this or am I trying to implement something not supported by ADF natively?
I found a work-around is just wrapping the string in a series of replace() statements, but hoping something simpler exists.
Can you try below query in the Dynamic Content text box:
#concat('SELECT * FROM ',pipeline().parameters.SchemaName,'.',pipeline().parameters.TableName)
I created an array variable and tried to pass that to max math function in ADF but i'm getting error in it. So how to using max function there?
Array is one of the datatypes supported in ADF with both parameters and variables, so if you have a valid array then max will work either. A simple example:
create a valid parameter of the Array datatype:
Create a variable and add a Set Variable activity. Use this expression as the Value argument:
#string(max(pipeline().parameters.pInputArray))
NB I'm using the max function directly on the array and then string as only the String, Array and Boolean datatypes are supported for variables (at this time).
I am trying to run spark query where I am creating curated table from a source table based upon values in parameter file.
properties_file.properties contains below key values:
substatus,allow,deny
SparkQuery is
//Code to load property file in parseConf
spark.sql(s"""insert into curated.table from source.table where
substatus='${parseConf.substatus}'""")
Above works with single value in substatus. But Can someone help what shall i do if I need to use substatus in ${parseConf.substatus} for multiple values from param as below.
spark.sql(s"""insert into curated.table from source.table where substatus in '${parseConf.substatus}'""")
To resolve my problem, I updated my property file as:
substatus,'allow'-'deny'
Then in scala code, I implemented below logic:
val subStatus=(parseConf.substatus).replace('-',',')
spark.sql(s"""insert into curated.table from source.table where substatus in ('${subStatus}')""")
Above strategy helped in breaking the values in string to muliple parameters of IN clause.
Equalto operator expects 1 value to be passed other than directly reading the value from parameter file who make in pass a one string. You need to break the values and then use IN clause inplace of equalto(=).
I’m wondering if someone can help us out here as we are going round in circles.
In short we are trying to;
Read a value from a SQL table via a stored procedure in a lookup task.
We want to use that value to set a variable so we can use it in a copy data task.
However our set variable task using (#activity('Lookup1').output) returns the value but its wrapped in lots of JSON (see attached).
In the attached we are only interested in the TokenGUID value, not the rest of the JSON.
So can someone please point us in the direction of how we would set a variable to be a string value.
Thanks,
Nic
You can use this expression to get TokenGUID:
#activity('Lookup1').output.firstRow.TokenGUID
My test:
Output of Lookup activity
Output of Set variable activity
In ADFv2 I'm looking up a date and passing it to an Azure Function. I can pass just the data like so:
#activity('GetLastDateProcessed').output.firstRow.LastDateProcessed
However if I embed this into a JSON string like this:
{"lastProcessDate":"#activity('GetLastDateProcessed').output.firstRow.LastDateProcessed"}
I get this {"lastProcessDate":"#activity('GetLastDateProcessed').output.firstRow.LastDateProcessed"} instead of {"lastProcessDate":"2019-11-13"} as input into function.
Last I've tried to use a parameter with no success also.
#concat('{"lastProcessDate":"', string(pipeline().parameters.lastProcessDate), '"}')
The problem here is the parameter was not set. I set the parameter like this:
#activity('GetLastDateProcessed').output.firstRow.LastDateProcessed
However this is a default value and is never dynamically updated. If I can update this string then the #concat method will work, but haven't been able to figure out how to dynamically update a parameter for the pipeline.
Another option could be a pipeline variable, but I don't know how to reference the variable.
How do I concat strings together with dynamic content?
I think what you are missing is that when you use the at-sign '#' in the json string you should follow it with a curly bracket '{'
In your example it will look something like this:
{"lastProcessDate":"#{activity('GetLastDateProcessed').output.firstRow.LastDateProcessed}"}
here is the source (found it in the comments):
https://azure.microsoft.com/en-us/blog/azure-functions-now-supported-as-a-step-in-azure-data-factory-pipelines/#:~:text=Azure%20Data%20Factory%20(ADF)%20is,in%20your%20data%20factory%20pipelines.
I was able to get this to work by creating a second pipeline. This is not optimal, but works for people running into this same issue. Hopefully someone finds a better solution than this!
From the first pipeline I set the second pipelines parameter with this:
#activity('GetLastDateProcessed').output.firstRow.LastDateProcessed
I named the parameter in the second pipeline lastProcessDate so then this worked:
#concat('{"lastProcessDate":"', string(pipeline().parameters.lastProcessDate), '"}')
This is not straight forward and can't be how Microsoft is expecting us to solve this!
I was able to achieve this with command.
{
"storedprocedure":"storedProcName",
"params":"#{variables('currentDt')}"
}