I'm writing a pipeline, where I fetch SQL queries from a metadata database in a lookup with the hope to execute those later on in the pipeline. Imagine a string stored in a database:
"SELECT * FROM #{pipeline().parameters.SchemaName}.#{pipeline().parameters.TableName}"
My hope was when passing this string to another Lookup activity, it would pick up the necessary parameters. However it's being passed to the activity as-is, without parameter substitution and I'm getting errors as a result. Is there any clean fix for this or am I trying to implement something not supported by ADF natively?
I found a work-around is just wrapping the string in a series of replace() statements, but hoping something simpler exists.
Can you try below query in the Dynamic Content text box:
#concat('SELECT * FROM ',pipeline().parameters.SchemaName,'.',pipeline().parameters.TableName)
Related
At the end of the pipeline I wanted to write the below query
INSERT INTO [TestTable] (Job_name, status) VALUES (Job_name, current_timestamp()).
Jobname will be passed as a parameter.
Can this be written in Lookup, please let me know.
Definitely possible, but should you write massive inserts/scripts within a lookup... probably not a great idea, but see below (Truncate Example, but will work the same with an insert)
I use this method for small things like truncating a table, but never with big code that should be stored as source in the DB.
EDIT:
If you need to pass parameters or Variables into the lookup you should use string interpolation like so:
INSERT INTO dbo.MyTable
SELECT '#{variables('YourVariable')}' as Variable1,
'#{pipeline().Pipeline} as PipelineName
I am trying to run spark query where I am creating curated table from a source table based upon values in parameter file.
properties_file.properties contains below key values:
substatus,allow,deny
SparkQuery is
//Code to load property file in parseConf
spark.sql(s"""insert into curated.table from source.table where
substatus='${parseConf.substatus}'""")
Above works with single value in substatus. But Can someone help what shall i do if I need to use substatus in ${parseConf.substatus} for multiple values from param as below.
spark.sql(s"""insert into curated.table from source.table where substatus in '${parseConf.substatus}'""")
To resolve my problem, I updated my property file as:
substatus,'allow'-'deny'
Then in scala code, I implemented below logic:
val subStatus=(parseConf.substatus).replace('-',',')
spark.sql(s"""insert into curated.table from source.table where substatus in ('${subStatus}')""")
Above strategy helped in breaking the values in string to muliple parameters of IN clause.
Equalto operator expects 1 value to be passed other than directly reading the value from parameter file who make in pass a one string. You need to break the values and then use IN clause inplace of equalto(=).
I'm trying to pass last modified date and time to my data flow as parameter. Can anyone tell me what is a correct way to pass it as parameter. I've tried multiple things like. passing utcnow() from activity throws error saying file not found whereas passing directly from dataflow works fine.
I figured out using dataflow expression works fine for ucnow() whereas pipeline expressions fails.
The Pipeline expression language is a different and a little more limited than the Data Flow expression language. While Data Flow supports a richer variable type system, Pipelines only support String, Boolean, and Array types. Since there is no Date or Timestamp types, the date functions in the Pipeline expression language return strings:
If you want to use the UTC value from the pipeline instead of the data flow, you will need define a string parameter on the Data Flow:
Then pass the string of utcnow() to the Data Flow as a Pipeline Expression:
In the expression, use the utcnow() function to get the string value:
In the Data Flow, use Derived Column to convert it to the desired type:
In ADFv2 I'm looking up a date and passing it to an Azure Function. I can pass just the data like so:
#activity('GetLastDateProcessed').output.firstRow.LastDateProcessed
However if I embed this into a JSON string like this:
{"lastProcessDate":"#activity('GetLastDateProcessed').output.firstRow.LastDateProcessed"}
I get this {"lastProcessDate":"#activity('GetLastDateProcessed').output.firstRow.LastDateProcessed"} instead of {"lastProcessDate":"2019-11-13"} as input into function.
Last I've tried to use a parameter with no success also.
#concat('{"lastProcessDate":"', string(pipeline().parameters.lastProcessDate), '"}')
The problem here is the parameter was not set. I set the parameter like this:
#activity('GetLastDateProcessed').output.firstRow.LastDateProcessed
However this is a default value and is never dynamically updated. If I can update this string then the #concat method will work, but haven't been able to figure out how to dynamically update a parameter for the pipeline.
Another option could be a pipeline variable, but I don't know how to reference the variable.
How do I concat strings together with dynamic content?
I think what you are missing is that when you use the at-sign '#' in the json string you should follow it with a curly bracket '{'
In your example it will look something like this:
{"lastProcessDate":"#{activity('GetLastDateProcessed').output.firstRow.LastDateProcessed}"}
here is the source (found it in the comments):
https://azure.microsoft.com/en-us/blog/azure-functions-now-supported-as-a-step-in-azure-data-factory-pipelines/#:~:text=Azure%20Data%20Factory%20(ADF)%20is,in%20your%20data%20factory%20pipelines.
I was able to get this to work by creating a second pipeline. This is not optimal, but works for people running into this same issue. Hopefully someone finds a better solution than this!
From the first pipeline I set the second pipelines parameter with this:
#activity('GetLastDateProcessed').output.firstRow.LastDateProcessed
I named the parameter in the second pipeline lastProcessDate so then this worked:
#concat('{"lastProcessDate":"', string(pipeline().parameters.lastProcessDate), '"}')
This is not straight forward and can't be how Microsoft is expecting us to solve this!
I was able to achieve this with command.
{
"storedprocedure":"storedProcName",
"params":"#{variables('currentDt')}"
}
I have an SSIS package that obtains a list of new GUIDs from a SQL table. I then shred the GUIDs into a string variable so that I have them separated out by comma. An example of how they appear in the variable is:
'5f661168-aed2-4659-86ba-fd864ca341bc','f5ba6d28-7283-4bed-9f11-e8f6bef225c5'
The problem is in the data flow task. I use the variable as a parameter in a SQL query to get my source data and I cannot get my results. When the WHERE clause looks like:
WHERE [GUID] IN (?)
I get an invalid character error so I found out the implicit conversion doesn't work with the GUIDs like I thought they would. I could resolve this by putting {} around the GUID if this were a single GUID but there are a potential 4 or 5 different GUIDs this will need to retrieve at runtime.
Figuring I could get around it with this:
WHERE CAST([GUID] AS VARCHAR(50)) IN (?)
But this simply produces no results and there should be two in my current test.
I figure there must be a way to accomplish this... What am I missing?
You can't, at least not using the mechanics you have provided.
You cannot concatenate values and make that work with a parameter.
I'm open to being proven wrong on this point but I'll be damned if I can make it work.
How can I make it work?
The trick is to just go old school and make your query via string building/concatenation.
In my package, I defined two variables, filter and query. filter will be the concatenation you are already performing.
query will be an expression (right click, properties: set EvaluateAsExpression to True, Expression would be something like "SELECT * FROM dbo.RefData R WHERE R.refkey IN (" + #[User::filter] + ")"
In your data flow, then change your source to SQL Command from variable. No mapping required there.
Basic look and feel would be like
OLE Source query