I have an SSIS package that obtains a list of new GUIDs from a SQL table. I then shred the GUIDs into a string variable so that I have them separated out by comma. An example of how they appear in the variable is:
'5f661168-aed2-4659-86ba-fd864ca341bc','f5ba6d28-7283-4bed-9f11-e8f6bef225c5'
The problem is in the data flow task. I use the variable as a parameter in a SQL query to get my source data and I cannot get my results. When the WHERE clause looks like:
WHERE [GUID] IN (?)
I get an invalid character error so I found out the implicit conversion doesn't work with the GUIDs like I thought they would. I could resolve this by putting {} around the GUID if this were a single GUID but there are a potential 4 or 5 different GUIDs this will need to retrieve at runtime.
Figuring I could get around it with this:
WHERE CAST([GUID] AS VARCHAR(50)) IN (?)
But this simply produces no results and there should be two in my current test.
I figure there must be a way to accomplish this... What am I missing?
You can't, at least not using the mechanics you have provided.
You cannot concatenate values and make that work with a parameter.
I'm open to being proven wrong on this point but I'll be damned if I can make it work.
How can I make it work?
The trick is to just go old school and make your query via string building/concatenation.
In my package, I defined two variables, filter and query. filter will be the concatenation you are already performing.
query will be an expression (right click, properties: set EvaluateAsExpression to True, Expression would be something like "SELECT * FROM dbo.RefData R WHERE R.refkey IN (" + #[User::filter] + ")"
In your data flow, then change your source to SQL Command from variable. No mapping required there.
Basic look and feel would be like
OLE Source query
Related
I want to perform a full-text-search on 2 columns with partial queries included.
I've tried multiple options and this one seems the best to me:
Add <-> between the words of the query and :* at the end
Execute query
The problem is, that I have to execute the query in TypeORM. So when I use to_tsquery(:query) there might be invalid syntax in the query, which produces an error.
The function plainto_tsquery() would be perfect since it prevents invalid syntax in the argument, but at the same time it prevents the partial queries, which I can do as described.
Any idea how I could combine the best of the to worlds?
You could try something like
SELECT to_tsquery(quote_literal(query) || ':*')
This will add <-> between word and :* at the end of every word, while quote_literal should protect you from syntax issues by escaping the text.
Disadvantage of this method however is that the generated query might behave unexpectedly when encountering queries with symbols, e.g. o'reilley as query will yield 'o':* <-> 'reilley':* as tsquery, which likely won't give back the expected result. Unfortunately, the only solution I know for this is cleaning both the input and text data of any symbols.
I'm writing a pipeline, where I fetch SQL queries from a metadata database in a lookup with the hope to execute those later on in the pipeline. Imagine a string stored in a database:
"SELECT * FROM #{pipeline().parameters.SchemaName}.#{pipeline().parameters.TableName}"
My hope was when passing this string to another Lookup activity, it would pick up the necessary parameters. However it's being passed to the activity as-is, without parameter substitution and I'm getting errors as a result. Is there any clean fix for this or am I trying to implement something not supported by ADF natively?
I found a work-around is just wrapping the string in a series of replace() statements, but hoping something simpler exists.
Can you try below query in the Dynamic Content text box:
#concat('SELECT * FROM ',pipeline().parameters.SchemaName,'.',pipeline().parameters.TableName)
At the end of the pipeline I wanted to write the below query
INSERT INTO [TestTable] (Job_name, status) VALUES (Job_name, current_timestamp()).
Jobname will be passed as a parameter.
Can this be written in Lookup, please let me know.
Definitely possible, but should you write massive inserts/scripts within a lookup... probably not a great idea, but see below (Truncate Example, but will work the same with an insert)
I use this method for small things like truncating a table, but never with big code that should be stored as source in the DB.
EDIT:
If you need to pass parameters or Variables into the lookup you should use string interpolation like so:
INSERT INTO dbo.MyTable
SELECT '#{variables('YourVariable')}' as Variable1,
'#{pipeline().Pipeline} as PipelineName
Objective
All I am trying to do is retrieve a single record from a specific table where the primary key matches. I have a feeling I'm greatly over complicating this as it seems to be a simple enough task. I have a theory that it may not know the variable value because it isn't actually pulling it from the Python code but instead trying to find a variable by the same name in the database.
EDIT: Is it possible that I need to wrap my where clause in an expression statement?
Attempted
My Python code is
def get_single_record(name_to_search):
my_engine = super_secret_inhouse_engine_constructor("sample_data.csv")
print("Searching for " + name_to_search)
statement = my_engine.tables["Users"].select().where(my_engine.tables["Users"].c.Name == name_to_search)
# Print out the raw SQL so we can see what exactly it's checking for
print("You are about to run: " + str(statement))
# Print out each result (should only be one)
print("Results:")
for item in my_engine.execute(statement):
print(item)
I tried hard coding a string in its place.
I tried using like instead of where.
All to the same end result.
Expected
I expect it to generate something along the lines of SELECT * FROM MyTable WHERE Name='Todd'.
Actual Result
Searching for Todd
STATEMENT: SELECT "Users"."Name", ...
FROM "Users"
WHERE "Users"."Name" = ?
That is an actual question mark appearing my statement, not simply my own confusion. This is then followed by it printing out a collection of all the records from the table, as though it successfully matched everything.
EDIT 2: Running either my own hard coded SQL string or the generated query by Alchemy returns every record from the table. I'm beginning to think the issue may be with the engine I've set up not accepting the query.
Why I'm Confused
According to the official documentation and third party sources, I should be able to compare to hardcoded strings and then, by proxy, be able to compare to a variable.
I have a parameter called Analyst group in this format :
[Dimension].[Analyst Group].&[Nl.Workplace.Foundation]
I want to pass this parameter to another report, to filter data. Its a multi value parameter. But the other report only accepts it in this format : [KanBan].[Analyst Group].&[Nl.Workplace.Foundation]
So im trying to isolate the "Nl.Workplace.Foundation", so i can do the following thing in the Go To Report parameter expression :="[KanBan].[Analyst Group].&["& --Isolated analyst group-- &"]" to create the desired format.
So what i need is to extract the part between .&[ and ]
But i really have no idea how to isolate that part of the string.
Found a solution! If i just use the Parameter.label instead of Parameter.value it automatically does what i want!
A different solution has been found, but I will still answer the initial question. It could help.
So what i need is to extract the part between .&[ and ]
You could use a regex.
This may not be the fastest way but it can handle most of the situations.
So let's assume you have a string containing:
[Dimension].[Analyst Group].&[Nl.Workplace.Foundation]
And you want to get the following string:
Nl.Workplace.Foundation
Just use the following expression:
=System.Text.RegularExpressions.Regex.Match("[Dimension].[Analyst Group].&[Nl.Workplace.Foundation]", "\.&\[(?<NWF>[^]]+)\]").Groups("NWF").Value
In the expression, replace the input string with your dynamic values, like for example:
=System.Text.RegularExpressions.Regex.Match(Fields!Dimension.Value & "." & Fields!AnalystGroup.Value, "\.&\[(?<NWF>[^]]+)\]").Groups("NWF").Value
I'm keeping the formula as simple as possible so that you can easily adapt it, with, say, handling the case where an input string will not have a match (with the above query it will return #Error).
You could do this by adding an IIF() or better, use a custom function that you can reuse in several places and will reduce the length of your expression.