I am writing some long SQL queries in a Terraform file, the query would be like:
"CREATE TABLE example_${var.name} (id string, name string..........................................)"
To make the query readable, I hope the query would be the format like the following, and cross multiple lines
CREATE TABLE example_$(var.name) (
id string,
name string,
................................
)
Is there a line continuation character for a long single line string to be written as multiple lines. Just like we could use backslash \ in Python for long string?
I have tried use heredoc, but it does not work when running the query. Thanks
It sounds like your goal is to have a long SQL query defined in Terraform, but across multiple lines so you don't need to horizontal scroll to infinity and beyond.
In my team we use heredoc to achieve this although you said it's not possible in your case.
Another idea my team use when heredoc isn't possible is to join an array of strings.
E.g.
locals {
sql = join(",", [
"id string",
"name string",
"address string",
"renter string",
"profession string"
])
}
Results in
> local.sql
id string,name string,address string,renter string,profession string
I hope I've understood your question correctly but if not please let me know.
PS: There's an open issue for multiline strings in Terraform
To make a multi-line strings in Terraform using the heredoc string syntax.
locals {
sql = <<EOT
CREATE TABLE example_$(var.name) (
id string,
name string,
................................
)
EOT
}
Related
I have a column in string format like below:
["name": "XXX","active": true,"locale": "EN","Channel":["1","2"]]
I would like to explode them like below in spark sql(preserving the quotes in string values).
This is code I used:
SELECT EXPLODE(from_json(col, 'map<string, string>>'))
FROM XXX;
I am not able to preserve the quotes in "XXX" and "EN" after exploding.
This is what I want:
key
value
name
"XXX"
active
true
locale
"EN"
Channel
[1,2]
The quotes are part of the JSON representation of the data and not the data itself. If there were embedded quotes in the data it would look like:
"\"SOME DATA\""
If you need to add quotes on strings, you can always concatenate them to the specific columns. You can use the concat operator to accomplish this, https://spark.apache.org/docs/latest/api/sql/index.html#concat
Alternatively, you can use get_json_object, which allows you to extract specific parts of a JSON object. https://spark.apache.org/docs/3.1.2/api/python/reference/api/pyspark.sql.functions.get_json_object.html
When editing a Stream Analytics transformation query in the Portal, you can format it for readability across multiple lines...e.g.
SELECT
INTO [Output1]
FROM [Input1]
PARTITION BY PartitionId
WHERE etc etc etc
When putting this into an ARM template for CI/CD, this is entered as one massive long string and would end up displaying in the portal as...
SELECT * INTO [Output1] FROM [Input1] PARTITION BY PartitionId WHERE etc etc etc to infinity....
The official documentation is pretty useless and doesn't give any clues for the query part of the template, just that it is a "string"...
https://learn.microsoft.com/en-us/azure/templates/microsoft.streamanalytics/2016-03-01/streamingjobs/transformations
There is a Microsoft sample template that is the only example I could find with a transform query specified...
https://github.com/Azure/azure-quickstart-templates/blob/master/101-streamanalytics-create/azuredeploy.json
...and it looks like it is trying to do spacing...
"query": "SELECT\r\n *\r\nINTO\r\n [YourOutputAlias]\r\nFROM\r\n [YourInputAlias]"
...but failing badly - see screenshot
Has anyone managed to do this?
Also does anyone know why you can see the transformation query in the Azure Resource Explorer (https://resources.azure.com/)? Or that it cannot be exported from the portal with the rest of the Stream Job? (done at Resource Group level)
Thanks in advance
I know it is a full year later and perhaps you've figured this out already, however, this is what I did:
In my Parameters file, I used an array of strings, for example:
"StreamAnalyticsJobQueryMultiline": {
"value": [
"WITH allData AS ( ",
" SELECT ",
" *, ",
" GetMetadataPropertyValue([%%INPUTSTREAMNAME%%], '[User].[EventNamespace]') AS EventNamespace ",
" FROM [%%INPUTSTREAMNAME%%] Partition By PartitionId ",
"SELECT ",
" *, ",
" 'EventHubWriterv1' AS EventType ",
"INTO ",
" [%%OUTPUTSTREAMNAME%%] ",
"FROM ",
" allData Partition By PartitionId "
]
When the array is concatenated, and output as a string, it produces something like this, where each item in that array is still enclosed by the quotation marks and the entire thing is contained within square braces (see: https://learn.microsoft.com/en-us/azure/azure-resource-manager/templates/template-functions-string#concat)
["Foo","Bar","Blah"]
So additional transformation in order to turn it into something readable in the Stream Analytics output is required.
Also note here the %%INPUTSTREAMNAME%% and %%OUTPUTSTREAMNAME%%, as both my input and output streams are also parameters, and using the typical inline [parameter('ParamName')] did not work nicely with the rest of the transformation needed.
In my Template file, I take the StreamAnalyticsJobQueryMultiline parameter and use the variables field to do this transformation:
"QueryStringRaw": "[string(concat(parameters('StreamAnalyticsJobQueryMultiline')))]",
// Update the end-of-lines by removing the doublequote and comma, and replacing it with a newline character
"QueryStringIter1": "[replace(variables('QueryStringRaw'), '\",', '\n')]",
// Update the beginning-of-lines by removing the doublequote
"QueryStringIter2": "[replace(variables('QueryStringIter1'), '\"', '')]",
// Update the InputStreamName and OutputStreamName values
"QueryStringIter3": "[replace(variables('QueryStringIter2'), '%%INPUTSTREAMNAME%%', parameters('InputStreamName'))]",
"QueryStringIter4": "[replace(variables('QueryStringIter3'), '%%OUTPUTSTREAMNAME%%', parameters('OutputStreamName'))]",
// Produce the final output for the query string by trimming the leading and trailing square brackets
"QueryStringFinal": "[substring(variables('QueryStringIter4'), 1, sub(length(variables('QueryStringIter4')), 2))]"
Then I reference that in the transformation portion of the Microsoft.StreamAnalytics/streamingjobs properties:
"transformation": {
"name": "Transformation",
"properties": {
"streamingUnits": "[parameters('StreamAnalyticsStreamingUnitsScale')]",
"query": "[variables('QueryStringFinal')]"
}
}
I need to create a dynamic query based on two string parameters:
description = "This is the description"
comment = "This is the comment"
query = "insert into case(desc, comm) value(description, comment)"
Note:
there might be single quote and double quotes in both description and comment.
How do I use formatted %s to generate the query string?
Thank you very much.
UPDATE:
Thanks to Green Cloak Guy (his/her answer has minors to be corrected), the right query is:
query = f"insert into case(description, comment) value(\'{description}\', \'{comment}\')"
Use an f-string.
query = f"insert into case({description}, {comment}) value({description}, {comment})"
Don't use any type of string formatting to do actual database queries - that leads to you having a SQL Injection problem. Use a database library instead, that properly sanitizes the data.
But if all you need to do is parse some variables into a string, this is more flexible than the % formatting that other languages tend to use (and that's technically still available in python via "some_string %s %s %s" % (str1, str2, str3)")
join works BUT i want to keep the double quotes join gives me this
[ben,linda,john]
BUT i want this
["ben", "linda", "john"]
this is getting crazy, spent over 2 hours trying to fix this
i want to pass in a list as a string variable
why can't terraform just take in my list as a string? why is this so difficult?
so i have
name = ["ben", "linda", "john"]
and i want to pass this to variable used in terrform
var.name
why can't terrform take this as is?
i get the error saying epxtected a string and i can not find a solution online after sarching everywhere
i have been able to get
[ ben,linda,john ] using join(",", var.name) but i want ["ben", "linda", "john"]
$ terraform --version
Terraform v0.12.18
+ provider.aws v2.42.0
+ provider.template v2.1.2
Conversion from list to string always requires an explicit decision about how the result will be formatted: which character (if any) will delimit the individual items, which delimiters (if any) will mark each item, which markers will be included at the start and end (if any) to explicitly mark the result as a list.
The syntax example you showed looks like JSON. If that is your goal then the easiest answer is to use jsonencode to convert the list directly to JSON syntax:
jsonencode(var.names)
This function produces compact JSON, so the result would be the following:
["ben","linda","john"]
Terraform provides a ready-to-use function for JSON because its a common need. If you need more control over the above decisions then you'd need to use more complex techniques to describe to Terraform what you need. For example, to produce a string where each input string is in quotes, the items are separated by commas, and the entire result is delimited by [ and ] markers, there are three steps:
Transform the list to add the quotes: [for s in var.names : format("%q", s)]
Join that result using , as the delimiter: join(", ", [for s in var.names : format("%q", s)])
Add the leading and trailing markers: "[ ${join(",", [for s in var.names : format("%q", s)])} ]"
The above makes the same decisions as the JSON encoding of a list, so there's no real reason to do exactly what I've shown above, but I'm showing the individual steps here as an example so that those who want to produce a different list serialization have a starting point to work from.
For example, if the spaces after the commas were important then you could adjust the first argument to join in the above to include a space:
"[ ${join(", ", [for s in var.names : format("%q", s)])} ]"
This appears to be pretty basic but I am unable to find a suitable pipeline expression function to achieve this.
I have set an array variable VAR1 with the following value, which is an output from a SQL Lookup activity in an ADF pipeline:
[
{
"Code1": "1312312"
},
{
"Code1": "3524355"
}
]
Now, I need to convert this into a comma separated string so I can pass it to a SQL query in the next activity - something like:
"'1312312','3524355'"
I am unable to find an expression function to iterate over the array elements, nor convert an array to a string. The only pipeline expression functions I see are to convert string to array and not the other way around.
Am I missing something basic? How can this be achieved?
Use 'join' function present in 'collection' functions in 'Add dynamic content'. For example:
join(variables('ARRAY_VARIABLE'), ',')
I had this same issue and was not totally satisfied just using the join function because it keeps the keys from the json object. Also, using an iterator approach can work but is needlessly expensive and slow if you have a long list. Here was my approach, using join and replace:
replace(replace(join(variables('VAR1'), ','), '{"Code1":', ''), '}', ''))
This will give you exactly the output you are looking for.
I got it working using a ForEach loop activity to iterate over my array and use a Set Variable task with a concat expression function to create my comma separated string.
Wish they had an iterator function in the expression language itself, that would have made it much easier.
In case, you just have two elements in the array, then you can do something like:
#concat(variables('variable_name')[0].Code1, ',', variables('variable_name')[1].Code1)