In Logstash I want to convert the JSON that comes from Input like RMQ and I would like to keep both JSON data to use it in a dynamic way for creating index/file names and need JSON data to be replaced with one double quote with two double quotes
for example:
logstash input gives data as below:
{"foo":"bar"}
Ouput:
{""foo"":""bar""}
Related
I have a csv which is somewhat like foo,bar. This must go into a single column but what I see is it goes into a separate column as it is comma delimited.
How do we handle this ? I also added double quotes "foo,bar" and put the escape character as double quotes(") in source but I get the data with the double quotes in the final data.
Please suggest how do we handle this situation in ADF ?
Handle comma data in csv file in Azure Data Factory
If your source file has data which contain comma(,) in it and you set the Column delimiter also as comma(,) then while reading file it will consider different column wherever it finds comma(,) in data.
To solve this there are two methods:
You can change the Column delimiter for file like Semicolon (;), Pipe (|), Tab (\t) etc; then it will not consider the comma in data as different column.
Add Quotes ("") to data where it contains comma
e.g, "foo,bar" and set Quote character as Double quote (") so it will consider it as single value
I also added double quotes "foo,bar" and put the escape character as double quotes(") in source but I get the data with the double quotes in the final data.
To handle this set Escape character as No escape character and Quote character as No quote character in the Sink dataset
Sink dataset setting:
Sample data:
Output
I tried converting data in table to a file and able to achieve. But, in the output file generated,there is "\" comming in. Can you please help how to eliminate or remove "\" from file.
Note: Have used Copy stage to move data from oracle sql table to storage account.
Output:
"\"we16\",\"ACTIVE\",\"we16\",01-JAN-22,\"Tester\""
"\"sb64\",\"ACTIVE\",\"sb64\",01-JAN-22,\"\"Operations"
"\"sb47\",\"ACTIVE\",\"sb47\",01-JAN-22,\"\"Developer"
"\"ud53\",\"ACTIVE\",\"ud53\",01-JAN-22,\"\"Manager"
"\"hk72\",\"ACTIVE\",\"hk72\",01-JAN-22,\"\"Tester"
"\"sk99\",\"ACTIVE\",\"sk99\",01-JAN-22,\"\"Tester"
I have reproduced the above with SQL as source and Blob as sink in copy activity.
This is my sample SQL data with Double Quotes on a column.
sink dataset:
When I set the Escape character to Backslash and Quote character to Double quote("), I got the same result as above.
In the above we have the data with double quotes and our Quote character also a double Quote, so that's why it is escaping the extra double quotes with the Escaping character.
As our table has values with double quotes, we need to set the Quote characters to any other values apart from double quotes.
So, when I have set it to single quotes('), I got proper result.
Result data of double quotes with single quotes as Quote characters:
When you are using this dataset in synapse or ADF, set it to the single Quote so that you can use the correct data.
Have text tab delimited file which I'm converting into csv in ADF
In which, some field having comma separated values ( Don't want to split into columns) -> want to treat it as single column
But in ADF trying to separate those values into multiple column
Although, I have set a delimiter as "TAB(\t)", still ADF taking Comma and tab both as delimiter
why?
As you can see in the above Ex,
I want to split this on basis of '\t' delimiter but ADF considering 'tab' as well as 'comma' as a delimiter(Want to treat [pp,aa] as single value/column)
Any solution? plz let me know.
Thank you
Attaching ADF conf as well.
Set the quote character to a single character instead of "No quote character" in source and sink datasets.
I tried with the sample and was able to get the data as expected.
Source:
Source Dataset:
Sink Dataset:
output:
Reference - Dataset Properties
Property
Description
quoteChar
The single character to quote column values if it contains column delimiter. The default value is double quotes ".
escapeChar
The single character to escape quotes inside a quoted value. The default value is backslash \.
In Azure Data Factory, I have Lookup activity. "Lookup" activity, which reads the JSON Data from SQL DB (more than 1 row) and bring into ADF Pipeline.
The lookup activity output contains escape character. Please see this:
{\ "resourceType\ ":\ "counter","id" :\ "9i5W6tp-JTd-24252\ "
How to remove escape character? Any help is appreciated.
Since your query result is an JSON String array, we need to do more to remove escape character.
Here is my steps:
Firstly, we can define two array type variables.
Here is the summary, the Lookup activity will return an JSON String array.
Here is the output of my Lookup activity.The data structure will affect the following expression at the Append variable inside the Foreach activity.
At the ForEach activity we can use #activity('Lookup1').output.value to get the JSON String array.
Inside the ForEach activity, we can use #json(item().value) to get the one object of the JSON String array and remove escape character.
In the end, we can use a SetVariable activity to check the result.
After I run debug. We can see the correct result:
My source file has nvarchar and numeric columns.
Numeric column has thousand separator, do identify the value comes with double quotes. When i use the quoteChar ("\"") in file format the numeric value works fine.
Same time the nvarchar column (Name) has multiple double quotes between the data, if i use the quoteChar the values are split into further more columns based on the number of double quotes.
Is there any fix/solution for this?
Based on the properties in Text format, only one character is allowed. So you can't separate the different data types by different delimiter.
You could try to use | column delimiter if your nvarchar doesn't have | character. Or maybe you have to parse your source file to remove the thousand separator in an Azure Function activity before the transfer. Then it could be identified by copy activity in adf.
ADF parser fails while reading text that is encapsulated with double quotes and also have a comma within the text, like "Hello, World". To make it work, set Escape Character and Quote Character properties value to double quote. This will save the whole text with double quotes into destination.