In Azure Data Factory, I have Lookup activity. "Lookup" activity, which reads the JSON Data from SQL DB (more than 1 row) and bring into ADF Pipeline.
The lookup activity output contains escape character. Please see this:
{\ "resourceType\ ":\ "counter","id" :\ "9i5W6tp-JTd-24252\ "
How to remove escape character? Any help is appreciated.
Since your query result is an JSON String array, we need to do more to remove escape character.
Here is my steps:
Firstly, we can define two array type variables.
Here is the summary, the Lookup activity will return an JSON String array.
Here is the output of my Lookup activity.The data structure will affect the following expression at the Append variable inside the Foreach activity.
At the ForEach activity we can use #activity('Lookup1').output.value to get the JSON String array.
Inside the ForEach activity, we can use #json(item().value) to get the one object of the JSON String array and remove escape character.
In the end, we can use a SetVariable activity to check the result.
After I run debug. We can see the correct result:
Related
I am using Azure Data Factory in which a data flow is used, I want to split my file in to two based on a condition. I am attaching an image with 2 lines, the first one is working but I want to use more programatic approach to achieve the same output:
I have a column named indicator inside my dataset, I want to use contains functionality to split the data, basically having 1 file where a string value inside indicator column has substring Weekly or does not.
Similar to what I would use in pandas:
df1 = df[df.indicator.str.contains('Weekly')]
df2 = df[~df.indicator.str.contains('Weekly')]
You can try the below expression as well in the Conditional split.
contains() expects an array. So first split the column content to create the array and give this to contains function.
contains(split(indicator, ' '),#item=='weekly')
This is my sample data.
Conditional split:
Weekly data in the output:
Remaining data:
If you are looking for the existing of a value inside of a string scalar column, use instr().
https://learn.microsoft.com/en-us/azure/data-factory/data-flow-expressions-usage#instr
I am appending variable with a key column inside for each loop for each record processed
first for each for create, the second time for an update, for delete.(sequentially)
First, two-time(inside two for each) variables appends with correct id, but inside third for each the variable append value with \n value \n\n.
Any idea why these \n are appending with value.
enter image description here
after third for each
There seem to be some escape characters in your data for this ID value.
You can remove the escape character using replace().
Example:
Option1: To replace the escape character, replace enter (press enter instead of \n) and replace it with ‘’.
#replace(variables('id'),'
','')
Option2: Edit the JSON code of the pipeline (which is right side in the pipeline with brackets {}) and replace ‘\n’ with ‘’.
Have text tab delimited file which I'm converting into csv in ADF
In which, some field having comma separated values ( Don't want to split into columns) -> want to treat it as single column
But in ADF trying to separate those values into multiple column
Although, I have set a delimiter as "TAB(\t)", still ADF taking Comma and tab both as delimiter
why?
As you can see in the above Ex,
I want to split this on basis of '\t' delimiter but ADF considering 'tab' as well as 'comma' as a delimiter(Want to treat [pp,aa] as single value/column)
Any solution? plz let me know.
Thank you
Attaching ADF conf as well.
Set the quote character to a single character instead of "No quote character" in source and sink datasets.
I tried with the sample and was able to get the data as expected.
Source:
Source Dataset:
Sink Dataset:
output:
Reference - Dataset Properties
Property
Description
quoteChar
The single character to quote column values if it contains column delimiter. The default value is double quotes ".
escapeChar
The single character to escape quotes inside a quoted value. The default value is backslash \.
I have a copy activity which takes the output of a procedure and writes it to a temp CSV file. I needed to have headers in double quotation mark so after that I have a Data Flow task that takes the temp file and adds the quote all in the sink settings. Yet the output is not what is expected. It looks like the last column is missing in some of the records due to comma in the data.
Is there a way to use only copy activity but still have the column names in double quotes?
When we set the column delimiter, data factory will consider the first row as the schema according the delimiter number. If your data which has the value which same with the column delimiter, then you will miss some columns.
Just for now in Data Factory, we can't solve it. The only way is that please se the different column delimiter, for example the '|':
Output example:
And we also can't make the header wrapped by double quote for the output .csv file. It's not supported in Data Factory.
HTH.
I need to create a long list of complex strings, containing the data of different fields in different places to create explanatory reports.
The only way I conceived, in Access 2010, is to save text parts in a table, together with field names to be used to compose the string to be shown (see line1 expression in figure). Briefly:
//field A contain a string with a field name:
A = "[Quantity]"
//query expression:
=EVAL(A)
//return error instead the number contained in field [Quantity], present in the query dataset
I thought doing an EVAL on a field (A), to obtain the value of the field (B) which name is contained in field A. But seems not working.
Any way exist?
Example (very simplified):
Sample query that EVAL a field containing other field names to obtain the value of the fields
Any Idea?
PS: Sorry for my english, not my mothertongue.
I found a interesting workaround in another forum.
Other people had same problem using EVAL, but found that it is possible to substitute a string with a field contents using REPLACE function.
REPLACE("The value of field Quantity is {Quantity}";"{Quantity}";[Quantity])
( {} are used only for clarity, not needed if one knows that words to be substituted do not compare in the string). Using this code in a query, and nesting as many REPLACE as many different fields one want to use:
REPLACE(REPLACE("<Salutation> <Name>";"<Salutation>";[Salutation]);"<Name>";[Name])
it is possible to embed fields name in a string and substitute them with the current value of that field in a query. Of course the latter example can be done more simply with a concatenation (&), but if the string is contained in a field instead that hardcoded, it can be linked to records as needed.
REPLACE(REPLACE([DescriptiveString];"[Salutation]";[Salutation]);"[Name]";[Name])
Moreover, it is possibile to create complex strings context-based as:
REPLACE(REPLACE(REPLACE("{Salutation} {Name} {MaidenName}";"{Salutation}";[Salutation]);"{Name}";[Name]);"{MaidenName}";IIF(Isnull([MaidenName]);"";[MaidenName]))
The hard part is to enumerate all the field's placeholders one wants to insert in the string (like {Quantity},{Salutation}, {Name}, {MaidenName}) in the REPLACE call, while with EVAL one would avoid this boring part, if only it was working.
Not as neat as I would, but works.