how to convert YYYYDDMMmmHHss to DDMMYYYYmmHHSS in Azure data factory file name output
Regards
Ravi
I tried this and my source and sink are both csv.
This is my file name: 20201706170905.csv.
First:create a getMetaData activity like this
Then:create a copy activity,and the sink dataset filename set like this:
The expression in file name(you can use concat and substring to convert what you want):
#concat(substring(activity('Get Metadata1').output.itemName,4,4),substring(activity('Get Metadata1').output.itemName,0,4),substring(activity('Get Metadata1').output.itemName,10,2),substring(activity('Get Metadata1').output.itemName,8,2),substring(activity('Get Metadata1').output.itemName,12,6))
Finally: run the pipeline(if you don't need the Original file,you can use delete activity delete it).
Result:
Hope this can help you.
Related
I am using ADF v2 DataFlow ativity to load data from a csv file in a Blob Storage into a table in Azure SQL database. In the Dataflow (Source - Blob storage), in Source options, there is an option 'After Completion(No Action/Delete Source file/ Move)'. I am looking to utilize the move option to save those csv files in a container renaming those files in concatenation with with today's date. How do I frame the logic for this? Can someone please help?
You can define the file name explicitly in both From and To-fields. This is not so well (if at all) documented, and I found it just trying different approaches.
You can also add dynamic content such as timestamps. Here's an example:
concat('incoming/archive/', toString(currentUTC(), 'yyyy-MM-dd_HH.mm.ss_'), 'target_file.csv')
You could parameter the source file to achieve that. Please ref my example.
Data Flow parameter settings:
Set the source file and move expression in Source Options:
Expressions to rename the source with "name + current date":
concat(substring($filename, 1, length($filename)-4),toString(currentUTC(),'yyyy-MM-dd') )
My full file name is "word.csv", the output file name is "word2020-01-26",
HTH.
I've been going round in circles trying to get what I thought would be a relatively trivial pipeline working in Azure Data Factory. I have a CSV file with a schema like this:
Id, Name, Color
1, Apple, Green
2, Lemon, Yellow
I need to transform the CSV into a JSON file that looks like this:
{"fruits":[{"Id":"1","Name":"Apple","Color":"Green"},{"Id":"2","Name":"Lemon","Color":"Yellow"}]
I can't find a simple example that helps me understand how to do this in ADF. I've tried a Copy activity, and a data flow, but the furthest I've got is a json object like this:
{"fruits":{"Id":"1","Name":"Apple","Color":"Green"}}
{"fruits":{"Id":"2","Name":"Lemon","Color":"Yellow"}}
Surely this is simple to achieve. I'd be very grateful if anyone has any suggestions. Thanks!
https://learn.microsoft.com/en-us/azure/data-factory/copy-activity-schema-and-type-mapping#tabularhierarchical-source-to-hierarchical-sink
"When copying data from tabular source to hierarchical sink, writing to array inside object is not supported"
But, if we put file pattern under Sink properties as 'Array of Objects', you can achieve somewhere till here:
[{"Id":"1","Name":" Apple","Color":" Green"}
,{"Id":"2","Name":" Lemon","Color":" Yellow"}
]
I have a basic requirement where I want to append time stamp to file extracted from sql db and put it in blob.i use utcnow() and it creates a timestamp with T and all which I dont need.
any format expression to get date and just time??
New to javascript expressions as I am from ssis background
Help appreciated
The only way you can do that is copy and create a new blob with a new name concat with the timestamp.
Data Factory doesn't support rename the blob.
I only succeed with one file.
You can follow my steps:
Using lookup activity to get the timestamp from SQL database.
Using Get metadata to get the blob name from Storage.
Using Copy data activity to copy and create new file name blob.
Pileline preview:
Lookup preview:
Get metadata and Source Dataset:
Copy data activity Source setting:
Copy data activity Sink setting:
Add parameter to set the new file name in source datasaet:
Using expression to create the new file with the filename and timestamp:
#concat(split(activity('Get Metadata1').output.itemName,'.')[0],activity('Lookup1').output.firstRow.tt)
Then check the output file in the Blob Storage:
Hope this helps.
You can use expression in the destination file name, in the sink.
toTimestamp(utcnow(), 'yyyyMMdd_HHmm_ss')
It's seem ADF v2 does not support writing data to TEXT file (.TXT).
After select File System
But don't see TextFormat at the next screen
So do we any method to write data to TEXT file ?
Thanks,
Thai
Data Factory only support these 6 file formats:
Please see: Supported file formats and compression codecs in Azure Data Factory.
If we want to write data to a txt file, the only format we can using is Delimited text, when the pipeline finished, you will get a txt file.
Reference: Delimited text: Follow this article when you want to parse the delimited text files or write the data into delimited text format.
For example, I create a pipeline to copy data from Azure SQL to Blob, choose DelimitedText format as Sink dataset:
The txt file I get in Blob Storeage:
Hope this helps
I think what you are looking for is DelimitedText dataset. You can specify extension as part of the file name
Why do I keep getting this error while using a folder from a blob container as source (which contains only one GZ compressed file) in copy activity in data factory v2 and as sink another blob storage (but I want the file decompressed)?
"message":"ErrorCode=UserErrorFormatIsRequired,
'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,
Message=Format setting is required for file based store(s) in this scenario.,Source=Microsoft.DataTransfer.ClientLibrary,'",
I know it means I need to specify explicitly the format for my sink dataset, but I am not sure how to do that.
I suggest using the copy data tool.
step 1
step 2
According you comment, I tried a lot times, unless you choose the compressed file as source dataset and import the schemas, Azure Data factory copy actives will not help you decompress the file.
If the files in the the compressed file don't have the same schema, the copy active also could be failed.
Hope this helps.
The easiest way to do this: go to the dataset, and click on the tab Schema, then Import Schema.
Hope this helped!!