speed up copy task in azure Data factory - azure

Question

I have a Copy job should copy 100 GB of excel files between two Azure DataLake.
"properties": {
"activities": [
{
"name": "Copy Data1",
"type": "Copy",
"policy": {
"timeout": "7.00:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false,
"secureInput": false
},
"typeProperties": {
"source": {
"type": "AzureDataLakeStoreSource",
"recursive": true,
"maxConcurrentConnections": 256
},
"sink": {
"type": "AzureDataLakeStoreSink",
"maxConcurrentConnections": 256
},
"enableStaging": false,
"parallelCopies": 32,
"dataIntegrationUnits": 256
},
"inputs": [
{
"referenceName": "SourceLake",
"type": "DatasetReference"
}
],
"outputs": [
{
"referenceName": "DestLake",
"type": "DatasetReference"
}
]
}
],
my throughput is about 4 MB/s. As I read here it should be 56 MB/s. What should I do to reach this throughput?

Answer 1

You can use the Copy actives Performance tuning to help you tune the performance of your Azure Data Factory service with the copy activity.
Summary:
Take these steps to tune the performance of your Azure Data Factory service with the copy activity.
Establish a baseline. During the development phase, test your pipeline by using the copy activity against a representative data sample. Collect execution details and performance characteristics following copy activity monitoring.
Diagnose and optimize performance. If the performance you observe doesn't meet your expectations, identify performance bottlenecks. Then, optimize performance to remove or reduce the effect of bottlenecks.
In some cases, when you run a copy activity in Azure Data Factory, you see a "Performance tuning tips" message on top of the copy activity monitoring page, as shown in the following example. The message tells you the bottleneck that was identified for the given copy run. It also guides you on what to change to boost copy throughput.
Your file is about 100 GB size. But test files for file-based stores are multiple files with 10 GB in size. The performance may be different.
Hope this helps.