System.Net.Http.HttpRequestException on download multiple files from Azure Datalake V2 - azure

Question

I am downloading a large amount of files >1000 from Azure Datalake V2 and I am continuously getting exception:
The SSL connection could not be established, see inner exception.
<--- Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..
<--- An existing connection was forcibly closed by the remote host.
Stacktrace:
System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..
---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.
--- End of inner exception stack trace ---
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token)
at System.Net.FixedSizeReader.ReadPacketAsync(Stream transport, AsyncProtocolRequest request)
at System.Net.Security.SslStream.EndProcessAuthentication(IAsyncResult result)
at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
The code:
var downloadTasks = job.Files.AsParallel().Select(x => Download(x));
await Task.WhenAll(downloadTasks);
private async Task Download(DownloadableFile file)
{
try
{
var options = new BlobRequestOptions
{
ParallelOperationThreadCount = 8,
DisableContentMD5Validation = true,
StoreBlobContentMD5 = false
};
var xzBlob = await _cloudBlobFileService.GetBlockBlobReference(file.FilePath);
await xzBlob.DownloadToFileAsync(file.LocalFilePath, FileMode.Create, null, options, null);
}
catch (Exception e)
{
_log.LogCritical(e, "Error downloading " + file.FilePath);
}
}
I have also added this:
ServicePointManager.DefaultConnectionLimit = Environment.ProcessorCount * 8;
ServicePointManager.Expect100Continue = false;
Using .Net core 3.1 and WindowsAzure.Storage 9.3.3
to program.cs main method in webjob
We used to have a blobstorage configured without datalake, but after switching to datalake this has appeared. It doesn't affect the application to much as skipped downloads are retried later. It would however be nice to know whats causing it.

Answer 1

You could start by trying the new storage SDK that hit GA in November, though I can't guarantee that would solve the issue. It is a complete rewrite
While its not possible to pinpoint exactly just from the error message, there are a couple things to look at:
Network errors. It is the by far most likely cause, though it is interesting that it works consistently with your old blob storage account. Increasing the timeout may lower the frequency of network errors, and retry logic will help overcome them.
Using unbounded parallelism is not recommended. ParallelOperationThreadCount is for uploads not downloads, so it isn't throttling the requests in this case. The default limit for server-side connections in .NET is 10, and it is recommended to increase this when using .NET Core, it is something to consider. If you are accessing the same blob or partition too many times concurrently you can start to bump into the concurrent connections limits in Storage.