When i do kubectl logs pod_name -n namespace, i find this below:
warn: FileSystemXmlRepository[60] Storing keys in a directory '/home/app/.aspnet/DataProtection-Keys' that may not be persisted outside of the container. Protected data will be unavailable when container is destroyed.
warn: XmlKeyManager[35] No XML encryptor configured. Key {e4463e4a-d257-44ba-932e-494413eae8fd} may be persisted to storage in unencrypted form.
warn: WarmupServicesStartupTask Warming up 14 controllers on http://localhost:3000
fail: WarmupServicesStartupTask Connection refused
System.Net.Http.HttpRequestException: Connection refused
---> System.Net.Sockets.SocketException (111): Connection refused
at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at MCS.Core.Web.Api.WarmupServicesStartupTask.<>c__DisplayClass4_0.<<StartAsync>b__2>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at MCS.Core.Web.Api.WarmupServicesStartupTask.StartAsync(CancellationToken cancellationToken)
Hosting environment: lab Content root path: / Now listening on:
http://[::]:3000 Application started. Press Ctrl+C to shut down. warn:
HttpsRedirectionMiddleware[3] Failed to determine the https port
for redirect.
Can someone help me to find a solution?
Thanks a lot,
Mauro.
I am working with Azure function app. I used a tool called the Durable Functions Monitor (see: https://github.com/microsoft/DurableFunctionsMonitor) to attempt to cancel an in-progress orchestrator function since I no longer wished for it to complete. To do this I found the instance in the Durable Functions Monitor UI (linked above) and first "terminated" the orchestration and then "purged" it. I did this a couple times.
Since then in the log stream on my function app I am seeing the following error message:
2022-12-22T23:38:44Z [Error] An error occurred while processing messages on XXXXXXXXXXXXX-workitems: DurableTask.AzureStorage.Storage.DurableTaskStorageException: The specified blob does not exist.
---> Microsoft.WindowsAzure.Storage.StorageException: The specified blob does not exist.
at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteAsyncInternal[T](RESTCommand`1 cmd, IRetryPolicy policy, OperationContext operationContext, CancellationToken token)
at Microsoft.WindowsAzure.Storage.Blob.CloudBlob.DownloadRangeToStreamAsync(Stream target, Nullable`1 offset, Nullable`1 length, AccessCondition accessCondition, BlobRequestOptions options, OperationContext operationContext, IProgress`1 progressHandler, CancellationToken cancellationToken)
at DurableTask.AzureStorage.Storage.AzureStorageClient.WrapFunctionWithReturnType(Func`3 storageRequest, OperationContext context, CancellationToken cancellationToken) in /_/src/DurableTask.AzureStorage/Storage/AzureStorageClient.cs:line 156
at DurableTask.AzureStorage.TimeoutHandler.ExecuteWithTimeout[T](String operationName, String account, AzureStorageOrchestrationServiceSettings settings, Func`3 operation, AzureStorageOrchestrationServiceStats stats, String clientRequestId) in /_/src/DurableTask.AzureStorage/TimeoutHandler.cs:line 133
at DurableTask.AzureStorage.Storage.AzureStorageClient.MakeStorageRequest[T](Func`3 storageRequest, String accountName, String operationName, String clientRequestId, Boolean force) in /_/src/DurableTask.AzureStorage/Storage/AzureStorageClient.cs:line 136
Is this error related to my actions? If so:
For the present time, how can I stop this error from logging. What process is looking for this blob and how to do tell it to stop doing so
How can I safely terminate an orchestrator function in the future to prevent this state of affairs.
Bonus question:
I noticed that sometimes terminating an orchestrator does not seem to also terminate a long-ish (~ 10 min) running activity function. Is there a way to terminate an orchestration and any activities it has called using any available tool?
I have a ASP.NET Core 6 API deployed to Azure using the F1 free App Plan. Calling any endpoint returns a status code of 500.
I used the Log stream feature. Below my signature I will post the whole thing but the highlights are these:
IIS was not able to access the web.config file for the Web site
The authenticated user does not have permission to use this DLL.
Install the .NET Extensibility feature if the request is mapped to a managed handler
My ASP.NET Core 6 API does not have a web.config file at all. It has appsettings.json file but not web.config file. Should I add one?
Below my sig I will add the full log. I appreciate any help!
Thanks, Dan
2022-11-08T14:52:02 Welcome, you are now connected to log-streaming service. The default timeout is 2 hours. Change the timeout with the App Setting SCM_LOGSTREAM_TIMEOUT (in seconds).
IIS Detailed Error - 500.0 - Internal Server Error
HTTP Error 500.0 - Internal Server Error
The page cannot be displayed because an internal server error has occurred.
Most likely causes:
IIS received the request; however, an internal error occurred during the processing of the request. The root cause of this error depends on which module handles the request and what was happening in the worker process when this error occurred.
IIS was not able to access the web.config file for the Web site or application. This can occur if the NTFS permissions are set incorrectly.
IIS was not able to process configuration for the Web site or application.
The authenticated user does not have permission to use this DLL.
The request is mapped to a managed handler but the .NET Extensibility Feature is not installed.
Things you can try:
Ensure that the NTFS permissions for the web.config file are correct and allow access to the Web server's machine account.
Check the event logs to see if any additional information was logged.
Verify the permissions for the DLL.
Install the .NET Extensibility feature if the request is mapped to a managed handler.
Create a tracing rule to track failed requests for this HTTP status code. For more information about creating a tracing rule for failed requests, click here.
Detailed Error Information:
Module AspNetCoreModuleV2Notification ExecuteRequestHandlerHandler aspNetCoreError Code 0x00000000 Requested URL https://FlowCastApi20221108072930:80/api/cohortPhysical Path C:\home\site\wwwroot\api\cohortLogon Method AnonymousLogon User Anonymous
More Information:
This error means that there was a problem while processing the request. The request was received by the Web server, but during processing a fatal error occurred, causing the 500 error.
View more information ยป
Microsoft Knowledge Base Articles:
2022-11-08 14:52:09.840 +00:00 [Error] Microsoft.EntityFrameworkCore.Database.Connection: An error occurred using the connection to database 'flowcast-dev' on server 'tcp:greenshoes-dev.database.windows.net,1433'.
2022-11-08 14:52:09.853 +00:00 [Error] Microsoft.EntityFrameworkCore.Query: An exception occurred while iterating over the results of a query for context type 'FlowCastApi.Model.DataContext'.Microsoft.Data.SqlClient.SqlException (0x80131904): Cannot open server 'greenshoes-dev' requested by the login. Client with IP address '20.49.104.46' is not allowed to access the server. To enable access, use the Windows Azure Management Portal or run sp_set_firewall_rule on the master database to create a firewall rule for this IP address or address range. It may take up to five minutes for this change to take effect.at Microsoft.Data.ProviderBase.DbConnectionPool.CheckPoolBlockingPeriod(Exception e)at Microsoft.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)at Microsoft.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)at Microsoft.Data.ProviderBase.DbConnectionPool.WaitForPendingOpen()--- End of stack trace from previous location ---at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenInternalAsync(Boolean errorsExpected, CancellationToken cancellationToken)at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenInternalAsync(Boolean errorsExpected, CancellationToken cancellationToken)at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenAsync(CancellationToken cancellationToken, Boolean errorsExpected)at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(AsyncEnumerator enumerator, CancellationToken cancellationToken)at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()ClientConnectionId:ec461d75-6cc2-4508-9541-47d546a25d0dError Number:40615,State:1,Class:14ClientConnectionId before routing:f9b3f656-f418-4e09-ac95-a3ab8ace69a6Routing Destination:e194283991a1.tr29051.eastus1-a.worker.database.windows.net,11042Microsoft.Data.SqlClient.SqlException (0x80131904): Cannot open server 'greenshoes-dev' requested by the login. Client with IP address '20.49.104.46' is not allowed to access the server. To enable access, use the Windows Azure Management Portal or run sp_set_firewall_rule on the master database to create a firewall rule for this IP address or address range. It may take up to five minutes for this change to take effect.at Microsoft.Data.ProviderBase.DbConnectionPool.CheckPoolBlockingPeriod(Exception e)at Microsoft.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)at Microsoft.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection) at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)at Microsoft.Data.ProviderBase.DbConnectionPool.WaitForPendingOpen()--- End of stack trace from previous location ---at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenInternalAsync(Boolean errorsExpected, CancellationToken cancellationToken)at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenInternalAsync(Boolean errorsExpected, CancellationToken cancellationToken)at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenAsync(CancellationToken cancellationToken, Boolean errorsExpected)at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(AsyncEnumerator enumerator, CancellationToken cancellationToken)at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()ClientConnectionId:ec461d75-6cc2-4508-9541-47d546a25d0dError Number:40615,State:1,Class:14ClientConnectionId before routing:f9b3f656-f418-4e09-ac95-a3ab8ace69a6Routing Destination:e194283991a1.tr29051.eastus1-a.worker.database.windows.net,11042
2022-11-08 14:52:09.872 +00:00 [Error] Microsoft.AspNetCore.Server.IIS.Core.IISHttpServer: Connection ID "15204152343613408971", Request ID "800006cc-0000-d300-b63f-84710c7967bb": An unhandled exception was thrown by the application.Microsoft.Data.SqlClient.SqlException (0x80131904): Cannot open server 'greenshoes-dev' requested by the login. Client with IP address '20.49.104.46' is not allowed to access the server. To enable access, use the Windows Azure Management Portal or run sp_set_firewall_rule on the master database to create a firewall rule for this IP address or address range. It may take up to five minutes for this change to take effect.at Microsoft.Data.ProviderBase.DbConnectionPool.CheckPoolBlockingPeriod(Exception e)at Microsoft.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)at Microsoft.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)at Microsoft.Data.ProviderBase.DbConnectionPool.WaitForPendingOpen()--- End of stack trace from previous location ---at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenInternalAsync(Boolean errorsExpected, CancellationToken cancellationToken)at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenInternalAsync(Boolean errorsExpected, CancellationToken cancellationToken)at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenAsync(CancellationToken cancellationToken, Boolean errorsExpected)at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(AsyncEnumerator enumerator, CancellationToken cancellationToken)at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()at Microsoft.EntityFrameworkCore.EntityFrameworkQueryableExtensions.ToListAsync[TSource](IQueryable`1 source, CancellationToken cancellationToken)at Microsoft.EntityFrameworkCore.EntityFrameworkQueryableExtensions.ToListAsync[TSource](IQueryable`1 source, CancellationToken cancellationToken)at FlowCastApi.Controllers.BaseController`1.GetAllAsync(Int32 count, Int32 skip, String searchTerm, String sortBy) in C:\_source\Greenshoes\api_dotnet\FlowCastApi\Controllers\Base\BaseController.cs:line 41at lambda_method5(Closure , Object )at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.AwaitableObjectResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask`1 actionResultValueTask)at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|20_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)at Microsoft.AspNetCore.Routing.EndpointMiddleware.g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContextOfT`1.ProcessRequestAsync()ClientConnectionId:ec461d75-6cc2-4508-9541-47d546a25d0dError Number:40615,State:1,Class:14ClientConnectionId before routing:f9b3f656-f418-4e09-ac95-a3ab8ace69a6Routing Destination:e194283991a1.tr29051.eastus1-a.worker.database.windows.net,11042
I have created ASP.Net CORE Web API and deployed to Azure App service. Able to access the endpoint without any issues.
Make sure you have checked the below options while creating the App.
My Program.cs
var builder = WebApplication.CreateBuilder(args);
builder.Services.AddControllers();
builder.Services.AddEndpointsApiExplorer();
builder.Services.AddSwaggerGen();
builder.Services.ConfigureSwaggerGen(setup =>
{
setup.SwaggerDoc("v1", new Microsoft.OpenApi.Models.OpenApiInfo
{
Title = "Weather Forecasts",
Version = "v1"
});
});
var app = builder.Build();
if (app.Environment.IsDevelopment())
{
app.UseSwagger();
app.UseSwaggerUI();
}
app.UseHttpsRedirection();
app.UseAuthorization();
app.MapControllers();
app.Run();
In Azure Portal => API Management service => API's, check whether you can see the created API.
My ASP.NET Core 6 API does not have a web.config file at all. It has appsettings.json file but not web.config file. Should I add one?
When we deploy .Net CORE Web App to Azure, web.config file will be generated by default. You can check it in KUDU DebugConsole.
Your Azure App Service=>Advanced Tools => Go => Debug Console
In web.config file , change modules="AspNetCoreModuleV2" to modules="AspNetCoreModule"
<add name="aspNetCore" path="*" verb="*" modules="AspNetCoreModule" resourceType="Unspecified" />
Your error indicates the issue while connecting to DB.
Make sure you have configured SQL Server Database.
Thanks Harsitha for the thorough response. My problem ended up being that I needed to mark my database server as allowing connections to Azure services.
I have an application (aspnet core app 3.1) where I load data during startup from azure blob storage with the following code:
BlobClient client = new BlobClient(loaderOptions.ConnectionString, loaderOptions.BlobContainerName, loaderOptions.BlobName);
var response = client.Download();
using (var memoryStream = new MemoryStream())
{
response.Value.Content.CopyTo(memoryStream);
return memoryStream.ToArray();
}
Locally, on my PC everything works fine. But when I publish this app to azure app service (windows) I get the following errors:
Unhandled exception. System.AggregateException: Retry failed after 6 tries. (No such host is known.) (No such host is known.) (No such host is known.) (No such host is known.) (No such host is known.) (No such host is known.)
---> Azure.RequestFailedException: No such host is known.
---> System.Net.Http.HttpRequestException: No such host is known.
---> System.Net.Sockets.SocketException (11001): No such host is known.
at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at Azure.Core.Pipeline.HttpClientTransport.ProcessAsync(HttpMessage message)
--- End of inner exception stack trace ---
A week ago everything worked fine. Now I have created new service plans and storage accounts (deleted old ones) and come across this problem.
What can be solution?
I am experiencing the same issue as this. Further analysis would indicate, Local DNS issues. If in doubt trace the address supplied and where needed add a temporary host record in c:\windows
I am downloading a large amount of files >1000 from Azure Datalake V2 and I am continuously getting exception:
The SSL connection could not be established, see inner exception.
<--- Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..
<--- An existing connection was forcibly closed by the remote host.
Stacktrace:
System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..
---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.
--- End of inner exception stack trace ---
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token)
at System.Net.FixedSizeReader.ReadPacketAsync(Stream transport, AsyncProtocolRequest request)
at System.Net.Security.SslStream.EndProcessAuthentication(IAsyncResult result)
at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
The code:
var downloadTasks = job.Files.AsParallel().Select(x => Download(x));
await Task.WhenAll(downloadTasks);
private async Task Download(DownloadableFile file)
{
try
{
var options = new BlobRequestOptions
{
ParallelOperationThreadCount = 8,
DisableContentMD5Validation = true,
StoreBlobContentMD5 = false
};
var xzBlob = await _cloudBlobFileService.GetBlockBlobReference(file.FilePath);
await xzBlob.DownloadToFileAsync(file.LocalFilePath, FileMode.Create, null, options, null);
}
catch (Exception e)
{
_log.LogCritical(e, "Error downloading " + file.FilePath);
}
}
I have also added this:
ServicePointManager.DefaultConnectionLimit = Environment.ProcessorCount * 8;
ServicePointManager.Expect100Continue = false;
Using .Net core 3.1 and WindowsAzure.Storage 9.3.3
to program.cs main method in webjob
We used to have a blobstorage configured without datalake, but after switching to datalake this has appeared. It doesn't affect the application to much as skipped downloads are retried later. It would however be nice to know whats causing it.
You could start by trying the new storage SDK that hit GA in November, though I can't guarantee that would solve the issue. It is a complete rewrite
While its not possible to pinpoint exactly just from the error message, there are a couple things to look at:
Network errors. It is the by far most likely cause, though it is interesting that it works consistently with your old blob storage account. Increasing the timeout may lower the frequency of network errors, and retry logic will help overcome them.
Using unbounded parallelism is not recommended. ParallelOperationThreadCount is for uploads not downloads, so it isn't throttling the requests in this case. The default limit for server-side connections in .NET is 10, and it is recommended to increase this when using .NET Core, it is something to consider. If you are accessing the same blob or partition too many times concurrently you can start to bump into the concurrent connections limits in Storage.