I have an application (aspnet core app 3.1) where I load data during startup from azure blob storage with the following code:
BlobClient client = new BlobClient(loaderOptions.ConnectionString, loaderOptions.BlobContainerName, loaderOptions.BlobName);
var response = client.Download();
using (var memoryStream = new MemoryStream())
{
response.Value.Content.CopyTo(memoryStream);
return memoryStream.ToArray();
}
Locally, on my PC everything works fine. But when I publish this app to azure app service (windows) I get the following errors:
Unhandled exception. System.AggregateException: Retry failed after 6 tries. (No such host is known.) (No such host is known.) (No such host is known.) (No such host is known.) (No such host is known.) (No such host is known.)
---> Azure.RequestFailedException: No such host is known.
---> System.Net.Http.HttpRequestException: No such host is known.
---> System.Net.Sockets.SocketException (11001): No such host is known.
at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at Azure.Core.Pipeline.HttpClientTransport.ProcessAsync(HttpMessage message)
--- End of inner exception stack trace ---
A week ago everything worked fine. Now I have created new service plans and storage accounts (deleted old ones) and come across this problem.
What can be solution?
I am experiencing the same issue as this. Further analysis would indicate, Local DNS issues. If in doubt trace the address supplied and where needed add a temporary host record in c:\windows
Related
When i do kubectl logs pod_name -n namespace, i find this below:
warn: FileSystemXmlRepository[60] Storing keys in a directory '/home/app/.aspnet/DataProtection-Keys' that may not be persisted outside of the container. Protected data will be unavailable when container is destroyed.
warn: XmlKeyManager[35] No XML encryptor configured. Key {e4463e4a-d257-44ba-932e-494413eae8fd} may be persisted to storage in unencrypted form.
warn: WarmupServicesStartupTask Warming up 14 controllers on http://localhost:3000
fail: WarmupServicesStartupTask Connection refused
System.Net.Http.HttpRequestException: Connection refused
---> System.Net.Sockets.SocketException (111): Connection refused
at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at MCS.Core.Web.Api.WarmupServicesStartupTask.<>c__DisplayClass4_0.<<StartAsync>b__2>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at MCS.Core.Web.Api.WarmupServicesStartupTask.StartAsync(CancellationToken cancellationToken)
Hosting environment: lab Content root path: / Now listening on:
http://[::]:3000 Application started. Press Ctrl+C to shut down. warn:
HttpsRedirectionMiddleware[3] Failed to determine the https port
for redirect.
Can someone help me to find a solution?
Thanks a lot,
Mauro.
I'm working through an online course on building Microservices in .NET - https://www.dotnetmicroservices.com/.
But while the instructor is running on Windows, I'm working on Linux - Linux Mint 20.1 (Ulyssa).
.NET core and .NET 5.0 are supposed to be cross-platform, and I've had no real issues in getting things to work, until now.
At this point, I have two webapi services, one providing identity services and one providing a catalog service.
The identity service is configured to use IdentityServer4, IdentityServer4.AspNetIdentity, and Microsoft.AspNetCore.Identity.UI to provide OAuth 2.0 and OpenID services.
The catalog service is configured to use Microsoft.AspNetCore.Authentication.JwtBearer, and requires a valid JWT to access the endpoints.
So, in Postman, I have a Request configured to access a simple GET endpoint in the catalog service. In the Authorization tab for the Request I choose OAuth 2.0, and enter the necessary data to make the request, including the Auth URL and Access Token URL (https://localhost:5003/connect/authorize and https://localhost:5003/connect/token).
When I click on Postman's "Get New Access Token" button, I get the login page, I log in, and get the "MANAGE ACCESS TOKENS" dialog. I can copy the Access Token or the id_token from the dialog and paste them into https://jwt.ms/ and they both look fine.
So that much is working.
But when I click on Postman's "Use Token" button, and then do a "Send" on the Request, I get an exception in the catalog service:
fail: Microsoft.AspNetCore.Authentication.JwtBearer.JwtBearerHandler[3]
Exception occurred while processing message.
System.InvalidOperationException: IDX20803: Unable to obtain configuration from: 'System.String'.
---> System.IO.IOException: IDX20804: Unable to retrieve document from: 'System.String'.
---> System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid because of errors in the certificate chain: PartialChain
at System.Net.Security.SslStream.SendAuthResetSignal(ProtocolToken message, ExceptionDispatchInfo exception)
at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm)
at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Boolean async, Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Boolean async, Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.SendAsyncCore(HttpRequestMessage request, HttpCompletionOption completionOption, Boolean async, Boolean emitTelemetryStartStop, CancellationToken cancellationToken)
at Microsoft.IdentityModel.Protocols.HttpDocumentRetriever.GetDocumentAsync(String address, CancellationToken cancel)
--- End of inner exception stack trace ---
at Microsoft.IdentityModel.Protocols.HttpDocumentRetriever.GetDocumentAsync(String address, CancellationToken cancel)
at Microsoft.IdentityModel.Protocols.OpenIdConnect.OpenIdConnectConfigurationRetriever.GetAsync(String address, IDocumentRetriever retriever, CancellationToken cancel)
at Microsoft.IdentityModel.Protocols.ConfigurationManager`1.GetConfigurationAsync(CancellationToken cancel)
--- End of inner exception stack trace ---
at Microsoft.IdentityModel.Protocols.ConfigurationManager`1.GetConfigurationAsync(CancellationToken cancel)
at Microsoft.AspNetCore.Authentication.JwtBearer.JwtBearerHandler.HandleAuthenticateAsync()
It looks like the problem is that dotnet's developer certificate isn't trusted.
The usual method for doing this is to use the dotnet cli:
dotnet dev-certs https --trust
But on Linux, this returns:
Trusting the HTTPS development certificate was requested. Trusting the
certificate on Linux distributions automatically is not supported. For
instructions on how to manually trust the certificate on your Linux
distribution, go to https://aka.ms/dev-certs-trust
So I went to https://aka.ms/dev-certs-trust, and it contains a lot of stuff, most of which clearly isn't relevant. The section on "Ubuntu trust the certificate for service-to-service communication" seemed apropos:
sudo dotnet dev-certs https -ep /usr/local/share/ca-certificates/aspnet/https.crt --format PEM
sudo update-ca-certificates
And while that ran without issue, it didn't not fix the problem.
As for the rest, the page says "Establishing trust is browser specific. The following sections provide instructions for the Chromium browsers Edge and Chrome and for Firefox."
And I don't see how either is relevant to the problem at hand.
What is dotnet core doing, when it is verifying certificates? What do I need to do to mark the dotnet developer's certificate as trusted?
Either you have to stop using HTTPS and use HTTP instead between the services.
I assume you use containers to host the services? and For HTTPS to work, you need a domain name and mapping that domain name to the containers that match the certificate and using localhost (the dev cert) to map to containers I feel is a bad idea.
Better to get a real certificate that you then using different techniques map to the containers. Like editing the hosts file to map the domain to the IP-addresses of your services.
In dev/production, I usually host my HTTPS certificate in Azure Key Vault and download and add it to ASP.NET Core at startup time.
When I work locally, I have created a certificate for the domain indentityservice.local, using the mkcert tool
mkcert.exe -install
mkcert -pkcs12 -p12-file identityservice.local.pfx identityservice.local
Then I copy the generated identityservice.local.pfx and add it to my project. Then if all works, I can contact my local IdentityServer using https://identityService.local
I also do add this entry to my local hosts file at C:\Windows\System32\drivers\etc
127.0.0.1 identityService.local
The following works on Linux:
Generate a local CA.
mkcert -install
Generate the certificate to be used in your service
mkcert -pkcs12 -p12-file myservice.local.pfx myservice.local
Add an entry for myservice.local in /etc/hosts
127.0.0.1 myservice.local
Use the p12 certificate in your dotnet service
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Net;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Hosting;
using Microsoft.Extensions.Hosting;
namespace myservice
{
public class Program
{
public static void Main(string[] args)
{
CreateHostBuilder(args).Build().Run();
}
public static IHostBuilder CreateHostBuilder(string[] args) =>
Host.CreateDefaultBuilder(args)
.ConfigureWebHostDefaults(webBuilder =>
{
#if DEBUG
webBuilder.UseStartup<Startup>().UseKestrel(options =>
{
options.Listen(IPAddress.Loopback, 5001, listenOptions =>
{
listenOptions.UseHttps("./myservice.local.pfx", "changeit");
});
}).UseUrls("https://myservice.local:5001");
#else
webBuilder.UseStartup<Startup>();
#endif
});
}
}
I am downloading a large amount of files >1000 from Azure Datalake V2 and I am continuously getting exception:
The SSL connection could not be established, see inner exception.
<--- Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..
<--- An existing connection was forcibly closed by the remote host.
Stacktrace:
System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..
---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.
--- End of inner exception stack trace ---
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token)
at System.Net.FixedSizeReader.ReadPacketAsync(Stream transport, AsyncProtocolRequest request)
at System.Net.Security.SslStream.EndProcessAuthentication(IAsyncResult result)
at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
The code:
var downloadTasks = job.Files.AsParallel().Select(x => Download(x));
await Task.WhenAll(downloadTasks);
private async Task Download(DownloadableFile file)
{
try
{
var options = new BlobRequestOptions
{
ParallelOperationThreadCount = 8,
DisableContentMD5Validation = true,
StoreBlobContentMD5 = false
};
var xzBlob = await _cloudBlobFileService.GetBlockBlobReference(file.FilePath);
await xzBlob.DownloadToFileAsync(file.LocalFilePath, FileMode.Create, null, options, null);
}
catch (Exception e)
{
_log.LogCritical(e, "Error downloading " + file.FilePath);
}
}
I have also added this:
ServicePointManager.DefaultConnectionLimit = Environment.ProcessorCount * 8;
ServicePointManager.Expect100Continue = false;
Using .Net core 3.1 and WindowsAzure.Storage 9.3.3
to program.cs main method in webjob
We used to have a blobstorage configured without datalake, but after switching to datalake this has appeared. It doesn't affect the application to much as skipped downloads are retried later. It would however be nice to know whats causing it.
You could start by trying the new storage SDK that hit GA in November, though I can't guarantee that would solve the issue. It is a complete rewrite
While its not possible to pinpoint exactly just from the error message, there are a couple things to look at:
Network errors. It is the by far most likely cause, though it is interesting that it works consistently with your old blob storage account. Increasing the timeout may lower the frequency of network errors, and retry logic will help overcome them.
Using unbounded parallelism is not recommended. ParallelOperationThreadCount is for uploads not downloads, so it isn't throttling the requests in this case. The default limit for server-side connections in .NET is 10, and it is recommended to increase this when using .NET Core, it is something to consider. If you are accessing the same blob or partition too many times concurrently you can start to bump into the concurrent connections limits in Storage.
I have a Service Fabric Setup and I have a microservice that registers on port 443 and uses https. I have a reverse proxy in my cluster setup. The reverse proxy is secured with a certificate.
I also use the same certificate when starting up my microservice:
new ServiceInstanceListener(serviceContext =>
new KestrelCommunicationListener(serviceContext, "EndpointHttps", (url, listener) =>
{
ServiceEventSource.Current.ServiceMessage(serviceContext, $"Starting Kestrel on {url}");
return new WebHostBuilder()
.UseKestrel(x =>
{
int port = serviceContext.CodePackageActivationContext.GetEndpoint("EndpointHttps").Port;
x.Listen(IPAddress.IPv6Any, port, listenOptions =>
{
listenOptions.UseHttps(transportCertificate);
listenOptions.NoDelay = true;
});
})
.ConfigureServices(
services => services
.AddSingleton<StatelessServiceContext>(serviceContext))
.UseContentRoot(Directory.GetCurrentDirectory())
.UseEnvironment(environment)
.UseStartup<Startup>()
.UseServiceFabricIntegration(listener, ServiceFabricIntegrationOptions.None)
.UseUrls(url)
.UseSerilog(Logger.Serilog)
.Build();
}))
};
Everything seems to work fine and my site is secure in the browser and the api works. However my log is getting filled up with the following statements (these are just debug messages but they are filling up my log):
133649 Failed to authenticate HTTPS connection. Debug System.IO.IOException: Authentication failed because the remote party has closed the transport stream.
133643 Connection id ""0HLPCETMP8DKM"" started. key='SourceContext'>Microsoft.AspNetCore.Server.Kestrel
133642 Connection id ""0HLPCETMP8DKL"" received FIN. key='SourceContext'>Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets
133641 Connection id ""0HLPCETLEPLQ7"" stopped. key='SourceContext'>Microsoft.AspNetCore.Server.Kestrel
133640 Connection id ""0HLPCETLEPLQ7"" sending FIN. key='SourceContext'>Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets
The exception is:
System.IO.IOException: Authentication failed because the remote party has closed the transport stream.
at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.PartialFrameCallback(AsyncProtocolRequest asyncRequest)
--- End of stack trace from previous location where exception was thrown ---
at System.Net.Security.SslState.ThrowIfExceptional()
at System.Net.Security.SslState.InternalEndProcessAuthentication(LazyAsyncResult lazyResult)
at System.Net.Security.SslState.EndProcessAuthentication(IAsyncResult result)
at System.Net.Security.SslStream.EndAuthenticateAsServer(IAsyncResult asyncResult)
at System.Net.Security.SslStream.<>c.<AuthenticateAsServerAsync>b__51_1(IAsyncResult iar)
at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.AspNetCore.Server.Kestrel.Https.Internal.HttpsConnectionAdapter.InnerOnConnectionAsync(ConnectionAdapterContext context)
Does anyone have any idea why it would do this? Any other logs I can look at or suggestions on how to debug?
Thanks.
This is very likely due to the fact that you're running Service Fabric with the standard template, which sets up a Azure Load Balancer with a health probe for the ports that you've set up to use. The standard probe is generally TCP and will probe the endpoint every 5 seconds. You can remove the probe if that's a viable option. I personally use a less aggressive probe interval.
Note that changing the load balancer rules generally takes a long time, anything from a few minutes to near half an hour.
I am having an intermittent issue with HttpClient. It randomly disposes SslStream ahead of time. It sometimes happens once an hour, sometimes more often. I am running my code on Azure Functions and it seems to be working flawlessly locally, but once I deploy it on Azure, it starts failing.
I know about the pitfalls of HttpClient and thus I am using the IHttpClientFactory in my implementation, which is injected in my API.
I am using microservices and this issue only affects calling another Azure Function app.
My HttpClient timeout is set to 5 minutes and Azure Functions timeout is set to max. 10 minutes, but that doesn't help.
The error message and stack is below:
The operation was canceled.. Inner Exception: The read operation failed, see inner exception.. Cannot access a disposed object.
Object name: 'SslStream'.
at System.Net.Security.SslStreamInternal.ReadAsyncInternal[TReadAdapter](TReadAdapter adapter, Memory`1 buffer)
at System.Net.Http.HttpConnection.FillAsync()
at System.Net.Http.HttpConnection.ReadNextResponseHeaderLineAsync(Boolean foldedHeadersAllowed)
at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at Huedash.Core.HttpClientService.GetAsync[T](String service, String endpoint, ValueTuple`2[] parameters) in C:\_Huedash\HuedashApiGit\HuedashApiCore\Huedash.Core\HttpClientService.cs:line 43
at HuedashConnectorsApp.WarehouseScheduleFunction.WarehouseDownload(Vendor vendor, IConfiguration configuration, IHttpClientService client, ILogger log) in C:\_Huedash\HuedashBackendGit\HuedashBackendCore\Huedash.ConnectorsApp\WarehouseScheduleFunction.cs:line 140