I am starting to see a lot of SocketException's in my Application Insights log, but from the actual machine/azure infrastruture that hosts and run my services. I am certain that it's the services that are somehow causing this but I no clue to how I can debug this any further.
The stacktrace is not really giving me a lot to work with considering it's coming from code I have no power over, since it's being thrown on the actual machine my services is hosted on:
System.Net.Http.HttpRequestException:
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Net.Http.HttpClient+<FinishSendAsyncBuffered>d__58.MoveNext (System.Net.Http, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at Microsoft.ServiceProfiler.Agent.FrontendClient.StampFrontendClient+<>c__DisplayClass9_0`1+<<HttpGetAsync>b__0>d.MoveNext (Microsoft.ServiceProfiler.Agent.FrontendClient, Version=2.6.1909.2701, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at Microsoft.ServiceProfiler.Agent.FrontendClient.StampFrontendClient+<HttpRequestAsync>d__12`1.MoveNext (Microsoft.ServiceProfiler.Agent.FrontendClient, Version=2.6.1909.2701, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at Microsoft.ServiceProfiler.Agent.FrontendClient.StampFrontendClient+<HttpGetAsync>d__9`1.MoveNext (Microsoft.ServiceProfiler.Agent.FrontendClient, Version=2.6.1909.2701, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at Microsoft.ServiceProfiler.Agent.FrontendClient.ProfilerFrontendClient+<GetEtlUploadAccessAsync>d__3.MoveNext (Microsoft.ServiceProfiler.Agent.FrontendClient, Version=2.6.1909.2701, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at Microsoft.ServiceProfiler.Uploaders.StampEtlUploader+<UploadEtlZipFileAsync>d__1.MoveNext (ApplicationInsightsProfiler, Version=2.6.1909.2701, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at Microsoft.ServiceProfiler.Collectors.DetailedTraceCollector+<UploadTraceAsync>d__41.MoveNext (ApplicationInsightsProfiler, Version=2.6.1909.2701, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at Microsoft.ServiceProfiler.Collectors.DetailedTraceCollector+<StopCollectingAndUploadTraceAsync>d__38.MoveNext (ApplicationInsightsProfiler, Version=2.6.1909.2701, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at Microsoft.ServiceProfiler.Engine+<>c__DisplayClass1_1+<<RunAsync>b__2>d.MoveNext (ApplicationInsightsProfiler, Version=2.6.1909.2701, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at Microsoft.ServiceProfiler.Agent.Orchestration.Orchestrator+<TraceAsyncOperation>d__18.MoveNext (Microsoft.ServiceProfiler.Agent.Orchestration, Version=2.6.1909.2701, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
Inner exception System.Net.WebException handled at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw:
at System.Net.HttpWebRequest.EndGetResponse (System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Net.Http.HttpClientHandler.GetResponseCallback (System.Net.Http, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
Inner exception System.Net.Sockets.SocketException handled at System.Net.HttpWebRequest.EndGetResponse:
at System.Net.Sockets.Socket.BeginConnectEx (System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Net.Sockets.Socket.UnsafeBeginConnect (System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Net.ServicePoint.ConnectSocketInternal (System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
If I look at the portal, all I see is a clue that it might come from the ApplicationInsightsProfiler, which is again nothing I have any touch with in my codebase, directly.
I am afraid I am exhausting the http connection pool somehow, but how can I further investigate it ?
To check your http connections and SNAT ports please from your app open "Diagnose and solve problems" then "Availability and Performance". From there you have 2 tabs to visit: "SNAT Port Exhaustion" and "TCP Connections".
There you can see your http connections and their targets.
Related
I have an app which works fine locally but when I try going to a specific page on my Azure hosted site I get an error.
Error:
[1] Ensure that NodeJS is installed and can be found in one of the PATH directories. The current PATH enviroment variable is: /opt/dotnetcore-tools:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/site/wwwroot. Make sure that the Node executable is in one of those directories, or update your PATH.
[2] Refer to the InnerException for further details. No such file or directory
Jering.Javascript.NodeJS.ConnectionException:
at Jering.Javascript.NodeJS.OutOfProcessNodeJSService.ConnectIfNotConnected (Jering.Javascript.NodeJS, Version=1.0.0.0, Culture=neutral, PublicKeyToken=3786d24ab42c57d6)
at Jering.Javascript.NodeJS.OutOfProcessNodeJSService+<TryInvokeCoreAsync>d__42`1.MoveNext (Jering.Javascript.NodeJS, Version=1.0.0.0, Culture=neutral, PublicKeyToken=3786d24ab42c57d6)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Jering.Javascript.NodeJS.OutOfProcessNodeJSService+<InvokeFromStringAsync>d__31`1.MoveNext (Jering.Javascript.NodeJS, Version=1.0.0.0, Culture=neutral, PublicKeyToken=3786d24ab42c57d6)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at ImageGenMvc.Controllers.HomeController+<LoadImage>d__14.MoveNext (Web, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: /home/runner/work/test/Controllers/HomeController.cs:146)
Inner exception System.InvalidOperationException handled at Jering.Javascript.NodeJS.OutOfProcessNodeJSService.ConnectIfNotConnected:
at Jering.Javascript.NodeJS.NodeJSProcessFactory.CreateProcess (Jering.Javascript.NodeJS, Version=1.0.0.0, Culture=neutral, PublicKeyToken=3786d24ab42c57d6)
at Jering.Javascript.NodeJS.NodeJSProcessFactory.Create (Jering.Javascript.NodeJS, Version=1.0.0.0, Culture=neutral, PublicKeyToken=3786d24ab42c57d6)
at Jering.Javascript.NodeJS.OutOfProcessNodeJSService.CreateAndSetUpProcess (Jering.Javascript.NodeJS, Version=1.0.0.0, Culture=neutral, PublicKeyToken=3786d24ab42c57d6)
at Jering.Javascript.NodeJS.OutOfProcessNodeJSService.ConnectIfNotConnected (Jering.Javascript.NodeJS, Version=1.0.0.0, Culture=neutral, PublicKeyToken=3786d24ab42c57d6)
Inner exception System.ComponentModel.Win32Exception handled at Jering.Javascript.NodeJS.NodeJSProcessFactory.CreateProcess:
at System.Diagnostics.Process.ForkAndExecProcess (System.Diagnostics.Process, Version=4.2.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Diagnostics.Process.StartCore (System.Diagnostics.Process, Version=4.2.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Diagnostics.Process.Start (System.Diagnostics.Process, Version=4.2.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Diagnostics.Process.Start (System.Diagnostics.Process, Version=4.2.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at Jering.Javascript.NodeJS.NodeJSProcessFactory.CreateProcess (Jering.Javascript.NodeJS, Version=1.0.0.0, Culture=neutral, PublicKeyToken=3786d24ab42c57d6)
If I Bash at https://sitename.scm.azurewebsites.net/ I get:
/home>node -v
v14.18.2
/home>whereis node
node: /usr/local/bin/node /opt/nodejs/14.17.6/bin/node /opt/nodejs/9.4.0/bin/node
/home>
So the error says the current path is
/usr/local/bin
And Bash says node exists in
/usr/local/bin/node
So why can't it find Node?
I have a .Net 6 Web API running on Azure App Service (Windows, Standard Service Plan (S1: 1))
Occasionally, I get this exception:
System.IO.IOException: There is not enough space on the disk. :
'D:\local\Temp\ASPNETCORE_7aca67c0-7d1f-40ba-bef3-9e58ea744487.tmp'
This is the stack trace for the last time it happened:
System.IO.IOException:
at System.IO.RandomAccess.WriteAtOffset (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.IO.Strategies.OSFileStreamStrategy.Write (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.IO.Strategies.OSFileStreamStrategy.Write (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.WebUtilities.PagedByteBuffer.MoveTo (Microsoft.AspNetCore.WebUtilities, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at Microsoft.AspNetCore.WebUtilities.FileBufferingWriteStream.Write (Microsoft.AspNetCore.WebUtilities, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at Microsoft.AspNetCore.WebUtilities.HttpResponseStreamWriter.Write (Microsoft.AspNetCore.WebUtilities, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at Newtonsoft.Json.JsonTextWriter.WritePropertyName (Newtonsoft.Json, Version=13.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed)
at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject (Newtonsoft.Json, Version=13.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed)
at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeList (Newtonsoft.Json, Version=13.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed)
at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.Serialize (Newtonsoft.Json, Version=13.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed)
at Newtonsoft.Json.JsonSerializer.SerializeInternal (Newtonsoft.Json, Version=13.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed)
at Microsoft.AspNetCore.Mvc.Formatters.NewtonsoftJsonOutputFormatter+<WriteResponseBodyAsync>d__13.MoveNext (Microsoft.AspNetCore.Mvc.NewtonsoftJson, Version=6.0.5.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.Mvc.Formatters.NewtonsoftJsonOutputFormatter+<WriteResponseBodyAsync>d__13.MoveNext (Microsoft.AspNetCore.Mvc.NewtonsoftJson, Version=6.0.5.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.Mvc.Formatters.NewtonsoftJsonOutputFormatter+<WriteResponseBodyAsync>d__13.MoveNext (Microsoft.AspNetCore.Mvc.NewtonsoftJson, Version=6.0.5.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker+<<InvokeResultAsync>g__Logged|22_0>d.MoveNext (Microsoft.AspNetCore.Mvc.Core, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker+<<InvokeNextResultFilterAsync>g__Awaited|30_0>d`2.MoveNext (Microsoft.AspNetCore.Mvc.Core, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow (Microsoft.AspNetCore.Mvc.Core, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.ResultNext (Microsoft.AspNetCore.Mvc.Core, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeResultFilters (Microsoft.AspNetCore.Mvc.Core, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker+<<InvokeFilterPipelineAsync>g__Awaited|20_0>d.MoveNext (Microsoft.AspNetCore.Mvc.Core, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker+<<InvokeAsync>g__Logged|17_1>d.MoveNext (Microsoft.AspNetCore.Mvc.Core, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker+<<InvokeAsync>g__Logged|17_1>d.MoveNext (Microsoft.AspNetCore.Mvc.Core, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.Routing.EndpointMiddleware+<<Invoke>g__AwaitRequestTask|6_0>d.MoveNext (Microsoft.AspNetCore.Routing, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at PropWorx.API.Middlewares.TenantIdentifier+<Invoke>d__4.MoveNext (PropWorx.API, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: C:\Users\fabsr\source\repos\PropWorx.API\PropWorx.API\Middlewares\TenantIdentifier.cs:50)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at PropWorx.API.Middlewares.WebSocketsMiddleware+<Invoke>d__2.MoveNext (PropWorx.API, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: C:\Users\fabsr\source\repos\PropWorx.API\PropWorx.API\Middlewares\WebSocketsMiddleware.cs:30)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.Authorization.Policy.AuthorizationMiddlewareResultHandler+<HandleAsync>d__0.MoveNext (Microsoft.AspNetCore.Authorization.Policy, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware+<Invoke>d__6.MoveNext (Microsoft.AspNetCore.Authorization.Policy, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware+<Invoke>d__6.MoveNext (Microsoft.AspNetCore.Authentication, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.Session.SessionMiddleware+<Invoke>d__8.MoveNext (Microsoft.AspNetCore.Session, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.Session.SessionMiddleware+<Invoke>d__8.MoveNext (Microsoft.AspNetCore.Session, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware+<Invoke>d__5.MoveNext (Swashbuckle.AspNetCore.SwaggerUI, Version=6.3.1.0, Culture=neutral, PublicKeyToken=4232c99127b3c254)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware+<Invoke>d__4.MoveNext (Swashbuckle.AspNetCore.Swagger, Version=6.3.1.0, Culture=neutral, PublicKeyToken=62657d7474907593)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware+<Invoke>d__9.MoveNext (Microsoft.AspNetCore.Diagnostics, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
This happens on various API endpoints (i.e. various action methods in various controllers).
The strange thing is that it's happening in methods where I'm not performing any file writing activities. It's simple GET request which performs a basic SELECT * FROM table query from a MySql database, and returns the results.
Now, if I go to Azure Portal, and then to my app service, and then to Quotas, I have plenty of free space left:
If I go to Kudu, and then Environment, I see this:
System info
System up time: 12:22:00.1560000
OS version: Microsoft Windows NT 10.0.14393.0
64 bit system: True
64 bit process: True
Processor count: 1
Machine name: RD2818786D33D4
Instance id: eeb07ed981018f5b52e2782779334447fffe71fbeeea9200f63be35f6a12f44a
Short instance id: eeb07e
CLR version: 4.0.30319.42000
System directory: D:\Windows\system32
Current working directory: D:\Program Files (x86)\SiteExtensions\ApplicationInsightsAgent\2.8.43\ExtensionManager
IIS command line: D:\Windows\system32\inetsrv\w3wp.exe -ap "~1propworx-api-san" -v "v4.0" -a "\\.\pipe\iisipmf473d9e2-218c-4cdb-abbe-f3fa8fd44d9a" -h "C:\DWASFiles\Sites\#1propworx-api-san\Config\applicationhost.config" -w "C:\DWASFiles\Sites\#1propworx-api-san\Config\rootweb.config" -m 0
D:\home usage: 51,200 MB total; 47,873 MB free
D:\local usage: 11,264 MB total; 10,894 MB free
If you look at the two last lines, I have plenty of space on both D:\home and D:\local
Any ideas?
Looks like it is explained here. The solution is to enable buffering on the requests. ASP.NET CORE writes request body to Windows TEMP folder
M$ doc for request buffering. https://learn.microsoft.com/en-us/dotnet/api/microsoft.aspnetcore.http.httprequestrewindextensions.enablebuffering?view=aspnetcore-6.0
Your application process experienced a crash. This could be caused by application code.
Suggestions for Monitoring Crash
Exception Code
1 crash due to (0xE0434352 - CLR Exception)
If the detector below does not have exception details for the crash, suggest the customer to enable Crash Monitoring by following these steps.
Go to App Service Diagnostics
Choose Diagnostic Tools .
Click on Crash Monitoring from menu options and follow the instructions.
Suggestions for HTTP Server Errors
ExceptionType Method Count
System.IO.IOException PropWorx.API.Services.Auth0Service+d__4.MoveNext 5
System.Threading.Tasks.TaskCanceledException PropWorx.API.Middlewares.TenantIdentifier+d__4.MoveNext 3
System.IO.IOException PropWorx.API.Services.Auth0Service+d__4.MoveNext 2
HTTP 500.0 error typically indicates an application code issue. An un-handled exception in the application code or an error in application is what typically causes this error. Please check the code.
The issue I found, please check it in your side
NLog and MySQL, please check it in your Application Exceptions
System.AggregateException: An error occurred while writing to logger(s). (Unable to connect to any of the specified MySQL hosts.)
Slow Request Execution
You need to check your AppInsights. You can find a lot of exceptions in it.
Summary
please contact your developer Teams, let them review the code according the exceptions.
Enable Local Cache
please read this article, it also will useful to you.
The Ultimate Guide to Running Healthy Apps in the Cloud
Shortly after posting this question, I deleted all the logs in the LogFiles folder on Azure (I did this via FTP / FileZilla). There were tons of log files going back a long time. I wasn't sure if these contributed towards, what I think could, be a limit on the available temporary storage space, but it's been nearly two weeks, and I haven't had any more "There is not enough space on the disk" exceptions.
I will continue to monitor it. I'm not really sure if this was the fix, but it appears to have helped. I'm half-reluctantly closing this post though as I don't want to keep it open indefinitely.
Thanks to everyone for your input.
I know Azure Functions 3.0 is in preview but so we better catch up bugs sooner than later.
My function is triggered by Service Bus queue messages. In my case, I have 2 processes sending a message every 15 minutes. So the Azure Func is not really busy.
Most of the time there are no problems, but about 2 times a day I get these errors:
Microsoft.Azure.ServiceBus.MessageLockLostException:
at Microsoft.Azure.ServiceBus.Core.MessageReceiver+d__99.MoveNext (Microsoft.Azure.ServiceBus, Version=4.1.1.0, Culture=neutral, PublicKeyToken=7e34167dcc6d6d8c)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.Azure.ServiceBus.RetryPolicy+d__19.MoveNext (Microsoft.Azure.ServiceBus, Version=4.1.1.0, Culture=neutral, PublicKeyToken=7e34167dcc6d6d8c)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.Azure.ServiceBus.RetryPolicy+d__19.MoveNext (Microsoft.Azure.ServiceBus, Version=4.1.1.0, Culture=neutral, PublicKeyToken=7e34167dcc6d6d8c)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.Azure.ServiceBus.Core.MessageReceiver+d__68.MoveNext (Microsoft.Azure.ServiceBus, Version=4.1.1.0, Culture=neutral, PublicKeyToken=7e34167dcc6d6d8c)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.Azure.ServiceBus.MessageReceivePump+d__16.MoveNext (Microsoft.Azure.ServiceBus, Version=4.1.1.0, Culture=neutral, PublicKeyToken=7e34167dcc6d6d8c)
I changed message TTL = 30 min, Lock duration = 2 min, Max Delivery Count = 10 (default). So these settings shouldn't affect anything I believe.
Can anyone tell me why do I get these errors and how to fix this?
MessageLockLostException indicates that your Function has taken longer than the defined LockDuration in the queue to process the message. You could increase the time up to 5 minutes, but that still might occur. I would recommend to log execution time and analyze how long did it take your Function when the exception is thrown.
Side note: TimeToLive and MaxDeliveryCount are not related to this issue.
I've recently received a mass of SocketExceptions in my Azure function when I try to access my KeyVault. I must also point out, that the issue doesn't happen every time, but rather there are short (1hr long) time frames where all calls seem fail.
The error doesn't originate directly from one of my calls, but rather from a call to a metadata endpoint.
I'm using the Microsoft.Azure.KeyVault library, and the error occurs at soon after using the GetSecretAsync method.
My code:
var keyVaultClient = new KeyVaultClient(new KeyVaultClient.AuthenticationCallback((new AzureServiceTokenProvider()).KeyVaultTokenCallback));
var x = await keyVaultClient.GetSecretAsync(<secreturi>);
This is all the information I have, which I extracted from Application Insights:
Exception: System.Net.Sockets.SocketException
Failed method: System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
Message: An attempt was made to access a socket in a way forbidden by its access permissions An attempt was made to access a socket in a way forbidden by its access permissions
Path: http://169.254.169.254/metadata/identity/oauth2/token?resource=https://vault.azure.net&api-version=2018-02-01
This was working without any issues until around 2 weeks ago. No deployments have since taken place which could have impacted this.
If any additional information is required please let me know.
Entire stacktrace:
System.Net.Http.HttpRequestException:
at System.Net.Http.ConnectHelper+<ConnectAsync>d__2.MoveNext (System.Net.Http, Version=4.2.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Threading.Tasks.ValueTask`1.get_Result (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1+ConfiguredValueTaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Net.Http.HttpConnectionPool+<CreateConnectionAsync>d__44.MoveNext (System.Net.Http, Version=4.2.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Threading.Tasks.ValueTask`1.get_Result (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1+ConfiguredValueTaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Net.Http.HttpConnectionPool+<WaitForCreatedConnectionAsync>d__49.MoveNext (System.Net.Http, Version=4.2.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Threading.Tasks.ValueTask`1.get_Result (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1+ConfiguredValueTaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Net.Http.HttpConnectionPool+<SendWithRetryAsync>d__39.MoveNext (System.Net.Http, Version=4.2.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Net.Http.RedirectHandler+<SendAsync>d__4.MoveNext (System.Net.Http, Version=4.2.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Net.Http.DiagnosticsHandler+<SendAsync>d__2.MoveNext (System.Net.Http, Version=4.2.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
Inner exception System.Net.Sockets.SocketException handled at System.Net.Http.ConnectHelper+<ConnectAsync>d__2.MoveNext:
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Net.Http.ConnectHelper+<ConnectAsync>d__2.MoveNext (System.Net.Http, Version=4.2.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
Reason 1 – You really need so much connections
In case you really need so much connections opened at once, you have to scale up you App Plan, according to limits table.
Reason 2 – You have bug in you app
Check, if your application closes all the opened connections. For example in my application, source of this issue were not-released sockets by HttpClient.
For more details, refer to this article.
My company has chat application built on ServiceStack SSE (v.4.5.14). Recently, we keep getting following exceptions which sometimes can make IIS application pool shut down.
System.Web.HttpException: The remote host closed the connection. The error code is 0x800704CD.
at System.Web.Hosting.IIS7WorkerRequest.RaiseCommunicationError (System.Web, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Web.Hosting.IIS7WorkerRequest.ExplicitFlush (System.Web, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Web.HttpResponse.Flush (System.Web, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Web.HttpWriter.WriteFromStream (System.Web, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.IO.Stream+<>c.<BeginWriteInternal>b__46_0 (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Threading.Tasks.Task`1.InnerInvoke (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Threading.Tasks.Task.Execute (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.IO.Stream.EndWrite (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.IO.Stream+<>c.<BeginEndWriteAsync>b__53_1 (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Threading.Tasks.TaskFactory`1+FromAsyncTrimPromise`1.Complete (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
After a quick research, we found this article that explains the reason behind this issue. However, we are not quite sure about the solution. Should we also add this compesssion setting, mentioned in article:
<urlCompression doStaticCompression="true" doDynamicCompression="false" />
Or could anyone share how to fix this issue?
Thank you!