Related
I have a .Net 6 Web API running on Azure App Service (Windows, Standard Service Plan (S1: 1))
Occasionally, I get this exception:
System.IO.IOException: There is not enough space on the disk. :
'D:\local\Temp\ASPNETCORE_7aca67c0-7d1f-40ba-bef3-9e58ea744487.tmp'
This is the stack trace for the last time it happened:
System.IO.IOException:
at System.IO.RandomAccess.WriteAtOffset (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.IO.Strategies.OSFileStreamStrategy.Write (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.IO.Strategies.OSFileStreamStrategy.Write (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.WebUtilities.PagedByteBuffer.MoveTo (Microsoft.AspNetCore.WebUtilities, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at Microsoft.AspNetCore.WebUtilities.FileBufferingWriteStream.Write (Microsoft.AspNetCore.WebUtilities, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at Microsoft.AspNetCore.WebUtilities.HttpResponseStreamWriter.Write (Microsoft.AspNetCore.WebUtilities, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at Newtonsoft.Json.JsonTextWriter.WritePropertyName (Newtonsoft.Json, Version=13.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed)
at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject (Newtonsoft.Json, Version=13.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed)
at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeList (Newtonsoft.Json, Version=13.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed)
at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.Serialize (Newtonsoft.Json, Version=13.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed)
at Newtonsoft.Json.JsonSerializer.SerializeInternal (Newtonsoft.Json, Version=13.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed)
at Microsoft.AspNetCore.Mvc.Formatters.NewtonsoftJsonOutputFormatter+<WriteResponseBodyAsync>d__13.MoveNext (Microsoft.AspNetCore.Mvc.NewtonsoftJson, Version=6.0.5.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.Mvc.Formatters.NewtonsoftJsonOutputFormatter+<WriteResponseBodyAsync>d__13.MoveNext (Microsoft.AspNetCore.Mvc.NewtonsoftJson, Version=6.0.5.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.Mvc.Formatters.NewtonsoftJsonOutputFormatter+<WriteResponseBodyAsync>d__13.MoveNext (Microsoft.AspNetCore.Mvc.NewtonsoftJson, Version=6.0.5.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker+<<InvokeResultAsync>g__Logged|22_0>d.MoveNext (Microsoft.AspNetCore.Mvc.Core, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker+<<InvokeNextResultFilterAsync>g__Awaited|30_0>d`2.MoveNext (Microsoft.AspNetCore.Mvc.Core, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow (Microsoft.AspNetCore.Mvc.Core, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.ResultNext (Microsoft.AspNetCore.Mvc.Core, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeResultFilters (Microsoft.AspNetCore.Mvc.Core, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker+<<InvokeFilterPipelineAsync>g__Awaited|20_0>d.MoveNext (Microsoft.AspNetCore.Mvc.Core, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker+<<InvokeAsync>g__Logged|17_1>d.MoveNext (Microsoft.AspNetCore.Mvc.Core, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker+<<InvokeAsync>g__Logged|17_1>d.MoveNext (Microsoft.AspNetCore.Mvc.Core, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.Routing.EndpointMiddleware+<<Invoke>g__AwaitRequestTask|6_0>d.MoveNext (Microsoft.AspNetCore.Routing, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at PropWorx.API.Middlewares.TenantIdentifier+<Invoke>d__4.MoveNext (PropWorx.API, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: C:\Users\fabsr\source\repos\PropWorx.API\PropWorx.API\Middlewares\TenantIdentifier.cs:50)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at PropWorx.API.Middlewares.WebSocketsMiddleware+<Invoke>d__2.MoveNext (PropWorx.API, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: C:\Users\fabsr\source\repos\PropWorx.API\PropWorx.API\Middlewares\WebSocketsMiddleware.cs:30)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.Authorization.Policy.AuthorizationMiddlewareResultHandler+<HandleAsync>d__0.MoveNext (Microsoft.AspNetCore.Authorization.Policy, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware+<Invoke>d__6.MoveNext (Microsoft.AspNetCore.Authorization.Policy, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware+<Invoke>d__6.MoveNext (Microsoft.AspNetCore.Authentication, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.Session.SessionMiddleware+<Invoke>d__8.MoveNext (Microsoft.AspNetCore.Session, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.Session.SessionMiddleware+<Invoke>d__8.MoveNext (Microsoft.AspNetCore.Session, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware+<Invoke>d__5.MoveNext (Swashbuckle.AspNetCore.SwaggerUI, Version=6.3.1.0, Culture=neutral, PublicKeyToken=4232c99127b3c254)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware+<Invoke>d__4.MoveNext (Swashbuckle.AspNetCore.Swagger, Version=6.3.1.0, Culture=neutral, PublicKeyToken=62657d7474907593)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware+<Invoke>d__9.MoveNext (Microsoft.AspNetCore.Diagnostics, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
This happens on various API endpoints (i.e. various action methods in various controllers).
The strange thing is that it's happening in methods where I'm not performing any file writing activities. It's simple GET request which performs a basic SELECT * FROM table query from a MySql database, and returns the results.
Now, if I go to Azure Portal, and then to my app service, and then to Quotas, I have plenty of free space left:
If I go to Kudu, and then Environment, I see this:
System info
System up time: 12:22:00.1560000
OS version: Microsoft Windows NT 10.0.14393.0
64 bit system: True
64 bit process: True
Processor count: 1
Machine name: RD2818786D33D4
Instance id: eeb07ed981018f5b52e2782779334447fffe71fbeeea9200f63be35f6a12f44a
Short instance id: eeb07e
CLR version: 4.0.30319.42000
System directory: D:\Windows\system32
Current working directory: D:\Program Files (x86)\SiteExtensions\ApplicationInsightsAgent\2.8.43\ExtensionManager
IIS command line: D:\Windows\system32\inetsrv\w3wp.exe -ap "~1propworx-api-san" -v "v4.0" -a "\\.\pipe\iisipmf473d9e2-218c-4cdb-abbe-f3fa8fd44d9a" -h "C:\DWASFiles\Sites\#1propworx-api-san\Config\applicationhost.config" -w "C:\DWASFiles\Sites\#1propworx-api-san\Config\rootweb.config" -m 0
D:\home usage: 51,200 MB total; 47,873 MB free
D:\local usage: 11,264 MB total; 10,894 MB free
If you look at the two last lines, I have plenty of space on both D:\home and D:\local
Any ideas?
Looks like it is explained here. The solution is to enable buffering on the requests. ASP.NET CORE writes request body to Windows TEMP folder
M$ doc for request buffering. https://learn.microsoft.com/en-us/dotnet/api/microsoft.aspnetcore.http.httprequestrewindextensions.enablebuffering?view=aspnetcore-6.0
Your application process experienced a crash. This could be caused by application code.
Suggestions for Monitoring Crash
Exception Code
1 crash due to (0xE0434352 - CLR Exception)
If the detector below does not have exception details for the crash, suggest the customer to enable Crash Monitoring by following these steps.
Go to App Service Diagnostics
Choose Diagnostic Tools .
Click on Crash Monitoring from menu options and follow the instructions.
Suggestions for HTTP Server Errors
ExceptionType Method Count
System.IO.IOException PropWorx.API.Services.Auth0Service+d__4.MoveNext 5
System.Threading.Tasks.TaskCanceledException PropWorx.API.Middlewares.TenantIdentifier+d__4.MoveNext 3
System.IO.IOException PropWorx.API.Services.Auth0Service+d__4.MoveNext 2
HTTP 500.0 error typically indicates an application code issue. An un-handled exception in the application code or an error in application is what typically causes this error. Please check the code.
The issue I found, please check it in your side
NLog and MySQL, please check it in your Application Exceptions
System.AggregateException: An error occurred while writing to logger(s). (Unable to connect to any of the specified MySQL hosts.)
Slow Request Execution
You need to check your AppInsights. You can find a lot of exceptions in it.
Summary
please contact your developer Teams, let them review the code according the exceptions.
Enable Local Cache
please read this article, it also will useful to you.
The Ultimate Guide to Running Healthy Apps in the Cloud
Shortly after posting this question, I deleted all the logs in the LogFiles folder on Azure (I did this via FTP / FileZilla). There were tons of log files going back a long time. I wasn't sure if these contributed towards, what I think could, be a limit on the available temporary storage space, but it's been nearly two weeks, and I haven't had any more "There is not enough space on the disk" exceptions.
I will continue to monitor it. I'm not really sure if this was the fix, but it appears to have helped. I'm half-reluctantly closing this post though as I don't want to keep it open indefinitely.
Thanks to everyone for your input.
I am starting to see a lot of SocketException's in my Application Insights log, but from the actual machine/azure infrastruture that hosts and run my services. I am certain that it's the services that are somehow causing this but I no clue to how I can debug this any further.
The stacktrace is not really giving me a lot to work with considering it's coming from code I have no power over, since it's being thrown on the actual machine my services is hosted on:
System.Net.Http.HttpRequestException:
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Net.Http.HttpClient+<FinishSendAsyncBuffered>d__58.MoveNext (System.Net.Http, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at Microsoft.ServiceProfiler.Agent.FrontendClient.StampFrontendClient+<>c__DisplayClass9_0`1+<<HttpGetAsync>b__0>d.MoveNext (Microsoft.ServiceProfiler.Agent.FrontendClient, Version=2.6.1909.2701, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at Microsoft.ServiceProfiler.Agent.FrontendClient.StampFrontendClient+<HttpRequestAsync>d__12`1.MoveNext (Microsoft.ServiceProfiler.Agent.FrontendClient, Version=2.6.1909.2701, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at Microsoft.ServiceProfiler.Agent.FrontendClient.StampFrontendClient+<HttpGetAsync>d__9`1.MoveNext (Microsoft.ServiceProfiler.Agent.FrontendClient, Version=2.6.1909.2701, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at Microsoft.ServiceProfiler.Agent.FrontendClient.ProfilerFrontendClient+<GetEtlUploadAccessAsync>d__3.MoveNext (Microsoft.ServiceProfiler.Agent.FrontendClient, Version=2.6.1909.2701, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at Microsoft.ServiceProfiler.Uploaders.StampEtlUploader+<UploadEtlZipFileAsync>d__1.MoveNext (ApplicationInsightsProfiler, Version=2.6.1909.2701, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at Microsoft.ServiceProfiler.Collectors.DetailedTraceCollector+<UploadTraceAsync>d__41.MoveNext (ApplicationInsightsProfiler, Version=2.6.1909.2701, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at Microsoft.ServiceProfiler.Collectors.DetailedTraceCollector+<StopCollectingAndUploadTraceAsync>d__38.MoveNext (ApplicationInsightsProfiler, Version=2.6.1909.2701, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at Microsoft.ServiceProfiler.Engine+<>c__DisplayClass1_1+<<RunAsync>b__2>d.MoveNext (ApplicationInsightsProfiler, Version=2.6.1909.2701, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at Microsoft.ServiceProfiler.Agent.Orchestration.Orchestrator+<TraceAsyncOperation>d__18.MoveNext (Microsoft.ServiceProfiler.Agent.Orchestration, Version=2.6.1909.2701, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
Inner exception System.Net.WebException handled at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw:
at System.Net.HttpWebRequest.EndGetResponse (System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Net.Http.HttpClientHandler.GetResponseCallback (System.Net.Http, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
Inner exception System.Net.Sockets.SocketException handled at System.Net.HttpWebRequest.EndGetResponse:
at System.Net.Sockets.Socket.BeginConnectEx (System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Net.Sockets.Socket.UnsafeBeginConnect (System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Net.ServicePoint.ConnectSocketInternal (System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
If I look at the portal, all I see is a clue that it might come from the ApplicationInsightsProfiler, which is again nothing I have any touch with in my codebase, directly.
I am afraid I am exhausting the http connection pool somehow, but how can I further investigate it ?
To check your http connections and SNAT ports please from your app open "Diagnose and solve problems" then "Availability and Performance". From there you have 2 tabs to visit: "SNAT Port Exhaustion" and "TCP Connections".
There you can see your http connections and their targets.
I know Azure Functions 3.0 is in preview but so we better catch up bugs sooner than later.
My function is triggered by Service Bus queue messages. In my case, I have 2 processes sending a message every 15 minutes. So the Azure Func is not really busy.
Most of the time there are no problems, but about 2 times a day I get these errors:
Microsoft.Azure.ServiceBus.MessageLockLostException:
at Microsoft.Azure.ServiceBus.Core.MessageReceiver+d__99.MoveNext (Microsoft.Azure.ServiceBus, Version=4.1.1.0, Culture=neutral, PublicKeyToken=7e34167dcc6d6d8c)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.Azure.ServiceBus.RetryPolicy+d__19.MoveNext (Microsoft.Azure.ServiceBus, Version=4.1.1.0, Culture=neutral, PublicKeyToken=7e34167dcc6d6d8c)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.Azure.ServiceBus.RetryPolicy+d__19.MoveNext (Microsoft.Azure.ServiceBus, Version=4.1.1.0, Culture=neutral, PublicKeyToken=7e34167dcc6d6d8c)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.Azure.ServiceBus.Core.MessageReceiver+d__68.MoveNext (Microsoft.Azure.ServiceBus, Version=4.1.1.0, Culture=neutral, PublicKeyToken=7e34167dcc6d6d8c)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.Azure.ServiceBus.MessageReceivePump+d__16.MoveNext (Microsoft.Azure.ServiceBus, Version=4.1.1.0, Culture=neutral, PublicKeyToken=7e34167dcc6d6d8c)
I changed message TTL = 30 min, Lock duration = 2 min, Max Delivery Count = 10 (default). So these settings shouldn't affect anything I believe.
Can anyone tell me why do I get these errors and how to fix this?
MessageLockLostException indicates that your Function has taken longer than the defined LockDuration in the queue to process the message. You could increase the time up to 5 minutes, but that still might occur. I would recommend to log execution time and analyze how long did it take your Function when the exception is thrown.
Side note: TimeToLive and MaxDeliveryCount are not related to this issue.
I've got a function which I've been playing with for a couple of months (even having some issue as can be seen here) and I've started running into issues when I start to scale up the usage from tests to "production".
The function takes in a 2 values, looks for a matching value in an Azure Table, removes it if it finds it, then adds the new values in together. This works fine in testing. As soon as I scale up, from a few calls every second, to 20-30 calls a second, it fails with the response mentioned above.
The actual issue, when I dive in using Insights, is that a System.InvalidOperationException exception is thrown. Here's the call stack:
System.InvalidOperationException:
at Microsoft.Azure.WebJobs.Script.WebHost.SecretManager+<PersistSecretsAsync>d__27`1.MoveNext (Microsoft.Azure.WebJobs.Script.WebHost, Version=1.0.0.0, Culture=neutral, PublicKeyToken=nullMicrosoft.Azure.WebJobs.Script.WebHost, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: C:\projects\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\Security\SecretManager.csMicrosoft.Azure.WebJobs.Script.WebHost, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: 440)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at Microsoft.Azure.WebJobs.Script.WebHost.SecretManager+<GetHostSecretsAsync>d__12.MoveNext (Microsoft.Azure.WebJobs.Script.WebHost, Version=1.0.0.0, Culture=neutral, PublicKeyToken=nullMicrosoft.Azure.WebJobs.Script.WebHost, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: C:\projects\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\Security\SecretManager.csMicrosoft.Azure.WebJobs.Script.WebHost, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: 104)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at Microsoft.Azure.WebJobs.Script.WebHost.WebJobsSdkExtensionHookProvider+<GetOrCreateExtensionKey>d__6.MoveNext (Microsoft.Azure.WebJobs.Script.WebHost, Version=1.0.0.0, Culture=neutral, PublicKeyToken=nullMicrosoft.Azure.WebJobs.Script.WebHost, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: C:\projects\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\WebHooks\WebJobsSdkExtensionHookProvider.csMicrosoft.Azure.WebJobs.Script.WebHost, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: 71)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at Microsoft.Azure.WebJobs.Script.WebHost.WebJobsSdkExtensionHookProvider.GetExtensionWebHookRoute (Microsoft.Azure.WebJobs.Script.WebHost, Version=1.0.0.0, Culture=neutral, PublicKeyToken=nullMicrosoft.Azure.WebJobs.Script.WebHost, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: C:\projects\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\WebHooks\WebJobsSdkExtensionHookProvider.csMicrosoft.Azure.WebJobs.Script.WebHost, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: 64)
at Microsoft.Azure.WebJobs.Script.WebHost.WebJobsSdkExtensionHookProvider.GetUrl (Microsoft.Azure.WebJobs.Script.WebHost, Version=1.0.0.0, Culture=neutral, PublicKeyToken=nullMicrosoft.Azure.WebJobs.Script.WebHost, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: C:\projects\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\WebHooks\WebJobsSdkExtensionHookProvider.csMicrosoft.Azure.WebJobs.Script.WebHost, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: 49)
at Microsoft.Azure.WebJobs.Host.Config.ExtensionConfigContext.GetWebhookHandler (Microsoft.Azure.WebJobs.Host, Version=2.3.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
at Microsoft.Azure.WebJobs.Extensions.EventGrid.EventGridExtensionConfig.Initialize (Microsoft.Azure.WebJobs.Extensions.EventGrid, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null)
at Microsoft.Azure.WebJobs.Host.Executors.JobHostConfigurationExtensions.InvokeExtensionConfigProviders (Microsoft.Azure.WebJobs.Host, Version=2.3.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
at Microsoft.Azure.WebJobs.Host.Executors.JobHostConfigurationExtensions.CreateStaticServices (Microsoft.Azure.WebJobs.Host, Version=2.3.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
at Microsoft.Azure.WebJobs.JobHost.InitializeServices (Microsoft.Azure.WebJobs.Host, Version=2.3.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
at Microsoft.Azure.WebJobs.Script.Utility.CreateMetadataProvider (Microsoft.Azure.WebJobs.Script, Version=1.0.0.0, Culture=neutral, PublicKeyToken=nullMicrosoft.Azure.WebJobs.Script, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: C:\projects\azure-webjobs-sdk-script\src\WebJobs.Script\Utility.csMicrosoft.Azure.WebJobs.Script, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: 362)
at Microsoft.Azure.WebJobs.Script.ScriptHost.LoadBindingExtensions (Microsoft.Azure.WebJobs.Script, Version=1.0.0.0, Culture=neutral, PublicKeyToken=nullMicrosoft.Azure.WebJobs.Script, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: C:\projects\azure-webjobs-sdk-script\src\WebJobs.Script\Host\ScriptHost.csMicrosoft.Azure.WebJobs.Script, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: 966)
at Microsoft.Azure.WebJobs.Script.ScriptHost.Initialize (Microsoft.Azure.WebJobs.Script, Version=1.0.0.0, Culture=neutral, PublicKeyToken=nullMicrosoft.Azure.WebJobs.Script, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: C:\projects\azure-webjobs-sdk-script\src\WebJobs.Script\Host\ScriptHost.csMicrosoft.Azure.WebJobs.Script, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: 299)
at Microsoft.Azure.WebJobs.Script.ScriptHostManager.RunAndBlock (Microsoft.Azure.WebJobs.Script, Version=1.0.0.0, Culture=neutral, PublicKeyToken=nullMicrosoft.Azure.WebJobs.Script, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: C:\projects\azure-webjobs-sdk-script\src\WebJobs.Script\Host\ScriptHostManager.csMicrosoft.Azure.WebJobs.Script, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: 178)
The message with it is:
Repository has more than 10 non-decryptable secrets backups (host).
I unfortunately have no idea what that means. Any searches for this result in just a few threads which talk about regenerating the keys, but again, I don't really know what that means. Some threads mention moving back to V1 of the functions, but I'm already on V1, so that's no an option.
What is going on with this function and how do I fix it?
For any Azure employees looking at this, my function ID is:
2019-01-18T15:52:18.658 [Info] Function started (Id=fc6850e8-7554-46d8-81ec-4d1697c7b572)
Deleting D:\home\data\Functions\secrets solved the issue.
In general I learned that whatever strange behavior you get from Azure Function, Kudu is always your best tool for investigation.
Little addition to what the solutions are mentioned above. I was facing this same issue and even after deleting secret files from D:\home\data\Functions\secrets folder on KUDU site, I was still getting this same error.
I was able to fix this by deleting host.*.snapshot.*.json files from azure-webjobs-secrets folder, which was there in my blob storage on azure. Basically there were 10 such snapshot files.
The message indicated that this is something to do with the host-level keys (secrets) in your Function.
So even though I don't have a clear fix to this (as I have never experienced this issue), I would suggest that you check the host.json in D:\home\data\Functions\secrets folder and see if anything unusual there, e.g. there are more than 10 keys - as the error message indicated.
If you have a custom Startup, there might be an error that blocks the host to start.
In the portal, go to the function's page (the one where you can see the content of function.json), there will be (likely) an error message. From this, you can get an idea where to look for the error. In my case, an exception was thrown when I tried to connect to the Azure Key Vault from the startup class, but List permission was not given to the function.
The reason for this exception is any one or both of below:
If you delete the app and recreate it with same name, it will use the same file share of the app and as the function app needs the keys, it will create a backup inside this folder as a snapshot.
If you have multiple function apps that uses the same storage account, this can cause this.
The maximum limit of the snapshots/backups that the platform will take is 10. Post that it will start throwing this error.
To solve this issue, you need to take the backup of this folder and delete “.snapshot..json” files or secret folder itself to regenerate secret files.
You can also delete the backup files from the storage account and navigate to the container and delete the existing snapshots.json file.
If runtime can’t decrypt secrets then they will be regenerated and non-decryptable secrets will be stored in “.snapshot..json” files. This Exception start firing if number of snapshots > 10.
You may have to kill the process and restart the Azure function after deleting the secret backup.
My company has chat application built on ServiceStack SSE (v.4.5.14). Recently, we keep getting following exceptions which sometimes can make IIS application pool shut down.
System.Web.HttpException: The remote host closed the connection. The error code is 0x800704CD.
at System.Web.Hosting.IIS7WorkerRequest.RaiseCommunicationError (System.Web, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Web.Hosting.IIS7WorkerRequest.ExplicitFlush (System.Web, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Web.HttpResponse.Flush (System.Web, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Web.HttpWriter.WriteFromStream (System.Web, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.IO.Stream+<>c.<BeginWriteInternal>b__46_0 (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Threading.Tasks.Task`1.InnerInvoke (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Threading.Tasks.Task.Execute (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.IO.Stream.EndWrite (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.IO.Stream+<>c.<BeginEndWriteAsync>b__53_1 (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Threading.Tasks.TaskFactory`1+FromAsyncTrimPromise`1.Complete (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
After a quick research, we found this article that explains the reason behind this issue. However, we are not quite sure about the solution. Should we also add this compesssion setting, mentioned in article:
<urlCompression doStaticCompression="true" doDynamicCompression="false" />
Or could anyone share how to fix this issue?
Thank you!