How to fix Azure functions "inotify limit" error - azure

I am trying to run a durable entity that I had described originally in this question. While trying to figure out that issue I have started getting the below error after a few runs.
The configured user limit (128) on the number of inotify instances has been reached, or the per-process limit on the number of open file descriptors has been reached.
Below is the full log of the error. Can someone point me to the possible cause and fix?
user:func1$ func start -p 8080
Found Python version 3.6.9 (python3).
Azure Functions Core Tools
Core Tools Version: 4.0.3971 Commit hash: d0775d487c93ebd49e9c1166d5c3c01f3c76eaaf (64-bit)
Function Runtime Version: 4.0.1.16815
Functions:
Entityfn: entityTrigger
KWhCalculator: activityTrigger
Orch: orchestrationTrigger
Starter: eventHubTrigger
For detailed output, run func with --verbose flag.
info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
Request starting HTTP/2 POST http://127.0.0.1:44825/AzureFunctionsRpcMessages.FunctionRpc/EventStream application/grpc -
info: Microsoft.AspNetCore.Routing.EndpointMiddleware[0]
Executing endpoint 'gRPC - /AzureFunctionsRpcMessages.FunctionRpc/EventStream'
[2022-01-24T10:58:42.582Z] Worker process started and initialized.
[2022-01-24T10:58:47.753Z] Host lock lease acquired by instance ID '000000000000000000000000A845906C'.
[2022-01-24T10:58:47.833Z] A host error has occurred during startup operation '9803b89f-6cf1-4fe1-a482-075e770a9fea'.
[2022-01-24T10:58:47.833Z] System.IO.FileSystem.Watcher: The configured user limit (128) on the number of inotify instances has been reached, or the per-process limit on the number of open file descriptors has been reached.
info: Microsoft.AspNetCore.Routing.EndpointMiddleware[1]
Executed endpoint 'gRPC - /AzureFunctionsRpcMessages.FunctionRpc/EventStream'
info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
Request finished HTTP/2 POST http://127.0.0.1:44825/AzureFunctionsRpcMessages.FunctionRpc/EventStream application/grpc - - 200 - application/grpc 5879.0529ms
info: Grpc.AspNetCore.Server.ServerCallHandler[14]
Error reading message.
System.IO.IOException: The request stream was aborted.
---> Microsoft.AspNetCore.Connections.ConnectionAbortedException: The HTTP/2 connection faulted.
--- End of inner exception stack trace ---
at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result)
at System.IO.Pipelines.Pipe.GetReadAsyncResult()
at System.IO.Pipelines.Pipe.DefaultPipeReader.GetResult(Int16 token)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2MessageBody.ReadAsync(CancellationToken cancellationToken)
at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
at Grpc.AspNetCore.Server.Internal.PipeExtensions.ReadStreamMessageAsync[T](PipeReader input, HttpContextServerCallContext serverCallContext, Func`2 deserializer, CancellationToken cancellationToken)
info: Microsoft.AspNetCore.Server.Kestrel[32]
Connection id "0HMEV51U1VS7R", Request id "0HMEV51U1VS7R:00000001": the application completed without reading the entire request body.
info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
Request starting HTTP/2 POST http://127.0.0.1:44825/AzureFunctionsRpcMessages.FunctionRpc/EventStream application/grpc -
info: Microsoft.AspNetCore.Routing.EndpointMiddleware[0]
Executing endpoint 'gRPC - /AzureFunctionsRpcMessages.FunctionRpc/EventStream'
[2022-01-24T10:58:49.902Z] Worker process started and initialized.
[2022-01-24T10:58:55.521Z] Host lock lease acquired by instance ID '000000000000000000000000A845906C'.
[2022-01-24T10:58:55.578Z] A host error has occurred during startup operation '3b77fea3-afd0-427c-9d79-3175d7f0b815'.
[2022-01-24T10:58:55.578Z] System.IO.FileSystem.Watcher: The configured user limit (128) on the number of inotify instances has been reached, or the per-process limit on the number of open file descriptors has been reached.
info: Microsoft.AspNetCore.Routing.EndpointMiddleware[1]
Executed endpoint 'gRPC - /AzureFunctionsRpcMessages.FunctionRpc/EventStream'
info: Grpc.AspNetCore.Server.ServerCallHandler[14]
Error reading message.
System.IO.IOException: The request stream was aborted.
---> Microsoft.AspNetCore.Connections.ConnectionAbortedException: The HTTP/2 connection faulted.
--- End of inner exception stack trace ---
at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result)
at System.IO.Pipelines.Pipe.GetReadAsyncResult()
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2MessageBody.ReadAsync(CancellationToken cancellationToken)
at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
at Grpc.AspNetCore.Server.Internal.PipeExtensions.ReadStreamMessageAsync[T](PipeReader input, HttpContextServerCallContext serverCallContext, Func`2 deserializer, CancellationToken cancellationToken)
info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
Request finished HTTP/2 POST http://127.0.0.1:44825/AzureFunctionsRpcMessages.FunctionRpc/EventStream application/grpc - - 200 - application/grpc 6153.6504ms
info: Microsoft.AspNetCore.Server.Kestrel[32]
Connection id "0HMEV51U1VS7S", Request id "0HMEV51U1VS7S:00000001": the application completed without reading the entire request body.
info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
Request starting HTTP/2 POST http://127.0.0.1:44825/AzureFunctionsRpcMessages.FunctionRpc/EventStream application/grpc -
info: Microsoft.AspNetCore.Routing.EndpointMiddleware[0]
Executing endpoint 'gRPC - /AzureFunctionsRpcMessages.FunctionRpc/EventStream'

Restarting VS Code and creating a new Azure Function project did not help.
Fix: I restarted my system and the error is gone.
Cause: Not sure, but looks like long time running locally with long JSON messages might have opened lot of listeners on the port

Related

Cloud Service (Classic) Roles and Instances reverts all updates and are not maintained by Azure

I have an issue with my roles and instances in Azure (cloud service classic). The Windows Server 2012R2 reverts the installed updates frequently. Also the Windows in not activated from the last reboot of the instances (environments).
We are using PaaS, which means the provider should maintain the OS and the health of our cloud services.
This is the error, which I receive:
Busy (Recovering role... Application startup task 0 is running.)
Other collected information:
[Information] (2:54 AM) Execute:Monitoring cycled started
[Warning] (2:54 AM) CaptureResourceStatus:Resource status is Down because Azure Instance State is not Ready or Stopped: Down and
[Warning] (2:54 AM) CaptureResourceMetrics:Instance Holden.eFox.Web_IN_0 status is BusyRole (expected ReadyRole). State Details: Recovering role... Failed to load role entrypoint. System.Reflection.ReflectionTypeLoadException: Unable to load one or more of the requested types. Retrieve the LoaderExceptions property for more information.
at System.Reflection.RuntimeModule.GetTypes(RuntimeModule module)
at System.Reflection.Assembly.GetTypes()
at Microsoft.WindowsAzure.ServiceRuntime.RoleEnvironment.GetRoleEntryPoint(Assembly entryPointAssembly)
--- End of inner exception stack trace ---
at Microsoft.WindowsAzure.ServiceRuntime.RoleEnvironment.GetRoleEntryPoint(Assembly entryPointAssembly)
at Microsoft.WindowsAzure.ServiceRuntime.RoleEnvironment.CreateRoleEntryPoint(RoleType roleTypeEnum)
at Microsoft.WindowsAzure.ServiceRuntime.RoleEnvironment.InitializeRoleInternal(RoleType roleTypeEnum)' [2022-09-01T02:53:36.000Z] Last exit time: [2022/09/01, 02:53:37.605]. Last exit code: 0.
[Warning] (2:54 AM) CaptureResourceStatus:Resource status is Down because Azure Instance State is not Ready or Stopped: Down and
[Information] (2:54 AM) CleanDuplicates:Removed 28 duplicate metrics. (Before cleaning: 44 metrics, after 16.)
[Warning] (2:54 AM) CaptureResourceMetrics:Metric SystemEventLogs: no values captured
[Warning] (2:54 AM) CaptureResourceMetrics:Metric ApplicationEventLogs: no values captured
Could you help me, how to resolve this problem or could you direct me whom to contact for some support? (I do not pay for technical support plans.)

Logstash Azure Event hub input , The client could not finish the operation within specified maximum execution timeout

Hi Im having this Warns when I try to get events from event hub.
com.microsoft.azure.storage.StorageException: The client could not finish the operation within specified maximum execution timeout.
after those warns I get this error: Out of retries creating lease store for this Event Hub
[ERROR] 2022-01-31 15: 26: 37.760 [azure_event_hubs-worker-11-fadc76f77bb4d96b880b8af84e7d4dc9af184d55efd5dfedeaaac4ad2f5a5836
] azureeventhubs - Event Hub encountered an error. {
:event_hub_name=>"<edited>",
:exception=>com.microsoft.azure.eventprocessorhost.PartitionManager$FinalException: java.util.concurrent.CompletionException: com.microsoft.azure.eventprocessorhost.ExceptionWithAction: Out of retries creating lease store for this Event Hub,
:backtrace=>[
"com.microsoft.azure.eventprocessorhost.PartitionManager.lambda$buildRetries$19(PartitionManager.java:278)",
"java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:930)",
"java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:907)",
"java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478)",
"java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)",
"java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)",
"java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)",
"java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)",
"java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)",
"java.base/java.lang.Thread.run(Thread.java:829)"
]
}
I can connect with telnet to the host and port <edited>.servicebus.windows.net:5671
I also can do telnet to ports 5672 and 443
and when I use wget:
wget https://<edited>.servicebus.windows.net
--2022-01-31 19:55:50-- https://<edited>.servicebus.windows.net/
Resolving <edited>.servicebus.windows.net (<edited>.servicebus.windows.net)... <edited IP>
Connecting to <edited>.servicebus.windows.net (<edited>.servicebus.windows.net)|<edited IP>|:443... connected.
HTTP request sent, awaiting response... 200 OK
I have other 4 pipelines working with Event Hub input pointing to diferents storage connections.
And another strange thing is that if I run the same losgtash version in a VM in my laptop, and the same pipeline, it work and the error is not shown
What may be the problem?

Need help investigating Azure App Service .NET Core 2.1 ArgumentOutOfRangeException and System.OverflowException

Today I've adjusted the App Service Log settings for my Azure App Serivce: a .NET Core 2.1 Web API. Soon after I changed the Application Logging (file system) setting from Information to Warning, my application stopped responding. I immediately kicked of a .NET Trace using the Diagnose and solve problems feature.
This Trace report showed the following:
system.private.corelib!
microsoft.aspnetcore.hosting!
System.Private.CoreLib!System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start
microsoft.aspnetcore.hosting.il!Microsoft.AspNetCore.Hosting.Views.ErrorPage.ExecuteAsync
microsoft.aspnetcore.hosting!
System.Private.CoreLib!System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start
microsoft.aspnetcore.hosting.il!Microsoft.Extensions.RazorViews.BaseView.ExecuteAsync
microsoft.aspnetcore.hosting!
microsoft.aspnetcore.hosting!
System.Private.CoreLib!System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1+<>c[System.Threading.Tasks.VoidTaskResult,Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol+<ProcessRequests>d__188`1[Microsoft.AspNetCore.Hosting.Internal.HostingApplication+Context]].<.cctor>b__9_0
system.private.corelib!
System.Private.CoreLib!System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[System.Threading.Tasks.VoidTaskResult,Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol+<ProcessRequests>d__188`1[Microsoft.AspNetCore.Hosting.Internal.HostingApplication+Context]].MoveNext
system.private.corelib!
system.private.corelib!
system.private.corelib!
system.private.corelib!
system.private.corelib!
Same time, the Azure Blob log showed these lines:
2019-07-06 14:04:06.476 +00:00 [Error] Microsoft.AspNetCore.Server.Kestrel: Connection id "0HLO233NHLK6Q", Request id "0HLO233NHLK6Q:000003BF": An unhandled exception was thrown by the application.
System.OverflowException: Arithmetic operation resulted in an overflow.
at System.IO.StreamWriter.Write(String value)
at Microsoft.Extensions.RazorViews.BaseView.Write(String value)
at Microsoft.AspNetCore.Hosting.Views.ErrorPage.ExecuteAsync()
at Microsoft.Extensions.RazorViews.BaseView.ExecuteAsync(HttpContext context)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)
2019-07-06 14:04:19.245 +00:00 [Error] Microsoft.AspNetCore.Server.Kestrel: Connection id "0HLO233NHLK6T", Request id "0HLO233NHLK6T:000000EF": An unhandled exception was thrown by the application.
System.ArgumentOutOfRangeException: Index and count must refer to a location within the buffer.
Parameter name: chars
at System.Text.EncoderNLS.GetBytes(Char[] chars, Int32 charIndex, Int32 charCount, Byte[] bytes, Int32 byteIndex, Boolean flush)
at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
at System.IO.StreamWriter.Write(String value)
at Microsoft.AspNetCore.Hosting.Views.ErrorPage.ExecuteAsync()
at Microsoft.Extensions.RazorViews.BaseView.ExecuteAsync(HttpContext context)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)
These stacktraces don't show any of my own code, they seem framework related.
Is there a message or a hint in these error stack traces that I can use to prevent these situations?
Many thanks in advance
I know it's too late but maybe my answer will help someone else.
I received the same exceptions as you. But it was not Azure App Service but IIS. The problem was in incorrect configuration somewhere in Startup.cs. You webhost tried to show error page but at the same time App Service (or IIS in my case) tried to shut down or restart it.
This thread helped me a lot https://github.com/aspnet/KestrelHttpServer/issues/1737

.netcore context.Request.EnableRewind() throwing System.IO.Exception once buffer exceeds

We have a production deployed application using netcoreapp2.0
We use context.Request.EnableRewind(); to read and log the request before processing.
Since few days intermittently we are getting below unhandled exception in kestrel hosted in AWS Docker ECS.
Basically our request size is more than 30 Kb that's why netcore trying to use filesystem buffer to rewind the request. And it is throwing File is readonly exception.
Is increasing the default buffer limit is the only way to fix the issue.
context.Request.EnableRewind([int bufferthreashold 30720],long? bufferLimit = null]);
Or there is any other way to configure request rewind ?
Option to upgrade to 2.1 is not possible for us right now. I see it is fixed in 2.1
2019-06-24 14:09:49[41m[30mfail[39m[22m[49m: Microsoft.AspNetCore.Server.Kestrel[13]
[41m[30mfail[39m[22m[49m: Microsoft.AspNetCore.Server.Kestrel[13]
2019-06-24 14:09:49 Connection id "0HLKCN30834Q0",
Request id "0HLKCN30834Q0:0003B777": An unhandled exception was thrown by the application.
Connection id "0HLKCN30834Q0", Request id "0HLKCN30834Q0:0003B777": An unhandled exception was thrown by the application.
2019-06-24 14:09:49System.IO.IOException: Read-only file system
System.IO.IOException: Read-only file system
Fixed it by increasing the default buffer size.

How do I prevent this IIS error that required a manual restart of the app pool from occurring again?

My website hosted in IIS became inaccessible. In the event log on the server, I saw this error:
Event code: 3005
Event message: An unhandled exception has occurred.
Event time: 01/01/2018 00:00:00 AM
Event time (UTC): 01/02/2018 6:00:00 AM
Event ID: 64b4d6533abd4e71b3ba30b9008d7d86
Event sequence: 1921230
Event occurrence: 1
Event detail code: 0
Application information:
Application domain: /LM/W3SVC/2/ROOT-1-131865989552967444
Trust level: Full
Application Virtual Path: /
Application Path: C:\inetpub\wwwroot\MyWebsite\
Machine name: MyWebServer
Process information:
Process ID: 3388
Process name: w3wp.exe
Account name: NT AUTHORITY\SYSTEM
Exception information:
Exception type: HttpException
Exception message: The remote host closed the connection. The error code is 0x80070057.
at System.Web.Hosting.IIS7WorkerRequest.RaiseCommunicationError(Int32 result, Boolean throwOnDisconnect)
at System.Web.Hosting.IIS7WorkerRequest.ExplicitFlush()
at System.Web.WebSocketPipeline.DoFlush()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Web.WebSocketPipeline.<ProcessRequestImplAsync>d__9.MoveNext()
Request information:
Request URL:
Request path:
User host address:
User:
Is authenticated: False
Authentication Type:
Thread account name: NT AUTHORITY\SYSTEM
Thread information:
Thread ID: 112
Thread account name: NT AUTHORITY\SYSTEM
Is impersonating: False
Stack trace: at System.Web.Hosting.IIS7WorkerRequest.RaiseCommunicationError(Int32 result, Boolean throwOnDisconnect)
at System.Web.Hosting.IIS7WorkerRequest.ExplicitFlush()
at System.Web.WebSocketPipeline.DoFlush()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Web.WebSocketPipeline.<ProcessRequestImplAsync>d__9.MoveNext()
Custom event details:
Restarting only the website caused the error to be logged again, but with a different code (0x800704CD instead of 0x80070057). The website was still inaccessible. Restarting the app pool resolved the issue.
What is this error, and how can I stop it from happening again?
Googling both error codes yields results from people who were encountering the error intermittently (post1, post2). The difference is that their websites don't appear to be brought down by this error, and they are seeing the error intermittently. I saw it twice, and had to manually intervene just to get the website running again.

Resources