I am having a function app which is running on consumption plan (Y1). as you can see below screen shots the execution count was suddenly drop to 0 for some time (~30min). in that time the app getting the requests as well. but giving 4xx errors.
why is this happening?
Thats an interesting little period you have in your graph there. Is it all http 4xx if you drill into app insight logs can see if its 400's or 404s by any chance. I have a feeling its because the client calling the function app was passing bad data etc resulting in a 400 Bad Request during that period.
If its something with the core function app you will get a 502 or a 500, was there a deployment or code change during that period that could have triggered client side bad requests.
Related
We have a service which pings our EP1 Premium service and yesterday we received 3 client side timeout errors after 2 minutes of waiting. When opening the trace in App insights, these requests which time out are not even logged and have no trace of ever being received Azure side, and therefore stay unanswered. By looking at the metrics provided in the Azure Functions app, I found out that 1-2 minutes after the request has been sent, the app loses all its ability to work as its Total App Domains falls to 0 as well as all connections, threads and so on and this state lasts until the next request is received, therefore "skipping" the request that happened beforehand. This is a big issue as I need to make sure requests get answered in a timely manner.
The client service sent HTTP requests to the Azure Functions app expecting an answer, only to time out while the Azure-side doesn't have any record of ever receiving the request.
I believe this issues is related to Consumption Plan of Azure Functions called Cold Start behaviour. The "skipping" mechanism is explained below:
Apps may scale to zero when idle, meaning some requests may have additional latency at startup. The consumption plan does have some optimizations to help decrease cold start time, including pulling from pre-warmed placeholder functions that already have the function host and language processes running.https://learn.microsoft.com/en-us/azure/azure-functions/functions-scale#cold-start-behavior
Please also consider of having look on this article, which explains the behaviour. https://azure.microsoft.com/en-us/blog/understanding-serverless-cold-start/
The errors are as shown in the image... I followed the guide and downloaded the newest tools, but the error still persists... I'm on school wifi right now, and since I'm new to this, I have no idea on how to change Azure's environment variables on an existing project... I cannot sign in in the first place. I was just following through the guide :(
Please check the below steps if they helps to work around:
Error in Azure portal is HTTP response code 503 Service Unavailable
This situation happens due to network connectivity or service available issues.
The better approach is to retry the operation and if the issue persists, contact Azure Support as referenced here.
Alternative ways to solve this error is finding the cause by navigating to Diagnose and solve problems to know the root cause of 503 error as there can be multiple reasons for this error.
Please check the below causes related to 503 error issue if they helps to work around:
request taking a long time
application crashing due to an exception.
average response time is long
Function App is also an app service so app service enforces limits on the number of outbound connections
Error in Browser / Postman is 502 - Web server received an invalid response while acting as a gateway or proxy server.
As this 502 error, you were addressed in MSFT Q&A.
Normally, 502 error occurs when HTTP is placed instead of HTTPS in the connection but I know that the Azure Functions endpoint look like in this format
http://<APP_NAME>.azurewebsites.net/api/<FUNCTION_NAME>
And 502 error occurs when maximum timeout exceeds the value. Please check the timeout value of the function app and the logs, metrics of requestTime, responseTime in Application Insights and if it is the cause, increase the timeout value.
References:
Troubleshooting Reason for a 502 Error
How do I fix this 502 Error on my Azure Function?
I have an app service (plan B2) running on Azure.
My integration tests running from docker container are calling some app service endpoints one by one and sometimes receive 500 or 502 error.
When I debug tests I make some pauses between calls and all requests work successfully. Also, when I scale up my app service, everything works properly.(I don't want to scale up because cpu and other params are low.)
In my tests I have only one HttpClient and I dispose it at the end so I don't think there should be any connections leaks.
Also, in TCP Connections I have around 60 total connections while in Azure docs the limit is 1,920.
This app is not accessed by any users but here it says that I had the maximum connections. Is there any way how can I track these connections? Why when I receive these 5xx errors I don't see anything in app insights? Also how 15 connections can exceed the limit when the limit is 1920? Are these connections related to my errors and how they can be fixed?
You don't see them in Application Insights because they're happening at IIS level which is breaking the request, and because of that, data is not being sent to Application Insights.
The place to look for information is "Diagnose and solve problems", then "Availability and Performance". More info in here:
https://learn.microsoft.com/en-us/azure/app-service/overview-diagnostics
PS: I do think the problem is related to the Dispose of your HTTPClient. It's a well known issue and the reason why they've introduced HttpClientFactory. More info in here:
https://www.stevejgordon.co.uk/httpclient-creation-and-disposal-internals-should-i-dispose-of-httpclient
https://stackoverflow.com/a/15708633/1384539
Every day at about 3:00PM-4:00PM GMT the response times start to increase (no memory increase or CPU increase)
There is a azure availability test going to server every 10 minutes.
As this is a dev site there is no traffic to it other than me (at the odd time) and the availability test
I log to a variable internally the startup time and this shows that the site is not restarting
The first request via a browser when this starts happening is very slow (2 minutes - probably some timeout).
After that it runs perfectly. That seems like the site is shutting down and then starting up on first request, but the pings are keeping it alive so the site is not shutting down (as far as I know)
On the odd log entry I get - I seem to be getting 502 errors - but I can't confirm this as the FEEB logs are usually off at this time.
FREB logs turn off automatically after 1 hour and as this is the middle of the night for me (NZDT) - I don't get a chance to turn on.
See attached images - as you can see the response times just increase at same time
Ignore the requests where they are above 20 - thats me going to it via browser
I always check the azure dashboard BEFORE viewing site in browser
Just got this error (from web browser randomly - keep accessing the same page:
502: The specified CGI application encountered an error and the server terminated the process.
Other relevant Info (Perhaps):
I initially had the availability test ping going to a ping endpoint /ping that only returned a 200 and empty string when I noticed this happening
It now points to the sites homepage to see if it changed anything - still the same.
Assuming the database is not the issue as the /ping endpoint doesn't touch the database - just a straight controller return.
Internal Exception handling is catching nothing
Service: Azure Free Web App (Development)
There are no web jobs or timed events on this site
Azure Dashboard Initial
Current tests:
Uploading as new site to a Basic 1 Small
Restarting dev site 12 hours before issues (usually 20 hours before)
Results:
Restarting free web-app 12ish hours before issue - same result at same time - so its not the app slowly overloading or it would me much later
Basic 1 Small: no problems - could it be something with the dev server ?
Azure Dashboard From Today
Observations:
Same behavior with /ping endpoint (just return empty string 200 Ok) and Main home page endpoint (database lookups [w/caching] / razer)
If anyone has any ideas what might be going on - I would very much appreciate it
:-)
Update:
It seems to of stopped (on its own) about 11/1/2016 1:50:49 AM GMT - my internal timestamp says it restarted - and then the errors started again same time as usual. Note: no-one is using the app. The basic 1 Small Server is still going fine.
Sorry I can't add anymore images (not enough rep)
By default, web apps are unloaded if they are idle for some period of time, which could cause the web site slow response during this period of time. Besides, this article is about troubleshooting HTTP "502 Bad Gateway" error or a HTTP "503 Service Unavailable" error in Azure web apps, you could read it. And from the article we could know scaling the web app could mitigate the issue.
This week we have started getting 502 errors on our Web App, these are random; some times they happen when there is consistent load other times they happen with even a single request.
I have checked event viewer and there is no application crash, also don't have any really slow requests in IIS logs. I had auto-heal enabled, which is now disabled. I have also enabled auto scale and even with 4 instances running I get 502 error every once in a while. There is no log entry for this 502 in IIS logs, so I am guessing something upstream is returning this response, I just don't know why its doing that and why its so random.