We have several APIs running as Azure App Services and have been experiencing intermittent service outages where 10-25% of the calls to those App Services return a 503.64 while the other calls (often to the same endpoint) return normal responses. None of the 503 failing requests appear in App Insights or in the App Service's web server logs, and we can only see them through tracing and logging from the calling services (App Gateway logging or other App Services making internal REST calls).
We haven't been able to determine a pattern to when this problem occurs. Between disruptions, we have seen as little as an hour and as much as 2-3 days. The issue has lasted anywhere from 2-30 minutes at a time. There is no correlation with traffic or system load, and it happens at any time of day or night. We have seen this on both I1 and I2 tier plans. We have tried scaling out our services well beyond necessary capacity, but the problem still occurred.
Looking at the 64 sub-status code, it suggests a rewrite issue.
<!-- Antares Rewrite provider codes-->
<error id ="64" description="Exception in rewrite provider (probably SQL)" />
However, I'm confused by the fact that some of the rewritten calls succeed during the same period that others fail.
For completeness, here is the rewrite for one of the failing services. A public API endpoint is being rewritten to an internal request address, and all other APIs are to remain as-is and pass through to the Controller:
<rewrite>
<rules>
<rule name="LegacyGetStatus">
<match url="Mapping/fw/1.0/Status(.*)" />
<action type="Rewrite" url="api/Status{R:1}" />
</rule>
</rules>
</rewrite>
We submitted a support ticket a week ago, and in the meantime we are trying to figure out what could be causing this and what we can do to mitigate the issue.
The 503.64 response seems to have been a red herring. It appears that there was required outbound network traffic that was being blocked by our network firewall. The advice from Azure was to double check the settings and allowed traffic according to the Azure documentation or to migrate from App Service Environment v2 to v3 where this traffic appears to be allowed automatically (or not needed).
Related
We host our Asp.Net app as Azure App Service and using scaling-out capabilities. The issue is that when we add new instances these new instances start to server requests almost immediately - before warming up process is done. I am using application initialization module with the following configuration:
<applicationInitialization remapManagedRequestsTo="/warmup.html">
<add initializationPage="/start.ashx" />
</applicationInitialization>
And once we scale-out the plan e.g. from 3 instances to 4 and requesting the site, approximately 1 of 4 times I see my warmup.html - therefore I assume that newly created instance considered by the balancer before it is actually warmed up.
Please note: our site does not require HTTPS, so this should not be an issue.
The question: how can we prevent warming up instances from being requested until the are ready to serve requests?
You may add add a section to your web.config that tells IIS to ping this route (initializationPage) for status, this should help in making the instance available until the warm-up is completed.
<system.webServer>
<applicationInitialization remapManagedRequestsTo="/warmup.html">
<add initializationPage="/wait-for-init.php" hostName="appinit-site.azurewebsites.net"/>
</applicationInitialization>
<system.webServer>
Check this blog link: https://ruslany.net/2015/09/how-to-warm-up-azure-web-app-during-deployment-slots-swap/
See: https://www.iis.net/downloads/microsoft/application-initialization
Also, you may use a external cache provider like Azure Cache for Redis, your new instance can just point to this external cache without having to reload the data.
Also check this blog might be helpful.
I setup a Sitecore infrastructure on azure (I created the same before and it worked) and when I connect to the web apps directly, they all work. I configured access to two of the apps via Application Gateway - while one works, the other one gives 502 on the health status.
I checked on the application and there it shows me that the error is "ERROR_INTERNET_CONNECTION_ABORTED".
When I completely deactivate Sitecore (rename default.aspx and web.config) and put an index.html page, the application gateway can access it.
As mentioned - I have a running deployment that I did with the same ARM template. I also deleted everything and redeployed - same issue.
As mentioned - direct access to the web app works perfectly.
I have NO network restrictions on the web app yet.
It might be the case for Sitecore initial startup to take ages (more than health probe from service gate).
Considering Service Gate does not receive response from WebApp, it assumes application to be unhealthy, and might cache the 502 code.
What does your web app diagnostics say?
I finally found the issue. It was not the startup time and not an issue with the Application Gateway:
<ipSecurity allowUnlisted="false" denyAction="AbortRequest">
<clear />
<add ipAddress="0.0.0.0" subnetMask="0.0.0.0" allowed="true" />
</ipSecurity>
this is in the web.config for newer Sitecore Versions. No idea why it worked for the other installation - but removing this solved the issue for me.
We've developed a NodeJS/Express web app locally and added some new features that utilize Server Sent Events (SSE). When testing locally, clients are able to connect to the server, register, and receive updates from the server via SSE. However, when we deploy the web app to Azure, the features that require SSE are non-functional. The clients are able to initiate the connection, but and the server sees the connection, but then the connection is immediately closed and the client receives a message with status 500 / internal server error. We are first trying to determine the exact point of failure, and then how to correct, as we assume something in either IIS or IISnode is eating the response.
We've been all over the internets, but we've nothing we've implemented has seemed to help. We've tried just about every combination of configurations we can find. I've excerpted the relevant bits of our web.config below, which we understand should pass control of websocket & related transports to the underlying nodejs server rather than having iis or iisnode handle it? We've also turned on web sockets and always on in the configuration, but to no avail.
web.config
<system.webServer>
<webSocket enabled="false" />
<handlers>
<add name="iisnode" path="server.js" verb="*" modules="iisnode" responseBufferLimit="0" />
.
.
.
<iisnode watchedFiles="*.js;node_modules\*;routes\*.js;views\*.jade" flushResponse="true" />
We would expect that the Azure deployment should work it does locally -- that clients connected to the server would receive all broadcasts via SSE, but instead they just receive the 500 / internal server error messages.
#tbr123psu the question is missing few details without which it is difficult to point in the right direction. For example I see that you are using the Windows offering to host your Node app, and you also indicated that the status code was 500. It is important to know what is the substatus as it provides pointers on where the problem might be. See this IISNODE http status and Substatus.
NOTE: On Windows App Service Plan, IIS is used as the underlying web server. For Node.JS application IIS uses the IISNode to process requests.
Additionally , I would recommend to turn on Application Logging for your Azure Web App. This option is available in the portal under Diagnostic Logs. Refer this: Azure Web App: How to enable diagnostics
Refer this blogpost on how to debug Node.JS apps hosted on Azure App Service: https://azureossd.github.io/2015/08/19/debug-node-js-web-apps-on-azure/
Once you have the logging enabled, the data emitted will point you in the direction of what is causing the failure.
I am attempting to host multiple websites in a single Azure WebApp rather than having multiple web apps each hosting a site. Each of these sites is rather small, but in order to meet the requirements of the SLA with Azure, we need to have the server scaled with more than one instance for our Production environment, thus the reasoning for combining sites.
After researching this topic, I have got the following setup with our Azure account.
Created the proper CNAME DNS records in our zone manager (complete and works).
Setup the Azure WebApp to respond to the subdomains (complete and works).
Setup a virtual directory for the second web application (complete and works)
At this point, both web applications function correctly and both subdomains are pointing at our Azure instance. We now have the following setup:
both www.mydomain.com and app.mydomain.com bring up the root application that I deployed.
going to www.mydomain.com/app2 and app.mydomain.com/app2 bring up the second application that I deployed to the virtual directory
What I would like to achieve:
Going to app.mydomain.com would bring up the application in the virtual directory.
Going to www.mydomain.com would bring up the application at the root of the azure instance.
However, what I cannot figure out is how to map a subdomain to a specific virtual directory. I have tried to update the Site URL to be the subdomain I want the application to respond to, however, the subdomain still brings up whatever I have in the root of the WebApp deployment.
Should I have some HttpHandler that sits in the site root and directs traffic to the proper virtual directory? Is there a setting in the portal that I am missing? Previously, we did this with Web Roles and tinkering with the ServiceDefinition file, but the tool sets for the Azure Web Apps in regards to publishing, integration with Source Control, etc seem to be a bit further along.
The answer posted by RuslanY will work (with some modifications to the rules slightly) however, after understanding more of the Azure portal and Web App configurations, it is not needed to host multiple sites within a single Web App (Its technically multiple web apps all sharing the resource plan you define, such as 2 instances of Standard Level 0 (S0))*.
As of today's Azure service offerings, the following is true. When you create a new Web App, you pecify the "App Service Plan" that the app falls into. If you have an App Service plan, lets say Standard with 2 instances, any Web App you deploy to that App Service plan shares those resources with other web apps in the same service plan, meaning you are not paying additional costs to host the additional web app if it is in the same App Service plan. I had assumed each web app was its own set of resources (it can be, but doesn't have to be). Given this, to accomplish what I need, I simply create a web app for each sub domain and place them all into the same App Service plan. I now am hosting multiple sites, not paying for 2 servers per site (what I wanted to avoid) and I don't have to use URL rewrites or HTTP Handlers.
I hope this write-up helps others understand the structure of the Azure Web Apps a little bit better. The current online documentation, from what I can tell, doesn't make this exactly clear.
This may be possible to do with URL rewrite rule which takes the hostname of the request and rewrites the request URL to start with the subdomain extracted from the hostname:
<system.webServer>
<rewrite>
<rules>
<rule name="Rewrite Subdomain To Directory">
<match url=".*" />
<conditions>
<add input="{HTTP_HOST}" pattern="^www\.mydomain\.com$" negate="true" />
<add input="{HTTP_HOST}" pattern="^(.+)\.mydomain.\com$" />
</conditions>
<action type="Rewrite" url="{C:1}/{R:0}" />
</rule>
</rules>
</rewrite>
</system.webServer>
To understand the syntax of the rewrite rules you can refer to the IIS.net documentation about URL rewrite module.
I have deployed one azure WCF hosted service. I created one service and one service contract. Everything works fine if I call it as below:
http://myexampleservice.cloudapp.net/TestSertvice.svc/Test1
Now, we want to catch all invalid request, which either do not have the correct service name or correct operation name. For example all requests of the form below:
http://myexampleservice.cloudapp.net/TestSertvice12.svc/Test1
Is there any way to do this?
If I call above invalid requests then service returns response status as 404. Is there any possibility that azure traffic manager will degrade the service if it gets too many such requests?
-Manish
This is actually pretty easy to do. So first you will need to catch 404 requests occurring on your instance(s):
<customErrors mode="On" >
<error statusCode="404" redirect="~/Errors/Error404.aspx" />
</customErrors>
Each time a 404 error occurs the user/consumer will be redirected to Error404.aspx where you can add this event to a local counter (a file or a static variable) or shared counter if you have multiple instances (SQL Azure, Table Storage, ...).
Take a look at the options you have when configuring the traffic manager:
You can setup a monitoring endpoint. This would point to a different page (like /Status/CanBeUsed.aspx). This page should return an HTTP status code different from 200 if it decides that the deployment should not be used (ie: if your local/shared counter contains too many 404 errors). The Traffic Manager will monitor this page and after 3 failed requests it will fail over to a different deployment.