Cannot warm up pages using applicationInitialization in webconfig - azure

Question

I have a simple Umbraco 7.7.2 application and I'm hosting it on Azure (app-service). When I restart the server it takes 20-40 seconds for first time requesting a page which is really annoying specially when the load is high and you are Scaling out to reduce the response times.
I've tried this setting in my webconnfig, but it doesn't seem to work.
<system.webServer>
<applicationInitialization>
<add initializationPage="/page1/?warmup=1" hostName="mydomain.com" />
<add initializationPage="/page1/page2/?warmup=1" hostName="mydomain.com" />
</applicationInitialization>
</system.webServer>
I might be trying it in a wrong way, but what I did was to restart the server and I've left it for 2-3 minutes without requesting any page.
I've checked my Umbraco logs and the application wasn't even started.
Then I've requested the home page and it took 40 seconds to come up.
Then I've tried mydomain.com/page1 and it also took 20 seconds since it was the first request to access it.
*P.S: after the first request, the site is very fast and each page takes less than 100 ms to load
Update
I've implemented a rewrite to stop next redirects as Kevin has suggested.
As a result, my Umbraco will start up, but still the requests doesn't reach to the pages.
On my master page, I've added a line to write a line in the logs if it has the warmup in the querystring and it works it the page is hitted from the browser:
if (!string.IsNullOrWhiteSpace( Request.QueryString["warmup"]))
{
var pageC = Model.Content;
logger.Info(pageC.UrlAbsolute()+" "+ Request.QueryString);
}
However, there is nothing in my logs after
2018-02-08 15:16:51,245 [P7036/D2/T1] INFO Umbraco.Core.CoreBootManager - Umbraco application startup complete (took 12727ms)
2018-02-08 15:16:54,911 [P7036/D2/T1] INFO MyNamespace.Web.CustomStartup - base config done!
Here is the confing that I've added based on Kevin's answer:
<rule name="No redirect on warmup request (request from localhost with warmup user agent)" stopProcessing="true">
<match url=".*" />
<conditions>
<add input="{REMOTE_ADDR}" pattern="127.0.0.*" />
</conditions>
<action type="Rewrite" url="{URL}" />
</rule>
Also, I've found another similar config on Microsoft:
<rule name="No redirect on warmup request (request from localhost with warmup user agent)" stopProcessing="true">
<match url=".*" />
<conditions>
<add input="{HTTP_HOST}" pattern="localhost" />
<add input="{HTTP_USER_AGENT}" pattern="Initialization" />
</conditions>
<action type="Rewrite" url="{URL}" />
</rule>

Answer 1

Note that azure will warm up the URL on http, so if you are forcing https using rewrite rules, the full site will not warm up, only the redirect module will. Then it needs to finish warming up Umbraco after azure adds it into the load balancer and the first https gets through to the umbraco code. We found that out by checking the http logs when scaling out.
We couldn't figure out how to tell azure to warmup using https, so we allowed Azure to access the site on http by making a rule before our force https rewrite to stopProcessing when {REMOTE_ADDR} matches 127.0.0.*.
<rule name="Allow localhost to warmup" stopProcessing="true">
<match url="(.*)"/>
<conditions>
<add input="{REMOTE_ADDR}" pattern="127.0.0.*" />
</conditions>
</rule>

Answer 2

There was so many reasons for the requests not reaching my site and thanks to Kevin and Twamley whom opened my eyes to the probable cause I could trace and find all of them.
First, as Kevin said HTTPS was one of the issues which I've fixed as below:
<rule name="No redirect on warmup request (request from localhost with warmup user agent)" stopProcessing="true">
<match url=".*" />
<conditions>
<add input="{HTTP_HOST}" pattern="localhost" />
<add input="{HTTP_USER_AGENT}" pattern="Initialization" />
</conditions>
<action type="Rewrite" url="{URL}" />
</rule>
Then I could see that Umbraco starts up but the requests didn't get to my pages.
<rule name="Redirect rquests to www.example.com" stopProcessing="true" enabled="true">
<match url="(.*)" />
<conditions >
<add input="{HTTP_HOST}" pattern="^example\.com$" />
</conditions>
<action type="Redirect" url="https://www.example.com/{R:0}" />
</rule>
I didn't expect my requests to get to this redirect since it was at the end of my rewrite rules and therefore it should be stoped at No redirect on warmup request but it didn't so I've added another condition to it: <add input="{HTTP_USER_AGENT}" pattern="Initialization" negate="true" />
<rule name="Redirect rquests to www.example.com" stopProcessing="true" enabled="true">
<match url="(.*)" />
<conditions logicalGrouping="MatchAll">
<add input="{HTTP_HOST}" pattern="^example\.com$" />
<add input="{HTTP_USER_AGENT}" pattern="Initialization" negate="true" />
</conditions>
<action type="Redirect" url="https://www.example.com/{R:0}" />
</rule>
Also, I have ipSecurity in my settings since it is my test environment and I didn't want it open to public. Turns out the Initialization cannot hit my site at all if I don't open up to 127.0.0.1.....
<security>
<ipSecurity allowUnlisted="false">
<add ipAddress="127.0.0.1" allowed="true" />
<add ipAddress="x.x.x.x" allowed="true" />

Answer 3

I like Kevin's idea of just stopping the processing as one of the first rewrite rules. I noticed his didn't have an action but you added one to yours. Maybe try his w/o an action? Is it the first rule in the file?
Another option we use is to add this condition to any problem rules (notice the negate).
<add input="{REMOTE_ADDR}" pattern="127\.0\.0\.1" negate="true"/>
Try temporarily clearing your rewrite rules until you're convinced your warmup is working, then add the rules back a few at a time to find and fix the problem redirect. Force SSL and Trailing Slash type rules will definitely cause problems.
Also, the hostName can get you in trouble easily. It should be a perfect match for your production environment. It doesn't use DNS to resolve it, it just talks to the local site and passes that in as the HOST header.
I don't have any querystrings in my warmup list. Maybe you should try dropping those. You don't really need them because you can change your logging code to:
if (Request.IsLocal && Request.UserAgent == "IIS Application Initialization Warmup") {
// log it
}
Logging them is a great idea because the warm up requests don't show up in the standard IIS logs. I have my servers mail me at the beginning and end of the warmup by hitting a special first and last entry that uses that if statement. Some useful details from Azure:
new {
WEBSITE_HOSTNAME = System.Environment.GetEnvironmentVariable("WEBSITE_HOSTNAME"),
WEBSITE_INSTANCE_ID = System.Environment.GetEnvironmentVariable("WEBSITE_INSTANCE_ID"),
WEBSITE_SITE_NAME = System.Environment.GetEnvironmentVariable("WEBSITE_SITE_NAME"),
COMPUTERNAME = System.Environment.GetEnvironmentVariable("COMPUTERNAME"),
USER_AGENT = Request.UserAgent,
URL = Request.Url,
WARM_UP_TIME = (DateTime.UtcNow - _start).ToString()
}