Azure Web Role warm up after Deploy - azure

As described in various other related questions here, I am also expecting long lasting (30 seconds) first call after (re)deploying a web role with pretty large EF6 Model and a plenty of referenced nuget-packages. After trying out different proposed solutions with preloadEnabled and serviceAutoStartProviders I am still consufed and decided to reopen this topic in hope, that somebody has come upon a better solution in the meantime.
The main goal for me is to make a web role answering the first request almost as fast as subsequent calls as soon as the role quits its busy state on a fresh deployment and gets accessible by load balancer and external clients. Unfortunately I experienced following problems with proposed solutions so far:
preloadEnabled:
I do add Application Initialization module via PKGMGR.EXE /iu:IIS-ApplicationInit in a startup task. So far so good.
When I then try to execute %windir%\system32\inetsrv\appcmd set site "MySiteName" -applicationDefaults.preloadEnabled:true it fails as at the time of execution of a startup script there are still no websites created in IIS on a fresh deployment.
If I try to set the preloadEnabled-setting via ServerManager-class in my Application_Start method instead, I cannot understand, how this code is intended to be executed before a first external call on the web role is made, as the preloadEnabled setting is defaulted to false after a fresh web role deploy and thus the Application_Start method in my understanding does not get a chance to be executed by the Application Initialization module?
serviceAutostartProviders:
here we need to put a name of our AutostartProvider implementing the IProcessHostPreloadClient interface in applicationhost.config i.e by using either appcmd script or the ServerManager class, BUT:
serviceAutostartProvider is like preloadEnabled a site related setting, so we have the same problem here as with %windir%\system32\inetsrv\appcmd set site "MySiteName" -applicationDefaults.preloadEnabled:true in 1.2 - at execution time of startup script after a fresh deployment the websites are not yet created in IIS and the script fails to execute properly
Another possibility would be to include applicationhost.config into the deployment package, but I did not find any solution to do this for a web role.
So how did you guys managed to ensure, that preloading assemblies and some initialization code (like filling memory caches) is run before the role gets it's first hit from outside?
We start to gain some traffic now and get approx. 1-2 requests per second on our WebApi, and thus a 30 second delay for preloading "visible" to clients after each update deployment is becoming a major issue now.
We are scheduling update deploys at low traffic times, but what if I need to make an urgent hotfix deploy?

Perhaps the best way to accomplish this is to use deployment slots. Deploy updates to your staging slot first. Before a switch from a staging slot to a production slot takes place, Kudu will hit the root of the staging slot with a request in order to warm up the application. After the request to the staging slot's root returns, the IP switch occurs and your slots are swapped.
However, sometimes you need to warm up other pages or services to get the app ready to handle traffic, and hitting the root with a warmup request is insufficient. You can update your web.config so that Kudu will hit additional pages and endpoints before the IP switch occurs and the slots are swapped.
These warmup request URLs should be in the tag. Here's an example:
<?xml version="1.0" encoding="utf-8"?>
<configuration>
<system.webServer>
<applicationInitialization>
<add initializationPage="/pagetowarmup1" />
<add initializationPage="/pagetowarmup2" />
<add initializationPage="/pagetowarmup3" />
</applicationInitialization>
</system.webServer>
</configuration>
You can read the Kudu docs on this issue here.

OK. Now I got it.
The point is to set preloadEnabled-property not inside of the Application_Start method in Global.asax (as it will not be hit before a first request to the Role anyway), but inside RoleEntryPoint.OnStart.
The difference is that RoleEntryPoint.OnStart is called directly after deployment package is extracted and everything is set up for starting the role. At this moment the azure instance is still in it's busy state and is not yet available from outside as long as some code inside RoleEntryPoint.OnStart is being executed.
So here is the code I came up so far for warming up the instance before it gets its first call from outside
public class WebRole : RoleEntryPoint
{
public override bool OnStart()
{
// set preloadEnabled of each Site in this deployment to true
using (var serverManager = new ServerManager())
{
foreach (var mainSite in serverManager.Sites)
{
var mainApplication = mainSite.Applications["/"];
mainApplication["preloadEnabled"] = true;
}
serverManager.CommitChanges();
}
// call my warmup-code which will preload caches and do some other time consuming preparation
var localuri = new Uri(string.Format("https://{0}/api/warmup", RoleEnvironment.CurrentRoleInstance.InstanceEndpoints["Endpoint1"].IPEndpoint));
try
{
var request = (HttpWebRequest)WebRequest.Create(localuri);
request.Method = "GET";
var response = request.GetResponse();
}
catch { }
// send this thread to sleep in order to give the warmup-request a chance to complete BEFORE the Role will get started
// and becomes available to the outside world
System.Threading.Thread.Sleep(60000);
return base.OnStart();
}
}

Related

App Settings not being observed by Core WebJob

I have a Core WebJob deployed into an Azure Web App. I'm using WebJobs version 3.0.6.
I've noticed that changes to Connection Strings and App Settings (added via the Azure web UI) are not being picked up immediately by the WebJob code.
This seems to correlate with the same Connection Strings and App Settings not being displayed on the app's KUDU env page straight away (although I acknowledge this may be a red herring and could be some KUDU caching thing which I'm unaware of).
I've deployed a few non-Core WebJobs in the past and have not come across this issue so wonder if it's Core related? Although I can't see how that might affect configs showing up KUDU though.
I was having this issue the other day (where the configs were not getting picked up by the WebJob or shown in KUDU) and was getting nowhere, so left it. When I checked back the following day, the configs were now correctly showing in KUDU and being picked up by the WebJob. So I'd like to know what has happened in the meantime which means the configs are now being picked up as expected.
I've tried re-starting the WebJob and re-starting the app after making config changes but neither seem to have an effect.
It's worth also noting that I'm not loading appSettings.json during the program setup. That being said, the connection string being loaded was consistenly the connection string from that file i.e. my local machine SQL Server/DB. My understanding was always that the anything in the Azure web UI would override any equivalent settings from config files. This post from David Ebbo indicates that by calling AddEnvironmentVariables() during the setup will cause the Azure configs to be observed, but that doesn't seem to be the case here. Has this changed or is it loading the configs from this file by convention because it can't see the stuff from Azure?
Here's my WebJob Program code:
public static void Main(string[] args)
{
var host = new HostBuilder()
.ConfigureHostConfiguration(config =>
{
config.AddEnvironmentVariables();
})
.ConfigureWebJobs(webJobConfiguration =>
{
webJobConfiguration.AddTimers();
webJobConfiguration.AddAzureStorageCoreServices();
}
)
.ConfigureServices((context, services) =>
{
var connectionString = context.Configuration.GetConnectionString("MyConnectionStringKey");
services.AddDbContext<DatabaseContext>(options =>
options
.UseLazyLoadingProxies()
.UseSqlServer(connectionString)
);
// Add other services
})
.Build();
using(host)
{
host.Run();
}
}
So my questions are:
How quickly should configs added/updated via the Azure web UI be displayed in KUDU?
Is the fact they're not showing in KUDU related to my Core WebJob also not seeing the updated configs?
Is appSettings.json getting loaded even though I'm not calling .AddJsonFile("appSettings.json")?
What can I do to force the new configs added via Azure to be available to my WebJob immediately?
The order in which configuration sources are specified is important, as this establishes the precedence with which settings will be applied if they exist in multiple locations. In the example below, if the same setting exists in both appsettings.json and in an environment variable, the setting from the environment variable will be the one that is used. The last configuration source specified “wins” if a setting exists in more than one location. The ASP.NET team recommends specifying environment variables last, so that the environment where your app is running can override anything set in deployed configuration files.
You can refer here for more details on Azure App Services Application Settings and Connection Strings in ASP.NET Core

Why should i use a LoadBalancerProbe instead of subscribing to the RoleEnvironment.StatusCheck Event?

I was fiddling with the options azure provides to balance the load between multiple webroles.
I found three possible ways to do this.
the first would be to do nothing at all and let the default ( round robin) implementation do the job.
the second possibility would be to define a custom LoadBalancerProbe in the ServiceDefinitionFile, which i tried and did not get to work: From my understanding the custom aspx page is called each time a status check is performed on the role. Depending on the http response code the role changes its status to busy. - but this is never happening.
Also, i couldnt really find any examples for defining a custom LoadBalancingProbe.
Thus i looked for an alternative way to do this.
Now iam subscribing to the RoleEnvironment.StatusCheck Event, which allows me to implement some logic and depending on the results set the role state to busy and available.
My Questions:
1) Supposing the Custom LoadBalancerProbe works as described in the MSDN, what is the difference between subscribing to the StatusCheckEvent and using a custom probe?
2) Why does my custom load balancer probe not work ? - iam just testing with the azure emulator for now and iam well aware that traffic still gets routed to the webrole instances allthough they are set to busy in the emulator.
But my Custom Probe does not change the status of the webroleinstances at all.
Here is the very rudimentary code, which should - to my knowledge set the status of webrole instance_n_0 to busy.
public class LoadBalanceController : Controller
{
public ActionResult Index()
{
WebOperationContext woc = WebOperationContext.Current;
if(RoleEnvironment.CurrentRoleInstance.Id.ToLower().Contains("_0"))
{
woc.OutgoingResponse.StatusCode = System.Net.HttpStatusCode.ServiceUnavailable;
}else
{
woc.OutgoingResponse.StatusCode = System.Net.HttpStatusCode.OK;
}
return View(); //not relevant
}
Ive also configered my servicedefinitionfile and set a Route to redirect to this controller/action when calling the healthcheck.aspx defined in the custom probe.
<LoadBalancerProbes>
<LoadBalancerProbe name="WebRoleBalancerProbeHttp" protocol="http" path="healthcheck.aspx" intervalInSeconds="5" timeoutInSeconds="100"/>
</LoadBalancerProbes>
...
<InputEndpoint name="EndpointWeb" protocol="http" port="80" loadBalancerProbe="WebRoleBalancerProbeHttp"/>
The Route:
routes.MapRoute(
name: "HealhCheck",
url: "healthcheck.aspx",
defaults: new { controller = "LoadBalance", action = "Index", id = UrlParameter.Optional }
);
Not sure why the custom probe isn't working, but for the differences: The health-check event lets you announce whether an instance is available, but you don't have any flexibility in terms of how often this is called. Also, you can't launch a separate service that listens on a custom port (or port type).
You have much more flexibility with custom probes, since you can create any type of port listener to determine health, even a separate exe.
With Virtual Machines, this is the only method of health probes, since Virtual Machines don't have the guest agent running and don't provide the health-check event.
In your service definition file, you have marked your probe to be hosted on an http endpoint (and not HTTPS).
Is that the case with your web app as well ? Is it exposing that endpoint on HTTP and not on HTTPS ? If yes, then also check if there is any automatic redirection to HTTPS happening.
I think you have mostly setup everything properly. Here is a post that has some example code and another question from SO, where they were successful in setting it up (should provide some insight into the csdef file).
I agree with David Makogon's points about the differences between the two.

Disabling request validation on Azure Websites

My node.js application runs nicely on my own machine, however, after pushing it up to an Azure WebSite using Git, I'm getting request validation problems.
Some of my requests are trapped by the IIS validation process. The fix is to add this to the web.config:
<httpRuntime requestPathInvalidCharacters=""/>
I downloaded the web.config from the site\wwwroot folder of my website using FTP, added it to the repo, added the new element in the XML, and pushed it back up to Azure.
My web.config file seems to have been overridden by the default.
Is there a way of changing this behaviour? Or another way of turning off request validation? Or am I just being thick?
UPDATE
I take it all back, it seems to be express which is blocking the request. However, the question still stands as 'how do I customise my web.config?'
Could you try setting a custom validation type? Like this:
<httpRuntime requestPathInvalidCharacters=""
requestValidationType="MyValidator, MyAssembly"/>
And the class:
public class MyValidator : RequestValidator
{
protected override bool IsValidRequestString(HttpContext context, string value, RequestValidationSource requestValidationSource, string collectionKey, out int validationFailureIndex)
{
validationFailureIndex = 0;
return true;
}
}

This controller won't show up in Web Api on Azure

namespace StorageRoleMVC4.Controllers
{
public class SearchController : ApiController
{
public Dictionary<string, string> Get([FromUri] string searchString, [FromUri] string searchObject)
{
var searchHelper = new SearchStorageHelper();
var objectList = searchHelper.Retrieve(searchString, searchObject);
return objectList;
}
}
}
Is there anything about this controller that makes it unreachable once it's deployed (to an Azure web role)? I just get a 404 error when I try to reach it. It works great on the local emulator.
The last 2 times I've deployed my project, all the controllers in my web service have returned 404 errors for several hours, until the project seems to fix itself. I'm not sure why, but it might be related.
UPDATE
There is a WARNING in the event log on the web role VM after I publish:
The application '/' belonging to site '1273337584' has an invalid AppPoolId 'ea7a2e15-9390-49e1-a16b-67ff1cdb7dcb' set. Therefore, the application will be ignored.
This is the id of my site, but the AppPoolId is not correct. Changing the app pool turns the 404 into a 502.
Also, after publishing, the World Wide Web Publishing Service is turned off. When I turn it on and do an IIS reset, after the reset it's turned off again.
When I reboot the web role VM, most of the controllers work again, and the World Wide Web Publishing Service is turned on. But still, this SearchController doesn't work. Or any other new controllers I've created since this problem started happening.
Well, after a ridiculous amount of unsuccessful troubleshooting, I just rolled back to an earlier version of the code and found that it didn't read the web service when I deployed it. So I started over with that version and re-built the delta.
This involved removing a few web.config entries and removing some libraries, creating a few classes and referencing them in the global.asax (I think that's where it was) in order to override te Authorize attribute...
If anyone has a better answer, I will switch the answer to what you post.

Add WebRole.cs - and have it called - in an existing ASP.NET MVC site converted to web role

I have an ASP.NET MVC 4 site running perfectly well in an Azure WebRole. The ASP.NET MVC project was started on its own, after which I added an Azure Cloud Service project to the solution and added the ASP.NET project/site as one of the 'roles' of the service (so it shows up in the 'Roles' folder).
My problem is that I would like to have working a WebRole.cs file within the ASP.NET MVC project, but no matter what I've tried to do, it appears that when deployed, it just never gets called. OnStart and the override of Run (which I know, must never leave the loop) -- these just apparently never get called.
But if you startup a new CloudService project and add, at that time from the start, an ASP.NET MVC project, it automatically has a WebRole.cs file in it, so my guess is that I need to configure something somewhere for the WebRole.cs (actually speaking, the WebRole class, which inherits RoleEntryPoint) to get called. What might that be?
using System;
using System.Web;
//using Microsoft.WindowsAzure.StorageClient;
using System.Diagnostics;
using System.Threading;
namespace Us.WebUI
{
public class WebRole : Microsoft.WindowsAzure.ServiceRuntime.RoleEntryPoint
{
public override bool OnStart()
{
return true; //return base.OnStart(); // CALL THIS???
}
public override void Run()
{
while (true) {
Thread.Sleep(TimeSpan.FromSeconds(30));
try {
EmailFuncs.SendEmailToUs("An email from our WebRole?????", "Email me this, email me that.");
}
catch { }
}
}
}
}
UPDATE: Thanks, the question has been answered. But I will add: On doing this, while it clearly was working (fully deployed and in emulator), that suddenly I was having problems doing a full publish of the site. After a azure publish took 3 hours:
Verifying storage account 'xyz'... > Uploading Package... > - Updating... [stayed here for 3 hours], it failed with this error: The server encountered an internal error. Please retry the request. So one thing I was wondering is, did I need to override OnStop in WebRole.cs?
UPDATE 2: Those previous problems were fixed, and had nothing to do with this issue. Actually, I've learned this: If you ever have any warnings generated in your build, Azure often will not work with them even when they don't cause problems locally or in other hosts. Since then, I've been much more studious to tackling build warnings (but critical to this is turning off with warning codes the many warning types you want to ignore!).
Adding a class to your Web Project which inherits from RoleEntryPoint is sufficient, it should just work. Did you try setting a breakpoint in the emulator?
What you might be experiencing is that EmailFuncs.SendEmailToUs requires info from the app/web.config and that this info is not available. You need to know that your WebRole class runs in a different process (not your web application), meaning it's not using your web.config. If you want the WebRole.cs to read info from the configuration file, you'll need to add these settings in WaIISHost.exe.config

Resources