Autoscaling Application Block not Increasing/Decreasing Instances - azure

I have implemented autoscaling using the Enterpise Library Autoscaling Block, pretty much as described in the tutorial.
So the first time when I am running load test for my site it's increasing the instance count by one.
And when I am running load again then CPU usage is 100% but it's not increasing the instance count. And when CPU usage is 0% it's not decreasing the instance count.
So what is wrong here?
I have following Rules and Service XML. Rules XML:
<?xml version="1.0" encoding="utf-8" ?>
<rules xmlns="http://schemas.microsoft.com/practices/2011/entlib/autoscaling/rules">
<constraintRules>
<rule name="default" enabled="true" rank="1"
description="The default constraint rule">
<actions>
<range min="2" max="5" target="ServiceWebRole"/>
</actions>
</rule>
</constraintRules>
<reactiveRules>
<rule name="ScaleUpOnHighUtilization" rank="10"
description="Scale up the web role" enabled="true">
<when>
<any>
<greaterOrEqual operand="WebRoleA_CPU_Avg_5m" than="60"/>
</any>
</when>
<actions>
<scale target="ServiceWebRole" by="1"/>
</actions>
</rule>
<rule name="ScaleDownOnLowUtilization" rank="11"
description="Scale up the web role" enabled="true">
<when>
<all>
<less operand="WebRoleA_CPU_Avg_5m" than="40"/>
</all>
</when>
<actions>
<scale target="ServiceWebRole" by="-1"/>
</actions>
</rule>
</reactiveRules>
<operands>
<performanceCounter alias="WebRoleA_CPU_Avg_5m"
performanceCounterName="\Processor(_Total)\% Processor Time"
source="ServiceWebRole" timespan="00:05:00" aggregate="Average"/>
</operands>
</rules>
Service XML:
<?xml version="1.0" encoding="utf-8" ?>
<serviceModel
xmlns="http://schemas.microsoft.com/practices/2011/entlib/autoscaling/serviceModel">
<subscriptions>
<subscription name="My subscription name"
certificateThumbprint="My subscription certificateThumbprint"
subscriptionId="My subscriptionId"
certificateStoreLocation="LocalMachine" certificateStoreName="My">
<services>
<service dnsPrefix="productionservice" slot="Production" scalingMode="Scale">
<roles>
<role alias="ServiceWebRole" roleName="ServiceWebRole"
wadStorageAccountName="targetstorage"/>
</roles>
</service>
</services>
<storageAccounts>
<storageAccount alias="targetstorage"
connectionString="DefaultEndpointsProtocol=https;AccountName=autoscale;AccountKey=storageaccountkey">
</storageAccount>
</storageAccounts>
</subscription>
</subscriptions>
</serviceModel>

Related

Remove Server header on response on 500 HTTP Error

I have a REST API service deployed as an Azure App Service and i been trying to remove the Server header from response.
I tried adding a custom outgoing rule to Rewrite module.
In web.config i have
<rewrite>
<outboundRules>
<rule name="Change Server Header">
<match serverVariable="RESPONSE_Server" pattern=".+" />
<action type="Rewrite" value="My Server" />
</rule>
</outboundRules>
</rewrite>
I also applied a applicationHost.xdt transformation to applicationHost.config in order to add the RESPONSE_Server variable
<?xml version="1.0"?>
<configuration xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">
<configSections>
<sectionGroup name="system.webServer" xdt:Locator="Match(name)">
<sectionGroup name="rewrite" xdt:Locator="Match(name)">
<section name="allowedServerVariables" overrideModeDefault="Allow" xdt:Locator="Match(name)" xdt:Transform="SetAttributes(overrideModeDefault)" />
</sectionGroup>
</sectionGroup>
</configSections>
<system.webServer>
<rewrite>
<allowedServerVariables>
<add name="RESPONSE_Server" xdt:Transform="InsertIfMissing" xdt:Locator="Match(name)" />
</allowedServerVariables>
</rewrite>
</system.webServer>
</configuration>
But i still see the Server response http header on a request like following which returns a 500 HTTP status code
https://api.internaltest.com/%3a/
The rewrite rule works in local IIS it removes the http header successfully... but not when doing it in Azure App Service.
Please tell me what i'm missing.
I mention that i have removeServerHeader="true" in web.config also but it only removes the server header on 404 bad request, not on 500 status code.
<security>
<requestFiltering removeServerHeader="true">
<requestLimits maxAllowedContentLength="2147483647" />
</requestFiltering>
</security>
Hope anyone can help me,
Thank you
I manged to make it work by adding the outbound rule in xdt file and not in web.config
Here is the correct application.Host.xdt hope it helps others... It replaces the server header with a value on 500 error HTTP response
<?xml version="1.0"?>
<configuration xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">
<configSections>
<sectionGroup name="system.webServer" xdt:Locator="Match(name)">
<sectionGroup name="rewrite" xdt:Locator="Match(name)">
<section name="allowedServerVariables" overrideModeDefault="Allow" xdt:Locator="Match(name)" xdt:Transform="SetAttributes(overrideModeDefault)" />
</sectionGroup>
</sectionGroup>
</configSections>
<system.webServer>
<rewrite>
<allowedServerVariables>
<add name="RESPONSE_Server" xdt:Transform="InsertIfMissing" xdt:Locator="Match(name)" />
</allowedServerVariables>
<outboundRules xdt:Transform="InsertIfMissing">
<rule name="Change Server Header" xdt:Transform="InsertIfMissing" xdt:Locator="Match(name)">
<match serverVariable="RESPONSE_Server" pattern=".+" />
<action type="Rewrite" value="My Server" />
</rule>
</outboundRules>
</rewrite>
</system.webServer>
</configuration>

Azure App Service applicationHost.xdt appears to have no effect

I'm trying to get a reverse proxy set up by using Azure Websites, roughly following this guide that explains how to modify ApplicationHost.config on such a website - but it doesn't work for me.
I've have this applicationHost.xdt:
<?xml version="1.0"?>
<configuration xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">
<system.webServer>
<proxy xdt:Transform="InsertIfMissing" enabled="true" preserveHostHeader="false" reverseRewriteHostInResponseHeaders="false" />
<rewrite>
<allowedServerVariables>
<add name="HTTP_X_ORIGINAL_HOST" xdt:Transform="InsertIfMissing" />
<add name="HTTP_X_UNPROXIED_URL" xdt:Transform="InsertIfMissing" />
<add name="HTTP_X_ORIGINAL_ACCEPT_ENCODING" xdt:Transform="InsertIfMissing" />
<add name="HTTP_ACCEPT_ENCODING" xdt:Transform="InsertIfMissing" />
</allowedServerVariables>
</rewrite>
</system.webServer>
</configuration>
I put it in the site directory of my web app.
The transforms appear to get executed (from the transform log):
2017-09-06T12:12:20 StartSection Executing InsertIfMissing (transform line 8, 50)
2017-09-06T12:12:20 on /configuration/system.webServer/rewrite/allowedServerVariables/add
2017-09-06T12:12:20 Applying to 'allowedServerVariables' element (no source line info)
2017-09-06T12:12:20 EndSection Done executing InsertIfMissing
I have indeed four of those blocks.
I still get 500s on setting the headers with rewrite. The detailed error message contains this:
<h3>HTTP Error 500.50 - URL Rewrite Module Error.</h3>
<h4>The server variable "HTTP_X_UNPROXIED_URL" is not allowed to be set. Add the server variable name to the allowed server variable list.</h4>
Not sure what to do at this point. Any ideas?
I faced the same issue with the TomSSL article, #David Ebbo's comment ultimately got me to the answer, but felt it was worth adding this to save people some time. It's because applicationHost.config is missing xdt:Locator="Match(name)":
<configuration xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">
<system.webServer>
<proxy xdt:Transform="InsertIfMissing" enabled="true" preserveHostHeader="false" reverseRewriteHostInResponseHeaders="false"/>
<rewrite xdt:Transform="InsertIfMissing">
<allowedServerVariables xdt:Transform="InsertIfMissing">
<add name="HTTP_X_ORIGINAL_HOST" xdt:Transform="InsertIfMissing" xdt:Locator="Match(name)"/>
<add name="HTTP_X_UNPROXIED_URL" xdt:Transform="InsertIfMissing" xdt:Locator="Match(name)"/>
<add name="HTTP_X_ORIGINAL_ACCEPT_ENCODING" xdt:Transform="InsertIfMissing" xdt:Locator="Match(name)"/>
<add name="HTTP_ACCEPT_ENCODING" xdt:Transform="InsertIfMissing" xdt:Locator="Match(name)"/>
</allowedServerVariables>
</rewrite>
</system.webServer>
</configuration>
The key to investigating these issues is to determine whether the problem is with the transform not doing the right thing, or with the applicationhost.config not working as you expect.
You can check the generated applicationhost.config in D:\local\Config from Kudu console.
See this page for more details about this.

Where should <allowedServerVariables> tag live in Azure Website applicationHost.config?

Technical Information
Azure Website
Installed IIS Manager Site Extension by shibayan
Scenario
I have implemented a reverse proxy on my Azure Website, however the receiving server doesn't get any indication of whether the initial request was over HTTPS or not.
What I want to do is send the HTTPS flag of ON/OFF from the initial request to the proxied server, via a custom HTTP Header.
In Theory
Using shibayan's IIS Manager Site Extension, I can edit the applicationHost.xdt file, give it a Transform to insert an <allowedServerVariables> tag and that should allow me to set a custom HTTP Header.
In Practise
I've configured my rewrite rule as such:
<rule name="Proxy" stopProcessing="true" xdt:Transform="Replace" xdt:Locator="Match(name)">
...
<serverVariables>
<set name="HTTP_X_USE_HTTPS" value="{HTTPS}" />
</serverVariables>
...
</rule>
And have attempted a few combinations of where to put the <serverVariables> tag...
Attempt one:
As described in this answer.
<?xml version="1.0" encoding="utf-8"?>
<configuration xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">
<system.webServer>
<proxy enabled="true" preserveHostHeader="false" reverseRewriteHostInResponseHeaders="false" xdt:Transform="Insert" />
<rewrite>
<allowedServerVariables>
<add name="HTTP_X_USE_HTTPS" xdt:Transform="Insert" />
</allowedServerVariables>
</rewrite>
</system.webServer>
</configuration>
Result:
HTTP Error 500.50 - URL Rewrite Module Error.
The server variable "HTTP_X_USE_HTTPS" is not allowed to be set. Add
the server variable name to the allowed server variable list.
Attempt two:
<?xml version="1.0" encoding="utf-8"?>
<configuration xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">
<location path="~1[app service name]" overrideMode="Allow">
<system.webServer>
<proxy enabled="true" preserveHostHeader="false" reverseRewriteHostInResponseHeaders="false" xdt:Transform="Insert" />
<rewrite>
<allowedServerVariables>
<add name="HTTP_X_USE_HTTPS" xdt:Transform="Insert" />
</allowedServerVariables>
</rewrite>
</system.webServer>
</location>
</configuration>
Result: HTTP 500.50
Attempt three:
<?xml version="1.0" encoding="utf-8"?>
<configuration xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">
<location path="" overrideMode="Allow">
<system.webServer>
<proxy enabled="true" preserveHostHeader="false" reverseRewriteHostInResponseHeaders="false" xdt:Transform="Insert" />
<rewrite>
<allowedServerVariables>
<add name="HTTP_X_USE_HTTPS" xdt:Transform="Insert" />
</allowedServerVariables>
</rewrite>
</system.webServer>
</location>
</configuration>
Result: HTTP 503
Attempt four:
<?xml version="1.0" encoding="utf-8"?>
<configuration xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">
<location path="[app service name]" overrideMode="Allow">
<system.webServer>
<proxy enabled="true" preserveHostHeader="false" reverseRewriteHostInResponseHeaders="false" xdt:Transform="Insert" />
<rewrite>
<allowedServerVariables>
<add name="HTTP_X_USE_HTTPS" xdt:Transform="Insert" />
</allowedServerVariables>
</rewrite>
</system.webServer>
</location>
</configuration>
Result: HTTP 503
I am aware that in the applicationHost.config file for an Azure Website there are a few places that <system.webServer> can be defined, such as under the following elements:
<configuration>
<configuration><location>
...however I've tried these combinations to no avail.
Questions
Is there another possible location?
Have I misconfigured my .xdt file in any way?
Am I missing something from my applicationHost.config?
You have to create a applicationHost.xdt file under the site folder d:\home\site\applicationHost.xdt with this content:
<?xml version="1.0"?>
<configuration xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">
<system.webServer>
<rewrite>
<allowedServerVariables>
<add name="HTTP_X_USE_HTTPS" xdt:Transform="InsertIfMissing" xdt:Locator="Match(name)" />
</allowedServerVariables>
</rewrite>
</system.webServer>
</configuration>
Now you can use the new variable in your web.config file
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<system.webServer>
<rewrite>
<rules>
<rule name="Proxy">
<serverVariables>
<set name="HTTP_X_USE_HTTPS" value="{HTTPS}"/>
</serverVariables>
</rule>
</rules>
</rewrite>
</system.webServer>
</configuration>
See also https://azure.microsoft.com/en-us/documentation/articles/web-sites-transform-extend/ or https://github.com/projectkudu/kudu/wiki/Xdt-transform-samples

Azure autoscaling based on length of queue

I want to autoscale my azure worker role based on the length of an azure queue. According to all the documentation I've seen this should be very straightforward, using the queueLength operand.
I've implemented the autoscaler, uploaded it to my service and added a bunch of elements to my queue, but the number of instances are not increasing.
What's the best way of troubleshooting? I've remote desktopped to the role and there isn't anything in the event log. Is there a log of autoscaling events/errors I can check?
Edit:
When I run the app in my dev environment, I see that the autscaler has successfully loaded the ServiceInfo.xml. There is a queue entry and a role entry. However, the rules don't appear to have been loaded out of the rules.xml file.
More edit:
When I removed the reactiverules and operands nodes from the rules.xml file, the constraint rule loads successfully. So the problem is in one of those nodes.
My serviceinfo xml file:
<?xml version="1.0" encoding="utf-8"?>
<serviceModel xmlns="http://schemas.microsoft.com/practices/2011/entlib/autoscaling/serviceModel">
<subscriptions>
<subscription name="MySubscription" subscriptionId="blah" certificateThumbprint="‎blah" certificateStoreName="My" certificateStoreLocation="CurrentUser">
<storageAccounts>
<storageAccount alias="targetstorage" connectionString="DefaultEndpointsProtocol=https;AccountName=blah; AccountKey="blah">
<queues>
<queue alias="auditqueue" queueName="auditqueue"/>
</queues>
</storageAccount>
</storageAccounts>
<services>
<service dnsPrefix="blah" slot="Production" scalingMode="Scale">
<roles>
<role alias="ScalingWebRole" roleName="ScalingWebRole" wadStorageAccountName="targetstorage" />
</roles>
</service>
</services>
</subscription>
</subscriptions>
</serviceModel>
My rules xml file:
<constraintRules>
<rule name="Default" enabled="true" rank="1">
<actions>
<range target="ScalingWebRole" min="1" max="10" />
</actions>
</rule>
</constraintRules>
<reactiveRules>
<rule name="Scale up when queue is long" enabled="true">
<actions>
<scale target="ScalingWebRole" by="1" />
</actions>
<when>
<greaterOrEqual operand="QueueLength_Avg" than="5" />
</when>
</rule>
<rule name="Scale down when queue is short" enabled="true">
<actions>
<scale target="ScalingWebRole" by="-1" />
</actions>
<when>
<less operand="QueueLength_Avg" than="5" />
</when>
</rule>
</reactiveRules>
<operands>
<queueLength alias="QueueLength_Avg" aggregate="Average" queue="auditqueue" timespan="00:01:00" />
</operands>
</rules>
Wasabi publishes a lot of diagnostic information, but you have to enable it. In the role where you are hosting the autoscaler, update the app.config to include the following:
<system.diagnostics>
<sources>
<source name="Autoscaling General" switchValue="All">
<listeners>
<add name="AzureDiag" />
</listeners>
</source>
<source name="Autoscaling Updates" switchValue="All">
<listeners>
<add name="AzureDiag" />
</listeners>
</source>
</sources>
<sharedListeners>
<add type="Microsoft.WindowsAzure.Diagnostics.DiagnosticMonitorTraceListener, Microsoft.WindowsAzure.Diagnostics, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35"
name="AzureDiag"/>
</sharedListeners>
<trace>
<listeners>
<add type="Microsoft.WindowsAzure.Diagnostics.DiagnosticMonitorTraceListener, Microsoft.WindowsAzure.Diagnostics, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35"
name="AzureDiagnostics">
<filter type="" />
</add>
</listeners>
</trace>
</system.diagnostics>
Then, if your role is configured to upload the log entries to the WADs table (Windows Azure Diagnostic table), you will see the entries there.
More info about the log entries that Wasabi generates here.
With the assistance provided by the diagnostics (thanks #Julian Dominguez), it became clear that the first time the autoscaling block was trying to hit the service (to check the queue length), it was failing. This is because the system was looking for the certificate in the CurrentUser store rather than the LocalMachine store.
Once I changed that setting in the configuration, it started to work.

Wasabi Azure constraint rules not enabled

I have a hosted service on Windows Azure and I am trying to use Wasabi Azure rules to affect scaling, however my rules are not being enforced. Both my rules.xml and serviceinformation-store.xml have been uploaded to my autoscaling Azure storage container. Here is my rules.xml
<rules xmlns="http://schemas.microsoft.com/practices/2011/entlib/autoscaling/rules" enabled="true">
<constraintRules>
<rule name="default" enabled="true" rank="1" description="The default constraint rule">
<timetable startDate="2012-11-05" startTime="05:34:00" utcOffset="-07:00" duration="02:00:00">
<daily/>
</timetable>
<actions>
<range min = "4" max="4" target="myWebService"/>
</actions>
</rule>
<rule name="ScaledownAtBusinessClose" enabled="true" rank="1" description="Scale down after hours">
<timetable startDate="2012-11-02" startTime="10:25:00" utcOffset="-07:00" duration="02:00:00">
<daily/>
</timetable>
<actions>
<range min ="1" max="1" target="myWebService" />
</actions>
</rule>
</constraintRules>
</rules>
Here is my service-information store
<serviceModel xmlns="http://schemas.microsoft.com/practices/2011/entlib/autoscaling/serviceModel">
<subscriptions>
<subscription name="Windows Azure MSDN - Visual Studio Ultimate" subscriptionId="<myid>" certificateThumbprint= "<mythumbprint>" certificateStoreName="My" certificateStoreLocation="CurrentUser" >
<services>
<service dnsPrefix="xxxxxxxxx" slot="Production" scalingMode="Scale">
<roles>
<role alias="myWebService" roleName="myWebService" wadStorageAccountName="<myaccount>"/>
</roles>
</service>
</services>
<storageAccounts>
<storageAccount alias="<myaccount>" connectionString="DefaultEndpointsProtocol=https;AccountName=<acctname>;AccountKey=<accountkey>">
<queues>
</queues>
</storageAccount>
</storageAccounts>
</subscription>
</subscriptions>
</serviceModel>
I start the web role with one instance but at 5:34 pm MST, the hosted service (myWebService) does not scale up to 4 instances.
In the project references I have Microsoft.WindowsAzure.Diagnostics, Microsoft.WindowsAzure.ServiceRuntime, Microsoft.WindowsAzure.StorageClient

Resources