Is there any details on how to achieve gradual rollout for application (running behind Azure Front door) based on origin health? Something similar to compute where we scale out/in based on some metric.
I am looking for dynamically increase the traffic to an origin based on
scenario 1 - Time
scenario 2: increase step/weighted value based on custom value computed by region. every X hr if region is still healthy increase traffic by Y %. and if health of region goes down, decrease traffic by Y %.
Related
I have a request pipeline that goes app-service > app-service > cosmosDB (then returns). We are currently in performance testing for said pipeline and hitting it with about 10k requests per minute. Something that we've noticed is that the longer the test goes on, the worse performance gets, so our response was to enable autoscale on the app services and on cosmosDB. Unfortunately, this hasn't impacted performance, and I'd like to see if the autoscale is even working.
TL;DR
Is there a log for how many instances of a resource are active [from autoscale] at a given time?
Is there a log for how many instances of a resource are active [from autoscale] at a given time?]
Thanks #MarkBrown for the Comment.
I have created Azure App Service and enabled auto scale based on the request’s metric rule.
The dotted line shows the maximum request that will enable additional instances of app service whenever the web app reaches request of 70.53%. Blue line shows the current requests of the web app.
Whenever the app reaches 70% increase the count by 1 with maximum instances up to 4
Added another rule for scale in
We can see the capacity is still 1. Run the load test on Web App. You can see App service is scaled up to 3 instances after load test.
Check the Auto Scale history.
View the Web apps activity history in Azure monitor =>Activity Log.
Autoscaling in Cosmos DB
In Azure Cosmos DB Number of Ru’s are scaled. We can mention the max number of RU’s required for our Cosmos DB.
Enabling Autoscale for the CosmosDB
Navigate to your Azure Cosmos DB Account => Scale => New Container.
Cosmos DB will scale from 5000 to 1/10th of its value (500-5000 RU's). If we work for more hours, scale will be up to the mark. When our workload is not running, we can scale down to less RU's.
Check the consumption of RU’s in Metrics
Currently the percentage is 14%, Normalized RU consumption chart varies between 0-100 % and it gives information on how much of the current throughput we have provisioned and is utilized.
100% means max RU that we have mentioned. If our consumption in the above chart is linear and above 66% on average, we don’t need autoscaling, if it is variable, we need to auto scale.
And how to check how many instances does cosmos dB has scaled to after enabling auto scale?
Provisioned Throughput is the instances or scaling that cosmos dB has done. In the chart below auto scale has provisioned 500 throughputs
Auto scale max throughput – 1k
Provisioned Throughput- 500k
We can also send Azure cosmos DB logs to log analytics to query and get additional insights.
I Pay-As-You-Go subscription with S3 plan
Having scale out settings like Min 2 instance Max 4. Based on CPU and Memory consumption it will add 1 more instance
Why first instance showing 82% CPU utilization while other is <40% CPU utilization ? I would expecting equal utilization.
based on R&D i come to know because application sticky session instance does not sharing equal utilization.
Automatic scaling is implemented in accordance with certain rules, and it is not clear which instance is accessed or used. You can achieve load balancing in the following ways, which should meet your needs.
Solution:
How Azure load balances scaled out App Service plan?
The reason of the problem
Please read below tips, when you configure the settings.
No metric rules defined; click Add a rule to scale out and scale in your instances based on rules. For example: 'Add a rule that increases instance count by 1 when CPU percentage is above 70%'.
I have a Web APP in Azure PAAS service App Service. App service plan uses a unit named as "ACU".
Is there any way by which i can calculate the number of ACU required by my web app.
As i have to do billing calculations so i need to prove that instance size chosen is based on some logical calulation.
Q. What is the Azure Compute Unit (ACU)?
A. The concept of the Azure Compute Unit (ACU) provides a way of comparing compute (CPU) performance across Azure SKUs.
The Azure Compute Unit (ACU) is used to help understand the
relative compute performance between different Azure series and size
VMs. It is based on the A0 (extra small) having a value of 50. A VM
with an ACU of 100 has twice the compute of a VM with an ACU of 50. A
VM with an ACU of 200 would be twice that of a VM with an ACU of 100
and so on.
So you should analyse the CPU Usage and Memory Usage.
For seeing that, you can go to your app service-->Diagnose and solve problems-->Availability and Performance.
The most important thing is NOT just considering ACUs for scalability and cost.
So you can Scale Up and Scale Out your App Service for meeting the demands during peak use and getting the final cost.
Vertical Scaling: For scaling up, since you don't initially know what kind of VM should suffice, you can start with a basic or intermediate one (not a very powerful one). General purpose Type VMs (50-210 ACUs) suffice in most cases. Link: https://learn.microsoft.com/en-us/azure/app-service/manage-scale-up
Horizontal scaling: It's good practice to also scale out your App service plan. You can autoscale and set up autoscaling rules or go the manual route.
Link: https://learn.microsoft.com/en-us/azure/azure-monitor/autoscale/autoscale-get-started
Next you should monitor your CPU Usage and Memory Usage as suggested by Doris.
Additionally you can also go to Run History and see your usage:
Based on your findings you can tweak the number and kind of VMs you need.
I have created a single instance of azure sql database with DTU-based purchase model. I am trying to setup some metric alerts for this database so once I use this Db for some application I can have better picture of how DTU's are actually being used, how many / much DTU's I will need etc. This information will help me to Configure right Pricing tier for this database.
Pricing tier info
Before you set the alerts, you should first use the Metric to monitor databases using the Azure portal.
In the Azure portal, you can monitor an individual databases utilization by selecting your database and clicking the Monitoring chart. This brings up a Metric window that you can change by clicking the Edit chart button. Add the following metrics:
CPU percentage
DTU percentage
Data IO percentage
Database size percentage
You can also configure alerts on the performance metrics. Click the Add alert button in the Metric window. Follow the wizard to configure your alert. You have the option to alert if the metrics exceed a certain threshold or if the metric falls below a certain threshold.
For example, if you expect the workload on your database to grow, you can choose to configure an email alert whenever your database reaches 80% on any of the performance metrics. You can use this as an early warning to figure out when you might have to switch to the next highest compute size.
The performance metrics can also help you determine if you are able to downgrade to a lower compute size. Assume you are using a Standard S2 database and all performance metrics show that the database on average does not use more than 10% at any given time. It is likely that the database will work well in Standard S1. However, be aware of workloads that spike or fluctuate before making the decision to move to a lower compute size.
Azure gives you the examples are setting the threshold value of the DTU/CPU metric 80%. You also can get this in this document: Create an alert rule on a metric with the Azure portal.
So you can reference this value on threshold. If your database CPU or DTU percentage is over 80% or with slow performance for a long time, you should consider to scale up the DTU-based price tier.
Hope this helps.
Last month I noticed a large increase of my Azure bill due to outgoing bandwidth. I used 1800GB of outgoing data vs ~200GB in previous periods. After some research I found that this was caused by the Azure Front Door service that I enabled last month and I was not aware of the additional indirect costs associated with the service.
I'll provide my analysis of the "issue" below to hopefully perevent others from making the mistake I made.
Azure Front Door allows for quick failover between groups (so called "pools") of web applications based on the health of the applications in the pool. A typical failover scenario would be between different regions. If one region has an issue, you failover to the other region.
The mechanism by which Front Door determines the health of an application is by sending a HTTP request where a 200 OK result is considered healthy.
The moment you enable Azure Front Door on your backend, it starts checking the health of your backend application and you potentially start paying so I performed some analysis and these are my findings:
[Range 4 hours, granularity 1 minute]
08:05 Started the webapp.
08:30 Enabled Front Door Service with default settings (Interval=30sec, Sample size=4, Successful samples required=2). Notice the immediate growth in number of requests from 0 to ~140 per minute.
09:03 Decreased the health probe interval from 30sec to 15sec. Notice the immediate growth in requests.
09:40 Quadrupled the body size of the health probe endpoint from 30KB to 119KB. Notice the immediate growth in bandwidth.
09:55 Reduced the body size of the health probe endpoint to 0KB. Notice the immediate drop in bandwidth.
10:08 Increased the health probe interval from 15sec to 90sec. Notice the immediate drop in requests.
It seems like the bandwidth is charged as outgoing bandwidth of the App Service (or whatever endpoint service is used) on top of the bandwidth of the Front Door service. I think this is because the Azure Front Door is a global service and therefore not region bound. These "hidden" charges are not mentioned on the pricing page
This default landing page of an Azure Function App is 126KB:
Takeaways
By default, Azure Front Door seems to visit your endpoint 140 times per minute and generates 20MB of traffic per minute (with a 30KB body). That is 27GB, or, EUR 1,90 per day (EU/US regions).
Don't use the default function app landing page as your health probe endpoint (or any large home page). I'm not sure about the best practices but I would think a custom endpoint that actually does some health checking and returns an empty body would be best.
Choose your interval wisely. Double the interval = double the bandwidth costs.
Set AzureWebJobsDisableHomepage to true in your app service config to remove the landing page from /, this saves on response size which saves on egress cost.