I have a request pipeline that goes app-service > app-service > cosmosDB (then returns). We are currently in performance testing for said pipeline and hitting it with about 10k requests per minute. Something that we've noticed is that the longer the test goes on, the worse performance gets, so our response was to enable autoscale on the app services and on cosmosDB. Unfortunately, this hasn't impacted performance, and I'd like to see if the autoscale is even working.
TL;DR
Is there a log for how many instances of a resource are active [from autoscale] at a given time?
Is there a log for how many instances of a resource are active [from autoscale] at a given time?]
Thanks #MarkBrown for the Comment.
I have created Azure App Service and enabled auto scale based on the request’s metric rule.
The dotted line shows the maximum request that will enable additional instances of app service whenever the web app reaches request of 70.53%. Blue line shows the current requests of the web app.
Whenever the app reaches 70% increase the count by 1 with maximum instances up to 4
Added another rule for scale in
We can see the capacity is still 1. Run the load test on Web App. You can see App service is scaled up to 3 instances after load test.
Check the Auto Scale history.
View the Web apps activity history in Azure monitor =>Activity Log.
Autoscaling in Cosmos DB
In Azure Cosmos DB Number of Ru’s are scaled. We can mention the max number of RU’s required for our Cosmos DB.
Enabling Autoscale for the CosmosDB
Navigate to your Azure Cosmos DB Account => Scale => New Container.
Cosmos DB will scale from 5000 to 1/10th of its value (500-5000 RU's). If we work for more hours, scale will be up to the mark. When our workload is not running, we can scale down to less RU's.
Check the consumption of RU’s in Metrics
Currently the percentage is 14%, Normalized RU consumption chart varies between 0-100 % and it gives information on how much of the current throughput we have provisioned and is utilized.
100% means max RU that we have mentioned. If our consumption in the above chart is linear and above 66% on average, we don’t need autoscaling, if it is variable, we need to auto scale.
And how to check how many instances does cosmos dB has scaled to after enabling auto scale?
Provisioned Throughput is the instances or scaling that cosmos dB has done. In the chart below auto scale has provisioned 500 throughputs
Auto scale max throughput – 1k
Provisioned Throughput- 500k
We can also send Azure cosmos DB logs to log analytics to query and get additional insights.
Related
I am following best practices document and trying to implement Auto Scaling and would like to know about pricing perspective.
Robust-Apps-for-the-cloud
I would like to utilize custom auto scale to use multiple instances. I have configured the rules as shown here:
With this, I would like more information on how this will affect the pricing for my app service plan.
Note: My App Service Plan is S2.
App Service Plans are priced based on the size and number of instances you run, and they are billed on a per second basis. For your case on the S2 plan, a single instance will cost $0.20/hour.
I see on your autoscale configuration that the minimum and default number of instances you will be running on this plan is two instances. With this, if the autoscale triggers are not hit, your App Service Plan would cost $0.40/hour.
With the configuration you shared this could run up to $0.80/hour, if maximum four instances are run after the autoscale triggers are met.
As App Service Plans are billed on a per second basis, the cost will be prorated on a per second basis for the number of instances you run.
For example:
if you were running two instances for 40 minutes, three instances for 10 minutes, and four instances for the last 10 minutes of an hour. The total cost of the App Service Plan for that hour would be roughly: $0.50 for that hour.
If you were to scale up or down your App Service Plan tier you can see more information about how this could affect pricing using this tool here:
App Service Pricing
I am using Azure SignalR service instance. SignalR service currently only supports 1000 concurrent connections per service instance per unit. If the number of concurrent SignalR connections exceed 1000, the service instances will have to be increased manually, and reduced manually as the users decrease.
Looking for a suitable solution to auto-scale (scale up and scale down) the SignalR service instances based on the demand.
If any idea, please share. Thanks.
Azure SignalR service doesn't support any auto-scaling capabilities out of the box.
If you want to automatically increase or decrease the number of units based on the current number of concurrent connections, you will have to implement your own solution. You may for example try to do this using a Logic App as suggested here.
The common approach is otherwise to increase the number of units manually using the portal, the REST API or the Azure CLI.
They solved the disconnection issue when scaling, according to https://github.com/Azure/azure-signalr/issues/1096#issuecomment-878387639
And for the auto-scaling feature they are working on it, and in the mean-time here are 2 ways of doing so:
Using Powershell function https://gist.github.com/mattbrailsford/84d23e03cd18c7b657e1ce755a36483d
Using Logic App https://staffordwilliams.com/blog/2019/07/13/auto-scaling-signalr-service-with-logic-apps/
Azure SignalR Service supports autoscale as of 2022 if you select premium pricing tear.
Go to Scale up on the SignalR Service and select Premium pricing
tear.
Go to Scale out and create a custom autoscale.
The examples says that you can scale up if the metric "Connection Quota Utilization" is over 70% (should be about 700 out of your 1000 connections for your first unit). You can also scale down with a similar rule. The examples says to scale down when the connection quota is under 20%.
20% from the example seems a bit restrictive, but I guess its to avoid unneeded scaling. The client connections should be closed and reconnected while scaling down, so doing so very frequently is probably a bad idea.
https://learn.microsoft.com/en-us/azure/azure-signalr/signalr-howto-scale-autoscale
Actually I am getting System.OutOfMemoryException for blob trigger azure function.Do i need to scale up or Scale out App Service Plan to fix this problem.
What is the difference between these two?
For your original question, if your function is running on consumption plan, Scale up App Service Plan of your Azure service.The plan you already have less Memory and if you have multiple functions running in App Service Plan then scale out.
From the docs,
Scale up means :
A scale up operation is the Azure Web Sites cloud equivalent of moving
your non-cloud web site to a bigger physical server. So, scale up
operations are useful to consider when your site is hitting a quota,
signaling that you are outgrowing your existing mode or options. In
addition, scaling up can be done on virtually any site without
worrying about the implications of multi-instances data consistency.
Two examples of scale up operations in Windows Azure Web Sites are:
Scale Out means:
A scale out operation is the equivalent of creating multiple copies of
your web site and adding a load balancer to distribute the demand
between them. When you scale out a web site in Windows Azure Web Sites
there is no need to configure load balancing separately since this is
already provided by the platform.
Digram depicting the difference between the two :
You need to scale up your app service plan.
"Scale up" means upgrade the capacity of the host where the app is hosted. Ex: Increase the memory from 1.75GB to 3.5GB.
"Scale out" means upgrade the capacity of the app by increasing the number of host instances.
In Short Scale Up is vertical scaling, where you add more resources to increase capacity of the underlaying Hardware/Infrastructure.
Where, Scale Out is horizontal scaling, where you add more instance of the same app to process/handle request simultaneously.
If you choose Scale Out, you will get more VMs and balance your workloads to those VMs. If you choose Scale Up, your VM will get more punch to handle current workloads. More VMs or more power to your current VM
I have created a single instance of azure sql database with DTU-based purchase model. I am trying to setup some metric alerts for this database so once I use this Db for some application I can have better picture of how DTU's are actually being used, how many / much DTU's I will need etc. This information will help me to Configure right Pricing tier for this database.
Pricing tier info
Before you set the alerts, you should first use the Metric to monitor databases using the Azure portal.
In the Azure portal, you can monitor an individual databases utilization by selecting your database and clicking the Monitoring chart. This brings up a Metric window that you can change by clicking the Edit chart button. Add the following metrics:
CPU percentage
DTU percentage
Data IO percentage
Database size percentage
You can also configure alerts on the performance metrics. Click the Add alert button in the Metric window. Follow the wizard to configure your alert. You have the option to alert if the metrics exceed a certain threshold or if the metric falls below a certain threshold.
For example, if you expect the workload on your database to grow, you can choose to configure an email alert whenever your database reaches 80% on any of the performance metrics. You can use this as an early warning to figure out when you might have to switch to the next highest compute size.
The performance metrics can also help you determine if you are able to downgrade to a lower compute size. Assume you are using a Standard S2 database and all performance metrics show that the database on average does not use more than 10% at any given time. It is likely that the database will work well in Standard S1. However, be aware of workloads that spike or fluctuate before making the decision to move to a lower compute size.
Azure gives you the examples are setting the threshold value of the DTU/CPU metric 80%. You also can get this in this document: Create an alert rule on a metric with the Azure portal.
So you can reference this value on threshold. If your database CPU or DTU percentage is over 80% or with slow performance for a long time, you should consider to scale up the DTU-based price tier.
Hope this helps.
My Aim is this:
My company has 5 webapps and some other resources running on Microsoft Azure. An e-mail is to be sent daily that would contain
CPU and memory utilization of the 5 webapps
DTU percentage of SQL DB
Observed capacity of Autoscale
Currently this is done manually by taking screenshots of the metrics. Could this be automated via API or something else? I looked into application insights API, but couldn't find the info for SQL databases and auto-scale metrics.
If someone could just lean me on the right path, that would be great. Thanks.
You can use "Azure Monitor REST API".
All metrics: Contains DTU/CPU percentage, DTU/CPU limit, physical data
read percentage, log write percentage, Successful/Failed/Blocked by
firewall connections, sessions percentage, workers percentage,
storage, storage percentage, and XTP storage percentage.
https://learn.microsoft.com/en-us/azure/sql-database/sql-database-metrics-diag-logging
and also the Azure Monitor REST API reference
https://learn.microsoft.com/en-us/rest/api/monitor/
Then simply you can use a logic app to send the email every day.