I created a pool in Azure Batch (from the Azure portal) with Auto scale activated.
I also defined a formula where the initial number of node is set to 0. This number will ramp up according to number of active tasks and will go back to 0 if there no task remaining.
My problem is that the minimum evaluation interval for the formula is 5 minutes, which means that in the worst case I have to wait at least 5 minutes (plus the time it takes for the nodes to boot and execute the start task) before a task can be assigned to a node.
I would like to apply the formula on the pool on demand by using the REST API (for instance right after adding a job).
According to the API documentation:
https://learn.microsoft.com/en-us/rest/api/batchservice/pool/evaluate-auto-scale
You can evaluate a formula but it will not be applied on the pool.
https://learn.microsoft.com/en-us/rest/api/batchservice/pool/enable-auto-scale
You can enable automatic scaling for a pool but if it is already enabled you have to specify a new auto scale formula and/or a new evaluation interval.
If you specify a new interval, then the existing auto scale evaluation schedule will be stopped and a new auto scale evaluation schedule will be started, with its starting time being the time when this request was issued.
You can disable and then re-enable the autoscale formula, but note the call limits on the enable API. However note that if you are trying to do this frequently on the order of less than the minimum evaluation period that thrashing a pool faster than the underlying infrastructure can allocate resources does not provide any benefits.
Related
With an Azure Data Factory "Tumbling Window" trigger, is it possible to limit the hours of each day that it triggers during (adding a window you might say)?
For example I have a Tumbling Window trigger that runs a pipeline every 15 minutes. This is currently running 24/7 but I'd like it to only run during business hours (0700-1900) to reduce costs.
Edit:
I played around with this, and found another option which isn't ideal from a monitoring perspective, but it appears to work:
Create a new pipeline with a single "If Condition" step with a dynamic Expression like this:
#and(greater(int(formatDateTime(utcnow(),'HH')),6),less(int(formatDateTime(utcnow(),'HH')),20))
In the true case activity, add an Execute Pipeline step executing your original pipeline (with "Wait on completion" ticked)
In the false case activity, add a wait step which sleeps for X minutes
The longer you sleep for, the longer you can possibly encroach on your window, so adjust that to match.
I need to give it a couple of days before I check the billing on the portal to see if it has reduced costs. At the moment I'm assuming a job which just sleeps for 15 minutes won't incur the costs that one running and processing data would.
there is no easy way but you can create two deployment pipelines for the same job in Azure devops and as soon as your winodw 0700 to 1900 expires you replace that job with a dummy job using azure dev ops pipeline.
Doing some tests, I could see that having an Azure Integration Runtime (AIR) allowed us to reduce considerably the amount of time required to finish a pipeline.
To fully understand the use of this configuration and its billing as well, I've got these questions. Let's assume I've got two independent pipelines, all of their Data Flow activities use the same AIR with a TTL = 10 minutes.
The first pipeline takes 7 minutes to finish. The billing will be (if I understand well):
billing: time to acquire cluster + job execution time + TTL (7 + 10)
Five minutes later, I trigger the second pipeline. It will take only 3 minutes to finish (I understand it will also use the same pool as the first one). After it concludes, the TTL is setting up to 10 minutes again or is equal to 2 minutes
10 - 5 -3 (original TTL - start time second pipe - runtime second pipe), in this case, what will happen if I trigger a third pipeline that could take more than 2 minutes?
What about the billing, how is it going to be calculated?
Regards.
Look at the ADF pipeline monitoring view and find all of your data flow activity executions.
Add up that total data flow activity execution time.
Now add the TTL value for that Azure IR you were using to that total.
That is the total time you will be billed.
I have an Azure Batch job where each node has a single task to run. Some of the tasks finish quickly, some run for much longer. I am using an auto-pool with around 100 nodes, so I want to give back the unneeded nodes ASAP.
Is there a built-in way to accomplish this pattern of node usage, or should I use auto-scaling where I set the number of dedicated/low-pri nodes to the number of pending tasks? If I do use auto-scaling, will Azure Batch always remove the idle nodes first? I don't want the active nodes to be disturbed. Thanks.
Use the auto scaling rules, this scenario is exactly what they're for. There is an option to only remove the node once it's tasks are complete - see NodeDeallocationOption. Note that the minimum autoscale rule evaluation interval is 5 minutes so you will have a slight delay between adding your tasks and the scaling to kick in.
Check out https://learn.microsoft.com/en-us/azure/batch/batch-automatic-scaling for some of the autoscale formula.
We would like to set up Azure auto-scaling based on specific time of the day. E.g. on 7:00 we would like to increase number of instances and at 17:00 we would like to decrease them.
We are aware that we can set to scale up by some other metrics (CPU, number of messages in queue, etc.), but this has some negative impacts for us - starting of a new instance takes some time and also w3wp warm-up takes some time too. And we need to have instances ready immediately when high load comes.
Is there any way to set auto-scaling on specific time of the day (from 7:00 to 17:00) and specific days of week (working days).
You could inculcate the following general guidelines based on your requirement:
Scale based on a schedule
In addition to scale based on CPU, you can set your scale differently for specific days of the week.
Click Add a scale condition.
Setting the scale mode and the rules is the same as the default condition.
Select Repeat specific days for the schedule.
Select the days and the start/end time for when the scale condition should be applied.
Scale differently on specific dates
In addition to scale based on CPU, you can set your scale differently for specific dates.
Click Add a scale condition.
Setting the scale mode and the rules is the same as the default condition.
Select Specify start/end dates for the schedule.
Select the start/end dates and the start/end time for when the scale condition should be applied.
Refer Get started with Autoscale in Azure for more details.
As general Autoscaling guidelines:
When you can predict the load on the application well enough to use scheduled autoscaling, adding and removing instances to meet anticipated peaks in demand. If this isn't possible, use reactive autoscaling based on runtime metrics, in order to handle unpredictable changes in demand. Typically, you can combine these approaches. For example, create a strategy that adds resources based on a schedule of the times when you know the application is most busy. This helps to ensure that capacity is available when required, without any delay from starting new instances. For each scheduled rule, define metrics that allow reactive autoscaling during that period to ensure that the application can handle sustained but unpredictable peaks in demand.
In the new Azure portal under
Cloud services > Scaling blade > Scale Profile > choosing Recurrence
There is a range of instances to choose, if I understood correctly.
Can someone explain the meaning of a "range", since in certain hour I would expect to have a fixed number of instances.
New portal
The range defines the minimum and maximum number of instances that will be running at any given time. For example, if you're configuring a scale profile based on CPU load you will have CPU percentage rules in place that will add/remove instances based on CPU thresholds you define. The scale profile will add/remove instances as needed within the range you specified.
The Range in the new Portal has 2 sliders Minimum and Max Number of Instances
Depending on the trigger you will set for this to Happen
Manual Scale : The Nummber of Instances will be set based on your Choice
CPU : You set the Minimum and the Max for Instances and Besed on the CPU Untalization Azure will Set the Number of instances
Other Measures : Same as CPU but for Queuing, Traffic, Disk I/O Etc.
There is "Scale Profile" and also a sub scaling call "rule".
The rule determine the scale within the range of the "Scale Profile".
target range for the metric. For example, if you chose CPU percentage, you can set a target for the average CPU across all of the instances in your service. A scale out will happen when the average CPU exceeds the maximum you define, likewise, a scale in will happen whenever the average CPU drops below the minimum
https://learn.microsoft.com/en-us/azure/monitoring-and-diagnostics/insights-how-to-scale