I am using azure YAML pipelines to deploy my .Net application. Generally, a deployment uses to complete in 30 min for one server.
But, for the past three days, it's taking almost 90 mins for deployment.
My org network is good and in the Azure status check pipelines are in advisory mode with below message "expect start time delays up to 30 minutes for macOS hosted pipelines during peak hours"
Is there any recent update from Microsoft related to YAML pipeline performance?
You could check this page: Azure DevOps Status, if Azure DevOps has event issue, we could check latest news in the Status page.
And I found this event issue: Advisory: expect start time delays up to 30 minutes for macOS hosted pipelines during peak hours
The latest news and workaround is: Customers will continue to see daily queueing during peak hours for macOS pools on Azure Pipelines due to a capacity shortage. We’re adding additional capacity over the next 3 months. Until the shortage is resolved, customers should expect queueing during peak hours.
As a workaround, customers can add self-hosted macOS agents to add new capacity for running their Pipelines: https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/v2-osx?view=azure-devops
Related
I need to deploy an Azure API Management using Arm Template. It took more than 60 minutes to deploy. I am using microsoft hosted agent.
I assume it's because the status of the API-M is still in activating status even after 60 minutes.
The problem is, the pipeline would throw an error for anything that takes more than 60 minutes. Is there any solution/ workaround to deploy API-M using azure pipeline so it does not throw any errors in the pipeline?
60 minutes sounds a bit long for APIM to activate. The documentation mentions 30-40 minutes, but I guess there are situations where it can take as long as you experience.
To avoid errors for long running jobs in Azure devops you would need to increase the timeout of the running jobs. The max timeout (also the default) you can set depends on what type of agents you use (self hosted or microsoft hosted) as well as what type of project you have (public or private):
To avoid taking up resources when your job is unresponsive or waiting too long, it's a good idea to set a limit on how long your job is allowed to run. Use the job timeout setting to specify the limit in minutes for running the job. Setting the value to zero means that the job can run:
Forever on self-hosted agents
For 360 minutes (6 hours) on Microsoft-hosted agents with a public project >and public repository
For 60 minutes on Microsoft-hosted agents with a private project or private repository (unless additional capacity is paid for)
It looks like you are using microsoft hosted agents from your screenshot, and since it times out after 60 seconds I guess you have a private repository. Your options then is either to
Use self hosted agents
Buy Microsoft-hosted parallel jobs., which will increase your limit to 360 minutes also for private repositories.
My dev team is using Azure Devops' Microsoft hosted build agent, but would like to have a higher performing build agent. The current build time for us is sometimes taking more than 5 mins which is not desirable.
We would prefer not to go self-hosted route as then we will require the task of managing our own build agents. We simply do not have the resources for such a task at this stage.
Is it possible to upgrade to a higher performing Microsoft-hosted build agent on Azure DevOps?
It could simply be that we need to optimise how we a structuring our builds, and then will improve the build time - but would like to know if such an option available.
Thank you!
The answer is No!
Please see below answer from Microsoft site.
We can't increase the memory, processing power, or disk space for Microsoft-hosted agents, but you can use a self-hosted agent that is hosted on a machine that has your desired specifications.
You can also consider using parallel jobs to build your pipeline as a workaround.
This blog Reducing Longer Build Times in CI Pipelines — Parallel Builds & Build Agents with Azure DevOps might be helpful to you.
Looking at docs there is no a way to get better machine:
Microsoft-hosted agents:
Have the above software. You can also add software during your build or release using tool installer tasks.
Provide at least 10 GB of storage for your source and build outputs.
Provide a free tier:
Public project: 10 free Microsoft-hosted parallel jobs that can run for up to 360 minutes (6 hours) each time, with no overall time limit per month. Contact us to get your free tier limits increased.
Private project: One free parallel job that can run for up to 60 minutes each time, until you've used 1,800 minutes (30 hours) per month. You can pay for additional capacity per parallel job. Paid parallel jobs remove the monthly time limit and allow you to run each job for up to 360 minutes (6 hours). Buy Microsoft-hosted parallel jobs.
Run on Microsoft Azure general purpose virtual machines Standard_DS2_v2
Run as an administrator on Windows and a passwordless sudo user on Linux
(Linux only) Run steps in a cgroup that offers 6 GB of physical memory and 13 GB of total memory
and what is important in your context:
If Microsoft-hosted agents don't meet your needs, then you can deploy your own self-hosted agents.
At the moment of writing this (04/06/2020) there is a partial solution for your needs. So this not not entirely Microsoft Hosted agents but this is sth betwwen Microsoft Hosted agents and self hosted agents. It is a scale set:
We are previewing a new feature called scale set agents which pairs the convenience and elastic capacity of the Microsoft-hosted agents with the control and flexibility of self-hosted agents. With this preview, we now enable you to manage agents to your specification, completely automated, in your Azure subscription. You may want to consider using scale set agents instead of Microsoft-hosted or self-hosted agents when you:
need more memory, more processor, more storage, or more I/O than what we offer in native Microsoft-hosted agents
do not want to whitelist a large number of IP addresses within your corporate firewall to enable Microsoft-hosted agents to communicate with your servers
need more Microsoft-hosted agents than we can provide to meet your large scale needs
need the ability to partition Microsoft-hosted parallel jobs to individual projects or teams in your organization
do not want to run dedicated agents around the clock but instead want to de-provision agent machines that are not being actively utilized
To use scale set agents, you will first create a VM scale set in your Azure subscription, and then create an agent pool in Azure Pipelines to point to that scale set. Azure Pipelines will automatically scale this pool based on the number of pending jobs and the number of idle machines that you wish to maintain at all times. Azure Pipelines will also install the agent for you on these virtual machines. For more information, see scale set agents. As you preview the feature, please include your feedback on the documentation page.
I have a Azure Scheduled Web Job on Azure WebApp. Inside web job, it downloads the excel file from azure blob and read (25 - 30 k records in a file), then it iterate through loop to insert's records in database. Now, when we schedule (Schedule time: 18 hrs) first time the job ran successfully and processed all records from file without any interruptions (between this time site was accessing). But, next time it fails (Aborted) without any reason (No one was accessed site. may be in idle state).
WebJob Abort Status
There is no any indications when i looked into logs. However, first schedule ran successfully and shows success status later all are in Aborted status, Why?
Success
Also done with below settings in webapp:
stopping_wait_time - 3600
WEBJOBS_IDLE_TIMEOUT - 3600
SCM_COMMAND_IDLE_TIMEOUT - 121
Per Azure's documentation,
Always On. By default, web apps are unloaded if they are idle for some
period of time. This lets the system conserve resources. In Basic or
Standard mode, you can enable Always On to keep the app loaded all the
time. If your app runs continuous web jobs, you should enable Always
On, or the web jobs may not run reliably.
So there is no promise from Azure Web Apps for your WebJob to continue running more than 20 minutes. If you need to rely on that you'll need to enable Always On. To enable it. See below.
In the Azure Portal, navigate to your App Services.
Click Application Settings in the SETTINGS menu.
Turn Always On to on, then click Save.
Here the picture:
If you are using the free tier of Azure, maybe this thread will help you: How can I keep my Azure WebJob running without "Always On".
I have one big task to do every day, with no need to scale, that takes about 30 minutes and is DB, processor and memory intensive.
This means actual 16h/month of computation time.
WebJobs require constantly running WebSite 744h/month
WebRole is also constantly running 744h/month
Azure Batch - suited for scaled storage input - storage output
processing (or that is how I understand it)
Stopped cloud service still cost you. Setting instance count to 0 is not available. And paying for 728h/month unused computation time looks like madness. Only thing I can imagine is automatic deployment of cloud service every day and automatic deletion of deployment once task is finished, but this also looks like madness.
Are there any options for this scenario in Azure?
Cloud service will be charged continuously until the deployment is deleted. Yes you can delete it every day and redeploy...
Azure VMs in Stopped (Deallocated) status, does not incur any charge. You can shut them down in portal or by script when you don't need them.
I think there is a large difference in billing if you only use it 62h/month. Would you consider switch this deployment to VM? WorkerRole and VMs can be placed on the same subnet, they can still connect to each other.
I could see primarily there are 3 options in Windows Azure to schedule jobs. Batch, scheduler and web jobs. Is there any link or video explaining what are the differences and what to use when and benefits?
Thanks in advance
So far I didn't see anything official from azure.com or msdn, so let me take a stab.
Azure Batch - is a way to run parallel (typically compute intensive) HPC style job on the cloud. Batch pitches the value of parallel job running as a service so you don't worry about provisioning/managing large cluster. A typical scenario is, go encoding those 10K H.264 videos from 1080p to 720p - instead of spinning up 200 VMs you just configure the command line and specify the location of those 10k videos (blobs).
Azure Scheduler is a way to run recurring job at specified time. It's Windows Task Scheduler in cloud. For example, start a cloud service 8AM every weekday and shut it down at 6PM.
Azure Web Job is focusing on doing background job for Azure Website. It's working daemon web server farm in cloud. An example is - compress all images uploaded from the webpage.
To add to Yiding answer, Azure Scheduler and Azure WebJobs actually work together and complete each other in that sense.
Azure WebJobs will host your code/executable that is doing the work.
Azure Scheduler will schedule when to run your work --> WebJob.
To start create a scheduled Azure WebJob which will create both resources.