We are using TFVC as VCS, our repo is located in Azure Repos, we use Azure DevOps for CI/CD.
There is a pipeline building changes of Main branch and "Gated check-in" option is enabled.
We use hosted agents deployed on our build servers. Currently there are 3 build environments and 1 agent is deployed to each build env. As there are many developers working on the project - many concurrent check-ins became a bottleneck, that's why we have now 3 agents in a single agent pool, the pipeline has 1 job and it is using agent pool with 3 agents.
The problem is that when several developers check-in changes only 1 build is running, all other in a queue have to wait until it finishes, despite the fact that other 2 agents are online and Idle in the same pool.
The build is queued with the message "Job is pending". Interesting thing is that if we run the pipeline from azure devOps they run concurrently, but when developers check-in changes from Visual Studio - they're queued.
This is due to the "gated checkin" option being enabled. Gated checkins do not run in parallel.
This answer provides more details.
In short, gated checkins take the latest changeset in the branch + the code being checked in and then build the code. If the build fails then the checkin will fail. If you want to have parallel builds then you need to disable gated checkins and use a CI build instead.
Related
I am working with 3 pipes based off 3 different repos within Azure DevOps. All 3 pipes deploy to the same cloud VM and all 3 have a set of E2E automated tests which are run after the deployment stage. I am currently having issues where 1 pull request is completed kicking off a deployment, then while the E2E tests are running a different pipe is kicked off, this causes services to stop on the VM during deployment making the E2E tests fail unusually.
Can I setup my pipe to only start a deployment + test if no other builds are already running against the target VM?
Cheers
If these stages in release pipeline are set to run one after another, there will be no issues.
However, new push will trigger a new release and break the ongoing deployment, we suggest that you review this doc: Release gates and approvals overview, so you can try to set up release deployment control using gates or define approvals and checks.
Does azure pipelines allow custom action like AWS codepipeline?
I want to create a job worker that will poll azure pipeline for job requests for this custom action, execute the job, and return the status result to azure pipeline.
Something similar to - https://docs.aws.amazon.com/codepipeline/latest/userguide/actions-create-custom-action.html
Tasks are the building blocks for defining automation in a build or release pipeline in Azure DevOps. There are many built-in tasks to enable fundamental build and deployment scenarios. If the existing tasks don't satisfy your needs, you can always build a custom task. Check Task types & usage for more details.
In addition, Visual Studio Marketplace offers a number of extensions; each of which, when installed to your subscription or collection, extends the task catalog with one or more tasks. Furthermore, you can write your own custom extensions to add tasks to Azure Pipelines.
Azure Pipeline Agents
When your pipeline runs, the system begins one or more jobs. An agent is computing infrastructure with installed agent software that runs one job at a time.
You have two options here to choose from: Microsoft-hosted agents or Self-hosted agents
An agent that you set up and manage on your own to run jobs is a self-hosted agent. Self-hosted agents give you more control to install dependent software needed for your builds and deployments. Also, machine-level caches and configuration persist from run to run, which can boost speed.
However, before you install a self-hosted agent you might want to see if a Microsoft-hosted agent pool will work for you. In many cases, this is the simplest way to get going.
With Microsoft-hosted agents, maintenance and upgrades are taken care of for you. Each time you run a pipeline, you get a fresh virtual machine. The virtual machine is discarded after one use. Microsoft-hosted agents can run jobs directly on the VM or in a container. Azure Pipelines provides a pre-defined agent pool named Azure Pipelines with Microsoft-hosted agents.
You can try it first and see if it works for your build or deployment. If not, you can use a self-hosted agent. Check this doc for more details.
I will pull the agent queue from my custom job worker and process the job. Is that possible in azure pipelines?
Based on my understanding of code pipeline and Azure devops, I am afraid what you said should be meaningless.
According to the document Create and add a custom action in CodePipeline, we could to know that:
AWS CodePipeline includes a number of actions that help you configure
build, test, and deploy resources for your automated release process.
If your release process includes activities that are not included in
the default actions, such as an internally developed build process or
a test suite, you can create a custom action for that purpose and
include it in your pipeline.
But for Azure devops, we do not need to create a job worker that will poll CodePipeline for job requests for this custom action. That because the whole process of build/release can be customized. We do not need to add a job worker for additional custom actions.
Azure devops provide a lot of templates when we create the pipeline, we could modify the pipeline directly in the pipeline to add/remove or update the task:
Even we can completely start with a blank pipeline and completely customize the entire build/release process.
So, we do not need to create a job worker for the custom action, just modify your pipeline directly.
The azure pipeline build task for github tends to build all the commits in a PR sent. I would like it to build only the last commit. That is I would like the other build jobs started for the previous commits in a PR (except for the last one) to be cancelled automatically without manual intervention. This is to reduce the number of build jobs queued at the same time for a given PR.
How do I achieve this in Azure Pipelines for CI/CD. I can do it in travis easily and would like to find out if it is available in Azure Pipelines for CI/CD and if so how to achieve it.?
I believe this behavior to cancel existing queued pipelines on PR update became the default behavior for Azure-DevOps with the Nov-12-2018 release with this feature. It doesn't look like it is flagged for 2019 On-Premise, so I'd expect this won't be available On-Premise for a while.
My team has an Azure App Services Web App where we house three major components:
Our main Node.js server and API, which is at the root
A secondary API, which is in a virtual directory
Our front-end web app (also served from a Node.js server), which is in another virtual directory
Each of those three components is maintained in its own git repo in VSTS. Additionally, the Web App has three slots: dev, ppe, and prod.
We are trying to move our build processes out of Azure and into VSTS. What we'd like to be able to do is the following:
When there's a new commit to master in any of the three repos, create a dev build and deploy it directly to the appropriate virtual directory in the dev slot.
When a component is ready to be released - whether that means a new commit in a special RELEASE branch or manually triggering a release process - create a production build, deploy it to ppe and, on user approval, swap the ppe and prod slots.
The complication here is that, when any component is deployed to ppe, we also need to deploy the latest released versions of all three components to ppe, since Azure does not have the ability to swap virtual directories independently.
What I currently have is the following:
A build process for each of the three repositories, which is triggered on commits to master or RELEASE. It creates both a development build and a production build and publishes them.
A dev release process that is triggered on any new builds of master in any of the three repositories. It takes the latest dev build from master from all three repos and deploys them to their appropriate virtual directories in dev.
A production release process that is triggered on any new build of RELEASE in any of the three repositories. It takes the latest production build from RELEASE from all three repos, deploys them to the appropriate virtual directories in ppe and, on user approval, swaps ppe and prod.
This works, but it seems pretty clunky, has a lot of wasted work, and it doesn't feel like we're exactly taking advantage of the power of the VSTS build/release pipeline. Is there a better or more accepted way of doing this?
There is Filters based on the artifacts feature in environment triggers of release, so you can base on the build tag(s) to trigger corresponding environments.
Regarding build, you can set the build tag(s) per to current branch by calling logging commands through PowerShell task (##vso[build.addbuildtag]build tag)
Regarding ppe scenario, I recommend that you can create a new CI build definition just for ppe related branch and build all components (get others source code by calling git clone command through Command Line task or other tasks (e.g. PowerShell)), then publish results of them, after that deploy them to corresponding slots and Virtual directory in release.
In my build definition I deploy to multiple Azure cloud services and would like to deploy in parallel, however the build definition in 2015 doesn't allow steps to be run this way.
Is there a way I can have three groups of steps (each with a Visual Studio Build and then Azure Cloud Service Deployment step) running in parallel.
This will save me huge amounts of time in our CI/CD builds allowing for faster feedback from the builds.
Instead of deploying from a build, deploy using the Release hub.
You can define multiple release environments, then use the "Deployment Conditions" option to set multiple environments to deploy at once. However, you would need one agent per environment so that the agents can run in parallel.
Currently, there are not parallel tasks in the VSTS build and release process. However, there is a uservoice item for it. Please vote for it!
https://visualstudio.uservoice.com/forums/330519-team-services/suggestions/13481574-add-ability-to-run-build-steps-in-parallel