Does azure pipelines allow custom action like AWS codepipeline?
I want to create a job worker that will poll azure pipeline for job requests for this custom action, execute the job, and return the status result to azure pipeline.
Something similar to - https://docs.aws.amazon.com/codepipeline/latest/userguide/actions-create-custom-action.html
Tasks are the building blocks for defining automation in a build or release pipeline in Azure DevOps. There are many built-in tasks to enable fundamental build and deployment scenarios. If the existing tasks don't satisfy your needs, you can always build a custom task. Check Task types & usage for more details.
In addition, Visual Studio Marketplace offers a number of extensions; each of which, when installed to your subscription or collection, extends the task catalog with one or more tasks. Furthermore, you can write your own custom extensions to add tasks to Azure Pipelines.
Azure Pipeline Agents
When your pipeline runs, the system begins one or more jobs. An agent is computing infrastructure with installed agent software that runs one job at a time.
You have two options here to choose from: Microsoft-hosted agents or Self-hosted agents
An agent that you set up and manage on your own to run jobs is a self-hosted agent. Self-hosted agents give you more control to install dependent software needed for your builds and deployments. Also, machine-level caches and configuration persist from run to run, which can boost speed.
However, before you install a self-hosted agent you might want to see if a Microsoft-hosted agent pool will work for you. In many cases, this is the simplest way to get going.
With Microsoft-hosted agents, maintenance and upgrades are taken care of for you. Each time you run a pipeline, you get a fresh virtual machine. The virtual machine is discarded after one use. Microsoft-hosted agents can run jobs directly on the VM or in a container. Azure Pipelines provides a pre-defined agent pool named Azure Pipelines with Microsoft-hosted agents.
You can try it first and see if it works for your build or deployment. If not, you can use a self-hosted agent. Check this doc for more details.
I will pull the agent queue from my custom job worker and process the job. Is that possible in azure pipelines?
Based on my understanding of code pipeline and Azure devops, I am afraid what you said should be meaningless.
According to the document Create and add a custom action in CodePipeline, we could to know that:
AWS CodePipeline includes a number of actions that help you configure
build, test, and deploy resources for your automated release process.
If your release process includes activities that are not included in
the default actions, such as an internally developed build process or
a test suite, you can create a custom action for that purpose and
include it in your pipeline.
But for Azure devops, we do not need to create a job worker that will poll CodePipeline for job requests for this custom action. That because the whole process of build/release can be customized. We do not need to add a job worker for additional custom actions.
Azure devops provide a lot of templates when we create the pipeline, we could modify the pipeline directly in the pipeline to add/remove or update the task:
Even we can completely start with a blank pipeline and completely customize the entire build/release process.
So, we do not need to create a job worker for the custom action, just modify your pipeline directly.
Related
I am looking for some best practices and suggestions:
My team is creating an Azure DevOps pipeline to deploy a complex infrastructure of VNets, VMs, Azure ML workspaces, SQL databases, etc.
The pipeline uses Terraform where possible, but Powershell or AZ CLI where needed.
The pipeline works, it is version controlled, it has proper unit tests and integration tests (or at least decent ones).
However, due to the instability of Azure resourcing sometimes the pipeline will fail because, for instance:
SQL server provisioning fails
AD join of VMs fails
or other activities which are not due to bad Infra as Code, but rather the stochasticity of the task. Provisioning resources is inherently unstable, similar to networking, etc.
I am not complaining about Azure. I am just asking:
How can I adjust the IaC pipeline so that when Azure fails occur, some sort of retry can automatically be triggered?
As a concrete example, is there an Azure or Terraform equivalent to Python's tenacity package or Java's Spring Retry?
How can I adjust the IaC pipeline so that when Azure fails occur, some sort of retry can automatically be triggered?
You could try to use Trigger Azure DevOps Pipeline task to trigger current build pipeline automatically when the build failed.
To be able to use the extension an Azure DevOps API endpoint needs to be created.
For the service connection to work as it should you need to configure the following parameters:
Organization Url: The URL of the organization.
(https://dev.azure.com/[organization])
Personal Access Token: The personal access token.
Besides, we need to set the condition for this task, which will trigger the current pipeline Only when a previous task has failed:
We use Azure Pipeline to implement our Continuous integration pipeline. The app is deployed in virtual machines that we need to provision and configure. There are tones of libraries, patches , configurations , and applications that we need to deploy on the target VM before we get our code into those.
The question is what is the best tool to provision and configure these virtual machines? I was thinking of using Ansible AWX. Basically Azure Pipeline would make a call to the AWX API, which would then take it from there and finalize things.
There is an Azure Pipeline Extension that allows me to execute a playbook https://github.com/microsoft/azure-pipelines-extensions/blob/master/Extensions/Ansible/Src/readme.md. But I would like to use AWX instead so that my ansible/deployment code is decoupled from my pipeline.
Any suggestions?
As far as I know, Ansible allows you to automate the deployment and configuration of resources in your environment. It could meet your needs.
As you said, Azure Pipeline supports to run the playbook in the Ansible task(Ansible extension).
So I think you can directly complete the VM Configuration and Code Deployment in the azure pipeline.
If you want to separate these two steps, you can split them into two pipelines (VM configure and Code Deployment). To avoid confusion between configuration and deployment code, you can also split them into two repos.
On the other hand, if you run the playbook in the azure pipeline, the azure pipeline also supports adding tasks to change the parameters in the playbook(e.g. Replace Token).
Here is an operation guide about using Ansible in Azure Pipeline.
By the way, if the Virtual Machine is Azure VM, you also could use ARM template to update the Azure VM resource.
Personally, I would drop the AWX requirement. It's something else to manage and maintain and an entirely separate interface too. Instead, just do your whole pipeline in one place... azure devops. Pick one or the other. Tower doesn't have a built in source control, so I recommend ADO over it, but they'll both run ansible and they'll both do it on your own control nodes. There's no reason to take an extra step with another tool. It adds way too much complexity.
I'm in the process of setting out Azure Devops for our organisation and although I have got things working to an extent, I still dont understand a couple of concepts that I think I should.
I have set up a Deployment Pool (Organization settings > Deployment Pools) and used the script to install and configure the Agents on my Dev, Test and Prod servers, and have been successfully able to deploy code to them
I have been able to build my projects using the Azure Hosted option for now, but I would really like to use the locally hosted option, but when looking at the Agent Pools(Organization settings > Agent Pools) I can only see my Production server and it wont let me use that build with. I clicked the button to "Download Agent" but it downloads the agent I already set up in the Deployment Pool stage using the same powershell command .config.cmd command
So as far as I can tell, there is no difference between an agent in an "Agent pool" vs a "Deployment pool", but I'm obviously missing something here as I cannot see the agent in the Agent pool.
Can anyone help me to understand what I might have missed, and why there are two totally different ways of downloading the same agent?
Many thanks!
Deployment pool are for deployment groups are a special agent configuration that are used specifically in release pipelines. They give some additional options for your release pipelines beyond the regular agents.
Deployment groups:
Specify the security context and runtime targets for the agents. As you create a deployment group, you add users and give them appropriate permissions to administer, manage, view, and use the group.
Let you view live logs for each server as a deployment takes place, and download logs for all servers to track your deployments down to individual machines.
Enable you to use machine tags to limit deployment to specific sets of target servers
It leverages the same pipeline agent but you are just specifying different configuration for the initialization. If you navigate to the Agent Pool page, there is a download link there with the configuration for setting up an agent that you can leverage in a build pipeline or in a release pipeline when you use an Agent Phase instead of a Deployment Group phase.
My source code is on GitHub.
I have an Azure Devops pipeline set up to build and deploy the application to an Azure subscription.
I also have the full azure environment defined in ARM templates.
I'd like to run the template deployment only when a specific folder changes in my GitHub repo.
Path triggers are only for Azure Devops repos.
Other possible solutions I investigated, but there is no clear documentation on how to achieve this exactly:
Custom condition on build or release task.
Pre-deployment conditions. Maybe artifact filters?
Pre-deployment Gates?
The ARM template deployment is idempotent, I know, but it takes a several long minutes to run even if there was no infrastructure change and I'd like to avoid that time wasted on every build.
Sounds like you have a single pipeline for both the infrastructure and application code. I have separate pipelines for each, one for infrastructure as code and other builds/pipelines for applications, NuGet package creation, etc. Perhaps split the pipeline and have the application deployment trigger after and separately from the infrastructure deployment pipeline. That way the application build and deployment can run in a more frequent cycle.
My team currently has a build definition in VSTS that handles our continuous build process.
The application is running on .NET Core 1.1.1.
After additional features have been added, the number of WebJobs have increased. Our current build definition requires a separate task to handle every WebJob.
Is there any way to create a single task to hold all our WebJobs?
The image shows our current build tasks in VSTS.
The settings for the individual WebJob task is seen here:
You can do it programming (e.g. PowerShell), then just add PowerShell task for webjobs.
On the other hand, you may custom build task to do it programmatically.