I am looking for some best practices and suggestions:
My team is creating an Azure DevOps pipeline to deploy a complex infrastructure of VNets, VMs, Azure ML workspaces, SQL databases, etc.
The pipeline uses Terraform where possible, but Powershell or AZ CLI where needed.
The pipeline works, it is version controlled, it has proper unit tests and integration tests (or at least decent ones).
However, due to the instability of Azure resourcing sometimes the pipeline will fail because, for instance:
SQL server provisioning fails
AD join of VMs fails
or other activities which are not due to bad Infra as Code, but rather the stochasticity of the task. Provisioning resources is inherently unstable, similar to networking, etc.
I am not complaining about Azure. I am just asking:
How can I adjust the IaC pipeline so that when Azure fails occur, some sort of retry can automatically be triggered?
As a concrete example, is there an Azure or Terraform equivalent to Python's tenacity package or Java's Spring Retry?
How can I adjust the IaC pipeline so that when Azure fails occur, some sort of retry can automatically be triggered?
You could try to use Trigger Azure DevOps Pipeline task to trigger current build pipeline automatically when the build failed.
To be able to use the extension an Azure DevOps API endpoint needs to be created.
For the service connection to work as it should you need to configure the following parameters:
Organization Url: The URL of the organization.
(https://dev.azure.com/[organization])
Personal Access Token: The personal access token.
Besides, we need to set the condition for this task, which will trigger the current pipeline Only when a previous task has failed:
Related
I am attempting to perform Resource Management inside an Azure YAML Pipeline.
The YAML file is checked into source code and can be edited by developers.
Code Reviews will not occur until the deployment is moving from UAT into Production.
I would like to make sure that a deployment to QA cannot perform Resource Management for the resource group associated with the Production Environment. ie: prod databases
Can I limit a Service Connection to only be available in certain release stages?
Is there another way to limit Resource Management to specific stages of a pipeline?
Azure Devops Pipeline Stages: Independent Security
I am afraid there is no such way to limit a Service Connection to only be available in certain release stages at this moment.
According to the document we could to know Service connections:
The service connection security is divided into three categories in the service connections new UI:
User permissions
Pipeline permissions
Project permissions
When your pipeline uses the classic UI mode, you could use User permissions to control who can create, view, use and manage the service connection with user permissions.
If your pipeline is YAML mode, you could use the Pipeline permissions, Pipeline permissions control which YAML pipelines are authorized to use this service connection.
So, there is no such way to limit Resource Management to specific stages of a pipeline.
For this request, you could add your request for this feature on our UserVoice site (https://developercommunity.visualstudio.com/content/idea/post.html?space=21 ), which is our main forum for product suggestions. Thank you for helping us build a better Azure DevOps.
Does azure pipelines allow custom action like AWS codepipeline?
I want to create a job worker that will poll azure pipeline for job requests for this custom action, execute the job, and return the status result to azure pipeline.
Something similar to - https://docs.aws.amazon.com/codepipeline/latest/userguide/actions-create-custom-action.html
Tasks are the building blocks for defining automation in a build or release pipeline in Azure DevOps. There are many built-in tasks to enable fundamental build and deployment scenarios. If the existing tasks don't satisfy your needs, you can always build a custom task. Check Task types & usage for more details.
In addition, Visual Studio Marketplace offers a number of extensions; each of which, when installed to your subscription or collection, extends the task catalog with one or more tasks. Furthermore, you can write your own custom extensions to add tasks to Azure Pipelines.
Azure Pipeline Agents
When your pipeline runs, the system begins one or more jobs. An agent is computing infrastructure with installed agent software that runs one job at a time.
You have two options here to choose from: Microsoft-hosted agents or Self-hosted agents
An agent that you set up and manage on your own to run jobs is a self-hosted agent. Self-hosted agents give you more control to install dependent software needed for your builds and deployments. Also, machine-level caches and configuration persist from run to run, which can boost speed.
However, before you install a self-hosted agent you might want to see if a Microsoft-hosted agent pool will work for you. In many cases, this is the simplest way to get going.
With Microsoft-hosted agents, maintenance and upgrades are taken care of for you. Each time you run a pipeline, you get a fresh virtual machine. The virtual machine is discarded after one use. Microsoft-hosted agents can run jobs directly on the VM or in a container. Azure Pipelines provides a pre-defined agent pool named Azure Pipelines with Microsoft-hosted agents.
You can try it first and see if it works for your build or deployment. If not, you can use a self-hosted agent. Check this doc for more details.
I will pull the agent queue from my custom job worker and process the job. Is that possible in azure pipelines?
Based on my understanding of code pipeline and Azure devops, I am afraid what you said should be meaningless.
According to the document Create and add a custom action in CodePipeline, we could to know that:
AWS CodePipeline includes a number of actions that help you configure
build, test, and deploy resources for your automated release process.
If your release process includes activities that are not included in
the default actions, such as an internally developed build process or
a test suite, you can create a custom action for that purpose and
include it in your pipeline.
But for Azure devops, we do not need to create a job worker that will poll CodePipeline for job requests for this custom action. That because the whole process of build/release can be customized. We do not need to add a job worker for additional custom actions.
Azure devops provide a lot of templates when we create the pipeline, we could modify the pipeline directly in the pipeline to add/remove or update the task:
Even we can completely start with a blank pipeline and completely customize the entire build/release process.
So, we do not need to create a job worker for the custom action, just modify your pipeline directly.
We use Azure Pipeline to implement our Continuous integration pipeline. The app is deployed in virtual machines that we need to provision and configure. There are tones of libraries, patches , configurations , and applications that we need to deploy on the target VM before we get our code into those.
The question is what is the best tool to provision and configure these virtual machines? I was thinking of using Ansible AWX. Basically Azure Pipeline would make a call to the AWX API, which would then take it from there and finalize things.
There is an Azure Pipeline Extension that allows me to execute a playbook https://github.com/microsoft/azure-pipelines-extensions/blob/master/Extensions/Ansible/Src/readme.md. But I would like to use AWX instead so that my ansible/deployment code is decoupled from my pipeline.
Any suggestions?
As far as I know, Ansible allows you to automate the deployment and configuration of resources in your environment. It could meet your needs.
As you said, Azure Pipeline supports to run the playbook in the Ansible task(Ansible extension).
So I think you can directly complete the VM Configuration and Code Deployment in the azure pipeline.
If you want to separate these two steps, you can split them into two pipelines (VM configure and Code Deployment). To avoid confusion between configuration and deployment code, you can also split them into two repos.
On the other hand, if you run the playbook in the azure pipeline, the azure pipeline also supports adding tasks to change the parameters in the playbook(e.g. Replace Token).
Here is an operation guide about using Ansible in Azure Pipeline.
By the way, if the Virtual Machine is Azure VM, you also could use ARM template to update the Azure VM resource.
Personally, I would drop the AWX requirement. It's something else to manage and maintain and an entirely separate interface too. Instead, just do your whole pipeline in one place... azure devops. Pick one or the other. Tower doesn't have a built in source control, so I recommend ADO over it, but they'll both run ansible and they'll both do it on your own control nodes. There's no reason to take an extra step with another tool. It adds way too much complexity.
My source code is on GitHub.
I have an Azure Devops pipeline set up to build and deploy the application to an Azure subscription.
I also have the full azure environment defined in ARM templates.
I'd like to run the template deployment only when a specific folder changes in my GitHub repo.
Path triggers are only for Azure Devops repos.
Other possible solutions I investigated, but there is no clear documentation on how to achieve this exactly:
Custom condition on build or release task.
Pre-deployment conditions. Maybe artifact filters?
Pre-deployment Gates?
The ARM template deployment is idempotent, I know, but it takes a several long minutes to run even if there was no infrastructure change and I'd like to avoid that time wasted on every build.
Sounds like you have a single pipeline for both the infrastructure and application code. I have separate pipelines for each, one for infrastructure as code and other builds/pipelines for applications, NuGet package creation, etc. Perhaps split the pipeline and have the application deployment trigger after and separately from the infrastructure deployment pipeline. That way the application build and deployment can run in a more frequent cycle.
I have several .NET applications that are hosted In Azure on virtual machines IIS. I want to automate deployment process directly from my machine or visual studio without using Azure portal because we are doing several deliveries each day. The biggest challenge for me now is to do backup before deployment. So I need to backup specific folder on VM remotely. Could anyone tell me how I can do it?
Are there any best practices for automation delivery from visual studio to Azure VM? Backup of app is required.
First, I'm assuming that you would like to back up some specific folders in your VM to an Azure Blobs or Azure Files storage. There are two parts you'd need to complete your continuous deployment:
Automate backing up some folders in your virtual machine.
Integrate task #1 into a CICD (Continuous Integration Continuous Deployment) tool, which I'd suggest VSTS (Visual Studio Team Services) for your beginning.
Approach #1
You can expose Windows Remote Management (WinRM) endpoint publicly and use PowerShell to perform a folder backup task. You will also need to invoke some scripts to write/copy your backup to Azure Blobs Storage. Your script must be authenticated silently so you don't need to key in Azure subscription admin or VM admin.
Once you have a PowerShell script, you can invoke this PowerShell by creating a new PowerShell task in your Build or Release definition. I'd suggest to define in Release definition.
Approach #2
It's more Cloud native and you don't have to expose WMI of your virtual machine which would lead to security threat. Instead, you utilize Azure Automation Worker to automate the entirely backup & copy to another Azure services (Blobs, Files, another backup/file server virtual machine..). In your run book, you need to use PowerShell DSC (Desired State Configuration) to interact with resources inside your virtual machine.
In VSTS, you have two ways to start your runbook
Invoke runbook's webhook: you can create a webhook for your runbook and call this webhook by creating a Http Task
Start runbook by PowerShell: similar to the approach #1, just create PowerShell task then use Start-AzureAutomationRunbook cmdlet with sample here.
Build an ARM template for your Runbook then define in Build definition. Here is the sample reference to deploy an ARM template in VSTS,
The reason I have to give several references because you might not have familiarity with CICD and DevOps concept, as well as some useful tools in Azure and VSTS which supports your continuous deployment. There are some awesome CICD solutions in the market you should also explore, such as Chef, Ansible, Puppet. They support CICD very well. Below are some references to get started with DevOps on Azure:
https://learn.microsoft.com/en-us/vsts/deploy-azure/
https://learn.microsoft.com/en-us/azure/automation/automation-dsc-overview
https://learn.microsoft.com/en-us/vsts/build-release/apps/cd/azure/azure-devops-project-aspnetcore