I was hoping to get some feedback on using Azure Pipelines and what the best practices are for my situation.
We have recently migrated from TFS 2017 and we are in the process of re-writing all our pipelines. We were using builds and releases prior to the upgrade in the legacy build tasks. We would like to setup more useful YAML pipelines.
Let me set the stage with what we currently have
10+ microservices
10 individual builds that trigger from a folder in the repo for each one
10 releases that get created on successful build
3 environments per release (Dev, QA, UAT)
So in summary... a build of a single microservice triggers off of a commit to a folder in the branch. The successful build then triggers a release to Dev. Dev completes and a user go go start a QA build by clicking the release.
In the new Azure Pipeline world... what would be the best approach to doing this model.
We would like to have all the builds happen in a single pipeline (each stage would be a microservice?)
How do we trigger only on a commit to that folder?
What would the CD look like? Should it be in the same pipeline and be a new stage?
How can we easily just add environments without having to just keep copy/pasting all the code for each environment? Ideally i would like to just be able to add a variable and a new environment can be deployed to
I am open to any suggestions here. I am ok if I am way off here, I am looking for the best practices and best approach to this.
TIA
Related
I am working for a company that develops a lot of apps and we use Azure devops portal for all of our pipeline releases etc.
So, the general flow is that a developer will create a branch to do the development on and change the code on that branch. They would like to deploy that branch before it is merged back into the development line from a pipeline, rather than from visual studio, which they currently do. So, it's about giving the developer to the option to choose which branch to deploy from.
Has anyone done something similar or point me in the right direction to how I could go about this?
When creating the release, currently there is no build-in feature to choose which branch to deploy from, the release pipeline works with the corresponding source Artifacts configuration.
A good approach for your scenario is referring to the build id on your build pipeline (instead of release pipeline).
When running the build pipeline, you could choose the target branch and record the build id for this run.
In your release pipeline, you could check whether the source Artifacts is from your target branch by checking the build id.
We are improving our CI/CD process and we understand that we need to use GitHub (or similar) for a better version control..
We would like to integrate SuiteCloud with Github to upload ALL our objects in order to track an monitor any changes. Also, we are planning to use branches to deploy code in our instances. Ex: Master = Production. Dev Branch = Sandbox1, Homologation = Sandbox2.
So... is possible to do it? And how? Any experiences to be shared?
Since SDF manipulates text (xml) files their management is no different than for any other git based CI/CD process.
Netsuite has java and node based CLI tools that can be incorporated in Git Actions for automated deployments.
You'll have some fun if you want to automate testing as that can be problematic with any live database.
I have a great experience of pipeline creating automation (in case of huge amount of repos).
For example, a project has 20 similar repos with Java app (like a microservice) and a pipeline for each of them is differing only by repo url (and a few more minor attributes). The CI/CD process for each of them is the same.
So, we can create a separated devops-repo with declaration configuration for our services. Also we can create a single pipeline which will pull the devops repo and create all needed pipelines for each repo in the configuration (this operation is going to be executed only once in the beginning and in case if we want to change the devops-configuration)
I have implemented that using Jenkins. Now, I am going to do so using GitLab CI. But I can't get how is it possible.
Is it possible to create a pipeline from another one (dynamically)?
Any suggestions?
You can use include and put the generic pipeline in your devops repo.
In your java repos you can include the devops pipeline and set the variables which are specific for the respective java repo.
So the pipeline for your java repos can be as short as this:
include:
- project: 'your-group/devops-repository'
file: '.generic-ci.yml'
variables:
FOO: bar
For my team where we have partner teams providing us SW pieces that need to be integrated on HW systems and tested together, our code footprint is small and hence churn is small, while number of changes from partner teams is frequent. In such a scenario, I see the need to trigger the release part of the yaml many more times than the build part. Is multi-stage pipelines the way to go? I want to trigger new release instances using RestAPI invoke only the Release stages on the YAML file, using AzureDevOps Rest API.
Regards,
You don't have to use multi-stage pipelines to be able to trigger repeated releases, it just makes the management of the pipeline cleaner.
It's possible to create a pipeline that include a build stage and release stages for each of your environments, trigger the build stage (manually or based on a CI trigger), and then from that Pipeline "Run", deploy as many times as you see fit to whatever environments you like. That can be done from API or portal.
It's also possible to create a pipeline that is "release-only" - that is, it gets created manually, or as the result of seeing a specified build having been run.
Personally, I like the multi-stage build because it's a little easier to see what build created the release that you're deploying around. It's not a requirement, though.
We have a self-hosted build agent on an on-prem server.
We typically have a large codebase, and in the past followed this mechanism with TFS2013 build agents:
Daily check-ins were built to c:\work\tfs\ (taking about 5 minutes)
Each night a batch file would run that did the same build to those folders, using the same sources (they were already 'latest' from the CI build), and build the installers. Copy files to a network location, and send an email to the team detailing the build success/failures. (Taking about 40 minutes)
The key thing there is that for the nightly build there would be no need to get the latest sources, and the disk space required wouldn't grow much. Just by the installer sizes.
To replicate this with Azure Devops, I created two pipelines.
One pipeline that did the CI using MSBuild tasks in the classic editor- works great
Another pipeline in the classic editor that runs our existing powershell script, scheduled at 9pm - works great
However, even though my agent doesn't support parallel builds what's happening is that:
The CI pipeline's folder is c:\work\1\
The Nightly build folder is c:\work\2\
This doubles the amount of disk space we need (10gb to 20gb)
They are the same code files, just built differently.
I have struggled to find a way to say to the agent "please use the same sources folder for all pipelines"
What setting is this, as we have to pay our service provider for extra GB storage otherwise.
Or do I need to change my classic pipelines into Yaml and somehow conditionally branch the build so it knows it's being scheduled and do something different?
Or maybe, stop using a Pipeline for the scheduled build, and use task scheduler in Windows as before?
(I did try looking for the same question - I'm sure I can't be the only one).
There is "workingDirectory" directive available for running scripts in pipeline. This link has details of this - https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/utility/command-line?view=azure-devops&tabs=yaml
The number '1' '2'...'6' of work folder c:\work\1\, c:\work\2\... c:\work\6\ in your build agent which stands for a particular pipeline.
Agent.BuildDirectory
The local path on the agent where all folders for a given build
pipeline are created. This variable has the same value as
Pipeline.Workspace. For example: /home/vsts/work/1
If you have two pipelines, there will also be two corresponding work folders. It's an except behavior. We could not configure pipelines to share the same build folde. This is by designed.
If you need to use less disk space to save cost, afraid stop using a Pipeline for the scheduled build, and use task scheduler in Windows as before is a better way.