I have separate existing code repositories- One for Angular UI and one for .NET 4.7 API layer.
In the manual process, The compiled UI code is placed in wwwroot folder after dotnet publish is executed, and the artifacts are deployed to an Azure App service.
While trying to implement CI using Azure DevOps, I had to create two build pipelines for UI and BackEnd.
In the release pipeline, it looks like I have to unzip the artifacts, write a copy step to UI artifacts into wwwroot and then again zip it before deploying to an App service.
I can only imagine this isnt the best approach. Given that I am new to Azure DevOps, I would like to know the best practices especially while handling relatively legacy code. I would have kept both the UI and API layer in a single repository if I could do that now. What is the best way to handle this?
UPDATE - Can I have multiple repositories built in a single build pipeline?
Based on the article - https://learn.microsoft.com/en-us/azure/devops/pipelines/repos/multi-repo-checkout?view=azure-devops
Thanks,
AK
Yes you can have multiple repo.
resources:
repositories:
- repository: MyGitHubRepo # The name used to reference this repository in the checkout step
type: github
endpoint: MyGitHubServiceConnection
name: MyGitHubOrgOrUser/MyGitHubRepo
- repository: MyBitbucketRepo
type: bitbucket
endpoint: MyBitbucketServiceConnection
name: MyBitbucketOrgOrUser/MyBitbucketRepo
- repository: MyAzureReposGitRepository # In a different organization
endpoint: MyAzureReposGitServiceConnection
type: git
name: OtherProject/MyAzureReposGitRepo
trigger:
- main
pool:
vmImage: 'ubuntu-latest'
steps:
- checkout: self
- checkout: MyGitHubRepo
- checkout: MyBitbucketRepo
- checkout: MyAzureReposGitRepository
- script: dir $(Build.SourcesDirectory)
And in terms of your original question you can publish your changes and without zipping them (set zipAfterPublish to false):
- task: DotNetCoreCLI#2
inputs:
command: publish
publishWebProjects: True
arguments: '--configuration $(BuildConfiguration) --output output_folder'
zipAfterPublish: False
workingDirectory: backend/app/
and then copying your wwwroot and zipped wwwroot and your backend app into two separate archives and publishem them.
I would create two build pipelines that publishes a package after each successful build to an Azure Artifact Feed.
Then 1 deploy pipeline that downloads both packages and apply further deploy logic.
Related
I have a repository in Github and want to integrate it into Azure-DevOps. I connected both the repositories in Github as well as Azure-devops.
When I commit some code into Github the changes are not getting updated automatically in Azure. Is there anyway that we can automatically pull the changes if there are any new changes in Github?
Any references/suggestions are much appreciated.
Update:
Azure DevOps doesn't have such a built-in feature to sync the Github repo to the DevOps repo now.
If you need the feature, you can upvote the feature request in this place:
https://developercommunity.visualstudio.com/t/automatically-sync-azure-devops-repository-with-gi/516057
When enough people request a new feature, Microsoft will include it in future product plans.
1, You need to use code/script to sync the repo and use the CI Trigger of the YAML pipeline to capture the changes in the Github repo.
trigger:
- <branch name>
pool:
vmImage: ubuntu-latest
variables:
- name: system.debug
value: true
steps:
- script: |
echo put the logic here
displayName: 'Push changes to DevOps repo'
The code you can refer to this page:
https://dileepveldi.medium.com/sync-azure-devops-repo-with-github-repo-35a958d7784e
2, Then after the above pipeline pushes the changes, you need to captrue the changes via the CI trigger of the YAML pipeline on DevOps side.
trigger:
- <branch name>
pool:
vmImage: ubuntu-latest
variables:
- name: system.debug
value: true
steps:
- script: |
echo xxx
displayName: 'Run a multi-line script'
Original Answer:
If you mean integrating Github repo and Azure DevOps pipeline, for example, you need continuous integration on main branch of your repo.
Then, follow the below steps.
1, For classic pipeline:
2, For YAML pipeline:
trigger:
- main
pool:
vmImage: ubuntu-latest
steps:
- script: |
echo Add other tasks to build, test, and deploy your project.
echo See https://aka.ms/yaml
displayName: 'Run a multi-line script'
If what you mean is not integrating Github repo and Azure DevOps pipeline, please clarify your requirements.
Hello my expert friends.
I know this question might sound really simple but I am seeking for some advice and best practices to follow/learn.
I have a testing infra in azure divided in 2 environment. One is Staging and the other is Production.
Those environment have the same configurations for on hand practices as I want to learn how to deploy specific docker images from staging to production.
At the current state, I have 1 web app in Staging and 1 web app in Production.
My Build pipeline for this lab is try to trigger the build pipeline in staging only if I push to GitHub a Tag, and I achieved this by setting my pipeline as follow:
trigger:
batch: true
tags:
include:
- '*'
branches:
exclude:
- Staging
This runs a docker build with some c# code and deploys it to a container registry.
But in my current pipeline, under the task Docker:
- task: Docker#2
name: 'dockerBuildAndPush'
displayName: 'docker - Build & Push'
inputs:
repository: $(imageRepository)
Dockerfile: $(dockerfilePath)
containerRegistry: ${{ variables.dockerRegistryServiceConnection }}
buildContext: ${{ variables.buildContext }}
tags: |
$(Build.BuildNumber)
latest
As you can see I have a tags that is based on the Build.BuildNumber
This of course during the build process it get the current date + build number, but I wanted to do is target the latest tag coming from GitHub and pass it to the build.
And this is where I got confuse and not really sure about the best practice to follow.
Assuming that on GitHub I push the update with the tag v1.0, is there a way how I can use the pipeline to pick the tag number and pass it automatically to the build? Or I have to update the Tag value manually in the pipeline every time before to push to GitHub?
So basically what I want to have in my container registry is as follow:
Github push tag v1.0
Azure Container registry have a build docker build:v1.0
In this way, I will be easier to detect quickly which docker image is running on Staging and Production later on.
Sorry if I couldn't explain my dilemma clearly and please if this is the case, don't hesitate to ask for more informations.
UPDATE:
Thank you so much Dave for your reply and you solution. I will look into it asap.
Right I was looking for something a bit easier to achieve to understand the full process and get confident with it.
At the current state I managed my pipeline in the following order.
parameters:
- name: tag
type: string
default: 'v1.1'
trigger:
batch: true
tags:
include:
- '*'
branches:
exclude:
- Staging
and in my docker task set the tag to the parameters
- task: Docker#2
name: 'dockerBuildAndPush'
displayName: 'docker - Build & Push'
inputs:
repository: $(imageRepository)
Dockerfile: $(dockerfilePath)
containerRegistry: ${{ variables.dockerRegistryServiceConnection }}
buildContext: ${{ variables.buildContext }}
tags: '${{ parameters.tag }}'
This process worked exactly how expected. In my container registry in the portal I have a version named 'v.1.1'
The issue I am facing is during the release phase of this image.
In the tag for the release I have the $(Build.BuildNumber) which of course as I am setting a parameters variable to build that image, I don't have a build number but a v1.1.
I have been reading around that I can override the BuildNumber with a specific variable name in the yaml file to map the Build.BuildNumber to a parameter and in the relelease pipeline I can leave the Build.BuildNumber as a reference. But is not 100% clear to me how this can be done.
You could use GitVersion to generate a version number from your repository tags.
This could be used from your pipeline as follows:
- task: UseDotNet#2
displayName: Use .NET Core CLI
inputs:
version: "6.x"
- task: DotNetCoreCLI#2
displayName: Install GitVersion
continueOnError: true
inputs:
command: "custom"
custom: "tool"
arguments: "install GitVersion.Tool --version 5.* --global"
- task: PowerShell#2
displayName: Set build version
inputs:
targetType: "inline"
script: |
Write-Host "##vso[build.updatebuildnumber]$(dotnet-gitversion /showvariable FullSemVer)"
The above example will set the Build.BuildNumber variable to a version string based on the last tag. You can, of course, set some other variable instead.
You can also customise the way GitVersion chooses a version number by adding a GitVersion.yml file to your repository.
This answer should serve as a starting point for you - it doesn't produce exactly the format you want, you will need to look at configuration options and such if you want a very specific format.
Also, look at the different modes that GitVersion can run in, since they will affect the version numbers generated for any commits inbetween tagged commits!
I have yaml file for the azure pipline in a repo. And I need to run powershell script from a different repo.
As far as I understood I can add side repo to resources section in yaml and then use task:ShellScript#2 with scriptPath parameter. But as I understood it works relatively for repo in which yaml is placed. And I'm not sure how can I access file from a different repo.
Yes, you have to use repository resource and checkout that repo as follows:
resources:
repositories:
- repository: devops
type: github
name: kmadof/devops-templates
endpoint: kmadof
steps:
- checkout: self
- checkout: devops
- task: ShellScript#2
inputs:
scriptPath: $(Agent.BuildDirectory)/devops/scripts/some-script.sh
I have a very weird (and I suppose easy to fix) problem :) I am trying to have a working CI/CD pipeline in Azure. For this purpose, I have a repository in Azure devops and build and release pipeline created. I am publishing docker images to Azure Container Registry and during release, I am pulling this image (or at least - I am trying because it doesn't work) and I am trying to publish it on Webapp for containers. The "app" in my case it is SingalR hub on .NET Core 3.1 (but I don't suppose it makes a difference in the problem I am having)
If somebody wants to know in details how did i configure it - here is the tutorial i did use:
https://wikiazure.com/devops/azure-devops-automate-your-release-pipeline-to-provision-a-docker-container-to-azure-web-app-for-containers/
There were some doubts/differences in the tutorial (for example - why initially in the tutorial web app is being configured on Docker hub, when in fact it is using ACR. And why to connect to ACR the tutorial uses Azure Resource Manager connection (And not dedicated Docker container --> ACR connetion) And why later on in build pipeline there is some weird id set for dockerRegistryServiceConnection (i am giving in this place name of my ACR docker service connection)
But the whole build pipeline is working. It is publishing image to ACR. Everything is fine till this step.
The problem starts when I want to publish Azure WebApp with this image. The problem is with ... TAGS :) They are mismatching. I have automatic CI/CD - so when i push some change to the repo i see that release pipeline is working. It is creating the image in the ACR. Then i see, that release pipeline is running. Everything is "correct" - meaning no error are seen and the release is green.
But when i go to App service and Container settings i see from logs:
2020-04-21 18:02:28.321 INFO - Pulling image: myAcrName.azurecr.io/mobile/signalr:c7aead0c46b66afc4131935efc7e6a51280dfb1a
2020-04-21 18:02:28.761 ERROR - DockerApiException: Docker API responded with status code=NotFound, response={"message":"manifest for myAcrName.azurecr.io/mobile/signalr:c7aead0c46b66afc4131935efc7e6a51280dfb1a not found: manifest unknown: manifest unknown"}
2020-04-21 18:02:28.761 ERROR - Pulling docker image myAcrName.azurecr.io/mobile/signalr:c7aead0c46b66afc4131935efc7e6a51280dfb1a failed:
2020-04-21 18:02:28.762 INFO - Pulling image from Docker hub: myAcrName.azurecr.io/mobile/signalr:c7aead0c46b66afc4131935efc7e6a51280dfb1a
2020-04-21 18:02:28.867 ERROR - DockerApiException: Docker API responded with status code=InternalServerError, response={"message":"Get https://myAcrName.azurecr.io/v2/mobile/signalr/manifests/c7aead0c46b66afc4131935efc7e6a51280dfb1a: unauthorized: authentication required"}
2020-04-21 18:02:28.870 ERROR - Image pull failed: Verify docker image configuration and credentials (if using private repository)
Very sophisticated error but the root cause is, that he is trying to get the image with non-existing tag, which is GIT COMMIT tag. And it suppose to get image by $(Build.BuildId) (this was my first attempt) or by $(Build.BuilNumber) (this was my second attempt)
Here is how this pipeline step (Deploy Azure App Service) looks like:
- task: AzureRmWebAppDeployment#4
displayName: 'Deploy Azure App Service'
inputs:
azureSubscription: mySubcsriptionARM
appType: webAppContainer
WebAppName: myProductsignalr
DockerNamespace: myAcrName.azurecr.io
DockerRepository: mobile/signalr
DockerImageTag: '$(Build.BuildNumber)'
When i go to Release pipeline logs as a "Deploy Azure App Service" log i see that
2020-04-21T18:41:01.6012767Z ##[section]Starting: Deploy Azure App Service
2020-04-21T18:41:01.6367124Z ==============================================================================
2020-04-21T18:41:01.6367787Z Task : Azure App Service deploy
2020-04-21T18:41:01.6368381Z Description : Deploy to Azure App Service a web, mobile, or API app using Docker, Java, .NET, .NET Core, Node.js, PHP, Python, or Ruby
2020-04-21T18:41:01.6368765Z Version : 4.163.5
2020-04-21T18:41:01.6369158Z Author : Microsoft Corporation
2020-04-21T18:41:01.6369603Z Help : https://aka.ms/azureappservicetroubleshooting
2020-04-21T18:41:01.6369976Z ==============================================================================
2020-04-21T18:41:03.8970184Z Got service connection details for Azure App Service:'myProductsignalr'
2020-04-21T18:41:04.5534864Z Trying to update App Service Configuration settings. Data: {"appCommandLine":null,"linuxFxVersion":"DOCKER|myAcrName.azurecr.io/mobile/signalr:1f283100"}
2020-04-21T18:41:05.5465725Z Updated App Service Configuration settings.
2020-04-21T18:41:05.5495890Z Trying to update App Service Application settings. Data: {"DOCKER_CUSTOM_IMAGE_NAME":"myAcrName.azurecr.io/mobile/signalr:1f283100"}
2020-04-21T18:41:06.2703349Z Updated App Service Application settings and Kudu Application settings.
2020-04-21T18:41:32.4715682Z Updated App Service Application settings and Kudu Application settings.
2020-04-21T18:41:33.4179962Z Successfully updated deployment History at https://myProductsignalr.scm.azurewebsites.net/api/deployments/111587494492765
2020-04-21T18:41:33.5945654Z App Service Application URL: http://myProductsignalr.azurewebsites.net
2020-04-21T18:41:33.6180118Z ##[section]Finishing: Deploy Azure App Service
What amazes me, that it is showing, that everything was ok - when it was far from "ok" :)
When i go to container settings after:
a) new code is published
b) build pipeline fires
c) release pipeline fires
i see it like this:
The tag is empty. If i would pick some tag manually:
And would choose: "SAVE" everything works correctly (SingalR is up and running correctly)
Clearly, I am missing something :/ Help me to see what;)
The root cause for me is that this fragment:
DockerImageTag: '$(Build.BuildNumber)'
should insert build number (as stated) and the info from container settings should be:
Pulling image: myAcrName.azurecr.io/mobile/signalr:20200421.09 (for BuildNumber 20200421.09) and it is inserting GIT COMMIT there as a tag and ends up with: Pulling image: myAcrName.azurecr.io/mobile/signalr:c7aead0c46b66afc4131935efc7e6a51280dfb1a Why o why?:)
[UPDATE 22.04 10:56]
I am posting build pipeline that i am using currently. I don't suppose it is important as it is working correctly, and the problem is more with deployment of correctly created docker image (on ACR), than with creating this image by the build pipeline. Nevertheless, here is the pipeline:
# Docker
# Build a Docker image
# https://learn.microsoft.com/azure/devops/pipelines/languages/docker
trigger:
- master
resources:
- repo: self
variables:
dockerRegistryServiceConnection: 'MyProductDockerACR'
imageRepository: 'mobile/signalr'
containerRegistry: 'myAcrName.azurecr.io'
dockerfilePath: '**/Dockerfile'
tag: '$(Build.BuildNumber)'
vmImageName: 'ubuntu-latest'
stages:
- stage: Build
displayName: Build and push stage
jobs:
- job: Build
displayName: Build
pool:
vmImage: $(vmImageName)
steps:
- task: Docker#2
displayName: Build and push image to container registry
inputs:
containerRegistry: $(dockerRegistryServiceConnection)
repository: $(imageRepository)
command: 'buildAndPush'
Dockerfile: $(dockerfilePath)
tags: |
$(tag)
I saw the release you are using is configured by UI. It's work logic much different with the one which configured by YAML.
In fact, here what you received just be the different performance produced while the running reason of the release are different.
I guess this release has the artifact source which targeting to Repos, right? You can confirm by checking its icon.
While the release source is coming from Repos, then the Build.BuildNumber would be the short part of the commit id(8 characters). And the Build.BuildId is the complete commit id.
If you want the release keep using the Build.Buildnumber value which the corresponding build(created/pushed image) was using, you must make sure the release source is targeting to this build. Also, this build need has artifacts generated. According to the YAML you shared, obviously, you haven't done that.
Only the release triggered by build along with artifact, then the Build.BuildNumber can be like 20200422.1 which the build was using.
So, please go your release definition, and re-configure its source to make sure it is coming from build artifact instead of repository.
Yes. You are right. You have mismatch in tags.
In Docker#2 task you can define tags:
steps:
- task: Docker#2
displayName: Login to ACR
inputs:
command: login
containerRegistry: devopsmanual-acr
- task: Docker#2
displayName: Build and Push
inputs:
repository: $(imageName)
command: buildAndPush
Dockerfile: build-docker-image/SampleAppForDocker/DOCKERFILE
tags: |
$(Build.BuildNumber)
- task: Docker#2
displayName: Logout of ACR
inputs:
command: logout
containerRegistry: devopsmanual-acr
Your definition should be pretty much like this one. Whre devopsmanual-acr is connection to your ACR.
.
I recently mad a blog post about creating docker images on Azure DevOps so maybe this will be also helpful for you.
If this won't be enough to solve your issue, please edit your question and show how you create and push your images.
I'm setting up several pipelines in Azure DevOps. To make my teams life easier, I'm using job templates.
These job templates are in a a proper repository, just for them.
For every pipeline I define the repository to get the templates from.
Some tasks in these templates run powershell code, and I want this code to be in a script file, to be reusable and stored in the same repo as the template.
When the pipelines runs, the template is embeded, it tries to locate the powershell script inside project repo actually being built/deployed.
How can i achieve this?
The workaround is to have inline code which I really don't want to have.
Any constructive answer will be very appreciated.
Thanks
After some digging I couldn't find any way to specify a script file as source to powershell task in a template.
Inside pipeline definition:
resources:
repositories:
- repository: templates
type: git
name: deploy-templates
variables:
artifactName: 'Trade Data ETL - $(Build.SourceBranchName)'
stages:
- stage: Build
displayName: Build
variables:
- group: DEV-Credential-Group
- group: COMMON-Settings-Group
jobs:
- template: ssis/pipelines/stage-build.yml#templates # Template reference
parameters:
artifactName: '$(artifactName)'
Inside template file:
- task: PowerShell#2
inputs:
filePath: ssis/pipelines/scripts/build-ssis-project.ps1
arguments: '-ProjectToBuild "tradedata-ldz-ssis/tradedata-ldz-ssis.dtproj'
pwsh: true
Update 2021
According to learn.microsoft.com, you can now also check out multiple repositories without custom scripting.
If you check out more than one repository, a separate folder containing the repository is created below $(Build.SourcesDirectory).
You can define multiple repositories like this:
resources:
repositories:
- repository: devops
type: git
name: DevOps
ref: main
- repository: infrastructure
type: git
name: Infrastructure
ref: main
And in the steps simply check them out as follows:
steps:
- checkout: self
- checkout: devops
- checkout: infrastructure
# List all available repositories
- script: ls
Original Answer
Currently the resources command only supports yml files in other repositories. However, you could simply checkout the repository in a task and then run the desired powershell script.
steps:
- task: PowerShell#2
inputs:
targetType: inline
script: |
git clone -b <your-desired-branch> https://azuredevops:$($env:token)#dev.azure.com/<your-organization>/<your-project>/_git/<your-repository> <target-folder-name>
./<target-folder-name>/foo.ps1
env:
token: $(System.AccessToken)
This script would checkout an arbitrary branch and execute a script foo.ps1 in the root of the target repository.
Call - checkout: templates inside the template file. This might only work when you insert a template but it successfully sees the repository resource and pulls it down.
You can copy the script files from source directory. Currently, you have not mentioned the root folder -
ssis/pipelines/scripts/build-ssis-project.ps1
Assuming, you are building on a repo where the powershell script resides -
Try -
- task: PowerShell#1
inputs:
scriptName: '$(ScriptsDir)/ssis/pipelines/scripts/build-ssis-project.ps1'
Pass the value of ScriptsDir where it could be the build source directory or build working directory