I get the following error:
problem picture
It says that the POM file can't be found. I have my yaml file within my github project like this:
Project picture
My yaml file looks like this:
In my github there is a pom file in a folder called dataops/poms/pom.xml so i don't know why it can't find it.
# Talend CI/CD on Azure DevOps
# Build Pipeline for building and publishing on Talend Cloud
trigger: none
pool:
vmImage: 'ubuntu-latest'
variables:
- group: Talend Configuration
- name: project_name
value: 'DataOps'
- name: job_name
value: 'Soccer_api_Call'
- name: job_version
value: '0.1'
steps:
- task: DownloadSecureFile#1
name: settings_xml
inputs:
secureFile: settings.xml
- task: DownloadSecureFile#1
name: license_txt
inputs:
secureFile: license.txt
- task: Maven#3
inputs:
mavenPomFile: '$(project_name)/poms/pom.xml'
mavenOptions: |
-Dlicense.path=$(license_txt.secureFilePath)
-Dupdatesite.path=$(UPDATESITE_PATH)
-Dservice.url=$(CLOUD_URL)
-Dcloud.publisher.screenshot=true
-Xmx3096m -Xmx1024m
options: '--settings $(settings_xml.secureFilePath) -Pcloud-publisher -pl jobs/process/$(job_name)_$(job_version) -am'
goals: 'deploy'
Any help is much appreciated!!
We recommend you can try to use the mavenPomFile: 'DATAOPS/poms/pom.xml' instead of the mavenPomFile: '$(project_name)/poms/pom.xml'.
On my test, I create a demo like your yaml, and it works well with the full path:
Please ignore the test result, it just uses to check if the task can find the pom.xml.
Related
I'm trying to edit an ADO YAML file down to the bare minimum in order to isolate another issue.
When I run Validate, it comes back with the following error:
No repository found by name templates
Here's the general gist of my YAML:
#resources:
# repositories:
# - repository: templates
# type: git
# name: TemplateProject/TemplateRepo
name: $(VersionName)
trigger:
branches:
include:
- main
batch: true
paths:
exclude: $(ListOfExclusions)
stages:
- template: core/setVersion.yml#templates
- stage: Build
pool: linux
jobs:
- job: BuildDocker
displayName: Build and Push Docker Image
pool: linux
steps:
- task: Docker#2
displayName: Build and push an image to container registry
inputs:
command: buildAndPush
repository: $(RepoName)
dockerfile: $(Build.SourcesDirectory)/Dockerfile
containerRegistry: $(dockerRegistryServiceConnection)
tags: |
$(Tag)
What could be going wrong? The error message makes me think the YAML isn't clean.
It turns out I caused a simple typo when commenting out the resources section of the YAML. I had a template part of the stage that also needed to be commented out, and I neglected to do this.
Once I updated the code to read:
stages:
# - template: core/setVersion.yml#templates
- stage: Build
pool: linux
jobs:
- job: BuildDocker
# etc...
Now my YAML validates with OK.
I am trying to grab a schema from another artifact build and then build and push the artifact. The error I am getting no builds currently exist in the pipeline definition, I know that is usually from not having a pipeline run on that branch yet. I have run the pipeline and pushed the artifact to the repo on that branch so there is an image already there. The schema was published from the other build. I am not sure why it is not grabbing the artifact. Any help would be appreciated.
jobs:
- job: Build_Push_Artifact
steps:
- task: Docker#2
inputs:
containerRegistry: $(azureContainerRegistry)
command: "login"
- task: DownloadPipelineArtifact#2
inputs:
source: specific
project: $(projectId)
pipeline: $(schemaPublishPipelineId)
runVersion: latestFromBranch
runBranch: $(Build.SourceBranch)
artifact: $(schemaArtifact)
patterns: $(schemaArtifactPattern)
path: $(Build.SourcesDirectory)/src
A possible situation that I can reproduce on my side:
If the build pipeline runs failed in some steps, then even the artifact be published successfully, nothing will be catched.
And please try this simple structure:
trigger:
- none
pool:
vmImage: ubuntu-latest
steps:
- task: DownloadPipelineArtifact#2
inputs:
buildType: 'specific'
project: '<Project Name>'
definition: '<Pipeline Name>'
buildVersionToDownload: 'latest'
targetPath: '$(Pipeline.Workspace)'
In my azure devops project, I have created a pipeline using template. This is the main yaml files for my build pipeline
name: Test-$(Date:yyyyMMdd)$(Rev:.r)
resources:
repositories:
- repository: api
type: git
name: porject/api
ref: master
- repository: front
type: git
name: project/front
ref: master
- repository: strapi
type: git
name: project/strapi
ref: master
trigger:
branches:
include:
- master
pool:
vmImage: 'ubuntu-latest'
workspace:
clean: all
variables:
- name: workingDir
value: project
- name: tfVersion
value: 0.12.29
- name: backendServiceGCP
value: test
- name: backendGCPBucketName
value: test
- name: tfpath
value: test
- name: env
value: dev
stages:
- stage: Terraform
jobs:
- job: Build
displayName: Build Terraform Infra
steps:
# Set and Export env var for api version to deploy
- template: templates/fetch-tag.yml
parameters:
repo: 'api'
envVar: TERRAFORM_API_TAG
# Set and Export env var for front version to deploy
- template: templates/fetch-tag.yml
parameters:
repo: 'front'
envVar: TERRAFORM_FRONT_TAG
# Set and Export env var for strapi version to deploy
- template: templates/fetch-tag.yml
parameters:
repo: 'strapi'
envVar: TERRAFORM_STRAPI_TAG
# Init Terraform providers
- template: templates/tf-init.yml
parameters:
backendServiceGCP: '$(backendServiceGCP)'
backendGCPBucketName: '$(backendGCPBucketName)'
workingDir: '$(workingDir)'
variableFilePath: $(buildSubscription)-common.tfvars
# Plan Terraform Infra to Deploy
- template: templates/tf-plan.yml
parameters:
backendServiceGCP: '$(backendServiceGCP)'
workingDir: '$(workingDir)'
variableFilePath: $(buildSubscription)-common.tfvars
# Publish Public Artifact with Terraform ressources to deploy
- template: templates/publish-artifact.yml
parameters:
tfpath: '$(tfpath)'
When I am trying to run the pipeline I have the following error:
Encountered error(s) while parsing pipeline YAML:
Job Build: The step name version appears more than once. Step names must be unique within a job.
Job Build: The step name version appears more than once. Step names must be unique within a job.
Job Build: The step name version appears more than once. Step names must be unique within a job.
Job Build: The step name version appears more than once. Step names must be unique within a job.
Job Build: The step name version appears more than once. Step names must be unique within a job.
I don't really understand why.
This is an example of templates I am using in the pipeline named publish-artifact.yml:
parameters:
tfPath: ''
steps:
- task: CopyFiles#2
inputs:
sourceFolder: ${{ parameters.tfpath }}
contents: |
tfplan
**/*.tf
**/*.tfvars
**/*.json
!**/.terraform
**/*.sh
targetFolder: $(Build.ArtifactStagingDirectory)
- task: PublishBuildArtifacts#1
inputs:
pathtoPublish: $(Build.ArtifactStagingDirectory)
artifactName: tf
What am I doing wrong with the job name ?
Azure Devops supports passing output variable from one step as next step's inputs. See this ticket:
We can name a step like this:
steps:
- script: echo test
name: ScriptName
- task: PowerShell#2
inputs:
targetType: 'inline'
script: |
Write-Host '##vso[task.setvariable variable=xxx;isOutput=true;]xxx'
name: PSName
The name must be unique so that we can use format $(referencename.variablename) to access output variable from specific step.
The error indicates that some steps in your templates have the same name version! And this is not supported. About why this issue occurs:
1.You called the same template several times, this is the main cause of your issue.
Devops expand templates first when processing the pipeline, so if your fetch-tag template has one step named version, the final expended azure-pipeline.yml would be:
stages:
- stage: Terraform
jobs:
- job: Build
displayName: Build Terraform Infra
steps:
- task: PowerShell#2
inputs:
targetType: 'inline'
script: |
Write-Host "Hello World"
name: version
...
- task: PowerShell#2
inputs:
targetType: 'inline'
script: |
Write-Host "Hello World"
name: version
...
- task: PowerShell#2
inputs:
targetType: 'inline'
script: |
Write-Host "Hello World"
name: version
...
2.You may also need to check if there're steps with same name in your different templates.
We can call same template more than once in one pipeline, but we can't call same templates in which there's named step. Cause the pipeline will expand the templates more than once and the final pipeline would contain many steps with same name. And this is not supported since the name should be unique.
Solution:
1.Remove the name element of your steps if you don't need to use output variables mentioned above.
2.Or you can make several copies of fetch-tag.yml and name them fetch-tag-api.yml, fetch-tag-front.yml and fetch-tag-strapi.yml. Rename the referenceName version in these three files to version1, version2 or what. Then you can run the pipeline with:
steps:
# Set and Export env var for api version to deploy
- template: templates/fetch-tag-api.yml
parameters:
repo: 'api'
envVar: TERRAFORM_API_TAG
# Set and Export env var for front version to deploy
- template: templates/fetch-tag-front.yml
parameters:
repo: 'front'
envVar: TERRAFORM_FRONT_TAG
# Set and Export env var for strapi version to deploy
- template: templates/fetch-tag-strapi.yml
parameters:
repo: 'strapi'
envVar: TERRAFORM_STRAPI_TAG
I have the following pipeline
variables:
azureSubscription: ...
stages:
- stage: Deploy
displayName: Deploy stage
dependsOn: Build
condition: succeeded()
jobs:
- deployment: Deploy
displayName: Deploy
environment: 'development'
strategy:
runOnce:
deploy:
steps:
- task: AzureResourceGroupDeployment#2
inputs:
action: 'Create Or Update Resource Group'
resourceGroupName: '...'
location: '...'
templateLocation: 'Linked artifact'
csmFile: '$(Pipeline.Workspace)/azure-deploy.json'
deploymentMode: 'Incremental'
The repo has the following files (at the root directory)
azure-pipelines.yaml
azure-deploy.json
and only a master branch.
I have tried:
azure-deploy.json
**azure-deploy.json
**/*azure-deploy.json
$(Build.SourcesDirectory)/azure-deploy.json
$(Pipeline.Workspace)/azure-deploy.json
$(System.DefaultWorkingDirectory)/azure-deploy.json
Having read:
Azure Pipeline Error: Could not find any file matching the template file pattern
VSTS Pipeline Deployment of ARM Error: Could not find any file matching the template file pattern
https://github.com/microsoft/azure-pipelines-tasks/issues/11520
to no avail. Any ideas?
Update: I have added a publish pipeline as suggested by #ShaykiAbramczyk
Now I get a Template file pattern matches a directory instead of a file: /home/vsts/work/1/azure-deploy.json
- stage: Build
displayName: Build stage
jobs:
- job: Build
displayName: Build
pool:
vmImage: $(vmImageName)
steps:
- task: PublishPipelineArtifact#1
inputs:
targetPath: '$(Pipeline.Workspace)'
artifact: 'azure-deploy.json'
publishLocation: 'pipeline'
"A deployment job doesn't automatically clone the source repo. You can checkout the source repo within your job with checkout: self."
Source: https://learn.microsoft.com/en-us/azure/devops/pipelines/process/deployment-jobs?view=azure-devops
Example from my setup, I put checkout: self as the first step and now my repository is cloned before before executing the Azure PowerShell:
strategy:
runOnce:
deploy:
steps:
- checkout: self
- task: AzurePowerShell#5
displayName: Setup Network
inputs:
It is a good strategy to go for the multi-stage pipelines for what you are doing.
Build is for composing your artifacts.
Deployment jobs is for the publishing part.
So you are on the right track.
If you need sources during the deployment jobs then use the checkout step to fetch the sources. ref. Repo Checkout docs.
Just my two cents
Because you use deployment job the sources from the master branch didn't download into the agent.
You need to publish the files in the build stage and consume them in the deployment - with pipeline artifacts.
Or, just run the AzureResourceGroupDeployment in a regular job, than the .json file will be in the agent.
My objective was to create a Blank and Starter Resource Group in Azure and nothing else. And this was repeatedly giving me the error
Template file pattern matches a directory instead of a file: /home/vsts/work/1/s
And thats what got me here.
I finally sorted that out with Stringfello's
- checkout self
step.
My full pipeline is as follows.
variables:
- name: finalBuildArtifactName
value: 'aspNetCoreDropFromYaml123'
- name: BuildParameters.RestoreBuildProjects
value: '**/*.csproj'
- name: BuildParameters.TestProjects
value: '**/*[Tt]ests/*.csproj'
- name: buildConfiguration
value: 'Release'
trigger:
- master
name: $(date:yyyyMMdd)$(rev:.r)
stages:
- stage: Build ##Basically prints out some vars and copies the template files.
jobs:
- job: buildWebApp
displayName: Build Stage for somepurpose
pool:
vmImage: ubuntu-latest
steps:
- script: |
echo build.artifactstagingdirectory and build.buildnumber are as follows.
echo $(build.artifactstagingdirectory) $(build.buildnumber)
echo $(projects)
echo $(BuildConfiguration)
echo Pipeline.Workspace is $(Pipeline.Workspace)
echo The current branch is - $(Build.SourceBranchName)!!.
echo $(finalBuildArtifactName)
echo "This is the build pipe line. This produces the necessary artifacts for subsequent release pipeline."
displayName: 'Command Line Script to write out some vars'
- powershell: |
# Write your PowerShell commands here.
Write-Host "This is from power shell command task"
Write-Host "This writes out the env vars"
get-childitem -path env:*
displayName: 'PowerShell script to write out env vars'
# The following task is needed. This copies the arm template files.
# Created these templates from Visual studio 2019 as follows.
# Right click your solution and Add -> New Project -> Azure Resource Group and gave the name Vivek_Aks_Rg
- task: CopyFiles#2
inputs:
SourceFolder: 'iac/ArmTemplates/Vivek_Aks_Rg/'
Contents: 'azuredeploy*.json'
TargetFolder: '$(build.artifactStagingDirectory)/ArmTemplates'
- task: PublishBuildArtifacts#1
displayName: Publish Artifact
condition: succeededOrFailed()
inputs:
PathtoPublish: '$(build.artifactstagingdirectory)'
ArtifactName: '$(finalBuildArtifactName)'
PublishLocation: 'Container'
- stage: DeployToDev
displayName: Deploy to Dev Env
jobs:
- deployment:
pool:
vmImage: ubuntu-latest
environment: Dev
strategy:
runOnce:
deploy:
steps:
- checkout: self
- task: AzureResourceManagerTemplateDeployment#3
displayName: 'Create Azure App Service in a Given Resource Group'
inputs:
deploymentScope: 'Subscription'
azureResourceManagerConnection: 'Pay-As-You-Go(123YourSubscriptionId)'
subscriptionId: '123YourSubscriptionId'
action: 'Create Or Update Resource Group'
resourceGroupName: 'YourResourceGroupName'
location: 'Central India'
csmFile: '$(Pipeline.Workspace)/$(finalBuildArtifactName)/ArmTemplates/azuredeploy.json'
csmParametersFile: '$(Pipeline.Workspace)/$(finalBuildArtifactName)/ArmTemplates/azuredeploy.parameters.json'
deploymentMode: 'Incremental'
I have a solution that contains the following
Several Asp.net Projects (Microservices and Gateway)
.Net Core + Angular 8 (Front End)
When I hit the build button in Visual Studio every project is built. I have created a repo and uploaded the solution. Now I'm trying to figure out how to setup the pipeline to build each microservice and deploy them to individual Azure Web Apps.
I have the following Pipeline for the Angular Project. Should I define separate tasks like this? Is there a way to replicate the Visual Studio build here?
# ASP.NET
# Build and test ASP.NET projects.
# Add steps that publish symbols, save build artifacts, deploy, and more:
# https://learn.microsoft.com/azure/devops/pipelines/apps/aspnet/build-aspnet-4
trigger:
- master
pool:
vmImage: 'windows-latest'
variables:
solution: '**/*.sln'
buildPlatform: 'Any CPU'
buildConfiguration: 'Release'
steps:
- task: NuGetToolInstaller#1
- task: Npm#1
inputs:
command: install
workingDir: 'd:\a\1\s\Ok.Web\ClientApp'
- task: Npm#1
inputs:
command: custom
workingDir: 'd:\a\1\s\Ok.Web\ClientApp'
customCommand: run build
- task: CopyFiles#2
inputs:
targetFolder: '$(Build.ArtifactStagingDirectory)'
- task: PublishBuildArtifacts#1
You may apply one of two approaches here:
One pipeline for whole repo
One pipeline for project
In both cases you may use templates to avoid repeating yourself; so you will define common tasks for building a .NET project and then use them in pipelines. I recently made a blog post about this, but please take a look at the documentation too.
To achieve this you need to first define a YAML file with common steps. For instance:
parameters:
- name: buildConfiguration # name of the parameter; required
default: 'Release'
- name: projectFolder
default: ' '
steps:
- task: DotNetCoreCLI#2
displayName: Restore nuget packages
inputs:
command: restore
projects: '*.csproj'
workingDirectory: '${{ parameters.projectFolder}}'
- task: DotNetCoreCLI#2
displayName: Build
inputs:
command: build
projects: '${{ parameters.projectFolder}}'
arguments: '--configuration ${{ parameters.buildConfiguration }}'
# You just added coverlet.collector to use 'XPlat Code Coverage'
- task: DotNetCoreCLI#2
displayName: Test
inputs:
command: test
projects: '*Tests/*.csproj'
arguments: '--configuration ${{ parameters.buildConfiguration }} --collect:"XPlat Code Coverage" -- RunConfiguration.DisableAppDomain=true'
workingDirectory: '${{ parameters.projectFolder}}'
- task: DotNetCoreCLI#2
inputs:
command: custom
custom: tool
arguments: install --tool-path . dotnet-reportgenerator-globaltool
displayName: Install ReportGenerator tool
- script: ./reportgenerator -reports:$(Agent.TempDirectory)/**/coverage.cobertura.xml -targetdir:${{ parameters.projectFolder}}/coverlet/reports -reporttypes:"Cobertura"
displayName: Create reports
- task: PublishCodeCoverageResults#1
displayName: 'Publish code coverage'
inputs:
codeCoverageTool: Cobertura
summaryFileLocation: ${{ parameters.projectFolder}}/coverlet/reports/Cobertura.xml
And then invoke this file from you main build file:
variables:
buildConfiguration: 'Release'
projectFolder: 'path to your project'
steps:
- template: build-and-test.yaml
parameters:
buildConfiguration: $(buildConfiguration)
- script: echo Some steps to create artifacts!
displayName: 'Run a one-line script'
In approach number 1 you will build all projects even if you change just one project, so I would recommend you use approach number 2. For this you may define multiple pipelines (one per project) and limit triggers to changes in specific folder. Please take a look here.
Here you have an example of how you can limit triggers to specific folder for master branch:
trigger:
branches:
include:
- master
paths:
include:
- gated-checkin-with-template/*
exclude:
- gated-checkin-with-template/azure-pipelines-gc.yml