Azure Pipeline- Copy files from one Repo to another Repo using YAML - azure

There is a folder in one of the repositories (Source Repo) that I like to copy to another repository (Destination Repo) using Azure Pipeline (as they needed to be in sync)
so far I can Copy a folder in the same repository using:
- task: CopyFiles#2
inputs:
SourceFolder: '$(Build.Repository.LocalPath)\MyFolder\'
Contents: |
**
!**\obj\**
!**\bin\**
TargetFolder: '$(Build.Repository.LocalPath)\DestFolder'
flattenFolders: false
CleanTargetFolder: true
OverWrite: true
preserveTimestamp: true
this is how I connect to another repository:
resources:
repositories:
- repository: SourceRepo
type: git
name: MyCollection/SourceRepo
but I don't know how to get files from the source repo and place them in the Destination Repo

after a lot of searching, this is the answer:
resources:
repositories:
- repository: SourceRepo
type: git
name: MyCollection/SourceRepo
steps:
- checkout: SourceRepo
clean: true
- checkout: self
persistCredentials: true
clean: true
- task: DotNetCoreCLI#2
displayName: "restore DestRepo"
inputs:
command: 'restore'
projects: '$(Build.Repository.LocalPath)/DestRepo/**/*.csproj'
feedsToUse: 'select'
- task: DotNetCoreCLI#2
displayName: "build DestRepo"
inputs:
command: 'build'
projects: '$(Build.Repository.LocalPath)/DestRepo/DestRepo/**/*.csproj'
configuration: Release
# configurations for using git command
- task: CmdLine#2
inputs:
script: |
cd $(Agent.HomeDirectory)\externals\git\cmd
git config --global user.email ""
git config --global user.name "$(Build.RequestedFor)"
- task: CmdLine#2
displayName: checkout
inputs:
script: |
git -C RootRep checkout $(Build.SourceBranchName)
- task: CmdLine#2
displayName: pull
inputs:
script: |
git -C DestRepo pull
- task: CopyFiles#2
inputs:
SourceFolder: '$(Build.Repository.LocalPath)\SourceRepo\SourceFolder'
Contents: |
**
!**\obj\**
!**\bin\**
TargetFolder: '$(Build.Repository.LocalPath)\DestRepo\DestFolder'
flattenFolders: false
CleanTargetFolder: true
OverWrite: true
# preserveTimestamp: true
- task: CmdLine#2
displayName: add
inputs:
script: |
git -C DestRepo add --all
- task: CmdLine#2
displayName: commit
continueOnError: true
inputs:
script: |
git -C DestRepo commit -m "Azure Pipeline Repository Integration"
- task: CmdLine#2
displayName: push
inputs:
script: |
git -C DestRepo push -u origin $(Build.SourceBranchName)

I was trying to find some solution related to this problem, but instead of using a copy file task, I found a better way and we can use any number of repositories are resources in the build pipeline and we don't need to check out all these.
This is how my build pipeline looks like.
As you can see I have used two variables
$(System.AccessToken), this variable is available in Azure DevOps aka PAT(Personal Access Token)
$(Build.Repository.Uri) URL of the repository (this could be the URL of any repo in resources).

Related

Azure Devops - Sign Mac OS .app and publish Artifact

I am trying to automate the process of signing a .app Binary file using Azure Devops CI pipelines. The current CI performs the following tasks:
CI builds a "setup" container containing the files to compile and pushes to our container registry
Runs a script against an Azure VM to pull this image down and compile against in an Ubuntu VM
The Ubuntu VM is required because the compiling process uses a GUI and a display on xhost is required for it to run.
Once it is done compiling we then zip the binary's and publish it to Azure Devops Artifacts.
What I wanted to do here is pull down the binary and then sign it and push it back up, but the pipeline just hangs indefinitely until it times out. My assumption is that it is waiting for some kind of prompt, and I have no idea how to pass it
name: $(Date:yyyyMMdd)$(Rev:.r)
trigger: none
pr: none
variables:
- group: MacOS
jobs:
- job: POC_Pipeline
pool:
vmImage: 'macOS-latest'
steps:
# - task: InstallAppleCertificate#2
# inputs:
# certSecureFile: '$(p12FileName)'
# certPwd: '$(p12Password)'
# keychain: 'temp'
# deleteCert: true
- task: DownloadSecureFile#1
name: AppleCertificate
displayName: 'Download Apple Certificate'
inputs:
secureFile: '$(p12FileName)'
- task: DownloadPackage#1
inputs:
packageType: 'upack'
feed: 'myfeed'
definition: 'mybinary'
version: '*' # Pulls latest
downloadPath: '$(System.ArtifactsDirectory)'
- script: 'security create-keychain -p password temp.keychain'
workingDirectory: '$(System.ArtifactsDirectory)'
displayName: 'Create Keychain'
failOnStderr: true
- script: 'security unlock-keychain -p password temp.keychain'
workingDirectory: '$(System.ArtifactsDirectory)'
displayName: 'Unlock Keychain'
failOnStderr: true
- script: 'security import $(AppleCertificate.secureFilePath) -k temp.keychain -P $(p12Password) -T /usr/bin/codesign'
workingDirectory: '$(System.ArtifactsDirectory)'
displayName: 'Install Apple Certificate'
failOnStderr: true
- script: 'security find-certificate temp.keychain'
workingDirectory: '$(System.ArtifactsDirectory)'
displayName: 'Find Certificate'
failOnStderr: true
- script: 'security find-identity -p codesigning -v keychain temp.keychain'
workingDirectory: '$(System.ArtifactsDirectory)'
displayName: 'Find Identity'
failOnStderr: true
- script: 'security default-keychain -s "/Users/runner/Library/Keychains/temp.keychain-db"'
workingDirectory: '$(System.ArtifactsDirectory)'
displayName: 'Set Default Keychain'
failOnStderr: true
- script: 'unzip -q myBinary.app.zip'
workingDirectory: '$(System.ArtifactsDirectory)'
displayName: 'Unzip myBinary'
failOnStderr: true
- script: 'xattr -rc myBinary.app'
workingDirectory: '$(System.ArtifactsDirectory)'
displayName: 'Extended Attributes'
failOnStderr: true
- script: 'sudo codesign -s Anasazi -f --deep myDinary.app'
workingDirectory: '$(System.ArtifactsDirectory)'
displayName: 'Codesign Dragon.app'
failOnStderr: true
- script: 'codesign -dv myBinary.app'
workingDirectory: '$(System.ArtifactsDirectory)'
displayName: 'Verify Codesign myBinary.app'
failOnStderr: true
Any assistance or recommendations how we can sign the .app file in the CI would be appreciated.

Running .CMD Script in YAML

I have recently needed to run a .CMD script using YAML in Azure DevOps. I can get it to run and put files in the root of $(System.DefaultWorkingDirectory) but I need it to put the files in $(System.DefaultWorkingDirectory)/teacher/website. Every time I try to do this In two different ways I come up short.
I have first tried with this method:
Batch script
Run a Windows command or batch script and optionally allow it to change the environment
- task: BatchScript#1
inputs:
filename: 'scripts/deploy.cmd'
arguments: # Optional
modifyEnvironment: False # Optional
workingFolder: $(Build.ArtifactStagingDirectory)/teacher/website # Optional
failOnStandardError: false # Optional
This way will just put the files in the $(Build.ArtifactStagingDirectory) which is not what I want, as I have some .Net Build files going there.
I then tried this way:
- script: 'scripts/deploy.cmd' # script path or inline
workingDirectory: '$(Build.ArtifactStagingDirectory)/teacher/website'
displayName: run deploy.cmd
#failOnStderr: #
#env: # mapping of environment variables to add
This way I get an error that says: ##[error]Container path not found: C:\azp\agent\_work\17\a\Teacher\website
I am trying to build all this into an artefact and then that artefact is later deployed to the web app. Here is my fully YAML file so you have an idea of what I am doing:
parameters:
- name: buildConfiguration
type: string
default: 'Release'
- name: project
type: string
default: 'Teacher.csproj'
- name: artifactName
type: string
default: 'Teacher'
jobs:
- job:
pool:
name: 'DotNet6_Terraform'
steps:
- checkout: Teacher
submodules: true
- task: CmdLine#2
displayName: make the Teacher Folder
inputs:
script: 'mkdir Teacher'
workingDirectory: $(Build.ArtifactStagingDirectory)
- task: CmdLine#2
displayName: make website directory
inputs:
script: 'mkdir website'
workingDirectory: $(Build.ArtifactStagingDirectory)/Teacher
- task: CmdLine#2
displayName: Check for folders
inputs:
script: |
echo '$(Build.ArtifactStagingDirectory)/Teacher'
echo '$(Build.ArtifactStagingDirectory)/Teacher/website'
- task: CmdLine#2
displayName: show directory tree
inputs:
script: |
cd '$(System.DefaultWorkingDirectory)/Teacher'
dir
- task: DotNetCoreCLI#2
displayName: dotnet restore
inputs:
command: restore
projects: 'Teacher/**/*.csproj'
- task: CmdLine#2
displayName: checking dotnet versions
inputs:
script: |
dotnet --list-sdks
dotnet --list-runtimes
- task: CmdLine#2
displayName: dotnet build
inputs:
script: |
dotnet build Teacher/Teacher.csproj --configuration Release
- task: DotNetCoreCLI#2
displayName: 'Publish Application'
inputs:
command: 'publish'
publishWebProjects: false
projects: '**/*.csproj'
arguments: '--configuration ${{ parameters.buildConfiguration }} --output $(Build.ArtifactStagingDirectory)/Teacher'
publishTestResults: false
zipAfterPublish: true
modifyOutputPath: false
workingDirectory: Teacher
- task: PowerShell#2
displayName: List Files Post Publish
inputs:
targetType: inline
script: Get-ChildItem -path '$(Build.ArtifactStagingDirectory)/Teacher'
- script: 'scripts/deploy.cmd' # script path or inline
workingDirectory: '$(Build.ArtifactStagingDirectory)/Teacher/website'
displayName: run deploy.cmd
#failOnStderr: #
#env: # mapping of environment variables to add
# Batch script
# Run a Windows command or batch script and optionally allow it to change the environment
#- task: BatchScript#1
# inputs:
# filename: 'scripts/deploy.cmd'
#arguments: # Optional
#modifyEnvironment: False # Optional
# workingFolder: $(Build.ArtifactStagingDirectory)/Teacher/website # Optional
# failOnStandardError: false # Optional
- task: PublishPipelineArtifact#1
displayName: 'Publish Artifacts'
inputs:
targetPath: '$(Build.ArtifactStagingDirectory)'
artifact: ${{ parameters.artifactName }}
publishLocation: 'pipeline'

Is it possible to copy build artifact from pipeline into repo

I'm trying to copy my published zip build into my azure repo for easy access. I have the following YML code. See inline comment with my question:
- task: ArchiveFiles#2
inputs:
rootFolderOrFile: '$(build.artifactstagingdirectory)\MyFolder'
includeRootFolder: true
archiveType: 'zip'
archiveFile: '$(Build.ArtifactStagingDirectory)\zipfolder/$(Build.BuildId).zip'
replaceExistingArchive: true
- task: PublishBuildArtifacts#1
displayName: 'Publish Artifact: drop'
inputs:
PathtoPublish: '$(Build.ArtifactStagingDirectory)\zipfolder'
TargetPath: '$(Build.SourcesDirectory)\MyFolder' // Here I would expect the code to copy the zip into my repo.
You could download the artifact and use the git command in command line task to push it to the repo, refer to the sample as below, it works for me.
# 'Allow scripts to access the OAuth token' was selected in pipeline. Add the following YAML to any steps requiring access:
# env:
# MY_ACCESS_TOKEN: $(System.AccessToken)
# Variable Group 'vargroup1' was defined in the Variables tab
resources:
repositories:
- repository: self
type: git
ref: refs/heads/testb2
jobs:
- job: Job_1
displayName: Agent job 1
pool:
vmImage: ubuntu-20.04
steps:
- checkout: self
persistCredentials: True
- task: ArchiveFiles#2
displayName: Archive README.md
inputs:
rootFolderOrFile: README.md
archiveFile: $(Build.ArtifactStagingDirectory)\zipfolder/$(Build.BuildId).zip
- task: PublishBuildArtifacts#1
displayName: 'Publish Artifact: drop'
inputs:
PathtoPublish: $(Build.ArtifactStagingDirectory)\zipfolder
- task: DownloadBuildArtifacts#1
displayName: Download Build Artifacts
inputs:
artifactName: drop
- task: CmdLine#2
displayName: Command Line Script
inputs:
script: >-
cd $(System.ArtifactsDirectory)\zipfolder
git config --global user.email "xxxxx"
git config --global user.name "xxxxx"
git init
git add .
git commit -m "123"
git remote add origin https://$(System.AccessToken)#dev.azure.com/orgname/testpro1/_git/testpro4
git push https://$(System.AccessToken)#dev.azure.com/orgname/testpro1/_git/testpro4
...
The alternative is to push to repo and then sync from the remote git/ - basically a automated way of pulling the file over.

Use template in Azure pipeline for repository refs

I have a project with multiple repositories and I would like to use a specific branch to checkout for RepoB, that matches the target branch of the PR
name: My nice pipeline
resources:
repositories:
- repository: RepoB
type: git
name: RepoB
# ref: ${{ replace($(System.PullRequest.TargetBranch), "refs/heads/", "")}} <<<< here is the issue
And the issue is that if I use just a plane $(System.PullRequest.TargetBranch) the input is accepted but the way ref: works, it will append to refs/heads/{{whatever the input was here}}, so I end up having refs/heads/refs/heads/target-branch. As a solution I though I can use replace but yeah thanks not the case...
Any ideas how I can grab just the target branch name without the prefix?
You can't use System.PullRequest.TargetBranch in template expression (which is {{}}). You have marked it here
And I can't find it at the moment, but from what I recall you can't use template expression in repository resource.
This is unfortunately not possible. You can't use expression here.
And if you try use this syntax ref:$(System.PullRequest.TargetBranch), you will get
/azure-pipelines-11.yml: Could not get the latest source version for repository kmadof/devops-templates hosted on https://github.com/ using ref refs/heads/$(System.PullRequest.TargetBranch). GitHub reported the error, "No commit found for SHA: refs/heads/$(System.PullRequest.TargetBranch)"
I had the same problem as you when I wanted to create a tag on a branch and specific commit in my pipeline. The best solution I found was just to make my agent do all the parts I would normally do and put it in a template :p this is how I solved it.
steps:
- checkout: none
clean: True
- task: PowerShell#2
displayName: Remove refs/heads from branch path and set pipeline variable
inputs:
targetType: 'inline'
script: |
$branchSourcePath = "$(Build.SourceBranch)" -replace "refs/heads/",""
Write-Host "##vso[task.setvariable variable=BRANCH_PATH;]$branchSourcePath"
Write-Host "Setting pipeline variable BRANCH_PATH to $branchSourcePath"
- task: PowerShell#2
displayName: Create temp folder
inputs:
targetType: 'inline'
script: 'New-Item -Path "$(Pipeline.Workspace)\temp" -ItemType Directory'
- task: PowerShell#2
displayName: Init git repo
inputs:
targetType: 'inline'
script: 'git init'
workingDirectory: '$(Pipeline.Workspace)\temp'
- task: PowerShell#2
displayName: Git remote add
inputs:
targetType: 'inline'
script: git remote add $(Build.Repository.Name) https://$(git-accesstoken)#{azure-repo-path}/_git/$(Build.Repository.Name)
workingDirectory: '$(Pipeline.Workspace)\temp'
- task: PowerShell#2
displayName: Fetch
inputs:
targetType: 'inline'
script: 'git fetch $(Build.Repository.Name) $(BRANCH_PATH)'
workingDirectory: '$(Pipeline.Workspace)\temp'
- task: PowerShell#2
displayName: Checkout commit
inputs:
targetType: 'inline'
script: |
Write-Host "Checking out commit $(Build.SourceVersion) \n"
git checkout -b $(BRANCH_PATH) $(Build.SourceVersion)
workingDirectory: '$(Pipeline.Workspace)\temp'
- task: PowerShell#2
displayName: Set user email
inputs:
targetType: 'inline'
script: git config --global user.email "$(Build.RequestedForEmail)"
workingDirectory: '$(Pipeline.Workspace)\temp'
- task: PowerShell#2
displayName: Set user name
inputs:
targetType: 'inline'
script: git config --global user.name "$(Build.RequestedFor)"
workingDirectory: '$(Pipeline.Workspace)\temp'
- task: PowerShell#2
displayName: Create tag
inputs:
targetType: 'inline'
script: 'git tag -a {tag-name} $(Build.SourceVersion) -m "{tag-description}"'
workingDirectory: '$(Pipeline.Workspace)\temp'
- task: PowerShell#2
displayName: 'Push tag'
inputs:
targetType: 'inline'
script: 'git push {repo-name} {tag-name}'
workingDirectory: '$(Pipeline.Workspace)\temp'
- task: DeleteFiles#1
displayName: Delete temp folder
inputs:
SourceFolder: '$(Pipeline.Workspace)'
Contents: 'temp'

WARN unable to find package.json for plottable

When running a build for storybook via an Azure devops pipeline I get the above mentioned error.
I've tried to completely remove this package(plottable) from my project but I keep getting this error and it causes my webpack build to get stuck.
This error doesn't occur locally.
My pipeline:
trigger:
batch: true
branches:
include:
- master
stages:
- stage: develop_build_deploy_stage
pool:
name: Default
demands:
- msbuild
- visualstudio
jobs:
- job: develop_build_deploy_job
steps:
- checkout: self
clean: true
persistCredentials: true
- task: NodeTool#0
displayName: Install Node
inputs:
versionSpec: '12.x'
- task: PowerShell#2
displayName: 'Install Dependencies'
inputs:
targetType: 'inline'
script: |
npm install
- task: PowerShell#2
displayName: 'Increment version'
inputs:
targetType: 'inline'
script: |
git checkout master
git pull origin master
git config --global user.email "d#gmail.com"
git config --global user.name "Build Agent"
npm version patch -m "Increment Version [skip ci]" --force
git push
- task: PowerShell#2
displayName: 'Build Project'
inputs:
targetType: 'inline'
script: |
npm run build-storybook
npm run build
- task: CopyFiles#2
displayName: 'Copy storybook-static Files'
inputs:
sourceFolder: '$(Build.SourcesDirectory)/storybook-static'
contents: '**'
targetFolder: '$(Build.ArtifactStagingDirectory)'
- task: PublishBuildArtifacts#1
displayName: 'Publish storybook-static Files to ArtifactStagingDirectory'
inputs:
pathToPublish: '$(Build.ArtifactStagingDirectory)'
artifactName: Storybook
- task: S3Upload#1
displayName: 'Upload storybook-static to S3'
inputs:
awsCredentials: 'my-s3'
regionName: 'us-east-1'
bucketName: 'my-s3-bucket'
sourceFolder: $(Build.ArtifactStagingDirectory)
- task: Npm#1
displayName: 'Publish to Feed'
inputs:
command: 'publish'
publishRegistry: 'useFeed'
publishFeed: '#####'
How would I go about resolving this problem?
I still don't know what caused this to be the case but for some reason the changes I had made on my branch weren't being picked up and an import that I had removed wasn't removed on the branch that was being built on.
So in the end it was trying to import a package which I had removed as a dependency.

Resources