How do I upload Artifacts depending on Release or Prerelease to Artifactory using GitVersion on Azure DevOps? - azure

I prefer to organize my artifacts in Artifactory in a hierarchy of Repo[dev|test|prod] -> Artifact Name -> Releases Artifacts go here -> Pre-Releases go into a sub-folder.
Why? So when I am navigating the Artifactory Repository Browser I don't have an exceedingly long tree. I can expand a repository and see the first level by artifact name and still not see any artifacts, then expand the artifact name leaf and then see my released artifacts. But, the top item underneath will be a sub-directory folder called "prerelease". This is done so I can easily manually delete all my pre-releases if I wish to do so in one action, or schedule to clean them up.
[My Repo]
|
+-\prerelease\
| |--artifact-1.2.3-ci0004.nupkg
| |--artifact-1.0.1-ci0002.nupkg
|--artifact-1.0.0.nupkg
|--artifact-1.0.1.nupkg
I know how to use the Artifactory filespec to upload the package to my repository:
** For Pre-Release
{
"files": [
{
"pattern": "$(build.artifactstagingdirectory)\*.nupkg",
"target": "myrepo-nuget-dev-local/$(PackageName)/prerelease/"
}
]
}
** For Release
{
"files": [
{
"pattern": "$(build.artifactstagingdirectory)\*.nupkg",
"target": "myrepo-nuget-dev-local/$(PackageName)/"
}
]
}
What I need to do is put each file spec into its own build step and then add conditions that will execute EITHER one build step OR the other, but never both. Why? Because the build artifact will ever be a pre-release or a release artifact but never both. I am using GitVersion and Git Tags along with Azure DevOps.
So the question: What does the Custom Condition need to be to get this working?

This logic should work for any CI system, but this syntax will work for Azure DevOps.
How to create these can be found here: https://learn.microsoft.com/en-us/azure/devops/pipelines/process/expressions?view=azure-devops
Here is what it looks like:
For Pre-Release:
and(succeeded(), not(startsWith(variables['GitVersion.BranchName'], 'tags/')), or(ne(variables['GitVersion.PreReleaseLabel'], ''),ne(variables['GitVersion.BuildMetaData'], '')))
This is saying all 3 conditions MUST be met:
if succeeding
GitVersion.BranchName does not start with 'tags/' (this makes sure this build event was not triggered by a tag), and,
GitVersion.PreReleaseLabel is not empty OR GitVersion.BuildMetaData is not empty
For Release:
and(succeeded(), or(and(eq(variables['GitVersion.PreReleaseLabel'], ''), eq(variables['GitVersion.BuildMetaData'], ''), eq(variables['GitVersion.BranchName'], 'master')), startsWith(variables['GitVersion.BranchName'], 'tags/')), ne(variables['Build.Reason'], 'PullRequest'))
This is saying all 3 conditions MUST be met:
if succeeding
(GitVersion.PreReleaseLabel is empty AND GitVersion.BuildMetaData is empty AND GitVersion.BranchName is 'master') OR (GitVersion.BranchName starts with 'tags/')
Build.Reason is not 'PullRequest'
Here is what it looks like:

Related

Unable to run Azure pipeline "A task is missing. The pipeline references a task called 'Cache'

I am setting up my fork of a Github project with the azure_pipeline.yaml configuration.
This seems to work just fine for everyone else in the community but when I setup the pipeline it gives me the following exception:
A task is missing. The pipeline references a task called 'Cache'. This usually indicates the task isn't installed, and you may be able to install it from the Marketplace: https://marketplace.visualstudio.com. (Task version 2, job 'compile_ci_build', step ''.)
A task is missing. The pipeline references a task called 'Cache'. This usually indicates the task isn't installed, and you may be able to install it from the Marketplace: https://marketplace.visualstudio.com. (Task version 2, job 'test_ci_build', step ''.)
A task is missing. The pipeline references a task called 'Cache'. This usually indicates the task isn't installed, and you may be able to install it from the Marketplace: https://marketplace.visualstudio.com. (Task version 2, job 'e2e_ci_build', step ''.)
Specifically,
Here is my Azure pipeline link
I am creating Flink CI build pipeline according to this instruction.
Which already have an azure-pipeline.yml in the repo
It uses the template to run the job parameterized in tools/azure-pipelines/jobs-template.yml
[UPDATE]
I modified the jobs-templates.yml and commented out all steps with Cache#2 and it runs fine.
Was able to get this working eventually.
Apparently for my Azure account. I am not allow to use Cache#2.
changing all lines with
- task: Cache#2
to
- task: CacheBeta#1
resolves all my problem.
By design, Azure DevOps does not automatically make all tasks available when you run a pipeline.
You have to add them manually as part of the pipeline.
'cache' is one such task.
I'm doing this in classic gui (non-yaml) mode as I find it easier to search for things
But what you do is in your pipeline add a new task and in the task search box type 'cache'.
This will bring up the task.
Click Add to include it in the pipeline.
For more information on this I would recommend reading:
https://learn.microsoft.com/en-us/azure/devops/pipelines/release/caching?view=azure-devops
I do not see you added cache task in your pipeline from your git repo.
It should look like the example here:
variables:
YARN_CACHE_FOLDER: $(Pipeline.Workspace)/.yarn
steps:
- task: Cache#2
inputs:
key: 'yarn | "$(Agent.OS)" | yarn.lock'
restoreKeys: |
yarn | "$(Agent.OS)"
yarn
path: $(YARN_CACHE_FOLDER)
displayName: Cache Yarn packages
- script: yarn --frozen-lockfile
Source
I was getting the same error A task is missing. The pipeline references a.. on one of azure task, PublishCucumberReport#1 . I resolved it by visiting https://marketplace.visualstudio.com/ and going to the task and then clicking on get free button, which installs it on your pipeline

Azure Devops - run build only in the folder that have changes

I have a some spark jobs which are written on scala and build using maven.
Right now the follow a path like this
/src/job1/<<build data>>
/src/job2/<<build data>>
.....
This build data holds the code, pom.xml, tests and other thinks needed for each job pass to maven.
We have a root folder that has a father pom.xml that builds all the code, but I don't want to build all the code every time. I want to be able to test and build only the code for the jobs that changes.
The only way that I think is possible is to run build an tests only in the folders that changes was detected but I can't seem to do that without create one azure devops build for each folder, any ideas?
TL:DR - How to pass the path of folder that have changes in azure devops to the build process?
How to pass the path of folder that have changes in azure devops to the build process?
We could add a powershell scripts to check the folder or path, like:
$editedFiles = git diff HEAD HEAD~ --name-only
echo "$($editedFiles.Length) files modified:"
$editedFiles | ForEach-Object {
echo $_
Switch -Wildcard ($_ ) {
'/src/job1' {
Write-Output "##vso[task.setvariable variable=job1]True"
}
'/src/job2' {
Write-Output "##vso[task.setvariable variable=job2]True" }
}
}
This script sets variables which are then referenced in custom conditions in the pipeline build task or in the build pipeline:
and(succeeded(), eq(variables['job1'], 'True'))
You could check this similar thread for some details.
Hope this helps.
You can work around this by creating a script that will do the following:
Get the changeset/commit in the build to check which files are changed.
Based on the files changes, you define the logic to decide which folder it should build, run tests, etc.
This solution will require you to build all the logic to decide which folders to build and test.
Thanks for your inputs. I was able to do it with your inputs on git, I realized that we are using gitflow workflow and every time that we have pull request to the master branch (meaning deployment) a commit is created with starting "Merged PR", this way I can compare the latest two commits and check what changed.
How I'm doing:
# This part uses this to get the latest two pull requests hashes
$last_pr_commit=git log --grep="^Merged PR" -n 1 --pretty=format:"%h" --abbrev=200
$second_last_pr_commit=git log -n 1 --grep="^Merged PR" --oneline --abbrev=200 $last_pr_commit^...HEAD~2 --pretty=format:"%h"
# This get the path of the files
$result=git diff $last_pr_commit $second_last_pr_commit --name-only
And then I just build the ones that I'm interested.
ForEach ($line in $($result -split "`r`n"))
{
# In my case I only need the two first parts of path
$folderPath=$line.split('/')[0]+'/'+$line.split('/')[1]
if ($line.contains('.scala')){
# Running the tests and packaging
mvn package $folderPath
}
}
This is powershell but it can work with bash changing the for loop.
Thanks

Gulp-compiled CSS folder missing from the Azure DevOps pipeline Build Artifact

A little background...
I have a small dotnet core application that is hosted on Azure and is being built and deployed using Azure DevOps Pipelines. Before we started using the DevOps Pipelines the CI was hooked up directly to Azure which compiled fine but took an actual lifetime to deploy, hence the decision to move.
However, the build pipeline no longer compiles or outputs the sass/css folder
Everything else works okay - I check in, the Build pipeline picks up my commits and has the following steps:
Restore [.NET Core]
Build [.NET Core]
Publish [.NET Core]
Publish Build Artifact
Part of step 3 (Publish) uses a Gulp task:
gulp.task('prod', function (callback) {
runSequence('clean','set-prod',
['icon-sprite', 'logo-sprite', 'images', 'sass', 'modernizr', 'mainjs', 'adminjs'],
callback);
});
And locally (and previously) this generated five folders:
icons
img
js
logos
css (now mysteriously missing in action)
Variations I've tried
I've tried deleting my local css folder and running the CLI dotnet publish exactly the same way the Pipeline does and that appears to work fine locally.
I've also stripped the sass task way back in case that was causing an issue somewhere in the pipeline, so that now looks like this:
return gulp.src('src/sass/style.scss')
.pipe(sass({outputStyle: 'compressed'}))
.pipe(gulp.dest('wwwroot/dist/css));
I can see all of the output in the console logs on the Pipeline and it successfully executes the sass task:
2019-01-02T14:43:51.3558593Z [14:43:51] Starting 'sass'...
2019-01-02T14:43:51.9284145Z [14:43:51] Finished 'sass' after 524 ms
There are no other errors or warnings in the build script and everything completes and fires off the Release pipeline (which copies the artifact up to the Azure site).
Speculation
I would expect an error somewhere... but nothing - all of the green ticks are downright cheerful... so I'm a little stumped at what may or may not be happening! I can only think that there must be some dependency or something missing in the Pipeline environment? Orrrrr maybe I'm missing a Pipeline step?
Any help or nudges or ideas would be greatly appreciated! Thank you for sticking it out through my small essay and for any help you can provide :)
Something I've done in this situation before is changing the Publish Build Artifact task to upload everything in the build folder. My guess is that right now the 'Path to Publish' value in that task is set to $(build.artifactStagingDirectory). Change it to $(build.SourcesDirectory). After running the build again you'll see that the entire build directory has was uploaded. This includes your source code and any other folders like you have on your local environment. From there you can figure out if the CSS folder is actually missing, or if it ended up in some other folder location.
If the folder ends up in a weird location you can either add a file copy task to move the CSS folder to the proper folder in $(build.artifactStagingDirectory) or make a change to the Gulp task. Whatever is better for your scenario.
Once you find the location, you can fix the Publish Build Artifact task.
I was having the exact same issue. I was able to get everything working locally without issue. gulp would generate the css folder just fine. dotnet publish -c release would do the same. However, when ran through the pipeline, no css folder.
The thing that I find the most strange, is that there is a sibling folder (scripts) that is used in the same way the css gulp task is used, but that folder makes it just fine. Here's my css task:
gulp.task('min', function() {
return gulp.src('wwwroot/css/**/*.css')
.pipe(cssnano({zindex:false}))
.pipe(gulp.dest('wwwroot/dist/css/'));
});
but, this task does works both locally and in the pipeline:
gulp.task('build-js', function() {
return gulp.src('wwwroot/scripts/**/*.js')
.pipe(concat('site.bundle.js'))
.pipe(uglify())
.pipe(gulp.dest('wwwroot/dist/scripts/'));
});
I ended up just giving up since this is legacy code anyways and settled on a workaround:
Add the Copy Files task right after your gulp task with the below configuration:
Or, if you like YAML:
steps:
- task: CopyFiles#2
displayName: 'Copy Files to: wwwroot/dist/css'
inputs:
SourceFolder: wwwroot/css
Contents: '*.css'
TargetFolder: wwwroot/dist/css

Concourse: Use a semver resource to control which artifact to use from s3

My pipeline contains a task with the following pre-requisites
- get: version
trigger: true
params: { bump: patch }
passed: ["trigger_job [CI]"]
- get: sdk-package
passed: ["package_generation_job"]
params:
version: {path: "artifact_[I want to put the version here]"}
version is a semver stored in git; sdk-package is a build artifact stored in s3 where each run of the pipeline puts a new artifact using the version number as part of the name.
What I would like to do is used the version input to determine which version of the artifact is pulled from S3. Based on this I suspect that Concourse doesn't allow this, but I couldn't find a definitive answer.
This is not currently possible, you will have to download the artifact you want in a task script. You can pass the version into that task.

Trigger build from bitbucket to Jenkins based on Tags (Declarative Syntax)

I am looking for a help in Jenkins to trigger a build when specific tags(release.*) are pushed.
So far I got the way of skipping the deployment if there are no proper tags (https://jenkins.io/blog/2018/05/16/pipelines-with-git-tags/)
stage('Deploy') {
when { tag "release-*" }
steps {
echo 'Deploying only because this commit is tagged...'
sh 'make deploy'
}
}
Looks for something which will help me to trigger the builds from feature branch whenever a developer will push a tag.

Resources