I want to be able to run automated tests from within Azure Test Plans and define everything in a yml file.
What I've done:
I have a repository in azure DevOps containing several tests written in .NET core. Those are associated to a testcase in azure test plans.
I also have created a azure pipeline by adding a .yml file to my repo. This pipeline consists of two stages, one for the build and one which actually runs all tests on a nightly schedule.
However, I want the manual testers to be able to trigger the automated tests from within Azure Test Plans too. The official docs say one must create a release pipeline to do so. So my questions are:
Why do I need a release pipeline? I already have my build pipeline, I won't do any release, I just want to be able to run my tests from within test plans by button click.
If I really have to create release pipeline, how can i define this in a yml file? The official docs only mention the "Classic", manual steps within azure DevOps, but I don't want to add pipelines manually, I want to have everything defined in yml files.
Is there a possibility to run my tests containerized? I do it already in my build pipeline, but since I couldn't find a way to define release pipelines in yml, I don't know how to run them in docker containers.
Azure Pipelines offers a compelling orchestration workflow to obtain test binaries as artifacts and run tests. This workflow shares the same concepts used in the scheduled testing workflow, meaning users running tests in scheduled workflow will find it easy adapt; for example, by cloning an existing scheduled testing release pipeline.
Another major benefit is the availability of a rich set of tasks catalog that enable a range of activates to be performed before and after running tests. Examples include preparing and cleaning test data, creating and cleaning configuration files, and more.
To define release pipeline in yml, you should go to the preview features page and turn on the toggle for “Multi-stage pipelines”.
Below is a simple sample.
stages:
- stage: Build
jobs:
- job: Build
pool:
vmImage: 'Ubuntu-16.04'
continueOnError: true
steps:
- script: echo my first build job
- stage: Deploy
jobs:
# track deployments on the environment
- deployment: DeployWeb
pool:
vmImage: 'Ubuntu-16.04'
# creates an environment if it doesn’t exist
environment: 'smarthotel-dev'
strategy:
# default deployment strategy
runOnce:
deploy:
steps:
- script: echo my first deployment
More detailed information, you can refer to https://devblogs.microsoft.com/devops/whats-new-with-azure-pipelines/
Related
New to Karate & Azure. Just created few API tests using Karate and easy simple. Want to take it further adding it Azure pipeline.
Found few links that just points to add pom.xml and maven from Microsoft. Also found Jekins integration but none for Azure.
Unknown / How to.
What file needs to be moved to the pipeline, eg, jar, war, xml etc.
How to create them i use intelliJ.
Any step by step tutorials available? any help appreciated.
To run Karate tests on azure devops pipeline, you can follow below general steps for building/testing java project.
1, First create your Karate tests projact with Maven. Add the related dependencies and plugins in the pom.xml. See example here.
2, Push your local source code(eg. .feature/.java/pom.xml etc) to github, or azure devop git repository. No need to push .jar dependencies, for the dependencies can be downloaded by the Maven task in the pipeline.
3, Create a azure pipeline, Follow this example to create a Yaml format pipeline. If you want to create a classic UI view pipeline, follow the example here.
4, Add Maven task in your pipeline to run the Karate test: See below example in Yaml.
steps:
- task: Maven#3
displayName: 'Maven Test'
inputs:
mavenPomFile: 'pom.xml'
goals: test
publishJUnitResults: false
If You use Microsoft cloud hosted agents to run your pipeline, you need to make sure the API tested by Karate can be accessed from the cloud hosted agents.(ie. API can be accessed publicly)
If the API server is hosted locally, you need to create self-hosted agent, and run your azure pipeline on your self-hosted agent.
#Devanathan
Better way is go for Publishing artifacts for both
a) your Cucumber Report
b) Your Karate HTML report.
Anyone can download both the artifacts (i.e reports from pipeline) and see the Report.
I tried to publish cucumber-html-reports (this is the exact name of folder that Karate generates) and was successful.
I intentionally kept report output directory in src/test/java for some reason [by using .reportDir("src/test/java/reports") in testParallel() method], you can try same by keeping reports in 'target' folder only.
I was not able to Publish Karate HTML report as somehow its not being picked in azure
Using multi project pipelines I want to run a specific job. Under deploy I have CI - PVT - Staging - Prod
I wish to deploy from a separate project simply Deploy Staging none of the other deploy stages.
I do have a paid for Gitlab so triggering etc. is enabled.
Is this possible please?
I am using azure build and release pipelines. I understand that build pipeline produces an artifact that can be used by release pipelines and deployment can be organized into stages with options to deploy/re-deploy stage.
However, I also notice that all tasks available in release pipeline are also available in the build pipeline. Also the documentation for build pipeline states that it can be used to build, test and deploy. So what exactly is the benefit of using release pipelines?
The pipelines under "Pipelines -> Releases", are classic deployment pipelines made with the GUI.
The new YAML pipelines are always under "Pipelines -> Pipelines" and allow both build and deployment jobs: https://learn.microsoft.com/en-us/azure/devops/pipelines/process/deployment-jobs?view=azure-devops.
By using deployment jobs instead of classic Releases, the deployments will show up under "Pipelines -> Environments".
So in this way it makes sense the same tasks are available on both sides since both allow defining deployment workflows.
YAML pipelines do not have the same capabilities as classic Releases yet though, I think release gates was still missing.
For classic UI-format build and release pipeline:
Usually we use Build Pipeline to build/test our project and output the tested software binaries.
Then we can use Release Pipeline to deploy the artifacts from Build Pipeline to more than one stages depending on we need. So that, we only need to run the Build Pipeline once to get the artifacts(containing the software binaries) and we can use Release Pipeline to reuse and deploy the artifacts to different environment. (Develop, Product...)
Of course we can use Build Pipeline to run some deploy tasks, and we can use Release Pipeline to run the build and test tasks. We can but we shouldn't(not recommended), configuring the Build and Release UI pipeline can make the workflow of how we build/test(more related to CI)=>deploy(more related to CD) better.
And if you're using Multi-stage pipeline, you don't need to use the Release(UI) Pipelines. You can define CI/CD steps in one xx.yml file.
It is the same tasks but not the same processes.
Nowadays, I would say Pipelines are responsible for Continuous Integration, and Release Pipelines are more about Continuous Delivery. When all deployment features will arrive to YAML pipelines it will not be the case.
As you said, Azure Release Pipelines can use and automatically trigger on result of YAML or classic pipelines and you can use different agents. Agent for YAML or classic pipelines need all tools to compile, test and assembly the code when Agent for Release, need the tools for deployment and correct access rights to target environment.
In Release pipelines you can organize your deployments by environments/stages more easily, add steps which could be launched separately with out need to relaunch the whole pipeline (like application recovery or switch to production).
The workflow of Release Pipelines is more visual than in YAML or Classic Build pipelines.
From my experience the main difference is in Approvals and Gates features, for now.
You can try to implement the whole CI/CD process just on YAML pipelines but it could be much easier to use both Azure Pipelines capabilities to separate the logic of CI and CD, at the moment, as deployment features of YAML pipelines are still in development.
in principle there isn't. its just that usually release pipelines are geared towards deploying stuff and build pipelines towards building stuff. jenkins, for example, only had builds. and you'd be releasing with builds. it really depends on the solution you are using. octopus cannot really build stuff, but can release stuff and got tools to make releasing easier.
in ADO you can have stages in release pipelines, so you can do promotes easily, you have approval gates, etc
we've got a GitLab CI build / test / deployment pipeline and need to insert a manual approval between deployment to Test and promoting to Prod. I can't figure out how to do that.
Ideally we would like to have a button like in GoCD or in AWS CodePipeline. However for our current project we use GitLab EE (ver 12.3.5-ee) hosted on our servers, not using gitlab.com, but I guess the .gitlab-ci.yml should be the same.
This is a part of my current .gitlab-ci.yml:
stages:
# lint, build, test, ...
- deploy_test
- approval
- deploy_prod
deploy_test:
stage: deploy_test
only:
refs:
- prod
script:
...
wait_for_approval:
stage: approval
# how do I do this???
deploy_prod:
stage: deploy_prod
only:
refs:
- prod
script:
...
Any idea?
Use when: manual in your stage.
To be sure the manual action is blocking (and no optionnal), add also allow_failure: false (is set to true by default)
More info in the documentation
Note : if you created the stage only for the approval, I could advice you to remove it and put the when: manual in the deploy_prod stage.
deally we would like to have a button like in GoCD or in AWS CodePipeline
In addition, of when:manual, you also have GitLab 14.9 (March 2022), which comes with (still for GitLab Premium/Ultimate only):
Deployment Approval on the Environments page
We are excited to introduce the Deployment Approval capability in the GitLab interface. In GitLab 14.8, we introduced the ability to approve deployments via the API.
Now, deployment approvers can view a pending deployment and approve or reject it conveniently directly in the Environments page. This update continues our work to enable teams to create workflows for approving software to go to production or other protected environments.
With this update, we are now upgrading the feature to beta.
See Documentation and Issue.
And GitLab 14.10 (April 2022) comes with:
Multiple approval rules for deployment approvals API
Previously, deployment approvals supported a simple model where the ability to execute a deployment and approve a deployment were both controlled with a single list of users.
With this update, you have more flexibility and granularity with these rules and can specify multiple levels of control using the API.
One type of model that can now be supported is where there is separation of duties between deployment executors and approvers in your organization.
Another supported model is where approval is needed separately from multiple levels, such as a QA tester group and a security group.
See Documentation and Issue.
You can combine that with GitLab 15.2 (July 2022) which adds (non-CE editions only):
Group-level UI for protected environment settings
Previously, if you wanted to configure group-level settings for protected environments, you had to use the API.
With this release, you can now view and edit these settings in the UI.
This change allows you to more easily set policies for which users and groups can deploy to environments across projects within a group.
See Documentation and Issue.
I can't run a plan test with automation tests associated, because the release pipeline dropdown doesn't show the available releases pipelines.
I have a test associated with my automation test.
When I tried to run the plan test
Then I tried to set up my release pipeline
The dropdown doesn't show my current available release pipelines
Thanks #Matt I had to add the artifact (from the Build pipeline) to my Release Pipeline