I am running my Selenium tests from Azure Test Plans in DevOps. When any of my test fails, I have the option in DevOps to create a bug for it**(attaching the screenshot of the option to create a bug manually once the test case fails)** but I want the bug to be created automatically as soon as the test case fails. Is there a way to configure it so that my work item(bug) gets created automatically?
There is an option to create a work item on failure in build pipeline as below, so you could enable it to create a bug when this build is failed.
In addition, if you run test plan in release pipeline, this option is not available in release pipeline, so you need to use API to create a work item in it. You could use Powershell Task to run Rest API: Work Items - Create to create a work.
Set the condition
Or you could directly use this Extension- Create Bug on Release failure.
For the YAML pipeline, you can use the CreateBug task:
Example task:
- task: CreateBug#2
inputs:
isyamlpipeline: true
custompaths: false
customrequestor: false
Link: https://marketplace.visualstudio.com/items?itemName=AmanBedi18.CreateBugTask
Related
Added a new VSTS extension to vsts marketplace https://marketplace.visualstudio.com/items?itemName=ManishSingh.soapuiazure&ssr=false#overview
I am able to install this extension on organization but not able to see this in task list to add in pipeline.
Got some way to fixed this on internet like uninstall and reinstall - Didn't worked
Disabled and enable - didn't worked.
The extension uses the deprecated PowerShell v1 execution handler for Azure Pipelines and needs to be upgraded to the Powershell3 execution handler. The v1 handler is from the era of the v1 agent that shipped in 2015 and was replaced by a complete rewrite in the summer of 2016.
##[warning]Task 'SoapUI on Azure' (2.0.6) is using deprecated task execution handler. The task should use the supported task-lib: https://aka.ms/tasklib
For me the task fails to show up in the YAML task helper, but it does show up in the classic UI based pipelines and releases.
You may be able to just make it work when using the Get YAML helper in the classic UI:
steps:
- task: ManishSingh.soapuiazure.5ad1e1bc-7ee6-49c8-b12f-7b7606c5b0e4.SoapUI on Azure#2
displayName: 'SoapUI project.xml'
Then simplify that to:
steps:
- task: SoapUI on Azure#2
displayName: 'SoapUI project.xml'
^^^^
Tested this and it works.
But it looks like this extension could use some love. I already see a few things wrong with it:
Using deprecated task handler
always downloads SOAPUI from blob storage (useful for Hosted agents, but not for persistent agents)
Manifest issues
space in Task Name (task.json)
guid in Task ID (in vss-extension.json)
Some of these are likely unfixable, cause the marketplace makes it very hard to change identifiers once the task is published publicly.
Test to install the same extension and it will show the task: Soap UI on Azure in the task list.
For example:
You can add the task to Agent Job or Deployment Job.
On the other hand, if you are not able to find the task after installing it, you can try to open a Browser InPrivate Window and use the same steps to search the task in Azure Pipeline.
SO far i can run the pipeline manually and it runs my testcafe tests with the "node myTests.js" command.
my pipeline run
My file myTests.js looks like this:
myTests.js
I followed this tutorial: https://learn.microsoft.com/en-us/azure/devops/test/run-automated-tests-from-test-hub?view=azure-devops
I tried to associated an test to my Testplan via the REST API, i guess theres some problem. because i can name it whatever i want and it just runs it without errors:my testcase association
When i run a testcase it says it found an automated test and it runs it without errors, the VsTest Job runs but with a warning:
2021-05-18T09:16:32.7619103Z Source filter: *test.dll,!*TestAdapter.dll,!\obj*
2021-05-18T09:16:32.7879061Z ##[warning]No test sources found matching the given filter '*,!\obj**'
Any ideas what im doing wrong? I just want to run my pipeline or my test when im running a testcase in my testplans.
Azure DevOps: How can i connect Testcafe-tests to a testcase?
I am afraid you could not connect the Testcafe-tests to a testcase at this moment.
According to the document Run automated tests from test plans:
You will need:
A Team Build pipeline that generates builds containing the test
binaries.
That is the reason why you get the error No test sources found matching the given filter '*test.dll,!*TestAdapter.dll,!\obj*.
We could build and test the Testcafe-tests in the azure devops pipeline.
Please check the document Integrate TestCafe with Azure DevOps for some more details.
I want to add some automation tests (it can be selenium c#/ java tests) to my existing azure pipeline. My pipeline is working correctly i.e. when I push changes to my master branch (GitHub), it triggers the build and it gets deployed to the live. But I want to integrate some automation test which will decide whether to deploy the build or not example:
After committing to the master branch:
run the automation test
if tests passed > deploy
if the test fails > stop deployment
Any suggestions or reference material would be helpful.
Thanks,
Depends on what kind of tests you are going to run - if they are unit tests, it is easy but selenium tests you would need to deploy it to a webserver before running the tests.
When edit the pipeline, and click "Show assistance" if is't not already done, you can see the tasks, including test dashboard etc.
For example if you add a VStests, it will add something like this to the yaml file.
- task: DotNetCoreCLI#2
inputs:
command: test
projects: '**/*Tests/*.csproj'
arguments: '-- no-build --configuration $(buildConfiguration)'
Above just executes 'dotnet test' command which will look for tests in the project and run them.
It is executing a command line, so you can run any tests as far as you can provide a command line using batch/powershell commands.
The Microsoft tutorial at https://learn.microsoft.com/en-us/learn/modules/run-quality-tests-build-pipeline/ explains how to use the test wigits etc.
I would suggest to start with unit tests which is simpler before trying selenium.
Krish
You can specify the conditions under which each stage, job, or step runs. By default, a job or stage runs if it does not depend on any other job or stage, or if all of the jobs or stages that it depends on have completed and succeeded. By default, a step runs if nothing in its job has failed yet and the step immediately preceding it has finished.
In another word, if test step/job/stage passed, the next step/job/stage can precede.
Currently I am trying to using Azure Test Plans to manage all test cases, but I just found how to build / generate manual test cases. I also make some research about Pipeline and I just write some "hello world" test scripts and using Pipeline to automatically test the code. My question is: How could I combine TestPlans with Pipeline to make the test in pipeline recordable? Thank you so much!
this is how the pipeline looks like
this is how the test code be checked
My question is: How could I combine TestPlans with Pipeline to make
the test in pipeline recordable?
I'm a bit confused about what the recordable mean in your question. To combine TestPlan with Pipeline, you should configure the Test Plan Settings. For example, set a build pipeline(build and generates the test binaries) and a release pipeline with test-related task in your Test Plan Settings.
The normal logic is:
A test plan containing your automated tests, which you have associated with automated test methods using Visual Studio 2017, or Visual Studio 2015 or earlier.
A Team Build pipeline that generates builds containing the test binaries.
The app to test. You can deploy the app as part of the build and release workflow and also use it for on-demand testing.
For more details please check Run automated tests from test plans.
Is any possibility to test created Azure Pipelines? From UI or your yaml definition of pipelines?
Mean that I have some yamls pipelines or pipelines defined from UI and I want to ensure by some tests(Unit Tests e.x.) that all have defined variables, build, test, and package parts or something else in each pipeline.
And verify pipelines configurations after some changes of them or after adding some new repos/branches if it's required.
Thanks...
Is any possibility to test created Azure Pipelines? From UI or your
yaml definition of pipelines?
If you want a out of box feature to achieve this, sorry to say, No, there hasn't.
BUT, the work around is using API to check them.
Client API.
You could write a simple script to get Builds definition with Client API.
In this simple script, you first get the whole definition:
List<BuildDefinitionReference> buildDefinitions = new List<BuildDefinitionReference>();
Then you could apply your customized check/test into this definition with scripts. In one word, write some test classes/methods. After the script complete, you can import it into VSTS, and then use task to run those tests part. Only this test succeed, then your builds could be executed.
So, at this time, it need you add 2 agent jobs into your pipeline, the first one is used to run your script test(names test agent job). And the second agent job is the one you want to check. In the second agent job, set its condition as:
At this time, only the test succeed, this current job can be ran.
Or, if you don't want the builds you want to check would be broken because of the test, please consider about using Build completion trigger. Set a separate pipeline to run the test. In the pipeline you want to check, set it can be run only when the test pipeline finished.
Rest API
You could use rest api with powershell which very similar with the above description. Use api to get builds definition, and then write some check powershell script.
I more recommend you to put the test at a separate pipeline. Then the API could only get the part you want check, not including the test part.