I'm running mocha tests as part of a deployment process and need to use mocha test parser for bamboo to know what failed (using reporter mocha-bamboo-reporter).
Mocha test parser task is only able to run during the build process (it can't be added as part of a deployment process). Is there a way to run it from a command, node.js or npm task?
Currently when tests fail bamboo is still saying that the deployment was ok.
Test run configuration:
config
Typically you would only want to run your tests as part of the build process and not the deployment process. This is why you do not see these options as part of the deployment. Generally, Bamboo deployment failures are the result of files not copying, connection errors, or errors in scripts.
Because you are running tests in the deployment, the test runner will return "0" indicating that the task to execute the tests ran fine. Bamboo allows you to do this so that you can run tests as a deployment and still deploy.
Instead of failing the deploy, add two tasks to the build to run the tests and parse the results. If the tests pass you can have the deploy trigger on the success of the build. This gives you the following advantages that you are currently missing:
Deployments only will start if tests pass.
Bamboo has a nice summary page for tests and will provide useful metrics such as how many times a specific test has failed.
It separates the deployment from the integration (i.e. build/test).
However, if you are dead set on running tests and parsing in a deployment, you could have a node.js, command, or script task that parses the results and then returns -1 (or a non-zero number) if tests failed.
Related
SO far i can run the pipeline manually and it runs my testcafe tests with the "node myTests.js" command.
my pipeline run
My file myTests.js looks like this:
myTests.js
I followed this tutorial: https://learn.microsoft.com/en-us/azure/devops/test/run-automated-tests-from-test-hub?view=azure-devops
I tried to associated an test to my Testplan via the REST API, i guess theres some problem. because i can name it whatever i want and it just runs it without errors:my testcase association
When i run a testcase it says it found an automated test and it runs it without errors, the VsTest Job runs but with a warning:
2021-05-18T09:16:32.7619103Z Source filter: *test.dll,!*TestAdapter.dll,!\obj*
2021-05-18T09:16:32.7879061Z ##[warning]No test sources found matching the given filter '*,!\obj**'
Any ideas what im doing wrong? I just want to run my pipeline or my test when im running a testcase in my testplans.
Azure DevOps: How can i connect Testcafe-tests to a testcase?
I am afraid you could not connect the Testcafe-tests to a testcase at this moment.
According to the document Run automated tests from test plans:
You will need:
A Team Build pipeline that generates builds containing the test
binaries.
That is the reason why you get the error No test sources found matching the given filter '*test.dll,!*TestAdapter.dll,!\obj*.
We could build and test the Testcafe-tests in the azure devops pipeline.
Please check the document Integrate TestCafe with Azure DevOps for some more details.
I want to add some automation tests (it can be selenium c#/ java tests) to my existing azure pipeline. My pipeline is working correctly i.e. when I push changes to my master branch (GitHub), it triggers the build and it gets deployed to the live. But I want to integrate some automation test which will decide whether to deploy the build or not example:
After committing to the master branch:
run the automation test
if tests passed > deploy
if the test fails > stop deployment
Any suggestions or reference material would be helpful.
Thanks,
Depends on what kind of tests you are going to run - if they are unit tests, it is easy but selenium tests you would need to deploy it to a webserver before running the tests.
When edit the pipeline, and click "Show assistance" if is't not already done, you can see the tasks, including test dashboard etc.
For example if you add a VStests, it will add something like this to the yaml file.
- task: DotNetCoreCLI#2
inputs:
command: test
projects: '**/*Tests/*.csproj'
arguments: '-- no-build --configuration $(buildConfiguration)'
Above just executes 'dotnet test' command which will look for tests in the project and run them.
It is executing a command line, so you can run any tests as far as you can provide a command line using batch/powershell commands.
The Microsoft tutorial at https://learn.microsoft.com/en-us/learn/modules/run-quality-tests-build-pipeline/ explains how to use the test wigits etc.
I would suggest to start with unit tests which is simpler before trying selenium.
Krish
You can specify the conditions under which each stage, job, or step runs. By default, a job or stage runs if it does not depend on any other job or stage, or if all of the jobs or stages that it depends on have completed and succeeded. By default, a step runs if nothing in its job has failed yet and the step immediately preceding it has finished.
In another word, if test step/job/stage passed, the next step/job/stage can precede.
I have created an Azure build pipeline using the 'classic editor' (i.e. not yaml). The build consists of two agent jobs:
Job 1 - Build code and deploy to test enviornment using a single agent.
Job 2 - Run tests against the test enviornment in parallel (using max 3 agents at a time).
My problem with this setup is if a build is triggered, and the tests are mid-run, if a second build is triggered, the code that is deployed to the test enviornment will be overwritten by the subsequent build, causing the test run in Job 2 of the first build to fail.
Is it possible to tell the build pipeline to only trigger builds sequentially?
I have figured out how to use Azure DevOps API to check if the latest build has completed, however Im not sure how I can use it in the pipeline. Is it possible to do something like:
1 - Invote REST API to check status of latest build.
2 - Success Criteria met (i.e. the build has completed)? If yes continue build, if not wait a minute and check again.
You have the option to control this in the Build options. Should work based on what you described.
Edit:
After looking again to your question, i noticed that you are running your tests after you deploy your app to test environment, so it means you run your tests during release, so you need to control the flow on your release and not on your buid. In order to do it, you should control on the maximum number of parallel deployments:
In Azure DevOps, is there any way that I can synchronize the results in Runs with test cases in Test Plans automatically?
My process is:
Release pipeline is set up to trigger Jenkins build to run automated tests.
JUnit test results will be downloaded and published once the build is completed.
Tasks in automation run stage of pipeline.
The process works as expected. I can see the JUnit results in Runs section as the image below. However, the outcome of the test results in Runs (all failed) is not synchronized with test cases in Test Plans (still be active). Can I have the outcome of test cases in Test Plans updated every time the automation build is run and then JUnit XML results are published to Runs?
Results in Test Runs.
Test cases in Test Plans.
So fat is there any settings of Azure DevOps to make it done automatically or I have to update the test outcome in Test Plans manually?
I am using Azure DevOps build pipeline, to run my selenium web automation tests (run by maven, inside docker container)
Now, even If one of my test scenarios fail, pipeline job is still considered as successful, how can I specify to look for certain log to fail it?
The 'pipeline job/task/phase' I am using to run my test is 'docker compose'
My second question is, is there possibility to filter pipeline output logs? Currently it is flooded with output from few services run in container:
The only thing I found is, possibility to search through logs, but no filtering, regards.
If your objective is to fail the build when one of more of your test has failed, I advise you to add one more step to your build process : Publish Test Results task
This is a necessary step for test running with another task than the default Visual Studio Test task, which consist in publishing your test result file to Azure DevOps, and enable your build to be aware of your tests results (and then let you decide what to do if one or more tests fail)
In your case, you will also probably have to find a way to extract test results file from your containers, as your test results might probably be produced and stored inside of your containers (and unavailable to the Publish Test Result task)
For your second question, I am not aware of any way to filter the output logs directly from the web interface, sorry :(
We ran into this with our cypress tests (you should ditch selenium for cypress, its sweet) and solved it by grabbing the exit code manually. We also found that AzureDevops will hang if there's a background process running, even if there's an error, so be sure to look out for that as well if you start up your web server like we do.
- bash: |
yarn test-ci:e2e 2> /dev/null
if [ $? -eq 0 ]
then
yarn stop
exit 0
else
yarn stop
exit 1
fi
displayName: 'Run Cypress Tests'
For anyone looking for a way to filter logs, if there are multiple services running, you can create new azure build pipeline task (Docker) that runs docker command:
docker logs -f NAME_OF_THE_SERVICE
This way you will only see logs from desired service.