We currently use TFS 2015 Update 3 to run nightly Releases with (only) CodedUI tests. In one Release definition, I have 4 environments that run these tests. 3 of them run fine, but one has consistently run into this error:
2018-09-21T19:04:23.8783809Z DistributedTests: Please use this link to analyze the test run : http://...
2018-09-21T19:04:23.8940074Z DistributedTests: Test run '70794' is in 'InProgress' state.
2018-09-21T19:04:23.8940074Z DistributedTests: Total Tests : 102, Passed Tests : 0
2018-09-21T19:04:33.9839005Z DistributedTests: Test run '70794' is in 'Aborted' state.
2018-09-21T19:04:33.9839005Z DistributedTests: Total Tests : 102, Passed Tests : 0
2018-09-21T19:04:44.0019357Z ##[warning]DistributedTests: Test run is aborted. Logging details of the run logs.
2018-09-21T19:04:44.0956639Z ##[warning]DistributedTests: New test run created.
2018-09-21T19:04:44.0956639Z ##[warning]Test Run queued for Project Collection Build Service (TEAM FOUNDATION).
2018-09-21T19:04:44.0956639Z ##[warning]DistributedTests: UnExpected error occured during test execution. Try again.
2018-09-21T19:04:44.0956639Z ##[warning]DistributedTests: Test run aborted. Test run id: 70794
2018-09-21T19:04:44.0956639Z ##[error]The test run was aborted, failing the task.
In the DTAlogs, I find this statement:
E, 8736, 10, 2018/09/21, 10:37:53.264, FALCONVANDV08\DTAExecutionHost, TestExecutionServiceCommandQueueListener.SendStopCommandToHandleException : An exception occured in queue listener. Microsoft.TeamFoundation.Test.WebApi.TestExecutionObjectNotFoundException: TestAgent 8984 not found.
I have a Visual Studio Test Agent Deployment task before the Run Functional Tests task (remember, 3 other environments that are the exact same handle this fine). The only difference is the VM this is running on, though all 4 of the VMs were created from the same image, and have the same configuration. On the VM, it looks like the vstf_testagent.exe exists and has a timestamp that's similar to when the environment was deployed. Any suggestions?
The issue seems similar as this one : vNext and new task RunDistributedTests - parallel testing
Please check atinb's comment:
Do you have your tests in 1 dll? If so, the parallelism today only
works across dlls. We are working on distributing tests within dlls
across machines and that should come shortly.
Related
I have a task of automation that include number of tests that running by category in ci/cd.
I'm trying to find something that I can set if certain tests or max tests fail then it will try to run them again before the task fails.
for now i have something like that in my code, any idea?
i working with VSTS (Azure - pipelines release).
thanks!
If you are using the Visual Studio Test task, the task has a built-in feature: Rerun failed tests.
According to your needs, you can specify rerunType(Do not rerun if test failures exceed specified threshold). You can specify % failures with basedOnTestFailurePercentage or # of failed tests as a threshold with basedOnTestFailureCount.
I have created an Azure build pipeline using the 'classic editor' (i.e. not yaml). The build consists of two agent jobs:
Job 1 - Build code and deploy to test enviornment using a single agent.
Job 2 - Run tests against the test enviornment in parallel (using max 3 agents at a time).
My problem with this setup is if a build is triggered, and the tests are mid-run, if a second build is triggered, the code that is deployed to the test enviornment will be overwritten by the subsequent build, causing the test run in Job 2 of the first build to fail.
Is it possible to tell the build pipeline to only trigger builds sequentially?
I have figured out how to use Azure DevOps API to check if the latest build has completed, however Im not sure how I can use it in the pipeline. Is it possible to do something like:
1 - Invote REST API to check status of latest build.
2 - Success Criteria met (i.e. the build has completed)? If yes continue build, if not wait a minute and check again.
You have the option to control this in the Build options. Should work based on what you described.
Edit:
After looking again to your question, i noticed that you are running your tests after you deploy your app to test environment, so it means you run your tests during release, so you need to control the flow on your release and not on your buid. In order to do it, you should control on the maximum number of parallel deployments:
I'm running mocha tests as part of a deployment process and need to use mocha test parser for bamboo to know what failed (using reporter mocha-bamboo-reporter).
Mocha test parser task is only able to run during the build process (it can't be added as part of a deployment process). Is there a way to run it from a command, node.js or npm task?
Currently when tests fail bamboo is still saying that the deployment was ok.
Test run configuration:
config
Typically you would only want to run your tests as part of the build process and not the deployment process. This is why you do not see these options as part of the deployment. Generally, Bamboo deployment failures are the result of files not copying, connection errors, or errors in scripts.
Because you are running tests in the deployment, the test runner will return "0" indicating that the task to execute the tests ran fine. Bamboo allows you to do this so that you can run tests as a deployment and still deploy.
Instead of failing the deploy, add two tasks to the build to run the tests and parse the results. If the tests pass you can have the deploy trigger on the success of the build. This gives you the following advantages that you are currently missing:
Deployments only will start if tests pass.
Bamboo has a nice summary page for tests and will provide useful metrics such as how many times a specific test has failed.
It separates the deployment from the integration (i.e. build/test).
However, if you are dead set on running tests and parsing in a deployment, you could have a node.js, command, or script task that parses the results and then returns -1 (or a non-zero number) if tests failed.
I am using Azure DevOps build pipeline, to run my selenium web automation tests (run by maven, inside docker container)
Now, even If one of my test scenarios fail, pipeline job is still considered as successful, how can I specify to look for certain log to fail it?
The 'pipeline job/task/phase' I am using to run my test is 'docker compose'
My second question is, is there possibility to filter pipeline output logs? Currently it is flooded with output from few services run in container:
The only thing I found is, possibility to search through logs, but no filtering, regards.
If your objective is to fail the build when one of more of your test has failed, I advise you to add one more step to your build process : Publish Test Results task
This is a necessary step for test running with another task than the default Visual Studio Test task, which consist in publishing your test result file to Azure DevOps, and enable your build to be aware of your tests results (and then let you decide what to do if one or more tests fail)
In your case, you will also probably have to find a way to extract test results file from your containers, as your test results might probably be produced and stored inside of your containers (and unavailable to the Publish Test Result task)
For your second question, I am not aware of any way to filter the output logs directly from the web interface, sorry :(
We ran into this with our cypress tests (you should ditch selenium for cypress, its sweet) and solved it by grabbing the exit code manually. We also found that AzureDevops will hang if there's a background process running, even if there's an error, so be sure to look out for that as well if you start up your web server like we do.
- bash: |
yarn test-ci:e2e 2> /dev/null
if [ $? -eq 0 ]
then
yarn stop
exit 0
else
yarn stop
exit 1
fi
displayName: 'Run Cypress Tests'
For anyone looking for a way to filter logs, if there are multiple services running, you can create new azure build pipeline task (Docker) that runs docker command:
docker logs -f NAME_OF_THE_SERVICE
This way you will only see logs from desired service.
For creating a Build-Deploy-Test Workflow for my Wpf application, I ma using TFS Build definition and MTM to create Environment.
I am using following Steps:
I am Using LabDefaultTemplate.11.xaml in Build Definition
In the Lab Process Settings, I have given the required details as:
Environment: I have created an environment using MTM on the VM
in the environment I have added a Desktop Client(the same VM only) in the machines List
Please note that my Test Controller, agent, MTM all are on the VM only
Build: I have selected an existing build definition and it's working fine.
Deploy: I have specidfied command for application installation(running of msi file)
also, In the deploy section only, at the end I have specified a batch file where I am calling my CodedUI tests via command line(using VSTEST.CONSOLE.EXE)
Test: Not using it i.e. Not checked the "Run these Tests in Environment"
When I run this build definition, build creation, and installation are working fine but failes to run the codedui tests. Getting following error:
Error Message:
Error calling Initialization method for test class CodedUITestProject1.AISTest: Microsoft.VisualStudio.TestTools.UITest.Extension.UITestException: To run tests that interact with the desktop, you must set up the test agent to run as an interactive process. For more information, see "How to: Set Up Your Test Agent to Run Tests That Interact with the Desktop" (http://go.microsoft.com/fwlink/?LinkId=255012)
If you are running the tests as part of your team build, you must also set up the build agent to run as an interactive process. For more information, see "How to: Configure and Run Scheduled Tests After Building Your Application" (http://go.microsoft.com/fwlink/?LinkId=254735)
Stack Trace:
at Microsoft.VisualStudio.TestTools.UITesting.Playback.Initialize()
at Microsoft.VisualStudio.TestTools.UITesting.CodedUITestExtensionExecution.BeforeTestInitialize(Object sender, BeforeTestInitializeEventArgs e)
at Microsoft.VisualStudio.TestTools.TestTypes.Unit.UnitTestExecution.RaiseBeforeTestInitialize(BeforeTestInitializeEventArgs args)
at Microsoft.VisualStudio.TestTools.TestTypes.Unit.UnitTestExecuter.RunInitializeMethod()
Test Run Failed.
Also, I have made sure that test agent is configured as an interactive process.
Any Help would be highly appreciated.
The issue got resolved after setting the build controller also to run as a interactive process.