For creating a Build-Deploy-Test Workflow for my Wpf application, I ma using TFS Build definition and MTM to create Environment.
I am using following Steps:
I am Using LabDefaultTemplate.11.xaml in Build Definition
In the Lab Process Settings, I have given the required details as:
Environment: I have created an environment using MTM on the VM
in the environment I have added a Desktop Client(the same VM only) in the machines List
Please note that my Test Controller, agent, MTM all are on the VM only
Build: I have selected an existing build definition and it's working fine.
Deploy: I have specidfied command for application installation(running of msi file)
also, In the deploy section only, at the end I have specified a batch file where I am calling my CodedUI tests via command line(using VSTEST.CONSOLE.EXE)
Test: Not using it i.e. Not checked the "Run these Tests in Environment"
When I run this build definition, build creation, and installation are working fine but failes to run the codedui tests. Getting following error:
Error Message:
Error calling Initialization method for test class CodedUITestProject1.AISTest: Microsoft.VisualStudio.TestTools.UITest.Extension.UITestException: To run tests that interact with the desktop, you must set up the test agent to run as an interactive process. For more information, see "How to: Set Up Your Test Agent to Run Tests That Interact with the Desktop" (http://go.microsoft.com/fwlink/?LinkId=255012)
If you are running the tests as part of your team build, you must also set up the build agent to run as an interactive process. For more information, see "How to: Configure and Run Scheduled Tests After Building Your Application" (http://go.microsoft.com/fwlink/?LinkId=254735)
Stack Trace:
at Microsoft.VisualStudio.TestTools.UITesting.Playback.Initialize()
at Microsoft.VisualStudio.TestTools.UITesting.CodedUITestExtensionExecution.BeforeTestInitialize(Object sender, BeforeTestInitializeEventArgs e)
at Microsoft.VisualStudio.TestTools.TestTypes.Unit.UnitTestExecution.RaiseBeforeTestInitialize(BeforeTestInitializeEventArgs args)
at Microsoft.VisualStudio.TestTools.TestTypes.Unit.UnitTestExecuter.RunInitializeMethod()
Test Run Failed.
Also, I have made sure that test agent is configured as an interactive process.
Any Help would be highly appreciated.
The issue got resolved after setting the build controller also to run as a interactive process.
Related
SO far i can run the pipeline manually and it runs my testcafe tests with the "node myTests.js" command.
my pipeline run
My file myTests.js looks like this:
myTests.js
I followed this tutorial: https://learn.microsoft.com/en-us/azure/devops/test/run-automated-tests-from-test-hub?view=azure-devops
I tried to associated an test to my Testplan via the REST API, i guess theres some problem. because i can name it whatever i want and it just runs it without errors:my testcase association
When i run a testcase it says it found an automated test and it runs it without errors, the VsTest Job runs but with a warning:
2021-05-18T09:16:32.7619103Z Source filter: *test.dll,!*TestAdapter.dll,!\obj*
2021-05-18T09:16:32.7879061Z ##[warning]No test sources found matching the given filter '*,!\obj**'
Any ideas what im doing wrong? I just want to run my pipeline or my test when im running a testcase in my testplans.
Azure DevOps: How can i connect Testcafe-tests to a testcase?
I am afraid you could not connect the Testcafe-tests to a testcase at this moment.
According to the document Run automated tests from test plans:
You will need:
A Team Build pipeline that generates builds containing the test
binaries.
That is the reason why you get the error No test sources found matching the given filter '*test.dll,!*TestAdapter.dll,!\obj*.
We could build and test the Testcafe-tests in the azure devops pipeline.
Please check the document Integrate TestCafe with Azure DevOps for some more details.
I have downloaded and installed in a windows server 2016 a vs test agent, which was installed by a tool called Test Agent Configuration Tool. I then followed this guide to configure the agent to run tests that interact with desktop - I want to run CodedUI Tests for web projects from whithin my release pipeline.
This is the output from the Test Agent Configuration Tool
When I run my release pipeline I get this error during the test phase:
Error calling Initialization method for test class
Captura.PDV.Web.Test.CodedUI.CodedUITest1:
Microsoft.VisualStudio.TestTools.UITest.Extension.UITestException: To
run tests that interact with the desktop, you must set up the test
agent to run as an interactive process. For more information, see "How
to: Set Up Your Test Agent to Run Tests That Interact with the
Desktop" (https://go.microsoft.com/fwlink/?LinkId=255012)
If you are
running the tests as part of your team build, you must also set up the
build agent to run as an interactive process. For more information,
see "How to: Configure and Run Scheduled Tests After Building Your
Application" (https://go.microsoft.com/fwlink/?LinkId=254735)
Here is my test task configuration:
Some notes and questions:
I can run the tests calling vstest.console.exe manually from within the server with success
My release pipeline runs on VSTS
I am not using Microsoft Test Manager - Is it really necessary?
Do I need to set up a test lab enviroment?
Do I need to have a controller for the agent?
I just want to run my tests in first place to see how it goes and then proceed to more advanced tests scenarios using MTM
You're mixed up between the old way of doing things and the new way.
You don't need to use MTM -- MTM is deprecated.
You don't need to set up a lab environment. Lab environments are deprecated.
You don't need special test controllers or agents.
All you should need to do in order to run UI tests is:
Set up a build/release agent that runs interactively.
Run the tests on that agent with the Visual Studio Test task with the "Test Mix Contains UI tests" check box checked.
That's it.
I'd also recommend against using Coded UI and instead use Selenium (or Appium for desktop applications); Microsoft is no longer investing in the Coded UI platform and recommends using Selenium as an alternative.
My web app depends on an environment variable.
I wrote a unit test to ensure the environment variable is not null. The test passes on my local machine because, of course, I put the environment variable in my system variables. I had to restart Visual Studio for it to recognize the environment variable. But the test fails when it is built using CI in the cloud.
The TFS Build is scheduled to deploy it to a staging environment, and the environment variable is present in the staging environment as well as the production environment. If it is successful, it will deploy it to the staging environment, and then auto-swap with production. But it doesn't get far because the unit test fails. Is the unit test run in an environment different from the actual hosted website? If so, I must change my unit test strategy to not test environment variables. I may make an application status page to check such things then. But I wondered if anyone knew what was up on this.
The custom environment variable which you have in your personal machine must also be on the build controller. TFS Builds run on controller (Agents internally) and the unit tests run post build of the binary (Integral part of the Build before packaging)
Check if you can add the environment variable in the Build controller. You probably would need to reach the Build team and get that environment variable added in the Build server.
You also have a choice to "Pass the build if the unit test fails"
You can also categorize the unit test and set priorities and run only unit tests of certain priority; like priority 1 alone. You ca eliminate the case which is failing to be part of a unit test priority which you want to run.
Hope this helps!!
When I'm running the test in Microsoft Test Manager I get an error: The build directory of the test run either does not exist or access permission is required.
What does it mean? Where can I check build directory?
I suppose you are running automated tests.
If you are running automated test in Microsoft Test Manageg (MTM) you have Test Case(s) automated by your test implementation.
So MTM need to know where is the drop location of the build building your tests.
That means you have to:
Create a Build definition that build the code implementing your tests.
Configure MTM to use the drop location of this build definition.
You can check which build MTM is currently using here:
(I suppose you will have "undefined" standing there.)
You can configure which build is to use here:
Just check here how to configure your Test Plan which build to use when running tests: Creating and Defining a Test Plan
Additionally I would highly recommend you this free e-book: Testing for Continuous Delivery with Visual Studio 2012
I would like some comments on the possibility or alternatives to the following workflow.
The players:
JimBob - Tester
APP_Test - Virtual machine configured with Lab, Build and Test agent. (shows testing and workflow capabilities active in the environment viewer)
Manual Test Environment - Environment consisting of one virtual machine "APP_Test"
Functional Tests - A test plan configured with the Test Environment for Manual runs being "Manual test Environment"
Build-Deploy - A TFS Build definition that restores a clean checkpoint and then deploys the latest version of the software to the "APP_Test" virtual machine
Build-Deploy-Test - A TFS Build definition that restores a clean checkpoint and then deploys the latest version of the software to the "APP_Test" virtual machine and then runs coded ui tests contained in the solution.
John - Tester/Dev
The scenario:
JimBob uses the TFS web interface to queue a new workflow build of the software ("Build-Deploy"). As part of the build the latest release of the software is deployed to the virtual machine "APP_Test". JimBob opens Microsoft Test Manager 2010 navigates to the Testing Centre, chooses the Test tab and highlights a Test from a previously defined Test Suite in the test plan "Functional Tests". When JimBob Runs the test the environment viewer opens viewing the "APP_Test" virtual machine. Jim performs the test and creates an action recording on the "APP_Test" virtual machine.
John then takes the action recording and creates a coded ui test, adds the test to the solution and when he queues the "Build-Deploy-Test" build it runs the coded ui test in the same environment that the original action recording was created in.
Questions:
I have created the environments, VM, Test plan and Test Suites outlined in the above scenario. What I can't seem to get working is that when a manual test is run it always runs the test on the Local machine not in a VM environment. This will not give me the ability to deploy so that my environment is always in a known state with no interference. I guess my question boils down to: Is it possible to use test management to create action recordings on virtual machines that belong to an environment? I have had success running tests on machines in environments but not making the initial recording. It seems to me that my testers should be able to always perform their tests (automated and manual) in a snapshotted known environment. What am I missing?
Maybe I didn't understand well, but since you want to perform recording you are talking about CodedUI I guess. I believe that the tester can record in his own workstation and perform any tweaks needed in the code and then deliver you an automated test case. Then you can run this automated test case in any virtual environment that satisfies your preconditions.
You cannot do action recording on a remote machine in the environment, see this link
Instead I installed MTM on the VM so I have a clean environment to build my tests in.