I have created a coded web performance test. Now, I'm running the test from visual studio. I need to run it as a application or as a exe file.
Web performance test (coded or not) are only available for MS test execution engine. The only solution I found is to invoke the test through mstest.exe.
You can find more informations here
Related
We have a windows client that our QA team wrote coded UI tests for. I'm trying to get OpenCover to work so we can see how much of the app their tests are really hitting. They wrote their tests using a custom framework on top of the MSTest framework.
Their framework uses ApplicationUnderTest.Launch to start the application as different users to test security settings. I can capture coverage of the test dlls but the application it self.
I've forced the app to build in 32bit, made sure all pdbs are present in the folders and included the pdb directory in the targetdir as well.
Has anyone else seen this issue? I also tried replacing my batch file with a wrapper exe and got coverage for that but not the main application. Everything is running as the user they are starting the application as who is also an admin on the machine.
The command I'm running is:
C:\Users\kkindt.CORP\AppData\Local\Apps\OpenCover\OpenCover.Console.exe -register "-target:C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe" -output:C:\CodeCoverage\CollectionResults\CodedUICover.xml "-targetargs:C:\CodeCoverage\Tests\EllisWinAppTest.dll /Platform:x86 /Framework:framework40 /Tests:LaunchEllisTest"
I strongly suspect it is due to how you are launching the application under test
OpenCover is a .NET profiler and for .NET application to be launched with a profiler attached requires some environment variables to be available to the new process. I suspect that ApplicationUnderTest.Launch does not propagate all the current environment variables and so the profiler does not start and therefore does not report coverage.
To get this to work you should look into using the ApplicationUnderTest.Launch overload that uses a ProcessStartInfo and then you need to propagate the following environment variables
Cor_Profiler
Cor_Enable_Profiling
OpenCover_Profiler_Key
OpenCover_Profiler_Namespace
OpenCover_Profiler_Threshold
OpenCover_Profiler_TraceByTest (if available)
The Cor_* are required by the runtime to launch a profiler and the OpenCover_* entries are to allow the profiler and the host to find each other - a list of these environment variables are available on github
I have created a few coded ui tests and linked them to the test case, and they now appear as automated and you can see the dll they link to in the test case details.
Now that I want to run the tests, MTM refuses to even start the test unless a build is defined.
However: I want to run the tests against a statically installed application in the lab environment. This is an application that I manually install, and I get this application already compiled, so no need to play around building it.
So how can I take the build server out of the loop? I don't need the application built or deployed, I'm already doing that.
All I want is the tests to run on the lab environment specified against an application that is already preinstalled.
It's asking you to define the build of the test solution, assuming that it's different from your application under test. The test assembly will be deployed to the test environment after you specify it in MTM. This article may help you with the specifics.
It is asking you to create a build for your Coded UI test solution. It requires that the tests be built so that it has something to execute when you run the tests. Assuming that your tests were recorded using your statically deployed application then they will test that same application.
I have solution with a WPF application, its class libraries, WiX installer and numerous MSTest test projects.
When I build the solution the test runner (we are using Visual Studio 2012.3) correctly discovers all the tests and allows us to run them.
Now I have created a Coded UI test project with just 1 CUIT test in it. I have added the project to the solution; because it is a complete app test of the thing the solution builds this seems logically correct to me.
However the default behaviour is the new test projects CUIT test is being discovered by the test runner and so gets run when I run all tests. I do not want this I only want the other (unit) tests to be found and run this way.
Is there a way (test class attributes perhaps) to supress the DISCOVERY of the test by the test runner?
(Note this is a similar question to Testing an WPF app with CodedUI tests, should the coded ui test project share a solution or not? but asking the specific question that is;
If a solution contains some test how can I prevent those tests being discovered by the test runner?
Any solution should still allow the "undiscovered" test to be selected in a lab build for automation of MTM tests.
I simply unload the project until I want to run the tests.
Which projects are unloaded are saved in the .suo file, which can be excluded from source control if your lab build is being done automatically, or you can reload in manually if your lab building is created manually.
First some context; we are developing a large desktop WPF application in .NET 4.5 targeting 64 bit Windows 7 and 8. We are using Visual Studio 2012.2 (soon to be .3 then probably 2013!) and TFS 2012 (again .2 soon to be .3 then 2013).
Currently this product is all in a single large solution (just over 50 projects) yielding a WPF exe, a load of dlls and a nice MSI to install it.
We use TFS (gated and scheduled) to build the solution, its installer (WiX) and run its tests (SpecFlow for BDD and MSTest unit tests) and this is working very well.
I have a separate scheduled TFS build that deploys the MSI to physical test rig in a untrusted AD domain via a PowerShell script (see TFS2012 LabDefault.11 template deploy scripts fail with “Team Foundation Server could not complete the deployment task” for details of the challenges involved with that!)
OK so that's where I am, now I want to take things to the next step; CodedUI tests to drive full app integration test; I want to "Smoke Test" my builds.
So being a simple soul I added a new project to my products solution; a CodedUI test project.
This happily runs the locally installed product (rather then the just built one; as I ultimately want the CUIT to be running on a deployed test rig as a smoke test, and that rig has just installed the MSI I just built) and performs some UI tests with assertions.
Now my problem is with the CUIT project as part of the products solution a local test run finds and runs my CUIT tests, and this is undesired. I only want to run the CUIT tests in a lab builds test phase.
So is putting the CUIT project into the product solution a bad idea? or should it be a separate solution? Splitting them seems wrong somehow as they are related; the CUIT project is the full stack integration test for the solution's deployable application.
Can I include the CUIT in the products solution and stop the test runner seeing the tests? or is it better just to have two solutions?
What are the pros and cons folks?
Update
In the end we created a new solution containing a coded UI test project and ensured this was built with the same TFS build that built the UI solution. This allows us to load and run the coded UI tests locally without issues, the unit tests in the main UI project are left unmolested. Still seems a little disjointed but on a multiple person team per user test settings were too awkward splitting the coded UI into a different solution was simpler.
What I did was make one Solution and made a CUIT project within, I then made multiple Coded UI test's within that. This is good because using an orderedTest you can run them together and they also share a UIMap which helps too.
I also have/had this problem, because we are at the beginning of using CUIT. For now the CUIT remains in product solution. We do this because the tests should remain in memory of developers. When tests stay in on solution I'm afraid they get lost in oblivion. But indeed there is sometimes a bad feeling that the CUIT pollute the products solution, so i guess they will get their own solution after some time pass and the test become established.
Edit: If you use different Versions of Visual Studio you have to consider that for example a VS Prof. can’t build a solution with Code UI Tests. This means in “multi VS-version environments” you have to separate Coded UI Tests from “real” code.
We would like to be able to deploy our code to azure and then run integration/acceptance tests on the deployed instances to validate the functionality, as using the emulator does not always give realistic results.
We would also like to have these tests generate code coverage reports which we could then merge in with the code coverage from our unit tests. We are using TeamCity as our build server with the built in dotcover as our code coverage tool.
Can we do this? Does anyone have any pointers on where to start?
Check out this video
Kudu can be extended to run Unit Tests and much more more using Custom Deployment Scripts.
http://www.windowsazure.com/en-us/documentation/videos/custom-web-site-deployment-scripts-with-kudu/