We have a windows client that our QA team wrote coded UI tests for. I'm trying to get OpenCover to work so we can see how much of the app their tests are really hitting. They wrote their tests using a custom framework on top of the MSTest framework.
Their framework uses ApplicationUnderTest.Launch to start the application as different users to test security settings. I can capture coverage of the test dlls but the application it self.
I've forced the app to build in 32bit, made sure all pdbs are present in the folders and included the pdb directory in the targetdir as well.
Has anyone else seen this issue? I also tried replacing my batch file with a wrapper exe and got coverage for that but not the main application. Everything is running as the user they are starting the application as who is also an admin on the machine.
The command I'm running is:
C:\Users\kkindt.CORP\AppData\Local\Apps\OpenCover\OpenCover.Console.exe -register "-target:C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe" -output:C:\CodeCoverage\CollectionResults\CodedUICover.xml "-targetargs:C:\CodeCoverage\Tests\EllisWinAppTest.dll /Platform:x86 /Framework:framework40 /Tests:LaunchEllisTest"
I strongly suspect it is due to how you are launching the application under test
OpenCover is a .NET profiler and for .NET application to be launched with a profiler attached requires some environment variables to be available to the new process. I suspect that ApplicationUnderTest.Launch does not propagate all the current environment variables and so the profiler does not start and therefore does not report coverage.
To get this to work you should look into using the ApplicationUnderTest.Launch overload that uses a ProcessStartInfo and then you need to propagate the following environment variables
Cor_Profiler
Cor_Enable_Profiling
OpenCover_Profiler_Key
OpenCover_Profiler_Namespace
OpenCover_Profiler_Threshold
OpenCover_Profiler_TraceByTest (if available)
The Cor_* are required by the runtime to launch a profiler and the OpenCover_* entries are to allow the profiler and the host to find each other - a list of these environment variables are available on github
Related
I recently read about ServiceFabric offering some kind of Autorefresh mode, considering its infrastructure and the cluster running independently.
So, according to what I've wrote, all I need to do it go to my project, check the properties and set the "Application Debug Mode" to Refresh Application.
Basically that is what I did now, but I don't quite see the difference. There's no repackaging happening at all.
Do I have to run some kind of cmdlet in the background, as it is the case with e.g. webpack watch?
The docs explains Refresh Application as follow.
Refresh Application This mode enables you to quickly change and debug
your code and supports editing static web files while debugging. This
mode only works if your local development cluster is in 1-Node mode.
This is the default Application Debug Mode.
On other application debug options, Visual Studio creates a package and deploy it to the cluster and register the application to run on Service Fabric, the package will contain all binaries compiled that are needed to run the application.
The main difference between the Refresh Application and the others, is that the package created is a symbolic link to the source in the Dev machine, you are not actually copying the package with the binaries, when you change the static files, it will be the same files used by the deployed application in SF, this will make more flexible to make changes without repackaging, registering and deploying the application on every change.
PS: It does not work the same way as the watch feature for nodeJs development, it is just to avoid the package deployment. You could just reload the page though.
This post explain in more details.
I'm in the process of writing some new C++/WinRT based components in order to replace some much older C++/CX code. The goal is to be able to use third-party C++ tools that don't understand CX (static code analyzers, etc).
However the first step in the journey is to ensure I can properly unit test my own code. Unit testing C++/CX code typically used the "C++ Unit Test App" project type, which is C++/CX based and has its own issues (lack of code coverage support, run all required before tests show up in the explorer, stability, etc)
Browsing through the available project types in Visual Studio 2017, I did not see a unit test project template for C++/WinRT based projects. Is my only option to use the "C++ Unit Test App" template with all its failings, or is there another way to build tests for a C++/WinRT library?
Perhaps there is a way to configure either the "Native Unit Test Project" or "Google Test" project templates to support what I'm looking for?
Ideally what I'm looking for is something that doesn't require launching a UI, is pure C++(/WinRT), and supports Visual Studio's Code Coverage Analysis.
There is no unit test project that is specific to C++/WinRT, much like there isn't one for other libraries like STL. I would recommend Catch2 as it supports C++17 (a requirement for C++/WinRT) and works well on Windows. It is also what we use for testing C++/WinRT itself. Catch2 is nice because it helps you create a simple console app that acts as the test driver that includes all of the tests.
For code coverage I don't have a strong recommendation, but if you are using Visual Studio then you might want to try VSInstr. It can be used for code coverage and produces a report that can be viewed with Visual Studio.
Make sure your code is built using the /profile linker option. This will ensure that profile hooks are included in a dedicated section of the PE file. Next, run vsinstr to instrument any of the binaries you're interested in (that were previously built with /profile):
vsinstr /coverage tests.exe
Now run vsperfcmd to begin collecting coverage data:
vsperfcmd /start:coverage /output:report
Run the code as normal. For Catch2, you can simply run the executable at the command line. Then you need to stop the collection as follows:
vsperfcmd /shutdown
And you're done. You can now view the report in Visual Studio:
devenv report.coverage
Hope that helps. Again, this is not specific to C++/WinRT and since C++/WinRT is a header-only library you are liable to get a lot of noise that is unrelated to your specific project. I haven't found a good way to deal with that yet.
Expanding on my comment to #KennyKerr's answer for those that are interested...
If you are planning on using Catch2 as recommended, then the C++/WinRT Windows Console Application template is a great starting point. Pretty much all you have to do is tweak the main() to setup Catch2 and start writing your test cases. My only complaint is that the C++/WinRT templates don't allow you to add Windows Runtime Component project references via the UI (must be done by editing the vcxproj). There is probably a similar problem adding NuGet package references.
As noted in my comment above, there is a Catch2 test adapter for Visual Studio 2017/2019 in the marketplace. Be aware that it requires a .runsettings file to enable the adapter and to tell it which projects are Catch2 test applications (via a regex). Without a properly configured runsettings, it will not find your tests. I also had to increase the discovery timeout, otherwise it "forgot" my tests occasionally.
With regards the code coverage, when using Visual Studio you can configure the code coverage to include/exclude functions in the .runsettings file. See Microsoft's Site for details. For myself I added the following in the CodeCoverage section and it works pretty well so far:
<Functions>
<Include>
<Function>.*YourNamespaceHere.*</Function>
</Include>
<Exclude>
<Function>winrt.*GetRuntimeClassName</Function>
<Function>winrt::impl.*</Function>
<Function>winrt::(?!YourNamespaceHere).*</Function>
</Exclude>
</Functions>
For those that are trying to test a C++/WinRT Windows Runtime Component like me, and have code that is not exposed as part of the WRC interface, here is what I did to make that testable...
Create a C++ Shared Items Project
Move all of the code for your Windows Runtime Component (WRC) project into the shared items project, and out of the WRC project. Going forward, only add/remove files from the shared project. That way you don't have to touch the WRC or Test projects when files are added/removed.
Add a reference to this shared items project in both your original WRC project, and your test project
Make sure your test project and WRC project are configured similarly with respect compile settings and project/NuGet references
Edit the test project and ensure the RootNamespace is configured the same as the WRC project (probably has to be done via your favorite editor). This is required otherwise the generated headers will be prefixed with the namespace, and thus won't be found by the shared code.
(Optional for Code Coverage) In the test project, enable profiling (Linker > Advanced > Profile > Yes)
You should now be able to write tests that exercise the private code. As to whether or not this is the best approach, I leave to the reader. It works for me, and the code I'm testing is simple enough that I'm not overly concerned with the project definitions not aligning perfectly. Your mileage may vary.
I will note that the above can also be used to make the "Native Unit Test Project" work with C++/WinRT, you just have the extra steps of integrating the C++/WinRT bits into the test project first.
i have created unit tests for my web project but i have come across an error whereby the tests are being ignored and Visual Studio 2012 is running my localhost instead. i cannot use localhost to run my tests as there are a lot of java resources and overlays which aren't displayed. Essentially that is the point to UI testing that you test the correct interface.
The code i used in a blank project - runs perfectly and completes the test with no issues but since i need to include this into my web project, i need a way to stop Visual Studio running the localhost and get it to execute the console application test so that my selenium webdriver can run the test properly.
using: Visual Studio 2012
Selenium (webdriver)
chrome driver (latest version)
c#.Net
example code:
IWebDriver _driver;
ChromeOptions options = new ChromeOptions();
options.AddArgument("--start-maximized");
_driver = new ChromeDriver(options);
_driver.Url = "http://theURLimTesting.com/";
var verificationErrors = new StringBuilder();
_driver.Navigate().GoToUrl("http://theURLimTesting.com/");
_driver.FindElement(By.Id("Username")).Clear();
if anyone could help me and provide a solution as to how to run these tests without excluding them from the project and without having to create a proxy - i would be very grateful, as i am very much a novice.
UPDATE: as #mutt 's answer helped steer me towards the right direction with being able to resolve my question i marked the answer as right - i have managed to configure the error and create a work around and tweaking some settings to get this to work and now i can run all tests inside of the web application properly and they all function properly with executing and closing themselves in the background when done.
Separate your unit tests from the web project and it should work. Since you have them together your webapp probably has a default start page so when you "play" it will load that and VS is scoped to that browser running on IIS Express instead of the regular browser.
Personally I would have thought it would still work since Selenium is hitting the driver package that is referencing the browser, but I'm not sure what all VS is doing when it runs the webapp. If you want it with the console then move all your unit tests to the console project and they should still work on the WebApp because they will be hitting the server version and not the locally run project.
Update:
It looks like it is process bound. So Selenium and visual studio are sharing the same process. VS2008 debugging with firefox as default browser - how to make the debugger stop/close on exit?
Update2:
It looks like you should be able to determine if the process is being utilized. Then the question would be can you just kill it in your unit test script so that it will be forced to create a new one... Programmatically determine if code is running under IIS Express
I have created a few coded ui tests and linked them to the test case, and they now appear as automated and you can see the dll they link to in the test case details.
Now that I want to run the tests, MTM refuses to even start the test unless a build is defined.
However: I want to run the tests against a statically installed application in the lab environment. This is an application that I manually install, and I get this application already compiled, so no need to play around building it.
So how can I take the build server out of the loop? I don't need the application built or deployed, I'm already doing that.
All I want is the tests to run on the lab environment specified against an application that is already preinstalled.
It's asking you to define the build of the test solution, assuming that it's different from your application under test. The test assembly will be deployed to the test environment after you specify it in MTM. This article may help you with the specifics.
It is asking you to create a build for your Coded UI test solution. It requires that the tests be built so that it has something to execute when you run the tests. Assuming that your tests were recorded using your statically deployed application then they will test that same application.
I have created a coded web performance test. Now, I'm running the test from visual studio. I need to run it as a application or as a exe file.
Web performance test (coded or not) are only available for MS test execution engine. The only solution I found is to invoke the test through mstest.exe.
You can find more informations here