Guys can anyone will explain me the use of the parameter's like automated test, run settings , analyze test impact, (each of them) etc present under Test in process section of Build Definition in TFS for Visual studio 2013. I am a fresher and new to this technology so can anyone explain me this in detail.
This is a very general and basic question. And most is related to the concept.
Such as run settings.
Unit tests in Visual Studio can be configured by using a
*.runsettings file.
analyze test impact
Used to determine which tests should be run since a previous build, only for manually test on TFS2013, more details please refer this tutorial.
Because of the limitation of length, no more tautology here. If you are a totally fresher to test in TFS, you could take some time go through all the topics in this link Testing the application. Which include all the answers you want and can help you quickly,comprehensively understand knowledge in this area.
Related
I'm in the process of writing some new C++/WinRT based components in order to replace some much older C++/CX code. The goal is to be able to use third-party C++ tools that don't understand CX (static code analyzers, etc).
However the first step in the journey is to ensure I can properly unit test my own code. Unit testing C++/CX code typically used the "C++ Unit Test App" project type, which is C++/CX based and has its own issues (lack of code coverage support, run all required before tests show up in the explorer, stability, etc)
Browsing through the available project types in Visual Studio 2017, I did not see a unit test project template for C++/WinRT based projects. Is my only option to use the "C++ Unit Test App" template with all its failings, or is there another way to build tests for a C++/WinRT library?
Perhaps there is a way to configure either the "Native Unit Test Project" or "Google Test" project templates to support what I'm looking for?
Ideally what I'm looking for is something that doesn't require launching a UI, is pure C++(/WinRT), and supports Visual Studio's Code Coverage Analysis.
There is no unit test project that is specific to C++/WinRT, much like there isn't one for other libraries like STL. I would recommend Catch2 as it supports C++17 (a requirement for C++/WinRT) and works well on Windows. It is also what we use for testing C++/WinRT itself. Catch2 is nice because it helps you create a simple console app that acts as the test driver that includes all of the tests.
For code coverage I don't have a strong recommendation, but if you are using Visual Studio then you might want to try VSInstr. It can be used for code coverage and produces a report that can be viewed with Visual Studio.
Make sure your code is built using the /profile linker option. This will ensure that profile hooks are included in a dedicated section of the PE file. Next, run vsinstr to instrument any of the binaries you're interested in (that were previously built with /profile):
vsinstr /coverage tests.exe
Now run vsperfcmd to begin collecting coverage data:
vsperfcmd /start:coverage /output:report
Run the code as normal. For Catch2, you can simply run the executable at the command line. Then you need to stop the collection as follows:
vsperfcmd /shutdown
And you're done. You can now view the report in Visual Studio:
devenv report.coverage
Hope that helps. Again, this is not specific to C++/WinRT and since C++/WinRT is a header-only library you are liable to get a lot of noise that is unrelated to your specific project. I haven't found a good way to deal with that yet.
Expanding on my comment to #KennyKerr's answer for those that are interested...
If you are planning on using Catch2 as recommended, then the C++/WinRT Windows Console Application template is a great starting point. Pretty much all you have to do is tweak the main() to setup Catch2 and start writing your test cases. My only complaint is that the C++/WinRT templates don't allow you to add Windows Runtime Component project references via the UI (must be done by editing the vcxproj). There is probably a similar problem adding NuGet package references.
As noted in my comment above, there is a Catch2 test adapter for Visual Studio 2017/2019 in the marketplace. Be aware that it requires a .runsettings file to enable the adapter and to tell it which projects are Catch2 test applications (via a regex). Without a properly configured runsettings, it will not find your tests. I also had to increase the discovery timeout, otherwise it "forgot" my tests occasionally.
With regards the code coverage, when using Visual Studio you can configure the code coverage to include/exclude functions in the .runsettings file. See Microsoft's Site for details. For myself I added the following in the CodeCoverage section and it works pretty well so far:
<Functions>
<Include>
<Function>.*YourNamespaceHere.*</Function>
</Include>
<Exclude>
<Function>winrt.*GetRuntimeClassName</Function>
<Function>winrt::impl.*</Function>
<Function>winrt::(?!YourNamespaceHere).*</Function>
</Exclude>
</Functions>
For those that are trying to test a C++/WinRT Windows Runtime Component like me, and have code that is not exposed as part of the WRC interface, here is what I did to make that testable...
Create a C++ Shared Items Project
Move all of the code for your Windows Runtime Component (WRC) project into the shared items project, and out of the WRC project. Going forward, only add/remove files from the shared project. That way you don't have to touch the WRC or Test projects when files are added/removed.
Add a reference to this shared items project in both your original WRC project, and your test project
Make sure your test project and WRC project are configured similarly with respect compile settings and project/NuGet references
Edit the test project and ensure the RootNamespace is configured the same as the WRC project (probably has to be done via your favorite editor). This is required otherwise the generated headers will be prefixed with the namespace, and thus won't be found by the shared code.
(Optional for Code Coverage) In the test project, enable profiling (Linker > Advanced > Profile > Yes)
You should now be able to write tests that exercise the private code. As to whether or not this is the best approach, I leave to the reader. It works for me, and the code I'm testing is simple enough that I'm not overly concerned with the project definitions not aligning perfectly. Your mileage may vary.
I will note that the above can also be used to make the "Native Unit Test Project" work with C++/WinRT, you just have the extra steps of integrating the C++/WinRT bits into the test project first.
I have been asked to write an automated test suite. The project manager has asked me to use Visual studio CodedUI, but this has raised issues with the development team who don't want me to have access to the source code, which I understand.
I normally use Selenium Webdriver for automation.
If I was to write tests in CodedUI would I need access to the source code and would my tests interfere with the source code i.e. code could go out of sync and add additional dependencies
or I should I be able kept my tests completely independent like I can with Selenium WebDriver?
You don't need access to the source code, just the application and visual studio.
I don't agree with the position on blocking access to the source code.
Not being able to build and test against the latest code will cause testing to lag far behind development. Development and testing should really happen at the same time.
Test early and often:
http://msdn.microsoft.com/en-us/library/vstudio/ee330950%28v=vs.110%29.aspx
I have a Visual studio solution, which is designed using c# 4.0 .
I want to check the code quality for my solution and generate report out of it.
I tried the FxCop and i also got the report but i need the report something like this(from the image).
The rules compliance is 85% but in FxCop it only showed me the critical, error, etc.
I was not able to even deploy my project into SONAR because I had some timeout issue
coming for one of my project in the solution.
please someone help me.
Thanks in advance.
Regards,
Roopini
I don't know if there's an equivalent of SonarQube for .NET projects, but if you really want such reporting (which I can understand, obviously!), you should rather ask questions on how to resolve your installation issue for SonarQube instead of searching for something else. There are plenty of organizations where big .NET solutions are successfully analyzed with SonarQube and the C# plugins, so there's no reason why it can't work for you!
You can find useful material on the net to help you on this. For instance, a blog post written by John M Wright about "setting up SonarQube for C# projects". John periodically updates his post, so the information should still be very relevant.
Have you tried the tool NDepend? It generates interactive reports about .NET code quality and code rules compliance. Here are some sample reports.
NDepend is also a tool integrated in Visual Studio (2017, 2015, 2013, 2012, 2010) that proposes a range of interactive features (graph, dependency matrix, code metrics visualization, code diff...). Another point about NDepend is that code rules are actually C# LINQ queries, so it is pretty easy to customize a default code rule or create your own code rules.
NDepend also integrates in VS Team Services and you'll get all code quality data from your VSTS UI instead of being redirected to a server.
I read that you have time-out problems analyzing your code base, maybe it is because your code base is pretty large. NDepend is optimized and it can analyze a very large code base and create a report in a few dozens of seconds (it takes around a minute to analyze the whole .NET Fx).
A 14 days full featured trial is available.
Disclaimer: I work in the NDepend team
If you haven't already, I would suggest taking a look at my blog post on setting up SonarQube for C# projects: http://www.wrightfully.com/setting-up-sonar-analysis-for-c-projects/
The key to fixing your issue will be determining what the system is doing when the timeout occurs. Take a look at your log files and see what the last lines were before it timed out. It could be that your code is complex and just needs more time, in which case you can adjust the timeout values for whichever tool is running at the time.
Otherwise, I would suggest running whichever analysis tool (fxcop, gendarme, sytlecop, etc) was running when the timeout occurred outside of SonarQube. That is, run the tool directly from the commandline to see if it still times out or provides any additional information on the console.
Also, assuming you're using the sonar-runner tool to execute the SonarQube analysis, you can add the -X argument to the commandline, which will run it with debug-level logging enabled. This will create a LOT more log messages which may shed some additional light on the issue.
First some context; we are developing a large desktop WPF application in .NET 4.5 targeting 64 bit Windows 7 and 8. We are using Visual Studio 2012.2 (soon to be .3 then probably 2013!) and TFS 2012 (again .2 soon to be .3 then 2013).
Currently this product is all in a single large solution (just over 50 projects) yielding a WPF exe, a load of dlls and a nice MSI to install it.
We use TFS (gated and scheduled) to build the solution, its installer (WiX) and run its tests (SpecFlow for BDD and MSTest unit tests) and this is working very well.
I have a separate scheduled TFS build that deploys the MSI to physical test rig in a untrusted AD domain via a PowerShell script (see TFS2012 LabDefault.11 template deploy scripts fail with “Team Foundation Server could not complete the deployment task” for details of the challenges involved with that!)
OK so that's where I am, now I want to take things to the next step; CodedUI tests to drive full app integration test; I want to "Smoke Test" my builds.
So being a simple soul I added a new project to my products solution; a CodedUI test project.
This happily runs the locally installed product (rather then the just built one; as I ultimately want the CUIT to be running on a deployed test rig as a smoke test, and that rig has just installed the MSI I just built) and performs some UI tests with assertions.
Now my problem is with the CUIT project as part of the products solution a local test run finds and runs my CUIT tests, and this is undesired. I only want to run the CUIT tests in a lab builds test phase.
So is putting the CUIT project into the product solution a bad idea? or should it be a separate solution? Splitting them seems wrong somehow as they are related; the CUIT project is the full stack integration test for the solution's deployable application.
Can I include the CUIT in the products solution and stop the test runner seeing the tests? or is it better just to have two solutions?
What are the pros and cons folks?
Update
In the end we created a new solution containing a coded UI test project and ensured this was built with the same TFS build that built the UI solution. This allows us to load and run the coded UI tests locally without issues, the unit tests in the main UI project are left unmolested. Still seems a little disjointed but on a multiple person team per user test settings were too awkward splitting the coded UI into a different solution was simpler.
What I did was make one Solution and made a CUIT project within, I then made multiple Coded UI test's within that. This is good because using an orderedTest you can run them together and they also share a UIMap which helps too.
I also have/had this problem, because we are at the beginning of using CUIT. For now the CUIT remains in product solution. We do this because the tests should remain in memory of developers. When tests stay in on solution I'm afraid they get lost in oblivion. But indeed there is sometimes a bad feeling that the CUIT pollute the products solution, so i guess they will get their own solution after some time pass and the test become established.
Edit: If you use different Versions of Visual Studio you have to consider that for example a VS Prof. can’t build a solution with Code UI Tests. This means in “multi VS-version environments” you have to separate Coded UI Tests from “real” code.
Does anyone know if it is possible to create a console application in c# that calls loads a webtest in a test project?
I have added a reference to the project but get stuck when trying to call the test from main(). I am using vs 2010 to do this.
Any ideas? Searched around but couldn't find anything on what I specifically was trying to do.
Cheers.
I don't know if you're willing to consider alternative solutions other than a Visual Studio Web Test. You could write your test using the free Telerik Testing Framework http://www.telerik.com/automated-testing-tools/free-testing-framework.aspx and it can be built into your own executable as demonstrated at http://www.telerik.com/automated-testing-tools/support/documentation/user-guide/code-samples/general/test-as-application.aspx.
If you want to take advantage of the recorder and record your tests, there's also a way of executing the recorded test from another executable. http://www.telerik.com/automated-testing-tools/support/documentation/online-api-reference/html/overload_artoftest_webaii_design_execution_runhelper_test.htm,