Visual Studio 2012 MSTest vs NUnit pros and cons - visual-studio-2012

Question

We have to decide which technology to use for our unit testing. Currently we use Visual Studio 2010 and not happy with MSTest that came with that. It is buggy, poor in deployment (E.g the test setting output directory is not recognized correctly), and have several issues when trying to test assemblies in 32bit and 64bit versions. To make matters worst MSTest does not have a good impedance match with our Jenkins build system. We therefore thought of moving into NUnit. However, no one in our team has a good exposure to NUnit. Also,we will be soon moving into Visual Studio 2012.
I need to know the pros and cons of Visual Studio 2012 MSTest vs Nunit latest version. Since most of the articles on stack overflow are related to older versions of VS they are not related to us. I guess Microsoft has improved MSTest a lot since 2010. Please provide an unbiased comparison with detail technical issues you have faced in both technologies (newer versions only)

Answer 1

I'm using both MSTest and NUnit at the moment. IMHO NUnit is still a better solution. If you have Visual Studio 2012 Premium edition then it's actually quite nice, except for the fact that you can't seem to group together tests. I like the fact it's integrated into Visual Studio, but the lack of grouping and the ability to run a subset of tests without manually selecting them is a huge problem.
The coverage analysis is also pretty neat in Premium. It's fast and gives you what you need quickly. It is a Premium feature though.
Since there are still lacking features in MSTest (even removed features since vs2010), I would still recommend using NUnit for unit tests. The benefits include test grouping by namespace, the ability to add test case annotations (running the same test multiple times with different parameters) and it works well with Opencover and Report Generator for coverage analysis. The main cited con is that it's not integrated like MSTest, so it really depends how much that matters to you as to whether or not that is a con.

Answer 2

#Biranchi: It doesn't matter anymore which unit test framework you use in Visual Studio 2012 (and upwards). See my blogpost here, the sequel to the one you refer to. http://blogs.msdn.com/b/visualstudioalm/archive/2012/11/20/part-2-using-traits-with-different-test-frameworks-in-the-unit-test-explorer.aspx
You can even mix and match tests from different frameworks, you can even do that down to the method level !!
This means you can even move legacy code from one to another with no bad sideeffects.
Also see this for how to use Nuget to install the NUnit adapter into the solution, freeing the developer for installing it herself.
http://blogs.msdn.com/b/visualstudioalm/archive/2013/06/11/part-3-unit-testing-with-traits-and-code-coverage-in-visual-studio-2012-using-the-tfs-build-and-the-new-nuget-adapter-approach.aspx
#Sriwantha: MSTest is a simpler framework than NUnit. NUnit (and also XUnit) give you more flexibility, which also leads to less code to write. One example: If you are using categories (and you should), MSTest require a category to decorate every method. NUnit allows you to decorate the class - that will take effect for all the methods in that class. NUnit also allow to you use strongly typed categories
public class Integration : Category {}
This is enough to declare a category that you can use instead of
Category("Integration");
where you risk spelling errors.
NUnit has much better support for data driven tests.
NUnit has also support for theories
to name a few.

Answer 3

VS2012 does allow categorization of tests into groups if you have "Update 1" or later:
http://msdn.microsoft.com/en-us/library/hh270865.aspx#BKMK_Grouping_and_filtering_the_test_list

Answer 4

Have you looked into Traits functionality of VS ?
http://blogs.msdn.com/b/visualstudioalm/archive/2012/11/09/how-to-manage-unit-tests-in-visual-studio-2012-update-1-part-1-using-traits-in-the-unit-test-explorer.aspx
Grouping is much better in mstest (2012 update 1) compared to nunit.

Answer 5

One more thing to add here. It looks like MSTEST engine doesn't work well with TFS Build in certain scenarios. If you are using TFS build, it won't report the skipped tests (marked with Ignore attribute). It will only include Assert.Inconclusive. If that's an issue, you should use NUnit instead of MSTest.