ENVIRONMENT
We are currently using 2 Chutzpah packages (this and this) in Visual Studio 2013 Update 4 to enable us to execute individual Jasmine unit tests (with Chutzpah context menu "Open in Browser") against an angular application we are developing.
We currently have to use Chrome developer tools to set breakpoints and debug the unit tests.
The current Jenkins build is running Moq Unit Tests against the C# application code and cross-browser Selenium tests with SauceLabs.
The next step for the JS tests is to execute them as part of our Jenkins build along with the other tests.
To achieve this the best tool currently is Intern.
OUR REQUIREMENTS
Execute JS Tests and calculate coverage as part of the Jenkins Build
on SauceLabs.
Execute and step through (debug) JS Unit Tests
Regarding 1, I can see from the Intern example that we will probably have to (and can) rewrite our tests for Intern to get them to execute on Jenkins and SauceLabs.
Regarding 2, the point of this post:
MY QUESTIONS
Can tests written in the manner of the Intern example be debugged in chrome with developer tools?
Is there a need for a Chutzpah-like tool to integrate Visual Studio
with Intern? (Is this the analogous tool?)
Related
I have added a release definition in Visual Studio Team Services to deploy an application to a Azure cloud service. I'm trying to run some integration tests after a successful deployment:
But unfortunately tests are not run:
Warning: No test is available in C:\a\301532a74\Build\Api.Web.IntegrationTests\Api.Web.IntegrationTests.dll. Make sure that installed test discoverers & executors, platform & framework version settings are appropriate and try again.
Information: Additionally, you can try specifying '/UseVsixExtensions' command if the test discoverer & executor is installed on the machine as vsix extensions and your installation supports vsix extensions. Example: vstest.console.exe myTests.dll /UseVsixExtensions:true
##[warning]No results found to publish.
I have installed the NUnit Test Adapter (nuget package) for all projects containing tests (unit tests and integration tests). The weird thing is that I am able to run the tests as part of a build but not as part of a release.
I also tried to set the Path to Custom Test Adapters but it doesn't seem to make a difference.
What I am missing here? Is there a better way to run some integration tests as part of a release?
This is usually caused by the NUnit test adapter cannot be found by VSTest.
Usually, In "Build" environment, NUnit test adapter is restored in working folder by nuget, so VSTest can find it during the testing. But in "Release" environment, NUnit test adapter does not exist by default. So you need to update your build definition to upload the NUnit test adapter files to "drop" and the NUnit test adapter will be avaiable in "Release" environment after the artifacts is downloaded. Then you can set the "Path to Custom Test Adapters" to "$(System.DefaultWorkingDirectory)\" and try again.
If you are using your own build agent, you can also place the NUnit Test Adapter in a permanent path that the build agent can access and set the "Path to Custom Test Adapters" to it. Or install NUnit test adapter extension for VS on the build agent and enter "/UseVsixExtensions:true" in "Visual Studio Test\Other console options".
I have found a solution for my problem, based on Eddie's suggestions:
Step 1 - Copy all NUnit assemblies to the output folder of the test project (set the Copy Local property to True)
Step 2 - Create an artifact that will contain all files of the bin folder of the test project. This will contain not only the test assemblies but also the NUnit assemblies
Step 3 - In the test task, set the Path to Custom Test Adapters to the artifacts folder that contains the test assemblies and the NUnit adapters assemblies
That's it! Next time you create a release and deploy to any environment tests will be executed.
My problem was that i didn´t have a testrunner for NUnit installed at all, since I´m using resharpers testrunner.
Therefore my solution was as simple as adding the nuget package, "NUnit3TestAdapter" version="3.4.0", to my test project.
In visualstudio.com (VS online) "Nuget restored adapters are automatically searched for." and you need not/should not specify a path for "Custom Test Adapters"
- I don´t know if this also works for NUnit test adapter version 2.
I have just added a specflow feature however the generated code is for NUnit. I would prefer to use Resharper to run the tests.
I installed specflow from NuGet. I did not install any runner projects.
Here is a sample of the generated .feature.cs
//
------------------------------------------------------------------------------ // <auto-generated> // This code was generated by SpecFlow (http://www.specflow.org/). // SpecFlow Version:1.9.0.77 // SpecFlow Generator Version:1.9.0.0 // Runtime Version:4.0.30319.18444 // // Changes to this file may cause incorrect behavior and will be lost if // the code is regenerated. // </auto-generated> //
------------------------------------------------------------------------------
#region Designer generated code
#pragma warning disable namespace Test.FeatureHanding {
using TechTalk.SpecFlow;
[System.CodeDom.Compiler.GeneratedCodeAttribute("TechTalk.SpecFlow", "1.9.0.77")]
[System.Runtime.CompilerServices.CompilerGeneratedAttribute()]
[NUnit.Framework.TestFixtureAttribute()]
[NUnit.Framework.DescriptionAttribute("Set Filter")
By default, all SpecFlow tests are in NUnit! SpecFlow simply generates the tests from the plain text for you.
You simply have to click on the icon to the left of the test method declaration in the generated YourFeature.feature.cs file, or you Resharpers test explorer to find it.
I think there is a terminology misunderstanding here.
NUnit is a testing framework (others are xUnit, mbUnit, MSTest). NUnit comes with a test runner which can run unit tests written with the NUnit framework.
Resharper has a test runner which can run tests from many testing frameworks (all of the above frameworks I believe). There are other test runners available. Visual studio (2012+ I believe) come with a test runner which can run MSTest tests without a plugin. NCrunch can run tests from any of the frameworks, and xUnit and mbUnit probably have runners provided which can run tests written in those frameworks.
Once you have built your feature files then Resharper should find the tests that SpecFlow has generated.
I've written a native unit test dll that uses VS2012's CppUnitTestFramework. I can run these tests in the VS2012's IDE in the Test Explorer window.
But, I'd like to also run these tests from our MsBuild script. I guess I need to launch some test runner exe with appropriate command line, but am struggling to find information on this.
Can you help point me in the right direction. Thanks.
The Visual Studio Test Runner is a simple command line tool which needs to be called in order to find and execute the tests. Creating a small msbuild task or using the standard exec task would be enough to invoke the tests after the build.
As long as the test runner is installed properly, it should just pick up these tests and execute them.
See this blog post explaining all the command line intricacies of the new vstest.console.exe.
So I've downloaded VS2012 RTM and upgraded my Coded UI test projects to the VS2012 versions.
The reason I have done this is because I was having issues with VS2010 Coded UI tools due to partial support of IE9.
So I ran the tests from my developer machine both locally and remotely (making use of the new Visual Studio agent tools), and the tests worked successfully. Presumably, the reason my tests now work is that VS2012 upgraded them so that they now work with IE9, right?
So - this is my problem. I want to automate these tests, by launching them from our build machine - but our build machine still runs on VS2010, and for the time-being this is not going to change.
So I checked in my new VS2012 Coded UI test solution into TFS and queued a new build - so the build machine built my solution. And the build was successful. All good there.
So, next I created a new test case in Microsoft Test Manager and associated with the ordered test list in my new solution. Then I launched the tests (using the existing VS2010 test controller) on my remote test environment (which has the existing VS2010 test agent tools).
But the tests failed - the same issues that affected my tests with the VS2010 Coded UI tests (due to lack of full support for IE9)
Why did they fail?
Do I need the new VS 2012 agent tools for my test controller and agents? Must I build my solution in VS2012?
Ideally, I don't want to have to install VS2012 RTM on my build machine - I want to do the minimum possible to get my tests working and automating.
Is there a way around this?
Your coded ui tests reference assemblies that come with VS installation or your agent's installation (the assemblies defininf the WpfControl, Mouse, Keyboard, Playback and other classes).
So if you run it on a build/test machine with older versions of the dll's the same issues are going to stay, the tests will not start using the newer fixed assemblies thatt were shipped with VS2012.
As a temporary workaround you can check what dll's you reference and make sure that the build process puts them in the same directory with your tests assembly. That way the lookup for them will find the in the current directory without using the /path and finding them in the VS installation directory.
i want to run a test project in ccnet,
can anyone help?
i am not using NUnit. I am using [Test Class()] attribute in my web project, and wanna run this project from ccnet
It appears the test framework would be MsTest
In order to run MsTest on the build server you have to install visual studio or do a work around or hack (alternate hack).
Then you can use MsTest.exe
Here is the official documentation on running MSTest tests.
Similar question and more hacks or work-arounds
If you're using NUnit, you may want to take a look at the documentation
I'd suggest looking in CCNet to run a msbuild task
There are a number of examples to run MSTest from MSBuild tasks.
This one looks prommesing:
http://harriyott.com/2005/07/running-mstest-from-msbuild.aspx?post=/2005/07/running-mstest-from-msbuild.blogpost
--
I myself am using NUnit and xUnit. Might you consider switching to NUnit for testing? There is a wider adoption of NUnit in combination with CCNet, which makes searching answers easier.