best way to run multi-threaded nunit tests - multithreading

Question

I currently trying to find a solution, how to ensure that a test fails if an exception occurs in a thread which is spawn by the test method.
I DON'T want to start a discussion about more than one thread in a unit test. => "unit test".Replace("unit","integration");
I already read a lot of threads in several forums, I know about CrossThreadTestRunner, but I'm searching for a solution whichs integrates into nunit, and does not require to rewrite a lot of tests.

Answer 1

The reason that exceptions on non-test threads (i.e. other spawned threads) do not cause tests to fail is that NUnit is configured by default to use legacyUnhandledExceptionPolicy which is a .Net process level setting which can be applied via app.config, i.e.:
<legacyUnhandledExceptionPolicy enabled="1"/>
Enabling this setting (i.e. setting it to "1") causes exceptions which do not occur on the main thread to be ignored.
I wrote an article which goes into more detail in reference to this issue with the ReSharper test runner, but it applies equally to the NUnit test runner:
https://web.archive.org/web/20101006004301/http://gojisoft.com/blog/2010/05/14/resharper-test-runner-hidden-thread-exceptions/)
ReSharper test runner – hidden thread exceptions
By Tim Lloyd, May 14, 2010 4:31 pm
We use the ReSharper test runner here at GojiSoft to run NUnit tests
from within Visual Studio. It’s a great test runner, but doesn’t play
nicely with multi-threaded components where exceptions may occur on
non-test threads. Unhandled exceptions on non-test threads are hidden
and tests which should fail, instead pass.
...
The problem lies in the fact that the ReSharper test runner is
configured to behave in the same way as .Net 1.0 and 1.1 apps where
unhandled exceptions on non-main threads were swallowed. The situation
improves from .Net 2.0 where all unhandled exceptions flatten the
process. However, Microsoft had to allow existing .Net 1.0 and 1.1
apps the option of behaving as before on new .Net frameworks. They
introduced the app.config setting: legacyUnhandledExceptionPolicy.
The ReSharper test runner is configured by default to use the .Net 1.0
and 1.1 policy, so if there is an unhandled non-test thread exception
it does not bubble up and cause the test to fail – the test passes,
and we get a false positive instead.
If unhandled exceptions on non-test threads should fail tests, the
app.config for the ReSharper test runner has to be updated.
...
Turn the legacy unhandled exception policy off by editing legacyUnhandledExceptionPolicy: <legacyUnhandledExceptionPolicy enabled="0" />
Now multi-threaded tests fail as expected when they raise exceptions on non-test threads:
Buyer beware…
There is a caveat to this. Exceptions on non-test threads will now
flattened the test runner and test suite execution will be halted when
they happen. This is in contrast to normal test runs where failed
tests are marked as failed, and the test runner continues. ...

Answer 2

I just had the same problem, my solution is to catch the exception and increment an exception counter, so the Test method just have to assert the exception counter is 0 to confirm no thread got an exception.
An extract of my test code once specific environment stuff is removed:
const int MaxThreads = 25;
const int MaxWait = 10;
const int Iterations = 10;
private readonly Random random=new Random();
private static int startedThreads=MaxThreads ;
private static int exceptions = 0;
…
[Test]
public void testclass()
{
// Create n threads, each of them will be reading configuration while another one cleans up
Thread thread = new Thread(Method1)
{
IsBackground = true,
Name = "MyThread0"
};
thread.Start();
for (int i = 1; i < MaxThreads; i++)
{
thread = new Thread(Method2)
{
IsBackground = true,
Name = string.Format("MyThread{0}", i)
};
thread.Start();
}
// wait for all of them to finish
while (startedThreads > 0 && exceptions==0)
{
Thread.Sleep(MaxWait);
}
Assert.AreEqual(0, exceptions, "Expected no exceptions on threads");
}
private void Method1()
{
try
{
for (int i = 0; i < Iterations; i++)
{
// Stuff being tested
Thread.Sleep(random.Next(MaxWait));
}
}
catch (Exception exception)
{
Console.Out.WriteLine("Ërror in Method1 Thread {0}", exception);
exceptions++;
}
finally
{
startedThreads--;
}
}
private void Method2()
{
try
{
for (int i = 0; i < Iterations; i++)
{
// Stuff being tested
Thread.Sleep(random.Next(MaxWait));
}
}
catch (Exception exception)
{
Console.Out.WriteLine("Ërror in Method2 Thread {0}", exception);
exceptions++;
}
finally
{
startedThreads--;
}
}

Answer 3

I solved the problem by creating an addin for nunit, which "installs" an ITestDecorator.