Processes stuck when more than four instances is created in IIS - c#-4.0

Question

Background:
I have a C#.Net (.Net 4.0) Website that calls a engine to grab data from an external database through a vendor executable file that I cannot modify. Whenever user click on a specific button, the webpage will instruct the engine to spawn a few threads, and each thread will spawn a process of the executable (let's called it ABC.exe) to grab the data. The executable will then run, and save the data grabbed into a CSV file on the server. The threads then read the CSV first and consolidate all the data, do some calculation and return to the website.
Problem:
When we deployed the website to IIS 7.5 on a web server (running Window 7, four virtual processors), if we spawn more than 4 threads, the processes will just stuck there.
Test done:
When we run the exact codes using Visual Studio on the same server, no error occurred. All the threads are spawned correctly, and all process is running correctly. This is very weird.
When two user click on the button and both click will spawn 4 threads, the first four threads will work correctly while the second four threads will just stuck there.
When we run the thread with different executables that do not perform any connections, the code works.
In our code, using Task or Thread doesn't make a difference.
We suspect that this might have something to do with the outbound connections allowed to a specific IP in IIS.
Can anyone shed some light on this? Let me know if any further information is required. Thanks!

Answer 1

A couple of thoughts:
Could this be tied to thread pool threads? The CLR tries to prevent the CPU from unnecessary spinning by not handing out (initially anyways) more threads on the thread pool thread than there are processors present. Are there 4 virtual processors present? Thread pool threads should resume eventually, though, so make sure that your threads are really stuck, and not running serial.
I know of a deadlock condition when redirecting standard output in C#. Double check the MSDN article to make sure this is not happening to you.
Additional thoughts:
At this point, it sounds like your ABC.exe is suspect. Since you have Visual Studio on the server, I recommend that you fire up a separate instance of Visual Studio, and attach to one of the ABC.exe processes to see where it is hanging. Also try to run something that you are reasonably sure would exit, e.g. cmd /c dir instead of your ABC.exe.
Edit: 5/29:
I find it hard to believe that IIS would restrict outbound connections this way. Try the following simple downloaded instead of ABC.exe:
class Program
{
public static void Main(string[] args)
{
using (var reader = new System.IO.StreamReader(
System.Net.HttpWebRequest.Create("http://www.google.com")
.GetResponse().GetResponseStream()))
{
System.Console.WriteLine(reader.ReadToEnd());
}
}
}