We have a .NET 2.0 Remoting server running in Single-Call mode under IIS7. It has two APIs, say:
DoLongRunningCalculation() - has a lot of database requests and can take a long time to execute.
HelloWorld() - just returns "Hello World".
We tried to stress test the remoting server (on a Windows 7 machine) in a worst case scenario by bombarding it randomly with the two API calls and found that if we go beyond 10 client requests, the HelloWorld response (which generally is less than 0.1 sec) starts taking longer and longer going into many seconds. Our objective is that we dont want to have the long-running remoting calls to block the short-running calls. Here are the performance counters for ASP.NET v2.0.50727 if we have 20 client threads running:
Requests Queued: 0
Requests Executing: (Max:10)
Worker Processes Running: 0
Pipeline Instance Mode: (Max:10)
Requests in Application Queue: 0
We've tried setting maxConcurrentRequestsPerCPU to "5000" in registry as per Thomas's blog: ASP.NET Thread Usage on IIS 7.0 and 6.0 but it hasn't helped. Based on the above data, it appears that the number of concurrent requests is stuck at 10.
So, the question is:
How do we go about increasing the concurrent requests? The main objective is that we don't want to have the long-running remoting calls to block the short-running calls.
Why are the Max Requests Executing always stuck at 10?
Thanks in advance.
Windows 7 has a 20 inbound connection limit. XP and prior was limited to 10 (not sure about Vista). This is likely the cause of your drop in performance. Try testing on an actual server OS that doesn't have an arbitrary connection limit.
Related
Our IIS website has its Maximum Concurrent Connections set to 4294967295. Our Web API application is logging all the requests it serves to Application Insights and the two do not appear to match up. A call which appears to get served quickly in Insights does not appear to complete quickly in IIS's logs.
What could cause this and is this an indication that IIS is running out of connections, even if the maximum is set ridiculously high?
Phrasing this another way (after reading #zakima's comment): What should I be looking for to identify requests which are getting delayed in IIS before or after they hit the application itself?
Maximum concurrent connections defaults to 4294967295, which is a staggering number. But it does not mean that the site can have the ability to execute 4294967295 concurrent connections.
Assuming that 4294967295 concurrent connections come at the same time, IIS does not immediately start 4294967295 threads to process, because this is unrealistic. For the processing of connections, IIS has the "Maximum concurrent worker threads" limit. From some sources, this number is related to the operating system. If IIS can only start 10 worker threads in the first time to process, then the other 4294967285 must queue.
In another word, 4294967295 means the maximum amount of allowed by default concurrent connections from http.sys module to the site. Then these request will hit each module of IIS and hit application at last.
If you want to check the real max concurrent connections of IIS, please refer to this article to use Performance Monitor.
Regard to how to monitor the request getting delayed in IIS before or after, I suggest you use failed request tracing. Here is the sample of failed request tracing log of my asp.net application.
In my application when i execute 2000 virtual users in thread(No: of threads) for 1 http request my response time was 30 sec , when i changed no of threads to 500 and instead of 1 http request I put 4 copies of same http request, RESPONSE TIME WAS 3 SEC . What is the difference? Is it the right way to reduce no of threads and increasing replicas of request? please help
Note: In each reqest i have changed the user id also
In terms of HTTP Request samplers your test must behave exactly like real browser behaves so artificially adding more HTTP Requests may (and will) break the logic of your workload (if it is in place).
In your case high response time seems to be caused by incorrect JMeter configuration, i.e. if JMeter is not properly configured to high load it simply will not be able to fire requests fast enough resulting in increased response time while your server will just be idle.
2000 threads sounds like quite a big number so make sure to:
Follow JMeter Best Practices
Follow recommendations from 9 Easy Solutions for a JMeter Load Test “Out of Memory” Failure especially these:
Increase JVM Heap size allocated for JMeter
Run your test in non-GUI mode
Remove all the Listeners from the Test Plan
Monitor baseline OS health metrics on the machine where JMeter is running (CPU, RAM, Disk, Network usage). You can use JMeter PerfMon Plugin for this. If you will notice the lack of any of the aforementioned resources, i.e. usage will start exceeding, say, 90% of total available capacity - JMeter is not acting at full speed and you will need to consider Distributed Testing.
To extend #Dmitri T answer, If your server response 10 times more on load, as you execute 2000 virtual users, it means there's a bottleneck that you need to identify.
Read JMeter's Best Practices
consider running multiple non-GUI JMeter instances on multiple machines using distributed mode
Also check Delay Thread creation until needed checkbox in Thread Group
JMeter has an option to delay thread creation until the thread starts sampling, i.e. after any thread group delay and the ramp-up time for the thread itself. This allows for a very large total number of threads, provided that not too many are active concurrently.
And set Thread Group Ramp-up to 2000
Start with Ramp-up = number of threads and adjust up or down as needed.
I'm troubleshooting performance / scalability issues with an asp.net app we've migrated to asp.net core 2.0. Our app is hosted on azure as an app service, and is falling over far too easily with any moderate traffic.
One thing that's puzzling me is how multiple concurrent requests are handled. From what I've read here, Kestrel uses multiple event loops to handle your requests. But the actual user code is handled on the .net thread pool (that's from here).
So, as an experiment - I've created a new asp.net core 2.0 MVC app, and added a rather nasty action method:
[AllowAnonymous]
public ActionResult Wait1()
{
System.Threading.Tasks.Task.Delay(1000).Wait();
return new StatusCodeResult((int)HttpStatusCode.OK);
}
Now, when I push this to azure, I'd expect that if I send say 100 requests at the same time, then I should be OK, because 100 requests sounds like minor load, right? And the waiting will happen on the thread pool threads, right?
So - I do just this and get some rather poor results - sample highlighted in red:
Hmm, not what I expected, about 50 seconds per request... If however I change the frequency so the requests are spaced a second apart, then the response time is fine - back to just over 1000ms as you'd expect. It seems if I go over 30 requests at the same time, it starts to suffer, which seems somewhat low to me.
So - I realise that my nasty action method blocks, but I'd have expected it to block on a thread pool thread, and therefore be able to cope with more than my 30.
Is this expected behaviour - and if so is it just a case of making sure that no IO-bound work is done without using async code?
Is this expected behaviour - and if so is it just a case of making sure that no IO-bound work is done without using async code?
Based on my experience, it seems that is as expected behaviour. We could get answer from this blog.
Now suppose you are running your ASP.Net application on IIS and your web server has a total of four CPUs. Assume that at any given point in time, there are 100 requests to be processed. By default the runtime would create four threads, which would be available to service the first four requests. Because no additional threads will be added until 500 milliseconds have elapsed, the other 96 requests will have to wait in the queue. After 500 milliseconds have passed, a new thread is created.
As you can see, it will take 100*500ms intervals to catch up with the workload.
This is a good reason for using asynchronous programming. With async programming, threads aren’t blocked while requests are being handled, so the four threads would be freed up almost immediately.
I recommand that you could use async code to improve the performance.
public async Task<ActionResult> Wait1()
{
await Task.Delay(TimeSpan.FromSeconds(15));
return new StatusCodeResult((int)HttpStatusCode.OK);
}
I also find another SO thread, you could refernce to it.
I have an application in nodejs, which I am testing against thousands of users. For 1000 users (when server is deployed at local), Jmeter fails for most test cases providing this:
org.apache.http.NoHttpResponseException: The target server failed to respond
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:95)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:61)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254)
at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289)
at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:191)
at org.apache.jmeter.protocol.http.sampler.MeasuringConnectionManager$MeasuredConnection.receiveResponseHeader(MeasuringConnectionManager.java:201)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127)
at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:715)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:520)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.executeRequest(HTTPHC4Impl.java:517)
at org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.sample(HTTPHC4Impl.java:331)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerProxy.sample(HTTPSamplerProxy.java:74)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1146)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1135)
at org.apache.jmeter.threads.JMeterThread.process_sampler(JMeterThread.java:434)
at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:261)
at java.lang.Thread.run(Thread.java:745)
Sometimes, I get this as well:
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:196)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:281)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:92)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:61)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254)
at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289)
at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:191)
at org.apache.jmeter.protocol.http.sampler.MeasuringConnectionManager$MeasuredConnection.receiveResponseHeader(MeasuringConnectionManager.java:201)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127)
at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:715)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:520)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.executeRequest(HTTPHC4Impl.java:517)
at org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.sample(HTTPHC4Impl.java:331)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerProxy.sample(HTTPSamplerProxy.java:74)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1146)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1135)
at org.apache.jmeter.threads.JMeterThread.process_sampler(JMeterThread.java:434)
at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:261)
at java.lang.Thread.run(Thread.java:745)
I tried all the steps in this link:
https://wiki.apache.org/jmeter/JMeterSocketClosed
None of them worked. (I am using Jmeter 2.13)
If you are running your app server locally (I assume with moderate level of HW i.e. dual/quad core CPU, 4/8 GB RAM etc.) and running Jmeter instance on same server then you should understand below things,
As a rule of thumb you can assume Jmeter alone (with all tuning settings applied and enough rampup)can create 500-1000 threads with that hw.
You are running your app server on same machine with Jmeter. This means your app server is getting less resources to use. From errors it looks like that target server/app server is unable to handle that load.
This behavior is obvious because of many reasons like insufficient memory, over CPU utilization, IO issues, Jmeter is not coping with server.
What you can do is,
Try to deploy app server on separate machine with equal or better hw.
Follow all Jmeter best practices mentioned in above link.
Run the test and monitor resource utilization on both servers just to correlate the values with results.
Check if your test passes or not. (Even if it doesn't then at least you know that its not because of Jmeter :) and with the help of resource util logs, Jmeter logs you can find out the bottleneck.)
Similar question I found was Why am I receiving Response code: Non HTTP response code: java.net.SocketException?.
when running maximum number of user, i hope you had taken care about various aspects of the run
user ramp up time
proper timers ( decent amount of think time must be provided)
and most of all , the server that is being utilized should have capability to respond
for a given simple server ( which does not support multi-threading), when we send concurrent requests continuously, at a given point of time, we must analyse and conclude a healthy load , at a point when there are more number of requests being sent, the usual behavior is the script would make you run in errors ;)
Please recheck and post additional info.
I am new to connection management in jboss and hibernate.I have an application using spring + hibernate running on jboss 7.I did some reading but have few doubts now:
How are connections and threads related to access a application.
Suppose i have maximum pool size of 10.Does it mean only 10 threads
can access my application to perform database operations at a time?
If no, what happens when there are more than 10 threads say 15 or 20 accessing it?
Does other threads wait for running threads to complete and they run next?
(or)
This results in a connection error like no managed connections available?