I am working on a project which is developed using plain JSF with webservice backend and our target runtime is WAS8.5. It’s an internet application and being used by more that 500 concurrent users.
We are performing NFT testing with 150% load ie. 750+ concurrent users. From Introscope, I could see web container thread pool size is set as min&max value of 50. We feel web container thread pool size is blocking the concurrent loads we are trying to achieve. We are expecting experts suggestion for ThreadPoolModule and webcontainer poolsize settings in our WAS servers. (FYI: We have two WAS servers in our cluster)
Related
Our IIS website has its Maximum Concurrent Connections set to 4294967295. Our Web API application is logging all the requests it serves to Application Insights and the two do not appear to match up. A call which appears to get served quickly in Insights does not appear to complete quickly in IIS's logs.
What could cause this and is this an indication that IIS is running out of connections, even if the maximum is set ridiculously high?
Phrasing this another way (after reading #zakima's comment): What should I be looking for to identify requests which are getting delayed in IIS before or after they hit the application itself?
Maximum concurrent connections defaults to 4294967295, which is a staggering number. But it does not mean that the site can have the ability to execute 4294967295 concurrent connections.
Assuming that 4294967295 concurrent connections come at the same time, IIS does not immediately start 4294967295 threads to process, because this is unrealistic. For the processing of connections, IIS has the "Maximum concurrent worker threads" limit. From some sources, this number is related to the operating system. If IIS can only start 10 worker threads in the first time to process, then the other 4294967285 must queue.
In another word, 4294967295 means the maximum amount of allowed by default concurrent connections from http.sys module to the site. Then these request will hit each module of IIS and hit application at last.
If you want to check the real max concurrent connections of IIS, please refer to this article to use Performance Monitor.
Regard to how to monitor the request getting delayed in IIS before or after, I suggest you use failed request tracing. Here is the sample of failed request tracing log of my asp.net application.
I'm currently trying SignalR and RabbitMQ in order to round-robin / load balance json webservice queries and I'm having troubles with the memory consumption by one of the application when it processes large (~ 300 - 2500 kb) messages.
I have a IIS server hosting a web application (named "Backend") that needs to query an another web application (name "Pricing") also hosted by a IIS server.
In order to keep a connection alive with my RabbitMQ server, I developped console application that are connected to Backend and Princing using SignalR.
So when Backend needs to query Princing, it asks its console to publish the message in the queue and the console attached to Pricing takes the message and give it to Pricing (with Invoke<> method). When Pricing finished its job, it asks its console to publish the reply message and the console attached to Backend takes it and give to Backend.
To sum up :
[Backend] -> [Console] -> [RabbitMQ] <- [Console] <- [Pricing]
And I have 2 Pricing taking messages from their console from the RabbitMQ queue.
This setup is to replace a traditionnal webservice query between the 2 IIS and benefit from the advantages of RabbitMQ (load balancer and asynchronous call in a micro/web services architecture)
I added
GlobalHost.Configuration.MaxIncomingWebSocketMessageSize = null;
in Startup.cs in both IIS in order to accept large messages.
When I take a look at Pricing's memory consumption in Windows Task Manager, it quickly grows from 500Mb to 1500Mb (in 5 minutes, dealing with neverending queries from Backend to test the setup).
I tried something else by writing the queries content in files in a shared folder and just publishing the name of the file in RabbitMQ's messages and the memory consumption of Pricing (with of course a code modification to load the file) doesn't move and stays around 500Mb.
So it seems that it has something to do with the message length that my console passes to the IIS.
I tried to disconnect the console from the IIS Hubs because I thought that it will maybe free some memory but nope.
Does anyone experienced this issue of memory consumption by large messages into Hubs ? How can I check if there's indeed a memory leak in my application ?
What about using SignalR and RabbitMQ in web/micro services environment ? Any feedback ?
Many thanks,
Jean-Francois
.NETFramework : 4.5
Microsoft.AspNet.SignalR : 2.4.1
So it seems that the version of SignalR I use (.NetFramework) allows to tune the number of messages per hub per connection kept in memory.
I fixed it to an arbitrary 50 in Startup.cs
GlobalHost.Configuration.DefaultMessageBufferSize = 50;
Its default value is 1000, meaning (if I understood it clearly) that IIS keep a circular buffer of 1000 messages in memory. Some of the messages were weighting 2.5Mo meaning that the memory used could go up to 2500Mo per connection.
As my IIS only has one connection (its console) and doesn't need to keep track of messages (as it works as webservice), it seems that 1000 messages is way too much.
With the limit of 50 messages, the memory used by the application in Windows Task Manager stays put (around 500Mo).
Is there any flaw in the way I'm using it ?
Thanks !
I am currently stress testing a .Net Core application, targeting netcoreapp2.2, that is hosted on Azure as a App Service connected to a P1V2 (210 ACU, 3.5GB memory) service plan with 2 instances.
The endpoint that I'm stress testing is very simple, it validates a Oauth2.0 token, gets the user and some info about the user from a P2 (250 DTU) Azure hosted database, total 4 db queries per request, and returns the string "Pong".
When running 15 concurrent users (or more) in 200 loops I see the stop(s) in processing seen in the image (between the high peaks). The service plan never hits more than around 20-35% CPU and the database never uses more than 2% load. Increasing the users decreases the average throughput.
When looking at the slow requests it is like it just randomly stops, never at the same place. When I look at the DB requests I never see a request that takes longer than a couple of 100 milliseconds while some requests can take upwards to 5-6s to process.
It feels like I reach some limit which results in something stopping for a period of time, but I can't figure out where the problem lies.
When running the same stress locally I don't see these stops.
I'm using jmeter cli to run the stress tests against both environments.
Any help is greatly appreciated, thanks!
This could be because of Azure DDOS protection behaviour.
If your application is being attacked by a DDOS attack, Microsoft will
stop all connections to your end point and in effect taking down your
service.
To avoid this you need to setup Web application firewall (WAF) to exclude healthy requests.
I am running a load test using JMeter on my Azure web services.
I scale my services on S2 with 4 instances and run JMeter 4 instances with 500 threads on each.
It starts perfectly fine but after a while calls start failing and giving Timeout error (HTTP status:500).
I have checked HTTP request queue on azure and found that on 2nd instance it is very high and two instances it is very low.
Please help me to success my load test.
I assume you are using Azure App Service. If you check the settings of your App, you will notice ARR’s Instance Affinity will be enabled by default. A brief explanation:
ARR cleverly keeps track of connecting users by giving them a special cookie (known as an affinity cookie), which allows it to know, upon subsequent requests, to which server instance they were talking to. This way, we can be sure that once a client establishes a session with a specific server instance, it will keep talking to the same server as long as his session is active.
This is an important feature for session-sensitive applications, but if it's not your case then you can safely disable it to improve the load balance between your instances and avoid situations like the one you've described.
Disabling ARR’s Instance Affinity in Windows Azure Web Sites
It might be due to caching of network names resolution on JVM or OS level so all your requests are hitting only one server. If it is the case - add DNS Cache Manager to your Test Plan and it should resolve your issue.
See The DNS Cache Manager: The Right Way To Test Load Balanced Apps article for more detailed explanation and configuration instructions.
Good Afternoon Everyone,
I am load testing my .NET Web API which is hosted on a Windows 2008 Server virtual machine. I am using Visual Studio 2012 Load Test. However, once my load test reaches 780 concurrent users, the CPU % starts to decrease as shown in the image attached. The load test reaches a maximum of 1000 concurrent users but the CPU % is still decreasing at the highest user load. I cannot explain why. Is any kind of IIS limit being reached? Why does this occur? Is the maximum user load reached for this function?
Just looking for an explanation to this result and some guidance.
Thank you
IIS does have separate output cache settings which are enabled by default which does start to make sense after considering how it handles dynamic content with static response and cache worthiness:
The IIS output caching feature targets semi-dynamic content. It lets
you cache static responses for dynamic requests and increase
scalability
Configuring Cache Worthiness:
Even if you enable output caching, IIS does not immediately cache a
request. It must be requested a few times before IIS considers a
request to be “cache worthy.” Cache worthiness can be configured via
the ServerRuntime section.
Two properties determine cache worthiness:
frequentHitTimePeriod
frequentHitThreshold
A request is only cached if more than frequentHitThreshold requests for a cacheable URL arrive within the frequentHitTimePeriod.
This was a good explanation: http://www.iis.net/learn/manage/managing-performance-settings/configure-iis-7-output-caching