Recently due to following problem my website stopped working. After restarting tomcat my issue is solved, but I want to know why and when tomcat generates maximum threads.
The problem was as follows:
Maximum number of threads (150) created for connector with address null and port 443
And suddenly my website stopped working.
Few Pointers :
Connectors are defined in server.xml file in $(TOMCAT_HOME)/conf directory. You can check the settings in this file and compare it with default connector setup.
Usually number of threads is equal to the number of incoming requests to the Server. You can check if there is some script which is triggering such requests.
you can also check if the request threads for the webapps are completing their processing normally and getting released for other requests.
if you are using an IDE like eclipse, etc. to start tomcat, then you shall be able to see which threads are being generated when running in debug mode.
Hope this helps.
Related
Is there any setting in IIS 6 where we can limit number of parallel requests coming from specific host / machine.
I have noticed when a specific web service is invoked from specific server with parallel thread, it times out when parallel thread are more than 10., where as similar works fine when number of parallel thread is limited to 10.
Similar behavior is observed even if requests are coming via Load Balancer to server.
Thanks in anticipation
yes you can, have a look here:
https://learn.microsoft.com/en-us/iis/configuration/system.applicationhost/sites/sitedefaults/limits
The limits element replaces the following IIS 6.0 metabase settings:
ConnectionTimeout
MaxBandwidth
MaxConnections
you may also want to check out this article detailing the steps specifically for IIS6:
https://www.microsoft.com/technet/prodtechnol/WindowsServer2003/Library/IIS/b2b550de-f655-4fb6-9bed-dfc9583b6700.mspx?mfr=true
I have an application in the Production environment which is Windows Server 2012/IIS 8 and is load balanced.
Recently out of nowhere, the website app pool suddenly started gettig disabled. The System Windows Logs logged the following error message by the Resource-Exhaustion-Detector ...
Application Pool 'x' is being automatically disabled due to a series of failures in the process(es) serving that application pool.
Windows successfully diagnosed a low virtual memory condition. The following programs consumed the most virtual memory: w3wp.exe (6604) consumed 5080641536 bytes, w3wp.exe (1572) consumed 477335552 bytes, and w3wp.exe (352) consumed 431423488 bytes.
Anyone got any idea how I figure out what is happening? We've never come across this issue before and the application has been running for a good couple of years.
Also, this isn't something that happens regularly but instead seems to happen one every day or so, and even that is at a random time. The Virtual Memory was initially 4GB but because of the issue above, we increased it to 8GB. Recently it spiked at using about 6.8GB out of 8GB, which it has no reason to do so.
Any help would be really appreciated!
The answer is easy here, obviously and certainly you have two issues here
1- You have a serious bug in your process/code that happens intermittently "you need to debug it to find how/when that happens" or at least run a ProcDump
such that you keep it listening on the server on the process W3WP till an exception happens and then analyze this dump to find where the code get stuck and consume that memory/otherwise just debug the code and see what changes were made in last few months "not days"
2- the application get stopped because you have configured/it is configured by default to get disabled break after a certain number of failure repeats, and that's a normal behavior but the main issue as I said is not the application pool itself, its inside the process
please let me know if you need a further explanation or help on this matter
I'm trying to stress test a server with JMeter. I followed the manual and successfully created the tests (Test are running ok and response is correct).
However even if I keep increasing the number of threads it never fails, but I keep reading that there must be limitations? So what am I doing wrong?
My CPU is running on +/-5% when I'm not running JMeter. Running 3000 threads I see the number of threads increase by 3000 and CPU usage goes to +/-15%. Also JMeter never complains something went wrong.
My JMeter configuration is:
Number of threads: 3000
Ramp-Up Period: 30
LoopCount: Forever (Let it run for over an hour and still nothing goes wrong)
The bottleneck now is my internet connection which simply can't handle this load and maxes out at 2.1Mbps. Is this causing the problem? It is increasing my latency from 10ms per thread to over 5000ms per thread, but threads are still running.
Assuming you have confirmed that you definitely aren't getting back any errors (e.g. using a results table listener, or logging/displaying only errors using a results graph listener) and your internet connection is running at capacity then yes, it does sound like your internet connection is the bottleneck. It doesn't sound like your server is being stressed at all.
If you can easily make use of other machines (e.g. servers in the same location as the server you are testing), you could try using JMeter remote (distributed) testing to sidestep the limitations of your internet connection. See http://jmeter.apache.org/usermanual/remote-test.html for details.
Alternatively, if it's easy (e.g. if you're using VM's in a cloud and can easily spin one up with your software on), you could try using the least-powerful server you can instead and stress testing that to see if you can make it struggle even with your internet connection (just as a sanity check).
If this doesn't help, more details on your server (hardware specifications, web server software and thread pool settings, language) and the site/pages you are testing (mostly static or dynamic? large requests/responses?) would be useful. I've certainly managed to make lower-powered machines (e.g. EC2 m1.small) struggle using JMeter over a 2Mbps connection, but it depends on the site you're testing.
When i use Java VisualVM to monitor my JBoss Application.
It shows
Live Threads as: 155
Daemon Threads as: 135
When i use JMX Web Console of JBoss.
It shows
Current Busy Threads as: 40
Current Thread Count as: 60
Why is there so much discrepancy between what Java Visual VM is reporting and what JMX Web Console shows. (How is Live Threads different from Busy Threads)
A live thread is one that exists and is not Terminated. (See Thread.State)
A busy thread is one that is actually working or, more precisely, Runnable.
JBoss's Web Console tends to report fewer threads because it is very non-invasive. In other words, it does not have to spawn additional threads just to render you a web page. It's already running a web server and it already allocated threads to handle web requests before you went into JMX Console.
Visual VM on the other hand, starts up several threads to support the JMX remoting (usually RMI) which comes with a little extra baggage. You might see extra threads like:
RMI TCP Connection(867)
RMI TCP Connection(868)
RMI TCP Connection(869)
JMX server connection timeout
Having said that, the discrepancy you are reporting is way out of line and makes me think that you're not looking at the same JVM.
The JMX Console is obvious :), so I would guess your Visual VM is connected elsewhere. See if you can correlate similar thread name (using the MBean jboss.system:type=ServerInfo listThreadDump operation), or browse the MBeans in Visual VM and inspect the JBoss MBeans. These mbeans are good ones to look at because they indicate a binding to a socket so they could not have the same values if they were not the same JVM process:
jboss.web:name=HttpRequest1,type=RequestProcessor,worker=http-0.0.0.0-18080
Of course, the other thing would be that if you start VisualVM first, have it running and then go to JMX Console and don't see as many threads, you're definitely in a different VM.
Cheers.
//Nicholas
I have a classic ASP application that has been stable for years and now we're having all kinds of problems with it. After moving the app between machines and wiping the original so we could have a fresh install of windows, we've come to the following "symptom". The app pools do not appear to allow for multiple simultaneous requests. Here's what we are seeing:
The app runs normally for most people, but when someone within one of the app pools accesses a long-running script (usually one with lots of DB access) all of the other users in the pool must wait for that script to complete. Once the script completes, everyone else's requests run. This initially made us suspect the DB connection string or something.
UNTIL... we noticed also that large file uploads into our system also cause the app pool to stop responding. What's interesting about this is that we're using the SAFileup COM+ object to do our uploads, which has a progress display in a pop-up window. When you go to upload the file, the progress display comes up, but then never refreshes to show upload progress. If you wait it out, however, the file will eventually upload and the other pending requests will process as normal.
Our app pools are in the default configuration, using the IWAM account to launch. I checked to ensure that the IWAM account has all the appropriate permissions. It does.
We've tried a variety of DB connection strings, none solved the problem (though I'm thinking it's not the DB connection string). Just in case someone thinks it is, here's our connection string: "Provider=SQLNCLI;Trusted_Connection=yes;Server=(local);Database=demo;". It couldn't be simpler. This string was previously not a problem.
I fussed with the web gardens thing and it does, indeed, make the system respond to multiple requests, but each worker thread in the garden has its own session state which causes our users to get booted when their request gets randomly assigned to a new worker thread. Only having a single worker process in the garden was never an issue before anyway.
I've used SQL Profiler and sp_who2 to see if during the long-running scripts there are any deadlocks or blocks on the SQL Server. There are not.
The issues initially started after we had installed some patches from Microsoft. We wiped a machine clean and installed Win2k3 server, then SP2, and then didn't patch anymore after that. The problem remained, so it doesn't appear to have been a patch.
I'm pretty much at a loss now... does anyone have any experience with similar issues? If so, how were they fixed?
Check that you don't have ASP debugging enabled on the server. This will force the ASP script engine to run on a single thread.
Sounds like an limit on the number of concurrent incoming requests to the IIS or the Windows Server..
Check out http://blogs.msdn.com/b/david.wang/archive/2006/04/12/howto-maximize-the-number-of-concurrent-connections-to-iis6.aspx and http://forums.iis.net/p/1152112/1880908.aspx#1880908 on how to tweak the settings.