I am working on application which is receiving high traffic. Each request takes around 100-500ms.
There is no memory leak.
When i enable garbage logging, i can see that when GC happens when i allocate 8 GB memory.
Log clearly clarify how much memory is reclaimed. But GC triggers stop the world event.
When i compare 8 GB allocation with 4 GB allocation, i find that with 4 GB GC occurs frequently but takes less time than 8 GB, which is expected.
But application response time is variable. For some request it could be higher.
I just want to know what is best deployment for application which receive high traffic.
Is running 2 tomcat with 4 GB is better or one tomcat with 8 GB is better.
Is there any other solution so that Tomcat always send response within time limit
I searched a lot but couldn't find a way to control stop the world event so that my response time is not affected for some of the request.
Related
I am testing node-webrtc project on 16 core cpu and 32 GB RAM.
I started process with pm2 and after some time node process stop responding.
Url returns not reachable, video streaming stopped.
What i noticed:
1) Every time it stopped at memory consumption 3.5 GB , CPU 900% but i tried to increase old memory size to 24 GB then it failed randomly after reaching 9 GB Memory and 1100 cpu..
2) In pm2 logs i found
"(node:3397) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 newBroadcast listeners added. Use emitter.setMaxListeners() to increase limit" but process keep running after this warning..
A) not sure this is memory leakage issue?
B) cpu consumption (900% out of 1600%) as i know node is single thread process so is there any chance thread assign to main node process reached to their peak point?
Please any suggestion how i can debug it..
concurrent users that time are around 110-120
Issue was server outbound bandwidth.
Server has maximum uplink speed 128 MB/s (~1 Gbps) and stream consuming maximum allowed bandwidth and after that connection to server goes unreachable...
It fixed by swtcihing our server to 500 MB/s bandwidth..
I have problems with RAM on servers which use varnish and only it (no other apps there). Each machine has 64GB RAM available for caching and has three separated varnish services for different backends. Currently, the sum of RAM allocated to varnish from all services is 24GB RAM on each server. I want to increase this value up to 48GB (75% of the whole memory available) but I have some problems.
When I tried to allocate 8GB more just for one service (32GB from all), the committed memory got a peak to 70GB RAM(?). What's more, the increased service has restarted a few times after getting 100% ram allocated to its limit (error msg: child not responding to CLI, killing it/died signal=6/Panic message: Assert error in vbf_fetch_thread()). In addition, services use a lot of VSZ (virtual memory size), it that okay?
This could be Transient memory which is uncapped by default and we'll use malloc as storage.
In Transinet stevedore Varnish stores object with a TTL < 10s, therefore if you have many of those that's what you see.
The solution is to either increase the TTLs or cap the Transient storage.
I've changed jemalloc parameters lg_dirty_mult and lg_chunk. Now I was able to assign 42 GB ram to varnish and committed memory is around 60 GB now. The main varnish task is to cash images and TTL is set to 365d.
I'm using gremlin-server (v3.02), with titan-hbase. I'm using the default configuration settings.The server is 8GB memory and 4-cores.
After few hours of work, the server stops responding to queries requests..
It must be said that the requests intensity on the server is NOT high, pretty much low-medium (few requests per hour, maybe less than that).
When cheking gremlin's last server log messages, I see it's about Hbase session timeout, and retries to reconnect the hbase again.
The server CPU and memory are 90-100% at this point.
JDK 1.8.0_45-b14 64bit on Redhat
Using jstat -gc I can all its time is spent in GC, also oldgen is 100%.
I have set "-Xmx 8g" but vitual memory in htop goes up to 12g, with a few tests with xmx I see that virtual memory always gets about "-Xmx + 4g ".
Jmap -histo gives me about 2g of [B (Byte[]) with a gig for CacheRelation and gig for CacheVertex.
After a restarting the gremlin-server, everything is back to normal, and works again.
Any ideas?
IIS hosted WCF service is consuming Large memory like 18 GB and the server has slowed down.
I Analyzed Mini dump file and it shows only 1 GB or active objects. I understand the GC is not clearing the memory and GC must be running in server mode in 64 bit System. Any idea why the whole computer is stalling and app is taking huge memory?
The GC was running on Server Mode it was configured for better performance. I Understand GC running in Server mode will have a performance improvement because the GC's will not be triggered frequently due to high available memory and in server mode it will have high limit on memory usage. Here the problem was when the high limit is reached for the process CLR triggered the GC and it was trying to clear the Huge 18 GB of memory in one shot, so it was using 90% of system resource and rest applications were lagging.
We tried restarting but it was forever going so We had to kill the process. and now with Workstation mode GC smooth and clean. The only difference is response time has some delay due to GC after 1.5 GB allocation.
One more info: .NET 4.5 version has revision regarding this which has resolved this issue in GC.
When I run my Java applications on a Window Azure's Ubuntu 12.04 VM,
with 4 by 1.6GHZ core and 7G RAM, I get the following out of memory error after a few minutes.
java.lang.OutOfMemoryError: GC overhead limit exceeded
I have a swap size of 15G byte, and the max heap size is set to 2G. I am using a Oracle Java 1.6. Increase the max heap size only delays the out of memory error.
It seems the JVM is not doing garbage collection.
However, when I run the above Java application on my local Windows 8 PC (core i7) , with the same JVM parameters, it runs fine. The heap size never exceed 1G.
Is there any extra setting on Windows Azure linux VM for running Java apps ?
On Azure VM, I used the following JVM parameters
-XX:+HeapDumpOnOutOfMemoryError
to get a heap dump. The heap dump shows an actor mailbox and Camel messages are taking up all the 2G.
In my Akka application, I have used Akka Camel Redis to publish processed messages to a Redis channel.
The out of memory error goes away when I stub out the above Camel Actor. It looks as though Akka Camel Redis Actor
is not performant on the VM, which has a slower cpu clock speed than my Xeon CPU.
Shing
The GC throws this exception when too much time is spent in garbage collection without collecting anything. I believe the default settings are 98% of CPU time being spent on GC with only 2% of heap being recovered.
This is to prevent applications from running for an extended period of time while making no progress because the heap is too small.
You can turn this off with the command line option -XX:-UseGCOverheadLimit