I'm having memory issues in my production environment webapp.
I have a Tomcat running in a AWS EC2 t2.medium instance (2 cores cpu 64 bits + 4gb ram).
This is some info from javamelody:
OS: OS Linux, 3.13.0-87-generic , amd64/64 (2 cores)
Java: Java(TM) SE Runtime Environment, 1.8.0_91-b14
JVM: Java HotSpot(TM) 64-Bit Server VM, 25.91-b14, mixed mode
Tomcat "http-nio-8080": Busy threads = 4 / 200 ++++++++++++
Bytes received = 8.051
Bytes sent = 206.411.053
Request count = 3.204
Error count = 70
Sum of processing times (ms) = 540.398
Max processing time (ms) = 12.319
Memory: Non heap memory = 130 Mb (Perm Gen, Code Cache),
Buffered memory = 0 Mb,
Loaded classes = 12,258,
Garbage collection time = 1,394 ms,
Process cpu time = 108,100 ms,
Committed virtual memory = 5,743 Mb,
Free physical memory = 142 Mb,
Total physical memory = 3,952 Mb,
Free swap space = 0 Mb,
Total swap space = 0 Mb
Free disk space: 27.457 Mb
And my application goes into:
There is insufficient memory for the Java Runtime Environment to continue.
Native memory allocation (mmap) failed to map 12288 bytes for committing reserved memory.
I had applied the following config options, but it seems to failing again.
export JAVA_OPTS="-Dfile.encoding=UTF-8 -Xms1024m -Xmx3072m -XX:PermSize=128m -XX:MaxPermSize=256m"
Is this config ok for my linux configuration?
For further information: my database and file system is running in another t2.medium instance (Windows 2 cores cpu + 4gb ram).
Thanks, and sorry for my english.
EDITED:
The problem is still going on. The weirdest thing is that at logs there was no big proccess running, and the time at it passed was at the very morning (so few people were connected to the application).
In the past I had the application in a Windows environment and non of this was going on. I thinked that a Linux instance would be better but I am driving crazy.
The log:
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000f2d80000, 43515904, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 43515904 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /home/ubuntu/hs_err_pid20233.log
Now my config is this at setenv.sh:
export CATALINA_OPTS="-Dfile.encoding=Cp1252 -Xms2048m -Xmx2048m -server"
And I don't know if makes any sense but the hs_err file have this line:
Memory: 4k page, physical 4046856k(102712k free), swap 0k(0k free)
Is this config ok for your linux configuration? Well, we don't know what else is running on the machine and how much memory will be used by other processes. Thus, the answer is "it depends". However, here's what you can do to figure out the correct setting by yourself:
If you want to know right ahead that the server has enough memory, set -Xmx and -Xms to the same value. This way you'd run into OutOfMemory conditions right when you start the server, not at some randome time in the future. Maybe your OS can just allocate 2.8G instead of 3. (same with the permsize parameters in case you're still running on Java7, otherwise remove them)
You might also want to add -XX:+AlwaysPreTouch to the list of your parameters, so that you can be sure the memory has been allocated right from the beginning.
And lastly, you don't want to set JAVA_OPTS (this will be used for any start of a JVM, including the shutdown command), instead use CATALINA_OPTS (which will only be used for starting up tomcat)
Related
I am building data models via an app called Sisense on Linux. Lately the process fails with an out of memory error. Running free -h I see that that the failure occurs when free memory is low, but before it actually reaches zero and even though there is still plenty of available memory.
Here is the exception:
Failed to build custom table: Rule_pre; BE#521691 SQL error: SafeModeException:
Safe-Mode triggered due to memory pressure. Pod physical memory: 5.31 GB available, 2.87 GB
used, 8.19 GB total. Server physical memory: 4.86 GB available, 28.67 GB used,
33.54 GB total. Application total virtual memory: 2.54 GB. The server exceeded 85% capacity
(28.67/33.54). Possible ways to reduce memory pressure: increase server memory, adjust data
modelling (M2M, un-indexed string fields, etc.), reduce number of simultaneous queries
And here is the output of free -h where you can see the declining memory in the center "free" column. Once free memory got below 235 MB I saw the above exception.
The free util man page has these definitions for free and available memory:
free Unused memory (MemFree and SwapFree in /proc/meminfo)
available
Estimation of how much memory is available for starting new applications, without swapping. Unlike the data provided by the cache or free fields, this field takes into account page cache and also that not all reclaimable memory slabs will be reclaimed due to items being in use (MemAvailable in /proc/meminfo, available on kernels 3.14, emulated on kernels 2.6.27+, otherwise the same as free
As I read on the internet there seems to be a casualness about low free memory. That it is not an issue. But the failure coincides with free memory getting to low. If I understand the man page, the available memory is for starting new applications. I am assuming then that available memory is not available to the existing application that fails, and that free memory is indeed what matters. But any confirmation form others or additional explanation would be appreciated. I'd also be curious about opinions on whether this may constitute a memory leak or if I should simply allocate more memory somehow perhaps at the Linux layer.
I think I have enough understanding here. Free memory never goes below 200MB whether a build fails or succeed. It does not appear to be an indicator of the issue. A successful build will also show a drop in free memory to 200MB.
Have a memory problem. Small app (server on few ports, TMS+Indy, Delphi 10.3) is using about 2 Gb of memory (to compare - only 6 Mb under Windows).
So, small test, almost empty console app:
var x: TThread;
begin
x := TThread.Create(true);
after Create (looking internally - after pthread_create) "pmap" in Ubuntu says that few memory blocks was added as: 4 kb + 132 kb + 8192 kb + 65404 kb
4 + 132 kb - not too large, ok
8 mb - looks as default stack size for a thread, ok
but what is 64 Mb????
The pmap entry is not the consumed memory, it is the reserved/mapped memory.
RAM is first reserved/mapped, but actually consumed per page (typically 4KB), the first time a page is accessed by the program.
IIRC under Linux, the Delphi memory manager is just a wrapper around the libc malloc free API calls. It is typical for this MM to reserve some per-thread memory pages for the thread-specific memory arena. I guess 64MB is a plausible value.
So don't be afraid about those numbers. They don't imply that your RAM chips are consumed eagerly. You can have a process reserving GB of memory, but actually only using a few 100MB of hardware RAM - the writeable/private value in pmap. And note that a Delphi app will consume typically lower than any GC program. :)
Update/Proposal after comments:
If you need to reduce memory, don't use threads, but non-blocking inet libraries, or a thread pool and a reverse proxy (e.g. nginx) to maintain the inet connections. You can have your server handle HTTP/1.0 in a thread pool, and your nginx proxy maintain HTTP/1.1 or HTTP/2 long standing connections. If you use Posix sockets between nginx and the service on the loopback, it is very efficient. We use this configuration with our mORMot services on high production Linux systems, with great stability.
Update/More Info:
This 64MB mapping seems indeed the per-thread-arena reserved mapped memory from malloc. Check https://codearcana.com/posts/2016/07/11/arena-leak-in-glibc.html and MALLOC_ARENA_MAX as detailed in https://siddhesh.in/posts/malloc-per-thread-arenas-in-glibc.html
WE have a 12 node cassandra cluster across 2 different datacenter. We are migrating the data from sql DB to cassandra through a net application and there is another .net app thats reads data from the cassandra. Off recently we are seeing one or the other node going down (nodetool status shows DN and the service is stopped on it). Below is the output of the nodetool status. WE have to start the service to again get it working but it stops again.
https://ibb.co/4P1T453
Path to the log: https://pastebin.com/FeN6uDGv
So in looking through your pastebin, I see a few things that can be adjusted.
First I'm reasonably sure that this is your primary issue:
Unable to lock JVM memory (ENOMEM). This can result in part of the JVM being swapped out,
especially with mmapped I/O enabled. Increase RLIMIT_MEMLOCK or run Cassandra as root.
From GNU Error Codes:
Macro: int ENOMEM
“Cannot allocate memory.” The system cannot allocate more virtual
memory because its capacity is full.
-Xms12G, -Xmx12G, -Xmn3000M,
How much RAM is on your instance? From what I'm seeing your node is dying from an OOM (Out of Memory error). My guess is that you're designating too much RAM to the heap, and there isn't enough for the OS/page-cache. In fact, I wouldn't designate much more than 50%-60% of RAM to the heap.
For example, I mostly build instances on 16GB of RAM, and I've found that a 10GB max heap is about as high as you'd want to go on that.
-XX:+UseParNewGC, -XX:+UseConcMarkSweepGC
In fact, as you're using CMS GC, I wouldn't go higher than 8GB for max heap size.
Maximum number of memory map areas per process (vm.max_map_count) 65530 is too low,
recommended value: 1048575, you can change it with sysctl.
This means you haven't adjusted your limits.conf or sysctl.conf. Check through the guide (DSE 6.0 - Recommended Production Settings), but generally it's a good idea to add the following to these files:
/etc/limits.conf
* - memlock unlimited
* - nofile 100000
* - nproc 32768
* - as unlimited
/etc/sysctl.conf
vm.max_map_count = 1048575
Note: After adjusting sysctl.conf, you'll want to run a sudo sysctl -p or reboot.
Is swap disabled? : false,
You will want to disable swap. If Cassandra starts swapping contents of RAM to disk, things will get really slow. Run a swapoff -a and then edit /etc/fstab and remove any swap entries.
tl;dr; Summary
Set your initial and max heap sizes to 8GB (heap new size is fine).
Modify your limits.conf an sysctl.conf files appropriately.
Disable swap.
It's also a good idea to get on the latest version of 3.11 (3.11.4).
Hope this helps!
I have some doubts how the JVM garbage collector would work with different values of Xmx and Xms and machine memory size:
How would garbage collector would work in following scenarios:
1. Machine memory size = 7.5GB
Xmx = 1024Mb
Number of processes = 16
Xms = 512Mb
I know 16*512Mb already exceeds the machine memory size. How would the garbage collector would work in this scenario. I think the memory usage would be entire 7.5GB in this case. Will the processes would be able to do anything in this? Or they all will be stuck?
2. Machine memory size = 7.5GB
Xmx = 320MB
Xms is not defined.
Number of Processes = 16
In this, 16*320Mb should be less than 7.5GB. But in my case, memory usage is again reaching 7.5GB. Is it possible? Or I have probably have a memory leak in my application?
So, basically I want to understand when does garbage collector runs? Does it run whenever memory used by the application reached exactly Xmx value? Or they are not related at all?
There's a couple of things to understand here and then consider in your situation.
Each JVM process has its own virtual address space, which is protected from other processes by the operating system. The OS maps physical ranges of addresses (called pages) to the virtual address space of each process. When more physical pages are required than are available, pages that have not been used for a while will be written to disk (called paging) and can then be reused. When the data of these saved pages is required again they are read back to the same or different physical page. By doing this you can easily run 16 or more JVMs all with a heap of 1Gb on a machine with 8Gb of physical memory. The problem is that the more paging to disk that is required the more you are going to degrade the performance of your applications since disk IO is orders of magnitude slower than RAM access. This is also the reason that the heap space of a single JVM should not be bigger than physical memory.
The reason for having -Xms and -Xmx options is so you can specify the initial and maximum size of the heap. As your application runs and requires more heap space the JVM is able to increase the heap size within these bounds. A lot of time these values are set to be the same to eliminate the overhead of having to resize the heap while the application is running. Most operating systems only allocate physical pages when they're required so in your situation making -Xms small won't change the amount of paging that occurs.
The key point here is it's the virtual memory system of the operating system that makes it possible to appear to be using more memory than you physically have in your machine.
Matlab will start to consume more and more resources and eventually this error occurs.
Exception in thread "Explorer NavigationContext request queue" java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.lang.String.concat(Uknown Sourc)
at com.matlab...
at com.matlab... etc
Exception in thread "Explorer NavigationContext request queue"
I'm running matlab R2015a
I'm on ubuntu 14.04 LTS - i5-2430M CPU # 2.4ghz x 4 500gb ssd 64bit
I get this error w/o running a single command. After launching matlab it spits this error out after about 30s of slowly consuming more and more resources. I can't figure this out and matlab hasn't responded to any of my posts.
I was getting the same error always. And solved by increasing the Java Heap Memory that is requested and allocated by MATLAB from the Operating System. Maybe this can help you:
Go to Home -> Preferences -> General -> Java Heap Memory and shift the bar (Memory size that can be used by MATLAB.)
I hope this also works for you!