JVM memory settings in docker container in AWS beanstalk - linux

Question

I run my java application in a docker container. I use AWS Beanstalk. The docker base image is CentOS. I run a container on an EC2 instance with 4gb of RAM on Amazon Linux AMI for Beanstalk.
How should I configure the container and JVM memory settings.
right now I have:
4GB on Amazon Linux Beanstalk AMI ec2 instance
I dedicated 3GB of 4 for a docker container
{
"AWSEBDockerrunVersion": 2,
"Authentication": {
"Bucket": "elasticbeanstalk-us-east-1-XXXXXX",
"Key": "dockercfg"
},
"containerDefinitions": [
{
"name": "my-service",
"image": "docker-registry:/myapp1.0.2",
"essential": true,
"memory": 3184,
"portMappings": [
{
"hostPort": 80,
"containerPort": 8080
}
]
}
}
JVM settings are
-Xms2560m
-Xmx2560m
-XX:+UseConcMarkSweepGC
-XX:+CMSParallelRemarkEnabled
-XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=70
-XX:+ScavengeBeforeFullGC
-XX:+CMSScavengeBeforeRemark
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=./heapdump.hprof
Can I dedicate whole ec2 instance memory for a docker container, 4gb, and set JVM to 4GB too?
I think JVM could crash when it is not able to allocate whatever the Xmx memory param says. If so what are the most optimal values I could use for the container and JVM?
Right now I left 1gb margin for Amazon linux itself and 512MB for CentOS running in a container. 1.5GB wasted. Can it be done better?

Answer 1

The real answer is very dependent on your Java app and usage.
Start with a JVM heap of 3.5G.
Set the container max to 3.8G to cover javas overhead.
Load test, repeatedly.
Java
-Xmx and -Xms only control Java's heap size so you can't allocate all your available memory to Java's heap and expect java and the rest of the system to run well (or at all).
You will also need to cater for:
Stack: -Xss * number of threads (Xss defaults to 1MB on 64bit)
PermGen/Metaspace: -XX:MaxPermSize defaults to 64MB. Metaspace to 21MB but this can grow.
Your app could also use a lot of shared memory, do JNI things outside of the heap, rely on large mmaped files for performance or exec external binaries or any number of other oddities outside the heap. Do some load testing at your chosen memory levels, then above and below those levels to make sure you're not hitting or getting close to any memory issues that affect performance.
Container
Centos doesn't "run" in the container as such, docker just provides a file system image for your JVM process to reference. Normally you wouldn't "run" any more than the java process in that container.
There's not a huge benefit to limiting a containers max memory when you are on a dedicated host with only one container but it doesn't hurt either, in case something in the native java runs away with the available memory.
The memory overhead here needs to cater for the possible native Java usage mentioned above and the extra system files for the jre that will be loaded from the container image (rather than the host) and cached. You also won't get a nice stack or heapdump when you hit a container memory limit.
OS
A plain Amazon AMI only needs about 50-60MB of memory to run plus some spare for the file cache, so say 256MB to cover the OS with some leeway.