I'm using the k8s v1.16 which is icp (ibm container platform).
I want to run some.exe files on the container.
So that I use the wineserver to run window based exe files.
But there is a problem with the memory usage.
Though I allocated 32GB of memory on the pod where the wineserver container will be running, the container does not use memory more than 3GB.
What should I do to make the wine container uses memory more than 3GB?
Related
I am currently working on a Kubernetes cluster, which uses docker.
This cluster allows me to launch jobs. For each job, I specify a memory request and a memory limit.
The memory limit will be used by Kubernetes to fill the --memory option of the docker run command when creating the container. If this container exceeds this limit it will be killed for OOM reason.
Now, If I go inside a container, I am a little bit surprised to see that the available system memory is not the one from the --memory option, but the one from the docker machine. (The Kubernetes Node)
I am surprised because a system with wrong information about available resources will not behave correctly.
Take for example the memory cache used by IO operations. If you write on disk, pages will be cached on the RAM before being written. To do this the system will evaluate how many pages could be cached using the sysctl vm.dirty_ratio (20 % by default) and the memory size of the system. But how this could work if the container system memory size is wrong.
I verified it:
I ran a program with a lot of IO operations (os.write, decompression, ...) on a container limited at 10Gi of RAM, on a 180Gi Node. The container will be killed because it will reach the 10Gi memory limit. This OOM is caused by the wrong evaluation of dirty_ratio * the system memory.
This is terrible.
So, my question is the following:
Is there a way to set the available resources of a docker container system using the docker container limit?
I am running Docker on a Linux server. By default only 2GB of memory and 0GB of Swap space are allocated. How can I change the memory and swap space in Docker?
From official documentation:
By default, a container has no resource constraints and can use as much of a given resource as the host’s kernel scheduler allows. Docker provides ways to control how much memory, CPU, or block IO a container can use, setting runtime configuration flags of the docker run command
You can use the -m or --memory option and set it to a desired value depending on your host's available memory.
i'm running spark using docker on DC/OS. When i submit the spark jobs, using the following memory configurations
Driver 2 Gb
Executor 2 Gb
Number of executors are 3.
The spark submit works fine, after 1 hour the docker container(worker container) crashes due to OOM (exit code 137). but my spark logs shows that 1Gb+ of memory is available.
The strange thing is the same jar which is running in the container , runs normally for almost 20+ hours in the standalone mode.
Is it the normal behaviour of the Spark contianers, or is there Something im doing wrong.Or are there any extra configuraton do I need to use for the docker container.
Thanks
It looks like I have a similar issue. Have you looked at the cache/buffer memory usage on the OS?
Using the command below you can get some info on the type of memory usage on the OS:
free -h
In my case the buffer / cache kept on growing until there was no more memory available in the Container. In my case the VM was a CentOS machine running on AWS and it crashed entirely when this happened.
Is your spark calling REST end point, if yes, try closing connections
I run my java application in a docker container. I use AWS Beanstalk. The docker base image is CentOS. I run a container on an EC2 instance with 4gb of RAM on Amazon Linux AMI for Beanstalk.
How should I configure the container and JVM memory settings.
right now I have:
4GB on Amazon Linux Beanstalk AMI ec2 instance
I dedicated 3GB of 4 for a docker container
{
"AWSEBDockerrunVersion": 2,
"Authentication": {
"Bucket": "elasticbeanstalk-us-east-1-XXXXXX",
"Key": "dockercfg"
},
"containerDefinitions": [
{
"name": "my-service",
"image": "docker-registry:/myapp1.0.2",
"essential": true,
"memory": 3184,
"portMappings": [
{
"hostPort": 80,
"containerPort": 8080
}
]
}
}
JVM settings are
-Xms2560m
-Xmx2560m
-XX:+UseConcMarkSweepGC
-XX:+CMSParallelRemarkEnabled
-XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=70
-XX:+ScavengeBeforeFullGC
-XX:+CMSScavengeBeforeRemark
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=./heapdump.hprof
Can I dedicate whole ec2 instance memory for a docker container, 4gb, and set JVM to 4GB too?
I think JVM could crash when it is not able to allocate whatever the Xmx memory param says. If so what are the most optimal values I could use for the container and JVM?
Right now I left 1gb margin for Amazon linux itself and 512MB for CentOS running in a container. 1.5GB wasted. Can it be done better?
The real answer is very dependent on your Java app and usage.
Start with a JVM heap of 3.5G.
Set the container max to 3.8G to cover javas overhead.
Load test, repeatedly.
Java
-Xmx and -Xms only control Java's heap size so you can't allocate all your available memory to Java's heap and expect java and the rest of the system to run well (or at all).
You will also need to cater for:
Stack: -Xss * number of threads (Xss defaults to 1MB on 64bit)
PermGen/Metaspace: -XX:MaxPermSize defaults to 64MB. Metaspace to 21MB but this can grow.
Your app could also use a lot of shared memory, do JNI things outside of the heap, rely on large mmaped files for performance or exec external binaries or any number of other oddities outside the heap. Do some load testing at your chosen memory levels, then above and below those levels to make sure you're not hitting or getting close to any memory issues that affect performance.
Container
Centos doesn't "run" in the container as such, docker just provides a file system image for your JVM process to reference. Normally you wouldn't "run" any more than the java process in that container.
There's not a huge benefit to limiting a containers max memory when you are on a dedicated host with only one container but it doesn't hurt either, in case something in the native java runs away with the available memory.
The memory overhead here needs to cater for the possible native Java usage mentioned above and the extra system files for the jre that will be loaded from the container image (rather than the host) and cached. You also won't get a nice stack or heapdump when you hit a container memory limit.
OS
A plain Amazon AMI only needs about 50-60MB of memory to run plus some spare for the file cache, so say 256MB to cover the OS with some leeway.
For a linux container, after it is created and some application are running in it, can CPU and memory be dynamically added to the container?