I would like to limit memory used by a process started through bash with the ulimit command on Linux. I was wondering what OS mechanism is used to support ulimit. In particular, is it based on cgroups?
The Linux API methods for getting and setting limits are getrlimit(2) and setrlimit(2)
Limits are managed within the process space. A child process will inherit the limits of its parent. Limits are part of the POSIX standard, so all POSIX compliant operating systems support them (Linux, BSD, OSX).
cgroups are Linux specific, and are not even required in a Linux install. I'm not sure if it is possible to manage limits with cgroups, but it would definitely be non-standard do to so.
"ulimit" is basically an anachronism. You shouldn't have any real limits out of the box if you need the resources, and there are better ways to establish quotes if you want to limit resources.
Here's a good overview:
http://www.gnu.org/software/libc/manual/html_node/Limits-on-Resources.html
Several man pages to look at include:
man 2 getrlimit
man 2 setrlimit
man 3 ulimit OBSOLETE!
I use softlimit, part of DJB's daemontools package.
By specifying something like softlimit -m 1048576 nautilus for example, the program (nautilus) will never exceed 1MiB of memory usage (which also causes it to fail immediately in this case).
Related
I have a question regarding STR (Suspend To RAM) in the Linux kernel.
I am working on a small embedded Linux (Kernel 3.4.22) and I want to implement a mechanism that will put the system into sleep (suspend to ram) while it has nothing to do.
This is done in order to save power.
The HW support RAM self-refresh meaning its content will stay persistence.
And I'll take care of all the rest things which should be done (e.g keeping CPU context etc…)
I want to trigger the Kernel PM (power management) subsystem from within the idle loop.
When the system has nothing to do, it should go into sleep.
The HW also supports a way to wake up the system.
Doing some research, I have found out that Linux gives an option for the user space to switch to STR by writing "echo "mem" > /sys/power/state".
This will trigger the PM subsystem and will perform the relevant callbacks.
My questions are:
Is there any other standard alternative to go into STR besides writing to the above proc?
Did anyone tried to put the system into STR from the idle loop code ?
Thanks,
Why would you need another method? Linux treats everything as a file. Is it any surprise that the contents of a psudo-file dictate the state of the system? Check for yourself. pm-utils is a popular tool set for managing the state of the system. All the commands are just calls to /sys files.
This policy is actually platform dependent. You would have to look at the cpuidle driver for your platform to understand what it is doing. For example, on atmel platforms, it is using both RAM self refresh and WFI.
I'm new to Linux and Terminal (or whatever kind of command prompt it uses), and I want to control the amount of RAM a process can use. I already looked for hours to find an easy-t-use guide. I have a few requirements for limiting it:
Multiple instances of the program will be running, but I only want to limit some of the instances.
I do not want the process to crash once it exceeds the limit. I want it to use HDD page swap.
The program will run under WINE, and is a .exe.
So can somebody please help with the command to limit the RAM usage on a process in Linux?
The fact that you’re using Wine makes no difference in this particular context, which leaves requirements 1 and 2. Requirement 2 –
I do not want the process to crash once it exceeds the limit. I want it to use HDD page swap.
– is known as limiting the resident set size or rss of the process, and it’s actually rather nontrivial to do on Linux, as is demonstrated by a question asked in 2010. You’ll need to set up Linux control groups (cgroups). Fortunately, Justin L.’s answer gives a brief rundown on how to do so. Note that
instead of jlebar, you should use your own Unix user name, and
instead of your/program, you should use wine /path/to/Windows/program.exe.
Using cgroups will also satisfy your other requirements – you can start as many instances of the program as you wish, but only those which you start with cgexec -g memory:limited will be limited.
What scheduling algorithms does Linux kernel use?
Where can I get more info about linux's kernel? (OS first course... student level)
The linux kernel has several different available scheduling algorithms both for the process scheduling and for I/O scheduling. Download it from www.kernel.org and call
make menuconfig
You will get a full list of all available options with a built-in help.
One guy that once came up with his O(1) scheduler is Con Kolivas. Definitively have to have a look at what he did. I was once a great break-through.
Note: As Abdullah Shahin noted, this answer is about IO queing scheduler, not for processes.
If you just want to check what scheduler your linux system is using and which are available you can run the following command:
cat /sys/block/sda/queue/scheduler
The one between the [] is the one it's using at the moment. The other ones are available.
To change it:
sudo bash -c 'echo deadline > /sys/block/sda/queue/scheduler'
Be carefull to set it back to default though, unless you know what you are doing and want.
Default (in newer Ubuntu distros at least) is CFQ (Completely Fair Scheduling):
http://en.wikipedia.org/wiki/CFQ
Interview with the creator (Jens Axboe):
http://kerneltrap.org/node/7637
As others have already mentioned, there are several scheduling algorithms available, according to the intended use.
Check this article if you want to learn more about scheduling in Linux.
i believe "completely fair scheduler" is in use with latest kernels. I think you can good amount of information if you just search for it in google.
link : http://en.wikipedia.org/wiki/Completely_Fair_Scheduler
A new addition to Linux Kernel is EDF (Earliest Deadline First) for guaranteed RealTime support
http://lkml.org/lkml/2009/9/22/186
http://www.evidence.eu.com/content/view/313/390/
I think the Linux kernel actually has a few different schedulers you can choose from at compile-time. To find out more about the Linux kernel, you can download the kernel source code (or browse it online) and look in the Documentation directory. For example, the scheduler subdirectory might be helpful. You can also just look at the code itself, obviously.
Modern GNU/Linux distributions use CFS (Completely Fair Scheduler). You may read more on that in the 4th chapter of this book:
Linux Kernel Development 3rd Edition by Robert Love
You will find many interesting and easy to understand explanations there. I enjoyed a lot.
Linux Kernel allows three different scheduling algorithms mainly
shortest job first
Round Robin Scheduling
Priority based preemptive scheduling algorithm.
The third scheduling method which it differs with lower version of Linux versions such as 2.4
Is there a way to tell Linux that it shouldn't swap out a particular processes' memory to disk?
Its a Java app, so ideally I'm hoping for a way to do this from the command line.
I'm aware that you can set the global swappiness to 0, but is this wise?
You can do this via the mlockall(2) system call under Linux; this will work for the whole process, but do read about the argument you need to pass.
Do you really need to pull the whole thing in-core? If it's a java app, you would presumably lock the whole JVM in-core. I don't know of a command-line method for doing this, but you could write a trivial program to call fork, call mlockall, then exec.
You might also look to see if one of the access pattern notifications in madvise(2) meets your needs. Advising the VM subsystem about a better paging strategy might work out better if it's applicable for you.
Note that a long time ago now under SunOS, there was a mechanism similar to madvise called vadvise(2).
If you wish to change the swappiness for a process add it to a cgroup and set the value for that cgroup:
https://unix.stackexchange.com/questions/10214/per-process-swapiness-for-linux#10227
There exist a class of applications in which you never want them to swap. One such class is a database. Databases will use memory as caches and buffers for their disk areas, and it makes absolutely no sense that these are ever put to swap. The particular memory may hold some relevant data that is not needed for a week until one day when a client asks for it. Without the caching/swapping, the database would simply find the relevant record on disk, which would be quite fast; but with swapping, your service might suddenly be taking a long time to respond.
mysqld includes code to use the OS / system call memlock. On Linux, since at least 2.6.9, this system call will work for non-root processes that have the CAP_IPC_LOCK capability[1]. When using memlock(), the process must still work within the bounds of the LimitMEMLOCK limit. [2]. One of the (few) good things about systemd is that you can grant the mysqld process these capabilities, without requiring a special program. If can also set the rlimits as you'd expect with ulimit. Here is an override file for mysqld that does the requisite steps, including a few others that you might need for a process such as a database:
[Service]
# Prevent mysql from swapping
CapabilityBoundingSet=CAP_IPC_LOCK
# Let mysqld lock all memory to core (don't swap)
LimitMEMLOCK=-1
# do not kills this process if low on memory
OOMScoreAdjust=-900
# Use higher io scheduling
IOSchedulingClass=realtime
Type=simple
ExecStart=
ExecStart=/usr/sbin/mysqld --memlock $MYSQLD_OPTS
Note The standard community mysql currently ships with Type=forking and adds --daemonize in the option to the service on the ExecStart line. This is inherently less stable than the above method.
UPDATE I am not 100% happy with this solution. After several days of runtime, I noticed the process still had enormous amounts of swap! Examining /proc/XXXX/smaps, I note the following:
The largest contributor of swap is from a stack segment! 437 MB and fluctuating. This presents obvious performance issues. It also indicates stack-based memory leak.
There are zero Locked pages. This indicates the memlock option in MySQL (or Linux) is broken. In this case, it wouldn't matter much because MySQL can't memlock stack.
You can do that by the mlock family of syscalls. I'm not sure, however, if you can do it for a different process.
As super user you can 'nice' it to the highest priority level -20 and hope that's enough to keep it from being swapped out. It usually is. Positive numbers lower scheduling priority. Normal users cannot nice upwards (negative nos.)
Except in extremely unusual circumstances, asking this question means that You're Doing It Wrong(tm).
Seriously, if Linux wants to swap and you're trying to keep your process in memory then you're putting an unreasonable demand on the OS. If your app is that important then 1) buy more memory, 2) remove other apps/daemons from the machine, or dedicate a machine to your app, and/or 3) invest in a really fast disk subsystem. These steps are reasonable for an important app. If you can't justify them, then you probably can't justify wiring memory and starving other processes either.
Why do you want to do this?
If you are trying to increase performance of this app then you are probably on the wrong track. The OS will swap out a process to increase memory for disk cache - even if there is free RAM, the kernel knows best (actauly the samrt guys that wrote the scheduler know best).
If you have a process that needs responsiveness (it's swapped out while not used and you need it to restart quickly) then nice it to high priority, mlock, or using a real time kernel might help.
I am a little concerned with the amount of resources that I can use in a shared machine. Is there any way to test if the administrator has a limit in the amount of resources that I can use? And if does, to make a more complete question, how can I set up such limit?
For process related limits, you can have a look in /etc/security/limits.conf (read the comments in the file, use google or use man limits.conf for more information). And as jpalecek points out, you may use ulimit -a to see (and possibly modify) all such limits currently in effect.
You can use the command quota to see if a disk quota is in effect.
You can try running
ulimit -a
to see what resource limits are in effect. Also, if you are allowed to change such limits, you can change them by the ulimit command, eg.
ulimit -c unlimited
lifts any limit for a size of a core file a process can make.
At the C level, the relevant functions (actually syscalls(2)...) could be setrlimit(2) and setpriority(2) and sched_setattr(2). You probably would want them to be called from your shell.
See also proc(5) -and try cat /proc/self/limits and sched(7).
You may want to use the renice(1) command.
If you run a long-lasting program (for several hours) not requiring user interaction, you could consider using some batch processing. Some Linux systems have a batch or at command.