I have a C program on Ubuntu. This program needs to open a lot of files. So I have to run it as the following:
ulimit -n 10000; ./xyz
Wonder if there is a way to do something within the program xyz itself to increase the limit. The program runs as root user, so it has the necessary privilege. In the C program (source code), I called
system("ulimit -n 10000");
But it doesn't work, which is not surprise.
See getrlimit(2)
Consider using the getrlimit(2) and setrlimit(2) routines. Note that only a process running with superuser privilege can increase a limit; any user can reduce their resource limits.
why not write a wrapper?
xyz_wrapper.sh
ulimit -n 10000; ./xyz
then simply run the wrapper
./xyz_wrapper.sh
Related
I have written a program that recursively reads files in one directory, modifies them and writes them into another directory. Everytime I run that program, it croakes after a couple of hundred iterations. I just run it again, and it seems to complete the task.
Either nodejs or Mac OS X or, most likely, nodejs-on-Mac-OS-X, seems to have some kind of limit on the number of files that can be opened at one time. Searching around, I see that a solution is to use something like ulimit -n 10480 and all will be well. Is that the right way? Instinctively, I'd rather not tinker with my system settings and rather modify my program to work within the limits.
An observation: Earlier I used to use Perl to do the task I've described above, and I never had a problem. I am assuming it was because I was opening, transforming, then closing the file, and then moving along. In nodejs, using async mode, I have no way of closing a file before going on to the next file. If I do the task in sync mode, it works fine.
You can use the async library with the limit commands to limit the number of files processed to a certain number. For example :
async.eachLimit(files, 1000, function (file, next) {
processFile(file, next);
}, done);
If you wish to process a single file before going to the next one just use eachSeries.
async.eachSeries(files, function (file, next) {
processFile(file, next);
}, done);
Yes, macOS (and possibly every UNIX variant) has a limit on the number of open files, and yes, Perl didn't have that problem for the reason that you mention.
ulimit is not a system setting the way that you seem to think about it. ulimit applies to the current process and is copied to its children processes when you start them, meaning that if you raise the limit in a process, you're not impacting the other processes, within the boundaries that if you're changing the limits on some globally-constrained resource like physical memory usage, you might be starving other programs. In other words, if you run ulimit -n 10480 in a shell, effects only last until you exit that shell.
On macOS, the actual upper ceiling of system-wide open files is given by the command sysctl kern.maxfiles. Regardless of ulimit settings, opening files will fail if you try to open more than that on your entire system at once. On my system, it's 12288. This is the "system setting" that tinkering with can have more lasting effects: raising it increases the static amount of memory that the kernel needs (by amounts unknown to me), and lowering it can starve processes from file descriptors.
If your script is relatively short-lived, raising the file descriptor limit using ulimit is probably not a problem.
I don't know about node.js though, and maybe (almost certainly) it has facilities to start just a number of async tasks at a time, so you could also do that.
I have a LINUX server running a process with a large memory footprint (some sort of a database engine). The memory allocated by this process is so large that part of it needs to be swapped (paged) out.
What I would like to do is to lock the memory pages of all the other processes (or a subset of the running processes) in memory, so that only the pages of the database process get swapped out. For example I would like to make sure that i can continue to connect remotely and monitor the machine without having the processes impacted by swapping. I.e. I want sshd, X, top, vmstat, etc to have all pages memory resident.
On linux there are the mlock(), mlockall() system calls that seem to offer the right knob to do the pinning. Unfortunately, it seems to me that I need to make an explicit call inside every process and cannot invoke mlock() from a different process or from the parent (mlock() is not inherited after fork() or evecve()).
Any help is greatly appreciated. Virtual pizza & beer offered :-).
It has been a while since I've done this so I may have missed a few steps.
Make a GDB command file that contains something like this:
call mlockall(3)
detach
Then on the command line, find the PID of the process you want to mlock. Type:
gdb --pid [PID] --batch -x [command file]
If you get fancy with pgrep that could be:
gdb --pid $(pgrep sshd) --batch -x [command file]
Actually locking the pages of most of the stuff on your system seems a bit crude/drastic, not to mention being such an abuse of the mechanism it seems bound to cause some other unanticipated problems.
Ideally, what you probably actually want is to control the "swappiness" of groups of processes so the database is first in line to be swapped while essential system admin tools are the last, and there is a way of doing this.
While searching for mlockall information I ran across this tool. You may be able to find it for your distribution. I only found the man page.
http://linux.die.net/man/8/memlockd
Nowadays, the easy and right way to tackle the problem is cgroup.
Just restrict memory usage of database process:
1. create a memory cgroup
sudo cgcreate -g memory:$test_db -t $User:$User -a $User:$User
2. limit the group's RAM usage to 1G.
echo 1000M > /sys/fs/cgroup/memory/$test_db/memory.limit_in_bytes
or
echo 1000M > /sys/fs/cgroup/memory/$test_db/memory.soft_limit_in_bytes
3. run the database program in the $test_db cgroup
cgexec -g memory:$test_db $db_program_name
I'd like to limit the memory usage for my Perl script, running on a Linux system. I've been trying to use BSD::Resource's setrlimit, but have been having problems. I'd appreciate any pointers. Thank you.
When you are developing code, it's easy to have your Perl program run away and consume all of memory. The machine will grind to a halt, until the program exhausts memory and dies. You can prevent this problem:
Use this code:
use BSD::Resource;
setrlimit(get_rlimits()->{RLIMIT_VMEM}, 1_000_000_000, -1) or die;
1;
I put this code in limit.pm (hence the "1;"). I can then say
use limit;
at the top of any program that I want to limit.
Scott Corely suggests setting ulimit before running the perl script.
I am a little concerned with the amount of resources that I can use in a shared machine. Is there any way to test if the administrator has a limit in the amount of resources that I can use? And if does, to make a more complete question, how can I set up such limit?
For process related limits, you can have a look in /etc/security/limits.conf (read the comments in the file, use google or use man limits.conf for more information). And as jpalecek points out, you may use ulimit -a to see (and possibly modify) all such limits currently in effect.
You can use the command quota to see if a disk quota is in effect.
You can try running
ulimit -a
to see what resource limits are in effect. Also, if you are allowed to change such limits, you can change them by the ulimit command, eg.
ulimit -c unlimited
lifts any limit for a size of a core file a process can make.
At the C level, the relevant functions (actually syscalls(2)...) could be setrlimit(2) and setpriority(2) and sched_setattr(2). You probably would want them to be called from your shell.
See also proc(5) -and try cat /proc/self/limits and sched(7).
You may want to use the renice(1) command.
If you run a long-lasting program (for several hours) not requiring user interaction, you could consider using some batch processing. Some Linux systems have a batch or at command.
I have a RHEL box that I need to put under a moderate and variable amount of CPU load (50%-75%).
What is the best way to go about this? Is there a program that can do this that I am not aware of? I am happy to write some C code to make this happen, I just don't know what system calls will help.
This is exactly what you need (internet archive link):
https://web.archive.org/web/20120512025754/http://weather.ou.edu/~apw/projects/stress/stress-1.0.4.tar.gz
From the homepage:
"stress is a simple workload generator for POSIX systems. It imposes a configurable amount of CPU, memory, I/O, and disk stress on the system. It is written in C, and is free software licensed under the GPL."
Find a simple prime number search program that has source code. Modify the source code to add a nanosleep call to the main loop with whichever delay gives you the desired CPU load.
One common way to get some load on a system is to compile a large software package over and over again. Something like the Linux kernel.
Get a copy of the source code, extract the tar.bz2, go into the top level source directory, copy your kernel config from /boot to .config or zcat /proc/config.gz > .config, the do make oldconfig, then while true; do make clean && make bzImage; done
If you have an SMP system, then make -j bzImage is fun, it will spawn make tasks in parallel.
One problem with this is adjusting the CPU load. It will be a maximum CPU load except for when waiting on disk I/O.
You could possibly do this using a Bash script. Use " ps -o pcpu | grep -v CPU" to get the CPU Usage of all the processes. Add all those values together to get the current usage. Then have a busy while loop that basically keeps on checking those values, figuring out the current CPU usage, and waiting a calculated amount of time to keep the processor at a certain threshhold. More detail is need, but hopefully this will give you a good starting point.
Take a look at this CPU Monitor script I found and try to get some other ideas on how you can accomplish this.
It really depends what you're trying to test. If you're just testing CPU load, simple scripts to eat empty CPU cycles will work fine. I personally had to test the performance of a RAID array recently and I relied on Bonnie++ and IOZone. IOZone will put a decent load on the box, particularly if you set the file size higher than the RAM.
You may also be interested in this Article.
Lookbusy enables set value of CPU load.
Project site
lookbusy -c util[-high_util], --cpu-util util[-high_util]
i.e. 60% load
lookbusy -c 60
Use the "nice" command.
a) Highest priority:
$ nice -n -20 my_command
or
b) Lowest priority:
$ nice -n 20 my_command
A Simple script to load & hammer the CPU using awk. The script does mathematical calculations and thus CPU load peaks up on higher values passwd to loadserver.sh .
checkout the script # http://unixfoo.blogspot.com/2008/11/linux-cpu-hammer-script.html
You can probably use some load-generating tool to accomplish this, or run a script to take all the CPU cycles and then use nice and renice on the process to vary the percentage of cycles that the process gets.
Here is a sample bash script that will occupy all the free CPU cycles:
#!/bin/bash
while true ; do
true
done
Not sure what your goal is here. I believe glxgears will use 100% CPU.
So find any process that you know will max out the CPU to 100%.
If you have four CPU cores(0 1 2 3), you could use "taskset" to bind this process to say CPUs 0 and 1. That should load your box 50%. To load it 75% bind the process to 0 1 2 CPUs.
Disclaimer: Haven't tested this. Please let us know your results. Even if this works, I'm not sure what you will achieve out of this?