Linux CROND resource limits - linux

Question

I have a machine with minified single user OS based on 64bit Fedora 24:
Vendor: Acer Veriton VN4640G
CPU: Intel(R) Core(TM) i5-6400T CPU # 2.20GHz
RAM: 4GB DDR4 2133 MHz
Storage: 32GB 2,5" ADATA SP600
I wrote a simple script /root/test.sh which run 10000 processes on background:
ulimit -a > /tmp/ulimit
i=1
while [ $i -le 10000 ]; do
echo $i
sleep 60 & disown
i=$(( $i + 1 ))
done
When I run this script directly from console, it runs 10000 sleep processes and print numbers as expected.
# bash test.sh
1
2
...
9999
10000
# ps ax | grep -c [s]leep
10000
Ulimit looks well
# cat /tmp/ulimit
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 15339
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 15339
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
BUT
if I run this script via cron (/etc/cron.d/custom), e.g.
0 8 * * * root bash /root/test.sh
I see in journalctl -e -o cat messages like:
(root) CMDOUT (494)
(root) CMDOUT (495)
(root) CMDOUT (496)
(root) CMDOUT (/root/test.sh: fork: retry: Resource temporarily unavailable)
(root) CMDOUT (/root/test.sh: fork: retry: Resource temporarily unavailable)
(root) CMDOUT (/root/test.sh: fork: retry: Resource temporarily unavailable)
(root) CMDOUT (/root/test.sh: fork: retry: Resource temporarily unavailable)
(root) CMDOUT (/root/proc.sh: fork: Resource temporarily unavailable)
So it run only about 500 processes and then cann't fork any other process even if there is still enough resources and user limits are the same as console case.
# free -h
total used free shared buff/cache available
Mem: 3,8G 472M 2,8G 62M 498M 3,0G
Swap: 0B 0B 0B
The count of running sleeps is always the same. Is there any recource limit for tasks run from cron?
P.S.: I did the test even on full Fedora 24 and result is the same...

Answer 1

Well, I found a solution during writing this question.
The main pointer to the problem was that I once saw in journalctl message
kernel: cgroup: fork rejected by pids controller in /system.slice/crond.service
So I checked the cron.service and found a parameter TasksMax.
# systemctl show crond.service
Type=simple
Restart=no
...
TasksMax=512
EnvironmentFile=/etc/sysconfig/crond (ignore_errors=no)
UMask=0022
LimitCPU=18446744073709551615
LimitCPUSoft=18446744073709551615
Solution
Add parameter TasksMax to the service configuration in /usr/lib/systemd/system/crond.service, e.g.:
Note: As Mark Plotnick wrote, better way is copy this service to /etc/systemd/system/ folder and modify this file to avoid rewriting service in /usr/ during upgrade.
# cat /usr/lib/systemd/system/crond.service
[Unit]
Description=Command Scheduler
After=auditd.service nss-user-lookup.target systemd-user-sessions.service time-sync.target ypbind.service
[Service]
EnvironmentFile=/etc/sysconfig/crond
ExecStart=/usr/sbin/crond -n $CRONDARGS
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
TasksMax=100000
[Install]
WantedBy=multi-user.target
Then reload systemd daemons
# systemctl daemon-reload
General solution
If you want avoid this problem with any systemd service you can change default value in /etc/systemd/system.conf, e.g.:
sed -i 's/#DefaultTasksMax=512/DefaultTasksMax=10000/' /etc/systemd/system.conf
And reload systemd daemons to apply the changes
# systemctl daemon-reload
But I don't know the exact consequences of this solution, so I can not recommend it.