I have a test program that crashes and I am expecting it to create a core file. When it crashes, it outputs (core dumped), however a core file is not created.
This is running on an amazon ec2 suse server.
./crashtest
Segmentation fault (core dumped)
uname -a
Linux ip-172-31-6-104 4.4.21-90-default #1 SMP Fri Dec 9 09:49:02 UTC 2016 (fc1fe0e) x86_64 x86_64 x86_64 GNU/Linux
ulimit -a
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 3675
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 3675
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
cat /proc/sys/kernel/core_pattern
|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %e
ls -l crashtest
-rwxrwxrwx 1 ec2-user users 6728 May 23 10:49 crashtest
Related
The error is coming only while printing reports which has more than
150 pages.
wkhtmltopdf version : 0.12.5 (with patched qt)
OS : Ubuntu 20.04.3 LTS
CPU(s) : 48
Memory(RAM): 94GB
ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 377646
max locked memory (kbytes, -l) 65536
max memory size (kbytes, -m) unlimited
open files (-n) 200000
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 200000
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
odoo configuration file
limit_memory_hard = 78114717696
limit_memory_soft = 65095598080
I have a Rails 6 application with Webpacker on a virtual host using Plesk. Node.js packages have been successfully installed with yarn.
When calling the website, Phusion Passenger fails with:
And the Stdout/stderr output of the failing subprocess just prints the first 65412 characters of my public/packs/js/application-ad2c73bce874600d5502.js file, without any more error details... What does that mean, and how can I get it running?
Passenger Core:
PID
27769
Backtrace
in 'bool Passenger::SpawningKit::HandshakePerform::checkCurrentState()' (Perform.h:238)
in 'void Passenger::SpawningKit::HandshakePerform::waitUntilSpawningFinished(boost::unique_lock<boost::mutex>&)' (Perform.h:213)
in 'Passenger::SpawningKit::Result Passenger::SpawningKit::HandshakePerform::execute()' (Perform.h:1752)
in 'Passenger::SpawningKit::Result Passenger::SpawningKit::DirectSpawner::internalSpawn(const AppPoolOptions&, Passenger::SpawningKit::Config&, Passenger::SpawningKit::HandshakeSession&, const Passenger::Json::Value&, Passenger::SpawningKit::JourneyStep&)' (DirectSpawner.h:211)
in 'virtual Passenger::SpawningKit::Result Passenger::SpawningKit::DirectSpawner::spawn(const AppPoolOptions&)' (DirectSpawner.h:261)
in 'void Passenger::ApplicationPool2::Group::spawnThreadRealMain(const SpawnerPtr&, const Passenger::ApplicationPool2::Options&, unsigned int)' (SpawningAndRestarting.cpp:95)
User and group
uid=0(root) gid=0(root) groups=0(root)
Ulimits
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 39266
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 39266
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
Environment variables
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
NOTIFY_SOCKET=/run/systemd/notify
LANG=C
PASSENGER_USE_FEEDBACK_FD=true
SERVER_SOFTWARE=Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips Apache mod_fcgid/2.3.9 Phusion_Passenger/6.0.8
Subprocess:
PID
3850
Stdout and stderr output
/var/www/vhosts/mydomain.com/httpdocs/myapp/public/packs/js/application-ad2c73bce874600d5502.js:2
[The first 65412 characters of the file content]
User and group
uid=10000(mthcgidu) gid=1003(psacln) groups=1003(psacln)
Ulimits
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 39266
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 39266
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
Environment variables
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
NOTIFY_SOCKET=/run/systemd/notify
LANG=C
PASSENGER_USE_FEEDBACK_FD=true
SERVER_SOFTWARE=Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips Apache mod_fcgid/2.3.9 Phusion_Passenger/6.0.8
IN_PASSENGER=1
PASSENGER_SPAWN_WORK_DIR=/tmp/passenger.spawn.XXXXoUtv1L
PYTHONUNBUFFERED=1
NODE_PATH=/usr/share/passenger/node
RAILS_ENV=development
RACK_ENV=development
WSGI_ENV=development
NODE_ENV=development
PASSENGER_APP_ENV=development
USER=mthcgidu
LOGNAME=mthcgidu
SHELL=/usr/local/psa/bin/chrootsh
HOME=/var/www/vhosts/mydomain.com
PWD=/var/www/vhosts/mydomain.com/httpdocs/myapp
GEOIP_ADDR=[...]
HTTPS=on
PASSENGER_COMPILE_NATIVE_SUPPORT_BINARY=0
PASSENGER_DOWNLOAD_NATIVE_SUPPORT_BINARY=0
PERL5LIB=/usr/share/awstats/lib:/usr/share/awstats/plugins
UNIQUE_ID=YVCbtpjnrt9WLCv4IWd-gAAAAMM
WEBPACKER_NODE_MODULES_BIN_PATH=/httpdocs/myapp/node_modules/.bin
The JS file was minified, thus containing a single line only. I downloaded the file, run some code formating, and uploaded the resulting content with ~25000 lines by replacing the original minified content. Then I could see the line responsible for the error, and also error message and backtrace.
I face a problem when running Microservices on K8s.
Sometimes this error is shown
java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
I was checking some config on the worker (all process running as root): ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 385975
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1048576
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 1048576
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
and cat /proc/sys/kernel/threads-max
771951
Memory is good
free -m
total used free shared buff/cache available
Mem: 96517 35835 45218 10 15462 60126
Swap: 0 0
Limits on each process is unlimited.
cat /proc/440583/limits | grep processes
Max processes unlimited processes
I created a script to counting the number of thread and write to file
while true; do ps -eo nlwp | tail -n +2 | awk '{ num_threads += $1 } END { print num_threads }' >> test.txt; date >> test.txt; sleep 10; done &
this is the log at the time of the error
Mon Jun 21 13:18:48 2021
34323
Mon Jun 21 13:18:52 2021
34325
Mon Jun 21 13:18:58 2021
34324
Mon Jun 21 13:19:02 2021
11945
Mon Jun 21 13:19:10 2021
11979
(the error occurs in 13:18)
All Microservices have good heap usage.
It's seems ulimit or thread-max not working as expected.
Could anyone help me?
I am using the Linux 12 SP2 . It always failed to create the thread when the thread is more than 420 .
Error log:
Exception in thread "main" java.lang.OutOfMemoryError: unable to create new native thread (424 threads running, rlimit: STACK 8192k, CORE 0k, NPROC 30654, NOFILE 16394, AS infinity, DATA infinity, CPU infinity, FSIZE infinity, MEMLOCK 64k , Memory: 4k page, physical 528062120k(345461208k free), swap 2097148k(2097148k free) )
From the log, you can see the server capacity is good enough for more thread creation. Actually there is quite a few thread is running under OS . Do you have any idea about this issue ?
See below details :
ps -aux | wc -l
379
cat /proc/sys/vm/max_map_count
65530
cat /proc/sys/kernel/pid_max
32768
cat /proc/sys/kernel/threads-max
4125328
ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 2062664
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 8192
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 30654
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
I am trying to get guitarix and jack running on the Raspberry Pi 2 (+ Cirrus audio card) with raspbian.
When starting jack via qjackctl, I get the errors
Cannot lock down 82278944 byte memory area (Cannot allocate memory)
Cannot use real-time scheduling (RR/10)(1: Operation not permitted)
It seems changes to /etc/security/limits.conf do not apply but to /etc/security/limits.d/audio.conf do.
I tried setting the memory lock size for the user and group:
#audio - rtprio 90 # maximum realtime priority
#audio - memlock unlimited # maximum locked-in-memory address space (KB)
#audio - nice -10
pi - rtprio 90
pi - memlock unlimited
pi - nice -10
From ssh I get a satisfactory result:
pi#raspberrypi ~ $ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 30
file size (blocks, -f) unlimited
pending signals (-i) 7349
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 90
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 7349
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
But from the desktop terminal I access via vnc I get
pi#raspberrypi ~ $ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 7349
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 7349
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
How can the same user have different settings and how do I get rt and memory allocation running on the desktop?
It seems to have been a problem with the PAM configuration as stated here.
Uncommenting the line in /etc/pam.d/su did the trick.
# session required pam_limits.so