At random moments our Cpanel server completely hangs itself. We see huge spikes in load (150), disk io's and IO wait 100%
Finally I've identified the responsible processes and it seems that Cpanel/WHM executes a lot of these:
xml-api - get_update_availability -json ./get_update_availability
xml-api - get_current_lts_expiration_s -json ./get_current_lts_expiration_status
Killing them one by one unlocks the server and it starts running again.
Does anybody have a idea what this is and what causes the spawn of so many processes?
CENTOS 7.3 x86_64 xen hvm
cPanel & WHM 64.0 (build 18)
also running csf/lfd
Related
I have a gitlab pipeline which executes it's jobs in my local machine, which has ubuntu 18.04 OS, using shell executor once the pipeline is triggered. It used to work fine with my ubuntu 20.04; once I downgraded my OS, my laptop is shutting down or turning into a black screen whenever the jobs are being executed.I have shared the final log messages from gitlab server!
Your best bet would be to check /var/log/syslog to see if or what errors are being thrown by your Runner. GitLab Runner logs to /var/log/syslog by default: GitLab Runner troubleshooting tips
Additionally, I'd check to see if there are any kernel failures at /var/log/kern.log (for whatever reason) considering you're having overall hardware issues when using a shell executor. That's definitely an odd issue.
I have a VPS with firewall and security notices enabled. I keep getting emails like this:
Time: Wed Jun 19 19:01:54 2019 -0500
Account: user
Resource: Process Time
Exceeded: 7248 > 3600 (seconds)
Executable: /opt/cpanel/ea-php72/root/usr/sbin/php-fpm
Command Line: php-fpm: pool domain_com
PID: 16374 (Parent PID:9915)
Killed: No
So for some reason with this example I have a script that has apparently been running for 2+ hours non-stop. I don't have anything that should be doing that.
I'm getting notices like this quite often. How can I use this info to track down what specifically is causing this?
Any information would be greatly appreciated. Thanks!
You can track which the exact process with the process ID mentioned.
lsof -p 16374
The alert which you are getting is from the LDF which is installed as a part of CSF. I think its normal for cPanel with php_fpm to have the process php_fpm run this long.
You can add the php-fpm to csf.pignore file to stop this warning.
You can also refer the below cPanel fourm thread.
https://forums.cpanel.net/threads/lfd-excessive-resource-usage-normal-for-php-fpm.592583/
To get more information on processes, I would use the Htop tool. This is a great article for learning about how to manage processes using htop and ps
Lsof (List open files) will tell you more information about what files the process is using.
You can get htop and lsof with
sudo apt install htop lsof -y
This article indicates that :
That message comes from the third-party CSF/LFD application and indicates a PHP-FPM process was running longer than the maximum time configured for the CSF/LFD detection period. It shows the process was not killed, thus you should not have traffic loss.
So you might want to check the PHP-FPM error log for the account in-question to see if you notice any particular error messages. It's located at:
/home/$username/logs/domain_tld.php.error.log
It looks like your specific issue has not been resolved on that form. So, you might want to try strace. It handles watching system calls made by a given process including all read-write operations and os function calls. You can activate it on the command line before the program you want to track or attach to a running process by hitting s on a process selected in htop.
We would like to monitor our CouchDB installation using the default pid file method with MONIT, however although couchdb is working fine there is no pid file generated under /var/run/couchdb, there is only a couch.uri file.
Permissions on /var/run/couchdb are good (couch:couch) and service couchdb stop and start work fine, although for MONIT to stop/start we would need the /etc/init.d/couchdb start/stop option (which again isn't present).
For info we just installed using apt-get install couchdb on Ubuntu 14.04.
Any advice appreciated.
Best regards
RichBos
I have done this with an older version (1.3) of CouchDB installed from source. Please check if this is working for you:
check process couchdb with pidfile
/usr/local/var/run/couchdb/couchdb.pid
group database
start program = "/etc/init.d/couchdb start -u couchdb"
stop program = "/etc/init.d/couchdb stop -u couchdb"
if failed host 127.0.0.1 port 5984 then restart
if cpu is greater than 40% for 2 cycles then alert
if cpu > 60% for 5 cycles then restart
if 10 restarts within 10 cycles then timeout
If you have installed it via a package manager, you will most likely find the pid in /var/run/couchdb/couchdb.pid
The place of the pid file did not change since 1.3. So chances are good, that it's working for you.
On a running linux system.
how to find out Active StartServers (running as of now), number of active child processes and idle workers using oS commands or any other tools ?
Appreciate your help!!
apache2ctl status
And
apache2ctl fullstatus
Or apachectl depending on your setup.
See http://httpd.apache.org/docs/2.2/programs/apachectl.html
You need to have mod_status enabled.
Since I install pending updates for my Ubuntu server as soon as possible, I have to restart my linux server quite often. I'm running an webapp on that server and would like to warn my users about the pending restart. Right now, I do this manually, adding an announcement before the restart, give them some time to finish their work, restart and remove the announcement.
I hope, shutdown -r +60 writes an file with all the information about the restart, which I can check on every access. Is there such a file? Would prefer a file in a virtual file system like /proc for performance reasons...
I'm running Ubuntu 10.04.2 LTS
If you are using systemd, the following command shows the scheduled shutdown info.
cat /run/systemd/shutdown/scheduled
Example of output:
USEC=1636410600000000
WARN_WALL=1
MODE=reboot
As remarked in a comment by #Björn, USEC is the timestamp in micro seconds.
You can convert it to a human friendly format dropping the last 6 figures and using date like this:
$ date -d #1636410600
Mon Nov 8 23:30:00 CET 2021
The easiest solution I can envisage means writing a script to wrap the shutdown command, and in that script create a file that your web application can check for.
As far as I know, shutdown doesn't write a file to the underlying files system, although it does trigger broadcast messages warning of the shutdown, which I suppose you could write a program to intercept .. but the above solution seems the easiest.
Script example:
shutdown.bsh
touch /somefolder/somefile
shutdown -r $1
then check for 'somefile' in your web app.
You'd need to add a startup link that erased the 'somefile' otherwise it would still be there when the system comes up and the web app would always be telling your users it was about to shut down.
You can simply check for running shutdown process:
if ps -C shutdown > /dev/null; then
echo "Shutdown is pending"
else
echo "Shutdown is not scheduled"
fi
For newer linux distributions versions you might need to do:
busctl get-property org.freedesktop.login1 /org/freedesktop/login1 org.freedesktop.login1.Manager ScheduledShutdown
The method of how shutdown works has changed
Tried on:
- Debian Stretch 9.6
- Ubuntu 18.04.1 LTS
References
Check if shutdown schedule is active and when it is
The shutdown program on a modern systemd-based Linux system
You could write a daemon that does the announcement when it catches the SIGINT / SIGQUIT signal.