I'm working on getting a Node server up with upstart and monit instead of using a cron job to run a script to check on things. I've built an admin dashboard for the server that uses the Node os module for things like os.loadavg() and os.totalmem(), etc...
The problem is, when monit is running, os.loadavg() always returns [0, 0, 0]. Has anyone else encountered this problem? Does monit create a lock or something that does not allow Node to read that property?
Thanks in advance for any help!
Monit Script
check process flinch
with pidfile "/var/run/flinch.pid"
start program = "/sbin/start flinch"
stop program = "/sbin/stop flinch"
if loadavg (1min) > 4 then alert
if loadavg (5min) > 2 then alert
if memory usage > 0% then alert
To give this question some closure, I removed monit from the system check and wrote a custom bash script that checks the process and it runs on during a cron job every minute. Monit seems to put a lock on the system stats when in use.
Related
I have a VPS with firewall and security notices enabled. I keep getting emails like this:
Time: Wed Jun 19 19:01:54 2019 -0500
Account: user
Resource: Process Time
Exceeded: 7248 > 3600 (seconds)
Executable: /opt/cpanel/ea-php72/root/usr/sbin/php-fpm
Command Line: php-fpm: pool domain_com
PID: 16374 (Parent PID:9915)
Killed: No
So for some reason with this example I have a script that has apparently been running for 2+ hours non-stop. I don't have anything that should be doing that.
I'm getting notices like this quite often. How can I use this info to track down what specifically is causing this?
Any information would be greatly appreciated. Thanks!
You can track which the exact process with the process ID mentioned.
lsof -p 16374
The alert which you are getting is from the LDF which is installed as a part of CSF. I think its normal for cPanel with php_fpm to have the process php_fpm run this long.
You can add the php-fpm to csf.pignore file to stop this warning.
You can also refer the below cPanel fourm thread.
https://forums.cpanel.net/threads/lfd-excessive-resource-usage-normal-for-php-fpm.592583/
To get more information on processes, I would use the Htop tool. This is a great article for learning about how to manage processes using htop and ps
Lsof (List open files) will tell you more information about what files the process is using.
You can get htop and lsof with
sudo apt install htop lsof -y
This article indicates that :
That message comes from the third-party CSF/LFD application and indicates a PHP-FPM process was running longer than the maximum time configured for the CSF/LFD detection period. It shows the process was not killed, thus you should not have traffic loss.
So you might want to check the PHP-FPM error log for the account in-question to see if you notice any particular error messages. It's located at:
/home/$username/logs/domain_tld.php.error.log
It looks like your specific issue has not been resolved on that form. So, you might want to try strace. It handles watching system calls made by a given process including all read-write operations and os function calls. You can activate it on the command line before the program you want to track or attach to a running process by hitting s on a process selected in htop.
I have node server that I want to restart whenever it stopped. For this case I setup system cron on ubuntu server to execute a simple bash script that will track node server every minute and log the server status. Now this cron trigger this bash script and logs relevant status every minute but node server doesn't execute(Using simple linux command I can check if node server running or not). When manually running that bash script node server starts but something happening when cron executes that script. I am trying to fix this meanwhile any help will be appreciated.
Thanks
instead of doing this with cron i think you have to use spervisor in order to keep the process running check this supervisor website
So I am relatively new to Centos, version 6.2. I have a service that needs to be mnonitored as a cron job, and if it stops needs to be restarted. I have a few ideas on how to monitor it, but when it comes to getting it restarted thats when I get stuck. I also know the PiD of the service I want to monitor.
You can use supervise for this: http://cr.yp.to/daemontools/supervise.html
Put it in your crontab to launch on system start:
#reboot supervise foo
Running my Java service using jsvc on Linux (Ubuntu) 10.04.4 LTS and when I stop service and requests and then hung, checked log to find jsvc exec error below.
14/03/2014 12:49:48 19831 jsvc.exec error: Still running according to PID file /home/user/tmp/example.pid, PID is 19728
14/03/2014 12:49:48 19830 jsvc.exec error: Service exit with a return value of 122
Any idea ?
Thanks,
I'm having a similar problem that happens at a log rotation. It appears that the system is shutting down, rotating the logs, then trying to start the system. I believe error 122 is telling you that it hasn't completed shutting down yet and can't restart. I believe the -wait parameter is needed in the start script.
http://commons.apache.org/proper/commons-daemon/jsvc.html
Also see http://freddyandersen.wordpress.com/2009/09/02/running-tomcat-as-a-service-on-linux/ for an example.
If this is happening due to logrotate, use the copytruncate option instead of restarting the service.
http://www.vineetmanohar.com/2010/03/howto-rotate-tomcat-catalina-out/
Try to run below command. Here 19728 is the process id.
pkill -9 19728
Now start your process. It will work.
This will solve your problem.
Thank you.
We are finishing development of a project, the client is already using it but occasionally some errors occur - crashing the server.
I know I could register a service as 'upstart' script on linux, in order to have my node service restart when it crashes.
But our server is running other stuff, so we can't restart it.
Well, actually, while writing, I realize I have two questions then:
Will 'upstart' work without having to reboot? Something is just whispering yes to me :)
If not, what other option would I have to 'respawn' my node server when it crashes?
Yes, upstart will restart your process without a reboot.
Also, you should look into forever.
PM2 is a Production process manager for Node.js app.
If your focus for automatic restart is an always running application, I suggest to use a process manager. Process manager, in general, handles the node process(es if cluster enabled), and is responsible for the process/es execution. PM leans on the operative system: your node app and the OS are not so strinctly chained because the pm is in the middle.Final trick: put the process manager on upstart. Here is a complete performance improvement path to follow.
Using a shared server and not having root privileges, I can't download or install any of the previously mentioned libraries. What I am able to do is use a simple infinite bash loop to solve my issue. First, I created the file ./startup.sh in the base directory ($ vim startup.sh):
#!/bin/bash
while:
do
node ./dist/sophisticatedPrimate/server/main.js
done
Then I run it with:
$ bash startup.sh
and it works fine. There is a downside to this, which is that is doesn't have a graceful way to end the loop (at least not once I exit the server). What I ended up doing is simply finding the process with:
$ ps aux | grep startup.sh
Then killing it with
$ kill <process id>
example
$ kill 555555