PHP-FPM processes gets stuck out of sudden - linux

While working on upgrading to PHP 5.5 I've faced a problem with php-fpm.
I've created a new web node with the new PHP version and I'm signing it in on the load balancer to test it in prod. Everything works out fine until a lot of php-fpm processes gets stuck out of sudden (after ~1 hour). Hereafter I receives timeouts.
I've changed distribution from Debian to Ubuntu and upgraded to PHP 5.5. Besides that the new server is the same as before (when everything worked/works).
I've tried signing in two new web nodes and they will get stuck at the same time.
I think it can be related to sudden slow responses from database (since that's the only thing they share).
However, I would like php-fpm not to crash, so it will work when database is OK again.
To fix it now I can restart php-fpm service but that's not a solid plan.
ps aux | grep php shows me:
root 13919 0.0 0.4 423472 19736 ? Ss Dec08 0:10 php-fpm: master process (/etc/php5/fpm/php-fpm.conf)
www-data 14087 0.0 3.1 474972 127284 ? S Dec08 0:54 php-fpm: pool www
www-data 14088 0.0 2.3 446944 94956 ? S Dec08 0:50 php-fpm: pool www
www-data 14089 0.1 2.2 443816 92644 ? S Dec08 1:03 php-fpm: pool www
www-data 14090 0.0 3.0 474208 124716 ? S Dec08 0:44 php-fpm: pool www
www-data 14092 0.0 3.1 478284 129060 ? S Dec08 0:41 php-fpm: pool www
www-data 14099 0.0 2.5 459700 105224 ? S Dec08 0:14 php-fpm: pool www
www-data 14111 0.0 2.0 439876 84952 ? S Dec08 0:06 php-fpm: pool www
www-data 14114 0.0 0.4 427220 18472 ? S Dec08 0:00 php-fpm: pool www
www-data 14115 0.0 0.4 427220 18380 ? S Dec08 0:00 php-fpm: pool www
www-data 14116 0.0 0.4 427184 18936 ? S Dec08 0:00 php-fpm: pool www
www-data 14117 0.0 0.4 427204 18904 ? S Dec08 0:00 php-fpm: pool www
www-data 14118 0.0 0.4 427220 18372 ? S Dec08 0:00 php-fpm: pool www
www-data 14119 0.0 0.4 427184 18832 ? S Dec08 0:00 php-fpm: pool www
www-data 14120 0.0 0.4 427188 18960 ? S Dec08 0:00 php-fpm: pool www
www-data 14121 0.0 0.4 427220 18384 ? S Dec08 0:00 php-fpm: pool www
www-data 14122 0.0 0.4 427200 18932 ? S Dec08 0:00 php-fpm: pool www
www-data 14123 0.0 0.4 427236 18488 ? S Dec08 0:00 php-fpm: pool www
www-data 14124 0.0 0.4 427220 18392 ? S Dec08 0:00 php-fpm: pool www
www-data 14125 0.0 0.4 427220 18384 ? S Dec08 0:00 php-fpm: pool www
www-data 14126 0.0 0.4 427204 18820 ? S Dec08 0:00 php-fpm: pool www
www-data 14127 0.0 0.4 427220 18484 ? S Dec08 0:00 php-fpm: pool www
www-data 14128 0.0 0.4 427204 18920 ? S Dec08 0:00 php-fpm: pool www
www-data 14129 0.0 0.4 427188 18904 ? S Dec08 0:00 php-fpm: pool www
www-data 14130 0.0 0.4 427220 18372 ? S Dec08 0:00 php-fpm: pool www
www-data 14131 0.0 0.4 427188 18920 ? S Dec08 0:00 php-fpm: pool www
www-data 14132 0.0 0.4 427220 18372 ? S Dec08 0:00 php-fpm: pool www
www-data 14133 0.0 0.4 427204 18936 ? S Dec08 0:00 php-fpm: pool www
www-data 14134 0.0 0.4 427220 18388 ? S Dec08 0:00 php-fpm: pool www
www-data 14135 0.0 0.4 427220 18368 ? S Dec08 0:00 php-fpm: pool www
www-data 14136 0.0 0.4 427184 18936 ? S Dec08 0:00 php-fpm: pool www
As you can see 24 processes has been running for 0 seconds (htop tells me it's around 2ms). They will stay like that forever. All of them appeared at the same time.
My config:
pid = /var/run/php5-fpm.pid
error_log = /var/log/php5-fpm.log
log_level = debug
user = www-data
group = www-data
listen = 127.0.0.1:9000
pm = dynamic
pm.max_children = 30
pm.start_servers = 5
pm.min_spare_servers = 3
pm.max_spare_servers = 7
pm.max_requests = 500
Also, I never see anything in the php-fpm.log (just NOTICE: configuration file /etc/php5/fpm/php-fpm.conf test is successful)
Anybody?
EDIT:
I have tried running both ondemand and dynamic process manager

I can't leave a comment, although you should check your log file related to php:
( /var/www/ )
( (hostname)/logs/error_log )
and
( /var/log/php5-fpm.log )

Related

Can you take a thread dump of a non-java process?

I have done quite a bit of research, and cant find any documentation on this. I may be using the wrong terminology with the term "thread dump", so I apologize for any lack of knowledge on this topic.
I am looking for a way to take a thread dump of an Apache webserver running on an RHEL AWS instance. The Apache webserver is running on the httpd process. From my understanding, tools like jstack and jmap are specific to taking thread dumps of JVM processes only, and httpd is not compatible.
When I use ps -aux | grep apache I receive the following output of processes.
ps -aux | grep apache
ssm-user 1539 0.0 0.0 112812 960 pts/1 S+ 17:26 0:00 grep apache
apache 1554 0.0 0.2 272752 15488 ? S 16:12 0:00 /usr/sbin/httpd -DFOREGROUND
apache 1713 0.0 0.1 270376 13452 ? S 16:13 0:00 /usr/sbin/httpd -DFOREGROUND
apache 1715 0.0 0.1 270376 13452 ? S 16:13 0:00 /usr/sbin/httpd -DFOREGROUND
apache 1956 0.0 0.1 270556 14080 ? S 13:16 0:00 /usr/sbin/httpd -DFOREGROUND
apache 5700 0.0 0.2 272752 15704 ? S May22 0:00 /usr/sbin/httpd -DFOREGROUND
apache 5701 0.0 0.2 272996 16064 ? S May22 0:00 /usr/sbin/httpd -DFOREGROUND
apache 5703 0.0 0.1 270676 14368 ? S May22 0:00 /usr/sbin/httpd -DFOREGROUND
apache 5704 0.0 0.2 272720 15636 ? S May22 0:00 /usr/sbin/httpd -DFOREGROUND
apache 11441 0.0 0.2 272776 15744 ? S 04:55 0:00 /usr/sbin/httpd -DFOREGROUND
apache 11442 0.0 0.2 272800 15832 ? S 04:55 0:00 /usr/sbin/httpd -DFOREGROUND
I am also unsure why there are so many of the same process, but I wanted to figure out if its possible to take a thread dump of one of the processes before I dive into solving the duplicate processes.
Are there any tools or commands I can use to accomplish a thread dump of an httpd process?

Centos: there are many 'httpd start' process

I found that my server is very slow,I used 'ps au' to show all the processes. There are a lot of /usr/local/apache/bin/httpd -k start running.
How could I fix this problem?
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 195536 10040 ? Ss 6月11 40:57 /usr/lib/systemd/systemd --switched-root --system --d
root 2 0.0 0.0 0 0 ? S 6月11 0:00 [kthreadd]
nobody 31295 0.0 0.0 97028 5044 ? S 11:02 0:00 /usr/local/apache/bin/httpd -k start
nobody 31296 0.0 0.0 97028 5044 ? S 11:02 0:00 /usr/local/apache/bin/httpd -k start
nobody 31297 0.0 0.0 97028 5044 ? S 11:02 0:00 /usr/local/apache/bin/httpd -k start
nobody 31299 0.0 0.0 97028 5044 ? S 11:02 0:00 /usr/local/apache/bin/httpd -k start
nobody 31303 0.0 0.0 97028 5044 ? S 11:02 0:00 /usr/local/apache/bin/httpd -k start
nobody 31310 0.0 0.0 97028 5044 ? S 11:02 0:00 /usr/local/apache/bin/httpd -k start
nobody 31312 0.0 0.0 97028 5044 ? S 11:02 0:00 /usr/local/apache/bin/httpd -k start
nobody 31314 0.0 0.0 97028 5044 ? S 11:02 0:00 /usr/local/apache/bin/httpd -k start
nobody 31315 0.0 0.0 97028 5044 ? S 11:02 0:00 /usr/local/apache/bin/httpd -k start
nobody 31316 0.0 0.0 97028 5044 ? S 11:02 0:00 /usr/local/apache/bin/httpd -k start
nobody 31320 0.0 0.0 97028 5044 ? S 11:02 0:00 /usr/local/apache/bin/httpd -k start
nobody 31324 0.0 0.0 97028 5044 ? S 11:02 0:00 /usr/local/apache/bin/httpd -k start
nobody 31326 0.0 0.0 97028 5044 ? S 11:02 0:00 /usr/local/apache/bin/httpd -k start
nobody 31328 0.0 0.0 97028 5044 ? S 11:02 0:00 /usr/local/apache/bin/httpd -k start
nobody 31329 0.0 0.0 97028 5044 ? S 11:02 0:00 /usr/local/apache/bin/httpd -k start
nobody 31330 0.0 0.0 97028 5044 ? S 11:02 0:00 /usr/local/apache/bin/httpd -k start
nobody 31331 0.0 0.0 97028 5044 ? S 11:02 0:00 /usr/local/apache/bin/httpd -k start
nobody 31332 0.0 0.0 97028 5044 ? S 11:02 0:00 /usr/local/apache/bin/httpd -k start
nobody 31333 0.0 0.0 97028 5044 ? S 11:02 0:00 /usr/local/apache/bin/httpd -k start
nobody 31335 0.0 0.0 97028 5044 ? S 11:02 0:00 /usr/local/apache/bin/httpd -k start
nobody 31338 0.0 0.0 97028 5044 ? S 11:02 0:00 /usr/local/apache/bin/httpd -k start
nobody 31339 0.0 0.0 97028 5044 ? S 11:02 0:00 /usr/local/apache/bin/httpd -k start
nobody 31341 0.0 0.0 97160 6208 ? S 11:02 0:00 /usr/local/apache/bin/httpd -k start
nobody 31346 0.0 0.0 97028 5044 ? S 11:02 0:00 /usr/local/apache/bin/httpd -k start
nobody 31364 0.0 0.0 97028 5044 ? S 11:02 0:00 /usr/local/apache/bin/httpd -k start
nobody 31370 0.0 0.0 97028 5036 ? S 11:02 0:00 /usr/local/apache/bin/httpd -k start
nobody 31373 0.0 0.0 97028 5040 ? S 11:02 0:00 /usr/local/apache/bin/httpd -k start
nobody 31374 0.0 0.0 97028 5044 ? S 11:02 0:00 /usr/local/apache/bin/httpd -k start
nobody 31416 0.0 0.0 97028 5048 ? S 11:03 0:00 /usr/local/apache/bin/httpd -k start
nobody 31418 0.0 0.0 97028 5044 ? S 11:03 0:00 /usr/local/apache/bin/httpd -k start
nobody 31422 0.0 0.0 97028 5044 ? S 11:03 0:00 /usr/local/apache/bin/httpd -k start
nobody 31423 0.0 0.0 97028 5044 ? S 11:03 0:00 /usr/local/apache/bin/httpd -k start
dovecot 31461 0.0 0.0 27940 2232 ? S 11:04 0:00 dovecot/auth
root 31525 0.0 0.0 82012 11368 ? S 11:04 0:00 /usr/local/cpanel/3rdparty/bin/perl /usr/local/cpanel/
nobody 31578 0.0 0.0 97028 4800 ? S 11:04 0:00 /usr/local/apache/bin/httpd -k start
nobody 31579 0.0 0.0 97028 5048 ? S 11:04 0:00 /usr/local/apache/bin/httpd -k start
You can use process control the worker by using the following directives:
ServerLimit 16
StartServers 2
MaxRequestWorkers 150
MinSpareThreads 25
MaxSpareThreads 75
ThreadsPerChild 25
Set values accordingly.

nginx is installed but site does not load

I installed LEMP with this tutorial I have done all it says but when I enter my server ip address in my browser it takes some time to load the page and it seems like my browser is doing some processes to load the page but at last the page is blank.
I'v checked all the installation again and they were all installed.
I'v installed apache at the past and I know for apache server if you enter server ip address it shows " apache is functioning " but It's my first time installing nginx ; I want to install DirectAdmin and I don't know nginx is installed or not.
If anyone is familiar with nginx and can help me, I'll be apreciate that.
Update: When I use service nginx status it shows nginx is running.
Update: Flowing are the results of my checkings:
Results of netstat -tulpn | grep :80 :
tcp 0 0 0.0.0.0:80 0.0.0.0:* LIST EN 10302/nginx
Results of ps aux | grep nginx :
nginx 8556 0.0 0.0 222416 3552 ? S 09:04 0:00 php-fpm: pool w ww
nginx 8557 0.0 0.0 222416 3552 ? S 09:04 0:00 php-fpm: pool w ww
nginx 8558 0.0 0.0 222416 3552 ? S 09:04 0:00 php-fpm: pool w ww
nginx 8559 0.0 0.0 222416 3552 ? S 09:04 0:00 php-fpm: pool w ww
nginx 8560 0.0 0.0 222416 3552 ? S 09:04 0:00 php-fpm: pool w ww
root 10302 0.0 0.0 96032 1980 ? Ss 09:11 0:00 nginx: master p rocess /usr/sbin/nginx -c /etc/nginx/nginx.conf
nginx 10304 0.0 0.0 96436 2692 ? S 09:11 0:00 nginx: worker p rocess
nginx 10305 0.0 0.0 96436 2692 ? S 09:11 0:00 nginx: worker p rocess
nginx 10306 0.0 0.0 96436 2660 ? S 09:11 0:00 nginx: worker p rocess
nginx 10307 0.0 0.0 96436 2672 ? S 09:11 0:00 nginx: worker p rocess
root 12564 0.0 0.0 103312 856 pts/0 S+ 09:59 0:00 grep nginx
FIIIIINNNNNNAAAAAALLLLLLLY I Found the problem!
The problem was with my firewall.
If anyone else has the same problem it may be the firewall settings problem like mine So Do the flowing to fix this:
vim /etc/sysconfig/iptables
in the opened file find this line :
-A INPUT -m state --state NEW -p tcp --dport 80 -j ACCEPT
Check that --dport 80 is set to the port that you entered in /etc/nginx/conf.d/default.conf if its something else like --dport 22 change it to your default port.
Update:
After you changed the port you should run this code:
service iptables restart
And this is the page shows nginx is running :

Multiple apache and finding its ports

The following command
[cuser#vdev-b7-5461-4f3c-9d98-7fc3ec6 kafka]$ ps aux|grep apache
apache 3967 0.0 0.0 175412 3192 ? S 03:40 0:00 /usr/sbin/httpd -k start
apache 3968 0.0 0.0 175412 3192 ? S 03:40 0:00 /usr/sbin/httpd -k start
apache 3969 0.0 0.0 175412 3192 ? S 03:40 0:00 /usr/sbin/httpd -k start
apache 3970 0.0 0.0 175412 3192 ? S 03:40 0:00 /usr/sbin/httpd -k start
apache 3971 0.0 0.0 175276 2700 ? S 03:40 0:00 /usr/sbin/httpd -k start
apache 3972 0.0 0.0 175276 2700 ? S 03:40 0:00 /usr/sbin/httpd -k start
apache 3974 0.0 0.0 175412 3188 ? S 03:40 0:00 /usr/sbin/httpd -k start
apache 3975 0.0 0.0 175276 2696 ? S 03:40 0:00 /usr/sbin/httpd -k start
clduser 9566 0.0 0.0 103244 844 pts/1 S+ 16:41 0:00 grep apache
There are multiple apache process running in my machine. Is there a way for me to determine the files/directory each of these process uses to run it?
Also, would like to know the ports that they are using?
[user#vdev-37-5461-4f3c-9d98-7af28dfc3ec6 kafka]$ netstat -tulpn|grep 80
(No info could be read for "-p": geteuid()=500 but you should be root.)
tcp 0 0 :::80 :::* LISTEN -
This above command shows port 80 is being used but I want to know what process is using it?
[duser#vpsq-dev-4ac0b737-5461-4f3c-9d98-7af28dfc3ec6 kafka]$ find / -iname apache 2>/dev/null
/opt/logstash/vendor/bundle/jruby/1.9/gems/jruby-kafka-1.4.0-java/lib/org/apache
/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-0.1.18-java/lib/org/apache
/opt/logstash/vendor/bundle/jruby/1.9/gems/jruby-kafka-1.1.2-java/lib/org/apache
/home/duser/logstash-1.5.0.rc2/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-0.1.18-java/lib/org/apache
/home/duser/logstash-1.5.0.rc2/vendor/bundle/jruby/1.9/gems/jruby-kafka-1.1.2-java/lib/org/apache

How to detect system power events such as suspend, hibernate, etc

I want to make a Linux application that send a message to a remote host when a local system (where the application runs) is going to suspend, hibernate or shutdown. I googled how to do this and found acpi_listen but it doesn't work.
Here is the result of ps aux | grep acpi on the system I tested (Ubuntu 10.04 LTS):
root 35 0.0 0.0 0 0 ? S Dec03 0:00 [kacpid]
root 36 0.0 0.0 0 0 ? S Dec03 0:00 [kacpi_notify]
root 37 0.0 0.0 0 0 ? S Dec03 0:00 [kacpi_hotplug]
root 934 0.0 0.0 2048 872 ? Ss Dec03 0:00 acpid -c /etc/acpi/events -s /var/run/acpid.socket
Communicating with acpid directly via a UNIX domain socket wouldn't work since that's the way acpi_listen works, although I didn't try it. Is there any other way?
Listen for them over D-Bus.

Resources