Intermittent Service not available Error in Elasticbeanstalk application - linux

We are using a webserver with ElasticBeanstalk from 2019.,
the platform is
tomcat 8.5 with java8 running on 64 bit Amazon Linux. httpd as proxy
recently (from Jan 30th) we started getting Service Unavailable issues if go to the endpoint from time to time. and if we refresh 2-3 times it will get resolved on its own.
then I download full logs. under elasticbeanstalk-error_log I can see
[Mon Feb 28 10:00:58.338035 2022] [proxy:error] [pid 14882:tid 139757313533696] (13)Permission denied: AH02454: HTTP: attempt to connect to Unix domain socket /var/run/httpd/ (localhost) failed
[Mon Feb 28 10:00:58.338078 2022] [proxy_http:error] [pid 14882:tid 139757313533696] [client <private-ip-here>:12566] AH01114: HTTP: failed to make connection to backend: httpd-UDS, referer: http://<custom-end-point>/1/<name.jsp>?s=sec$$4P!&refresh=300
[Mon Feb 28 10:43:40.663468 2022] [proxy:error] [pid 14882:tid 139757120071424] (13)Permission denied: AH02454: HTTP: attempt to connect to Unix domain socket /var/run/httpd/ (localhost) failed
[Mon Feb 28 10:43:40.663518 2022] [proxy_http:error] [pid 14882:tid 139757120071424] [client <private-ip-here>:21136] AH01114: HTTP: failed to make connection to backend: httpd-UDS
repeated multiple times from Jan30th.
and when I look at access.log
I can see 503 error log exactly at the same time when permission denied error logs in elasticbeanstalk-error_log
And I looked at the running process using ps -aux | grep HTTPd and ps -aux | grep tomcat
both are running from 2019 and have no restarts.
what more I can do to troubleshoot these issuesWe are running a web application written in Java(tomcat8) hosted in AWS ElastcBeanStalk
Some weeks back we started getting 503 error randomly
When we checked the elasticbeanstalk-erorr_logs
[Thu Mar 03 13:22:12.906144 2022] [proxy:error] [pid 14882:tid 139757338711808] (13)Permission denied: AH02454: HTTP: attempt to connect to Unix domain socket /var/run/httpd/ (localhost) failed
[Thu Mar 03 13:22:12.906202 2022] [proxy_http:error] [pid 14882:tid 139757338711808] [client 172.31.17.0:61382] AH01114: HTTP: failed to make connection to backend: httpd-UDS, referer: http://our-domain.com/1/callBackLog.jsp
The error logs are suggesting connection error with backend unix socket
When we checked in /var/run/httpd/ folder, there were no unix sockets(.sock files)
But in apache httpd config
<VirtualHost *:80>
<Proxy *>
Require all granted
ProxyPass / http://localhost:8080/ retry=0
ProxyPassReverse / http://localhost:8080/
ProxyPreserveHost on
ErrorLog /var/log/httpd/elasticbeanstalk-error_log
the proxy backend is ip address not unix socket
As per the config httpd should connect to backend ip address(localhost:8080) but why is it complaining about unix socket
Have anyone faced similar issues?
============= UPDATE
The error logs are suggesting connection error with backend unix socket
When we checked in /var/run/httpd/ folder, there were no unix sockets(.sock files)
But in apache httpd config
<VirtualHost *:80>
<Proxy *>
Require all granted
ProxyPass / http://localhost:8080/ retry=0
ProxyPassReverse / http://localhost:8080/
ProxyPreserveHost on
ErrorLog /var/log/httpd/elasticbeanstalk-error_log
the proxy backend is ip address not unix socket
As per the config httpd should connect to backend ip address(localhost:8080) but why is it complaining about unix socket
Have anyone faced similar issues?

Related

Deployed Flask API in production using WSGI not answering petitions

Having this API (prova.py) running in local:
app = Flask(__name__)
api = Api(app, version='1', title='title1', description='decription1')
ExtractDataParser = reqparse.RequestParser()
ExtractDataParser.add_argument('preprocessing', help='Enable image preprocessing', type=inputs.boolean)
ExtractDataParser.add_argument('onlyTopResults', help='Top results only', type=inputs.boolean)
ExtractDataParser.add_argument('image', help='image (jpg, jpeg, png, tiff, pbm, webp)', location='files', type=FileStorage)
ns = api.namespace('v1',
description='description2')
#ns.route('/Extract')
class DataExtractor(Resource):
#api.doc(parser=ExtractDataParser)
def post(self):
try:
return "Im working", 200
except Exception as ex:
return str(ex), 500
if __name__ == '__main__':
#just in local
app.run(debug=True, port=70)
#for deployment
#app.run()
When I run from terminal curl -X GET "http://127.0.0.1:70/" I get the generated swagger HTML code. But, when I deploy it using WSGI, it always timeouts with the following output:
[Thu Jun 30 13:21:46.971820 2022] [wsgi:error] [pid 70987:tid 139869931951872] Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve.
[Thu Jun 30 13:21:46.971824 2022] [wsgi:error] [pid 70987:tid 139869931951872] from .cv2 import *
[Thu Jun 30 13:22:46.014752 2022] [wsgi:error] [pid 70990:tid 139869940868864] [client 127.0.0.1:44438] mod_wsgi (pid=70990): Request data write error when proxying data to daemon process: The timeout specified has expired.
[Thu Jun 30 13:23:46.067793 2022] [wsgi:error] [pid 70990:tid 139869940868864] [client 127.0.0.1:44438] mod_wsgi (pid=70990): Request data write error when proxying data to daemon process: The timeout specified has expired.
[Thu Jun 30 13:24:46.112848 2022] [wsgi:error] [pid 70990:tid 139869940868864] [client 127.0.0.1:44438] Timeout when reading response headers from daemon process 'myapp': /var/www/Ebisu/Standalone/app.wsgi
This is my app.wsgi
from prova import app as application
And this is my app.conf
Define PROJECT_PATH /var/www/Ebisu/Standalone/
Listen 5000
<virtualhost *:5000>
# # wsgi settings
WSGIDaemonProcess myapp python-path=${PROJECT_PATH}:/var/www/Ebisu/Standalone/invoicenet3.8/lib/python3.8/site-packages
WSGIProcessGroup myapp
WSGIScriptAlias / ${PROJECT_PATH}/app.wsgi
# # map server side static directory to {ip or domain_name}/static
Alias /static ${PROJECT_PATH}/static
#
# # allow all requests to access this project file
<Directory ${PROJECT_PATH}/app.wsgi/>
Require all granted
</Directory>
#
# set log saved path
ErrorLog /var/www/Ebisu/log/error.log
CustomLog /var/www/Ebisu/log/access.log combined
</virtualhost>
What am I missing?

How to fix: Truncated or oversized response headers received from daemon process

I have a flask application that runs on an apache web server, and recently i added a feature using the p4python module. But every time the site calls this package, i get a 500 Internal Server Error. When i check the error log i get the error saying "Truncated or oversized response headers received from daemon process ".
Configuration
Ubuntu 14.04.6
Apache 2.4.7
Python 3.4.3
mod-wsgi 4.6.8
openssl 1.0.1f
Notable python packages are
Flask 1.0.4
psycopg2 2.8.3
p4python 2019.1.1858212
I have tried every solution suggested on the other threads here, and i have tried the debugging suggestions from https://modwsgi.readthedocs.io/en/develop/user-guides/debugging-techniques.html#tracking-request-and-response.
This includes setting WSGIApplicationGroup %{GLOBAL}, increasing the header size and updating mod-wsgi to the newest version using pip.
Unfortunately none of this has worked, and when i call a function using the p4python api the site shows an error.
.conf file
Listen 31415
<VirtualHost *:31415>
ServerName dev-collab
SSLEngine on
SSLCertificateFile /etc/ssl/private/********.cer
SSLCertificateKeyFile /etc/ssl/private/*********.key
WSGIDaemonProcess applicationDash user=www-data group=www-data threads=6 python-home=/var/www/applicationDash/venv
WSGIScriptAlias / /var/www/applicationDash/api.wsgi
ErrorLog "/var/log/apache2/applicationDash_error.log"
CustomLog "/var/log/apache2/applicationDash_access.log" common
LogLevel info
DocumentRoot "/var/www/applicationDash"
<Directory "/var/www/applicationDash">
WSGIProcessGroup applicationDash
WSGIScriptReloading On
WSGIApplicationGroup %{GLOBAL}
</Directory>
Alias /static /var/www/applicationDash/static
<Directory "/var/www/applicationDash/static">
Require all granted
</Directory>
</VirtualHost>
Response header
[wsgi:error] [pid 42349] [remote 10.4.24.191:52037] ('RESPONSE',
[wsgi:error] [pid 42349] [remote 10.4.24.191:52037] '200 OK',
[wsgi:error] [pid 42349] [remote 10.4.24.191:52037] [('Content-Type', 'text/html; charset=utf-8'), ('Content-Length', '5598')])
The actual error
[wsgi:info] [pid 42371] mod_wsgi (pid=42371): Attach interpreter ''.
[wsgi:info] [pid 42371] mod_wsgi (pid=42371): Imported 'mod_wsgi'.
[ssl:info] [pid 41529] [client 10.4.32.52:53304] AH01964: Connection to child 2 established (server dev-collab:443)
[ssl:info] [pid 41529] (70014)End of file found: [client 10.4.32.52:53304] AH01991: SSL input filter read failed.
[ssl:info] [pid 41530] [client 10.4.24.191:58651] AH01964: Connection to child 3 established (server dev-collab:443)
[wsgi:info] [pid 42371] [remote 10.4.24.191:58651] mod_wsgi (pid=42371, process='applicationDash', application=''): Loading Python script file '/var/www/applicationDash/api.wsgi'.
[wsgi:error] [pid 41530] [client 10.4.24.191:58651] Truncated or oversized response headers received from daemon process 'applicationDash': /var/www/applicationDash/api.wsgi, referer: https://dev-collab:31415/branch/3/AccessControl
This is all with LogLevel info.
Does anyone have an idea where my mistake lies?

HTTP: failed to make connection to backend: 0.0.0.0 - socket-js

I am running into an interesting problem in regards to running nodejs on port 8080. I have a new EC2 instance running ubuntu 16.04, I've configured apache2 to run on port 80 and have a reserve proxy setup to switch the port to the nodejs server running inside the /public directory to port 8080. This works great but, my bundle.js package calls the server in order to be updated: http://myamazonelasticipaddress/sockjs-node/info?t=1486698514348 This continually fails and I am left with the following error messages:
Fri Feb 10 02:28:51.358580 2017] [proxy:error] [pid 19100:tid 140639517771520] AH00940: HTTP: disabled connection for (0.0.0.0)
[Fri Feb 10 02:43:57.689148 2017] [proxy:error] [pid 19101:tid 140639568127744] (111)Connection refused: AH00957: HTTP: attempt to connect to 0.0.0.0:8080 (0.0.0.0) failed
[Fri Feb 10 02:43:57.689205 2017] [proxy:error] [pid 19101:tid 140639568127744] AH00959: ap_proxy_connect_backend disabling worker for (0.0.0.0) for 60s
[Fri Feb 10 02:43:57.689211 2017] [proxy_http:error] [pid 19101:tid 140639568127744] [client 192.55.192.52:56715] AH01114: HTTP: failed to make connection to backend: 0.0.0.0
I thought my firewall might be blocking this but I've allowed all connections to this port through. I've double checked my iptables configs and can't find anything. I have a vagrant machine that does this exact same routing and I have no problem.
I have to be missing something simple, any thoughts or ideas?
So my AWS security group configurations were indeed correct. The culprit, was that I needed to add a custom TCP type in the security group to allow port 8080 for the socketjs-node to connect.

Migrate web application from xmapp to lamp

I have developed web application using codeigniter on xampp server.
and want to deploy it on LAMP server. I have migrated database and app successfully but facing some problem.
When request web page using browser its not show anything not even error.
I have enabled selinux,
have root account and testing using root account
virtual host file config in /etc/httpd/conf.modules.d/hotelking.conf
<VirtualHost *:80>
ServerName www.hotelking.com
ServerAlias hotelking.com
DocumentRoot /var/www/hotelking/app/
Customlog "logs/www.hotelking.com.log" combined
ErrorLog "log/www.hotelking.com.error_log"
<Directory /var/www/hotelking/app/>
Require all granted
</Directory>
And error log have following error
[Wed Jan 06 03:20:14.515526 2016] [:error] [pid 6912] [client 127.0.0.1:40964] PHP Fatal error: Class 'MY_controller' not found in /var/www/hotelking/app/application/controllers/master.php on line 5
[Wed Jan 06 03:20:21.773841 2016] [:error] [pid 6908] [client 127.0.0.1:40965] PHP Fatal error: Class 'MY_controller' not found in /var/www/hotelking/app/application/controllers/master.php on line 5
I CHECKED IT HAS MY_controller at give location

Configure Apache web server to perform SSL authentication

I'm trying to perform SSL authentication in apache web server, using XAMPP in Linux. After I configure httpd.conf like this, Apache server is failing to start. Can some one help me to fix this ? What is wrong with my configuration ?
Alias /bitnami/ "/opt/lampp/apache2/htdocs/"
Alias /bitnami "/opt/lampp/apache2/htdocs"
<Directory "/opt/lampp/apache2/htdocs">
Options Indexes FollowSymLinks
AllowOverride All
Order allow,deny
Allow from all
</Directory>
Listen 443
<VirtualHost 127.0.0.1:443>
DocumentRoot "/opt/lampp/htdocs"
ServerName localhost.localdomain
SSLEngine on
SSLCertificateFile "/home/lahiru/Desktop/ucsc/security/ssl/server.cer"
SSLCertificateKeyFile "/home/lahiru/Desktop/ucsc/security/ssl/server.key"
SSLVerifyClient require
SSLVerifyDepth 10
SSLCACertificateFile "/home/lahiru/Desktop/ucsc/security/ssl/ca.cer"
</VirtualHost>
And the Apache error log says,
(98)Address already in use: AH00072: make_sock: could not bind to address [::]:443
[Sun Jun 21 07:42:47.000754 2015] [ssl:warn] [pid 1863] AH01909: localhost.localdomain:443:0 server certificate does NOT include an ID which matches the server name
[Sun Jun 21 07:42:47.001082 2015] [ssl:warn] [pid 1863] AH01906: www.example.com:443:0 server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)
[Sun Jun 21 07:42:47.001101 2015] [ssl:warn] [pid 1863] AH01909: www.example.com:443:0 server certificate does NOT include an ID which matches the server name
[Sun Jun 21 07:42:47.001183 2015] [suexec:notice] [pid 1863] AH01232: suEXEC mechanism enabled (wrapper: /opt/lampp/bin/suexec)
Bitnami developer here,
In XAMPP the SSL configuration is located at /opt/lampp/etc/extras/httpd-ssl.conf file, where there is a default VirtualHost already configured in port 443, and you are trying to bind again the same port.
Please, try to modify this file instead. You can check if there is any other process using port 443 running command sudo netstat -vnpa | grep 443
What do you mean with "Apache server is failing to start"? Using the /opt/lampp/ctlscript.sh script to restart Apache server didn't produce any error message on my side but the error_log told me the same that you mentioned.
Regards,
Gonzalo

Resources