I build a web server with flask appbuilder,i can run this project by command:
python3 run.py or fabmanage run,but it always No response when not interaction after some hours,So i try to run it with Nginx.
Here is my config:
uwsgi.ini:
[uwsgi]
base = /root/flask_spider/gttx_spider/web
all_base = /root/flask_spider/gttx_spider/
app = run
module = %(app)
chdir = %(base)
virtualenv = %(all_base)/venv
socket = %(all_base)/uwsgi_gttx_spider.sock
logto = /var/log/uwsgi/%n.log
master = true
processes = 500
chmod-socket = 666
vacuum = true
callable = app
nginx.conf
server {
listen 82;
server_name gttx_spider;
charset utf-8;
client_max_body_size 75M;
location / {
include uwsgi_params;
uwsgi_pass unix:/root/flask_spider/gttx_spider/uwsgi_gttx_spider.sock;
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
}
}
and modify run.py
from app import app
app.run(host='0.0.0.0')
#app.run(host='0.0.0.0', port=8080,debug=True)
and then:
sudo ln -s /root/flask_spider/gttx_spider/nginx.conf /etc/nginx/conf.d/
sudo /etc/init.d/nginx restart
uwsgi --ini uwsgi_gttx_spider.ini
When i access IP:82 and get this log in nginx:
[error] 11104#11104: *3 upstream timed out (110: Connection timed out) while reading response header from upstream
When i access IP:5000,
And this log in uwsgi:
2018-09-10 19:36:25,747:INFO:werkzeug: * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
2018-09-10 19:36:38,434:INFO:werkzeug:115.192.37.57 - - [10/Sep/2018 19:36:38] "GET / HTTP/1.1" 302 -
2018-09-10 19:36:38,466:INFO:werkzeug:115.192.37.57 - - [10/Sep/2018 19:36:38] "GET /home/ HTTP/1.1" 200 -
Also,i try this:
mv web/run.py web/run_bak.py
vi run.py
from flask import Flask
app = Flask(__name__)
#app.route("/")
def hello():
return "Hello World!"
if __name__ == "__main__":
app.run(host='0.0.0.0', port=8090)
And access IP:82,it will return 'Hello World' and everything is fine
The difference is werkzeug will run the flask appbuilder project on 5000,and how to run in uwsgi.sock,Please Help,thanks!
Related
I using flask socket io to implementing the notification to my multiple clients using my application.
The flask socket is working well on my local development but the problem happen I deployed it to use on the server. It most of the time returned status 400 frequently and down the server sometime. I tried digging into source to root cause but I could not.
Below is a snippet that I had a problem:
Here is an init file for initializing application
from __future__ import absolute_import
import os
import redis
import flask_sqlalchemy as sa
from flask import Flask
from flask_socketio import SocketIO
from flask_cors import CORS
from .core.constant import MKT_BLUEPRINT, admin
# Initialize the core application and configure the app
app = Flask(__name__, instance_relative_config=True)
CORS(app)
# Initialize the core application and configure the app
app.config.from_object('mkt.config')
# redis address: "redis://localhost:6379/0"
redis_add = "redis://localhost:6379/0"
async_mode = None
notify_socketio = SocketIO(app, cors_allowed_origins="*",
message_queue=redis_add,
async_mode=async_mode
)
db = sa.SQLAlchemy(app)
app.register_blueprint(admin, url_prefix='/%s'%MKT_BLUEPRINT)
And then on client side, I use this js script
$(document).ready(function() {
var domain = "{{notify_domain|safe}}"
var socket = io(domain);
socket.on('connect', function() {
var user = "{{user|safe}}"
console.log("im in connect", user)
socket.emit('join_room', {room: user});
});
});
Then on server I config the nginx as below:
server {
listen 443 ssl;
server_name xxx;
include /config/nginx/ssl.conf;
location / {
proxy_read_timeout 300s;
proxy_connect_timeout 300s;
proxy_send_timeout 300s;
proxy_pass http://xxx:8000;
}
location /socket.io {
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Access-Control-Allow-Origin *;
proxy_http_version 1.1;
proxy_buffering off;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
proxy_pass http://xxx:8000/socket.io;
}
}
And here are dependencies that I'm using in my application:
#for notification to client
Flask-SocketIO==5.1.1
#for notification to client
Flask-SocketIO==5.1.1
python-engineio==4.2.1
python-socketio==5.4.0
Without any misconfiguration, I logged in console and below
Till now I still could not know what caused this error 400 (bad request) and solved it. Please kindly help. Thank so much.
My original nginx.conf:
events {}
http {
server {
include credentials.conf;
listen 80;
location / {
proxy_set_header Authorization $credentials;
proxy_pass [website_of_choice];
}
}
}
My credentials.conf:
set $credentials 'Basic [long_encoded_login_details]';
But this wont work when nginx starts.
Using .conf as a file ending will not work in some cases. This should give you an error trying to reload nginx sudo nginx -s reload or nginx -t on configtest. This depends on your nginx.conf. Check if there is any include directive including anything *.conf from a given directory.
use this
credentials.include
set $credentials 'Basic [long_encoded_login_details]';
nginx.conf
events {}
http {
server {
include credentials.include;
listen 80;
location / {
proxy_set_header Authorization $credentials;
proxy_pass [website_of_choice];
}
}
}
I'm facing an issue with my current flask app setup and would really appreciate some input on this. Thank you!
Flow
user --> nginx --> uwsgi --> flask app --> https call to external system (response is processed and relevant data returned to client)
Workflow
Intent My flask view/route invokes another class, within which a https (GET) call is made to an external system to retrieve data. This data is then processed (analyzed) and an appropriate response is sent to the user.
Actual User receives 502 Bad Gateway from webserver upon invoking Flask Based endpoint. This is only happening when placing the nginx and uwsgi server in front of my flask application. Initial tests on the server directly with flask's in-built server appeared to work.
Note: That analytics bit does take up some time so I increased all relevant timeouts (to no avail)
Configurations
Nginx (tried with and without TLS)
worker_processes 4;
error_log /path/to/error.log;
pid /path/to/nginx.pid;
events {
worker_connections 1024;
}
http {
default_type application/json;
access_log /path/to/access.log;
sendfile on;
keepalive_timeout 0; [multiple values tried]
# HTTPS server
server {
listen 5555 ssl;
server_name my_host.domain.com;
ssl_certificate /path/to/server.crt;
ssl_certificate_key /path/to/server.key;
ssl_session_cache shared:SSL:1m;
ssl_session_timeout 5m;
ssl_ciphers HIGH:!aNULL:!MD5;
ssl_prefer_server_ciphers on;
location /my_route {
uwsgi_connect_timeout 60s;
uwsgi_read_timeout 300s;
client_body_timeout 300s;
include uwsgi_params;
uwsgi_pass unix:/path/to/my/app.sock;
}
}
}
uWSGI (threads reduced to 1 as part of troubleshooting attempts)
[uwsgi]
module = wsgi:app
harakiri = 300 [also added as part of troubleshooting steps]
logto = /path/to/logs/uwsgi_%n.log
master = true
processes = 1
threads = 1
socket = app.sock
chmod-socket = 766
vacuum = true
socket-timeout = 60
die-on-term = true
Code Snippets
Main Flask Class (view)
#app.route(my_route, methods=['POST'])
def my_view():
request_json = request.json
app.logger.debug(f"Request Received: {request_json}")
schema = MySchema()
try:
schema.load(request_json)
var1 = request_json["var1"]
var2 = request_json["var2"]
var3 = request_json["var3"]
var4 = request_json["var4"]
# begin
execute = AnotherClass(client_home, config, var1, var2, var3, var4, mime_type)
return jsonify(execute.result)
except ValidationError as exception:
error_message = json.dumps(exception.messages)
abort(Response(error_message, 400, mimetype=mime_type))
Class which executes HTTPS GET on external system
custom_adapter = HTTPAdapter(max_retries=3)
session = requests.Session()
session.proxies = self.proxies
session.mount("https://", custom_adapter)
try:
json_data = json.loads(session.get(process_endpoint, headers=self.headers, timeout=(3, 6)).text)
Errors
Nginx
error] 22680#0: *1 upstream prematurely closed connection while
reading response header from upstream, client: client_ip, server:
server_name, request: "POST /my_route HTTP/1.1", upstream:
"uwsgi://unix:/path/to/my/app.sock:", host: "server_name:5555"
User gets a 502 on their end (Bad Gateway)
uWSGI
2020-04-24 16:57:23,873 - app.module.module_class - DEBUG - Endpoint:
https://external_system.com/endpoint_details 2020-04-24 16:57:23,876 -
urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1):
external_system.com:443 !!! uWSGI process #### got Segmentation Fault
!!!
* backtrace of #### /path/to/anaconda3/bin/uwsgi(uwsgi_backtrace+0x2e) [0x610e8e]
/path/to/anaconda3/bin/uwsgi(uwsgi_segfault+0x21) [0x611221]
/usr/lib64/libc.so.6(+0x363f0) [0x7f6c22b813f0]
/path/to/anaconda3/lib/python3.7/lib-dynload/../../libssl.so.1.0.0(ssl3_ctx_ctrl+0x170)
[0x7f6c191b77b0]
/path/to/anaconda3/lib/python3.7/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so(+0x5a496)
[0x7f6c16de2496]
....
end of backtrace * DAMN ! worker 1 (pid: ####) died :( trying respawn ... Respawned uWSGI worker 1 (new pid: ####)
SOLVED
Steps taken
update cryptography
update requests
update urllib3
add missing TLS ciphers to Py HTTP Adapter (follow this guide)
I've been experiencing some issues when I try to upload a file trough my web api using python and flask, so whenever I try to do so I get the following error message (it happens with any kind of file and regardless the size):
2019/12/18 16:30:04 [error] 111206#111206: *12251 upstream prematurely closed connection while reading response header from upstream,
client: xxx.61.xxx.70, server: *mydomain.com*, request: "POST /upload HTTP/1.1", upstream: "uwsgi://unix:/home/climpia/tracking/tracking.sock:", host: "*api.mydomain.com*", referrer: "https://client.mydomain.com/?id_ruta=14130"
Connection reset by xxx.251.xx.158 port 22
I am not sure where is the issue since I have already modded some options on the uWSGI.ini and the nginex.config files, but I still get the same error message. I have checked the code (I am using python and flask) and it seems fine.
This is the uWSGI.ini:
[uwsgi]
module = wsgi
master = true
processes = 5
socket-timeout = 65
http-keepalive = 256
socket = tracking.sock
chmod-socket = 660
vacuum = true
#location of log files
logto = /tmp/%n.log
die-on-term = true
This is the nginex.config file for this project:
server {
server_name mydomain.com api.mydomain.com;
location / {
include uwsgi_params;
uwsgi_pass unix:/home/user/tracking/tracking.sock;
}
listen 443 ssl; # managed by Certbot
ssl_certificate /etc/letsencrypt/live/mydomain.com-0001/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/mydomain.com-0001/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
sendfile on;
client_max_body_size 512M;
keepalive_timeout 0;
}
server {
if ($host = api.mydomain.com) {
return 301 https://$host$request_uri;
} # managed by Certbot
if ($host = mydomain.com) {
return 301 https://$host$request_uri;
} # managed by Certbot
listen 80;
server_name mydomain.com api.mydomain.com;
return 404; # managed by Certbot
}
This is the python code:
# tracking.py
from flask import Flask, request, session, g, redirect, \
url_for, abort, render_template, flash, jsonify, \
make_response, send_from_directory
from flask_sqlalchemy import SQLAlchemy
from sqlalchemy import text
from flask_cors import CORS
from datetime import datetime
from werkzeug.utils import secure_filename
import os
app = Flask(__name__, static_folder='static')
app.config.from_object(__name__)
cors = CORS(app, resources={r"/*": {"origins": "https://client.mydomain.com"}})
#cors = CORS(app, resources={r"/*": {"origins": "*"}})
app.config.from_envvar('APP_CONFIG_FILE', silent=True)
DB_URL = 'postgresql+psycopg2://user:pwsd#127.0.0.1/NAME DB'
UPLOAD_FOLDER = 'static/upload'
ALLOWED_EXTENSIONS = {'qpj', 'cpg', 'prj', 'dbf', 'shx', 'shp', 'txt', 'json'}
app.config['UPLOAD_FOLDER'] = UPLOAD_FOLDER
app.config['SEND_FILE_MAX_AGE_DEFAULT'] = 0
app.config['SQLALCHEMY_DATABASE_URI'] = DB_URL
#app.config['SQLALCHEMY_TRACK_MODIFICATIONS'] = False # silence the deprecation warning
#app.config['SQLALCHEMY_ECHO'] = True
app.config['DEBUG'] = True
db = SQLAlchemy(app)
#app.route('/upload',methods = ['POST'])
def upload_file():
response_object = {'status': 'no se ha cargado nada'}
if request.method =='POST':
if 'flecheo' not in request.files:
print('no viene el FLECHEO')
files = request.files['flecheo']
for f in files:
print(f.filename)
filename = secure_filename(f.filename)
app.logger.info('FileName: ' + filename)
f.save(os.path.join(app.config['UPLOAD_FOLDER'], f.filename))
response_object['message'] = 'Subieron los archivos del shape file'
return jsonify(response_object)
This is the Json:
function uploadFiles(){
var capa = document.getElementsByName('capa');
var flecheo = document.getElementById('getFiles').files;
formData = new FormData();
for(i = 0; i < capa.length; i++) {
if(capa[i].checked)
formData.append('capa', capa[i].value);
}
for(i=0; i < flecheo.length; i++){
console.log(flecheo[i]);
formData.append('flecheo', flecheo[i]);
}
for (var key of formData.entries()) {
console.log(key[0] + ', ' + key[1]);
}
var contentType = {
headers: {
'content-type': 'multipart/form-data'
}
};
axios.post('https://api.mydomain.com/upload', formData, contentType)
.then(response => {
console.log(response);
}).catch(error => {
console.log(error);
});
}
Any help is very much appreciated!
Update 2
I configure my routes in configure_blueprints called from create_app. I don`t want put all my views handlers in uwsgi.py. They are stored in separate modules.
def configure_blueprints(app):
from .root import root_bp
...
blueprints = [
(root_bp, None),
...
]
for bp, endpoint in blueprints:
app.register_blueprint(bp, url_prefix=endpoint)
return app
def create_app(config_fn=None):
app = Flask(__name__, template_folder='../templates', static_folder='../static')
...
configure_blueprints(app)
return app
app/root/views.py
root_bp = Blueprint('root_bp', __name__)
#root_bp.route('/')
def root():
if not current_user.is_authenticated:
return redirect('/login/')
return render_template('index.html')
Here is output of SIMPLE_SETTINGS=app.config,instance.docker python3 manage.py list_routes
2017-03-01 06:48:11,381 - passlib.registry - DEBUG - registered 'sha512_crypt' handler: <class 'passlib.handlers.sha2_crypt.sha512_crypt'>
...
root_bp.root HEAD,OPTIONS,GET /
...
This is implementation of the list_routes command
#manager.command
def list_routes():
import urllib
output = []
for rule in flask.current_app.url_map.iter_rules():
options = {}
for arg in rule.arguments:
options[arg] = '[{0}]'.format(arg)
methods = ','.join(rule.methods)
try:
url = flask.url_for(rule.endpoint, **options)
except BaseException as e:
print('Exc={}'.format(e))
line = urllib.parse.unquote('{:50s} {:20s} {}'.format(rule.endpoint, methods, url))
output.append(line)
for line in sorted(output):
print(line)
I do not understand why routes should be placed to a file and can not be configured dynamically. If this does not work, what should I do?
Update
uwsgi.ini
[uwsgi]
env = SIMPLE_SETTINGS=app.config,instance.docker
callable = app
wsgi-file = /var/www/app/uwsgi.py
uid = www-data
gid = www-data
socket = /var/www/app/uwsgi.sock
chmod-socket = 666
logto = /var/log/uwsgi/app/app.log
chdir = /var/www/app
plugin = python3
master = true
processes = 1
/var/www/app/uwsgi.py
from app import create_app
app = create_app()
I configure blueprints inside create_app function. I think they should be available at the time of application start. Also I use Flask-script.
SIMPLE_SETTINGS=app.config,instance.docker python3 manage.py shell
The shell starts without errors.
Original post
I have studied all related questions. I could not solve my problem. I deploy my project through the docker-machine on the remote host. Here is my configuration:
Dockerfile
FROM ubuntu:latest
RUN locale-gen en_US.UTF-8
ENV LANG en_US.UTF-8
ENV LANG en_US.UTF-8
ENV LC_ALL en_US.UTF-8
RUN apt-get update && apt-get install -y \
python3 python3-pip git libpq-dev libevent-dev uwsgi-plugin-python3 \
nginx supervisor
COPY nginx/flask.conf /etc/nginx/sites-available/
COPY supervisor/supervisord.conf /etc/supervisor/conf.d/supervisord.conf
COPY . /var/www/app
RUN mkdir -p /var/log/nginx/app /var/log/uwsgi/app /var/log/supervisor \
&& rm /etc/nginx/sites-enabled/default \
&& ln -s /etc/nginx/sites-available/flask.conf /etc/nginx/sites-enabled/flask.conf \
&& echo "daemon off;" >> /etc/nginx/nginx.conf \
&& pip3 install -r /var/www/app/python_modules \
&& chown -R www-data:www-data /var/www/app \
&& chown -R www-data:www-data /var/log
WORKDIR /var/www/app
CMD ["/usr/bin/supervisord"]
docker-compose.yml
version: '2'
services:
db:
image: postgres
volumes:
- moderator-db:/var/lib/postgresql/data
redis:
image: redis
rabbitmq:
image: rabbitmq:3.6
api:
build: .
mem_limit: 1000m
ports:
- "80:80"
depends_on:
- redis
- db
- rabbitmq
volumes:
moderator-redis:
driver: local
moderator-db:
driver: local
supervisord.conf
[supervisord]
nodaemon=true
[program:nginx]
command=/usr/sbin/nginx
[program:uwsgi]
command=uwsgi --ini /var/www/app/uwsgi.ini
flask nginx conf
server {
server_name localhost;
listen 80 default_server;
charset utf-8;
sendfile on;
client_max_body_size 70M;
keepalive_timeout 0;
proxy_buffering on;
proxy_buffer_size 8k;
proxy_buffers 2048 8k;
proxy_ignore_client_abort on;
location / {
include uwsgi_params;
uwsgi_pass unix:///var/www/app/uwsgi.sock;
}
location /static {
root /var/www/app/static/;
}
}
$docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d9dec354d97e moderator_api "/usr/bin/supervisord" 13 seconds ago Up 9 seconds 0.0.0.0:80->80/tcp moderator_api_1
243c48dac303 postgres "docker-entrypoint..." 3 hours ago Up 14 seconds 5432/tcp moderator_db_1
23901a761ef1 redis "docker-entrypoint..." 3 hours ago Up 14 seconds 6379/tcp moderator_redis_1
7cc0683bfe18 rabbitmq:3.6 "docker-entrypoint..." 3 hours ago Up 16 seconds 4369/tcp, 5671-5672/tcp, 25672/tcp moderator_rabbitmq_1
tail -f /var/log/uwsgi/app/app.log
[pid: 24|app: 0|req: 1/1] 123.23.7.216 () {44 vars in 945 bytes} [Tue Feb 28 14:53:57 2017] GET / => generated 233 bytes in 10 msecs (HTTP/1.1 404) 3 headers in 311 bytes (1 switches on core 0)
If you need any additional information, please let me know. I tried different configurations: nginx-proxy image, gunicorn etc. I have same problem with the remote host. What am I still missing?
I spent a lot of time and was not able to solve the problem with the uwsgi socket. I decided to switch to the gunicorn HTTP uwsgi web server. I liked the simplicity of setup.
supervisord.conf
[supervisord]
nodaemon=true
environment=SIMPLE_SETTINGS="app.config,instance.docker,instance.prod"
[program:nginx]
command=/usr/sbin/nginx
[program:app]
command=gunicorn app:app -b 0.0.0.0:8000 --name app --log-level=debug --log-file=- --worker-class gevent
nginx flask.conf
server {
server_name your-domain;
listen 80;
charset utf-8;
sendfile on;
client_max_body_size 70M;
keepalive_timeout 0;
root /var/www/app/static/;
proxy_buffering on;
proxy_buffer_size 8k;
proxy_buffers 2048 8k;
proxy_ignore_client_abort on;
location / {
try_files $uri #proxy_to_app;
}
location #proxy_to_app {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_redirect off;
proxy_pass http://localhost:8000;
}
}
One last caveat. I used a flask snippet with minor modifications for the docker.
class ReverseProxied(ProxyFix):
def __init__(self, app, config, **kwargs):
self.config = config
super().__init__(app, **kwargs)
def __call__(self, environ, start_response):
script_name = environ.get('HTTP_X_SCRIPT_NAME', '')
if script_name:
environ['SCRIPT_NAME'] = script_name
path_info = environ['PATH_INFO']
if path_info.startswith(script_name):
environ['PATH_INFO'] = path_info[len(script_name):]
scheme = environ.get('HTTP_X_SCHEME', '')
if scheme:
environ['wsgi.url_scheme'] = scheme
# fix for docker to proxy server_name
server_name = self.config.get('SERVER_NAME')
if server_name:
environ['HTTP_X_FORWARDED_HOST'] = server_name
return super().__call__(environ, start_response)
...
app.wsgi_app = ReverseProxied(app.wsgi_app, app.config)
app/config.py
SERVER_NAME = 'localhost:8000'
instance/prod.py
SERVER_NAME = 'your-domain'
Now everything works fine.