I'm using nginx as a proxy for a Node server that's rate-limiting requests. The rate is one request every 30 seconds; most requests return a response fine, but if a request is kept open for an extended period of time, I get this:
upstream prematurely closed connection while reading response header from upstream
I cannot figure out what might be causing this. Below is my nginx configuration:
# For more information on configuration, see:
# * Official English Documentation: http://nginx.org/en/docs/
# * Official Russian Documentation: http://nginx.org/ru/docs/
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;
# Load dynamic modules. See /usr/share/nginx/README.dynamic.
# include /usr/share/nginx/modules/*.conf;
events {
worker_connections 1024;
}
http {
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
# Load modular configuration files from the /etc/nginx/conf.d directory.
# See http://nginx.org/en/docs/ngx_core_module.html#include
# for more information.
include /etc/nginx/conf.d/*.conf;
server {
listen 80 default_server;
listen [::]:80 default_server;
server_name _;
root /srv/www/main/htdocs;
# Load configuration files for the default server block.
include /etc/nginx/default.d/*.conf;
location /vcheck {
proxy_pass http://127.0.0.1:8080$is_args$query_string;
# proxy_buffer_size 128k;
# proxy_buffers 4 256k;
# proxy_busy_buffers_size 256k;
# proxy_http_version 1.1;
# proxy_set_header Upgrade $http_upgrade;
# proxy_set_header Connection 'upgrade';
# proxy_set_header Host $host;
# proxy_cache_bypass $http_upgrade;
# proxy_redirect off;
proxy_read_timeout 600s;
}
location ~ \.php$ {
include fastcgi.conf;
fastcgi_split_path_info ^(.+\.php)(/.+)$;
fastcgi_pass unix:/var/run/php-fpm/php-fpm.sock;
fastcgi_index routes.php$is_args$query_string;
}
location / {
if (-f $request_filename) {
expires max;
break;
}
if ($request_filename !~ "\.(js|htc|ico|gif|jpg|png|css)$") {
rewrite ^(.*) /routes.php last;
}
}
}
}
Is there a reason why Node could be closing the connection early?
EDIT: I'm using Node's built-in HTTP server.
Seems like You've to extend response timeout of nodejs application.
So if it's expressjs app so I can guess You try this one:
install: npm i --save connect-timeout
use:
var timeout = require('connect-timeout');
app.use(timeout('60s'));
But I recommend to not to keep connection waiting and fix issue in nodejs app, find why it's halting so long.
Seems like nodejs app has issues that cannot respond and request is getting lost keeping nginx waiting.
Related
I faced a problem case on linux nginx proxy server. I want to create a cache proxy server when original server is down. I determined proxy timeout 5s, but the main page loading have approximately 132 request. The proxy server is sending average 6 request per timeout period and waiting answers, so the page opens 132/6*5 second from cache. I tried with request zone, but i guess it represents the requests to nginx proxy cache server.
Are there any parameters to satisfy this request?
default.conf:
proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=my_cache:10m max_size=90g inactive=10d;
server {
listen 443 ssl;
server_name blabla.com;
location / {
#limit_req zone=mylimit;
#limit_req_dry_run on;
proxy_cache my_cache;
proxy_cache_methods POST GET HEAD;
proxy_connect_timeout 5s;
proxy_cache_key "$host$request_uri|$request_body";
proxy_cache_use_stale error timeout updating http_500 http_502 http_503 http_504;
proxy_cache_background_update on;
proxy_ignore_headers X-Accel-Expires Expires Set-Cookie Cache-Control;
proxy_set_header Host $host;
proxy_pass blabla.com;
proxy_buffering on;
proxy_cache_valid 200 304 301 10m;
proxy_cache_lock on;
}}
nginx.conf:
user nginx;
worker_processes 4;
error_log /var/log/nginx/error.log notice;
pid /var/run/nginx.pid;
events {
worker_connections 100;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
limit_req_zone $binary_remote_addr zone=mylimit:10m rate=150r/s;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
keepalive_timeout 65;
gzip on;
gzip_disable "msie6";
include /etc/nginx/conf.d/*.conf;
}```
I had a Node JS server running with Express, that is being used as a web server. It connects to my database to run queries for the end user.
I have a VPS set up on Digital Ocean, with a Node App running on port 3000. When I access the Node app on ip:3000 it runs fine and as fast as to be expected. If I set up a reverse proxy with nginx, or a firewall rule that forwards traffic from port 80 to port 3000, parts of the page seem to run extremely slowly, or not at all. I can't seem to find a link as to why, as some of the database queries run fine, but some don't load at all and cause the page to hang. If I access the site using port 3000, the site still continues to run fine, even with nginx running. It's only the access from port 80 that is slow.
My NGINX conf is:
user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;
events {
worker_connections 768;
# multi_accept on;
}
http {
##
# Basic Settings
##
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
# server_tokens off;
# server_names_hash_bucket_size 64;
# server_name_in_redirect off;
include /etc/nginx/mime.types;
default_type application/octet-stream;
##
# SSL Settings
##
ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
ssl_prefer_server_ciphers on;
##
# Logging Settings
##
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
##
# Gzip Settings
# server_names_hash_bucket_size 64;
# server_name_in_redirect off;
include /etc/nginx/mime.types;
default_type application/octet-stream;
##
# SSL Settings
##
ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
ssl_prefer_server_ciphers on;
##
# Logging Settings
##
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
##
# Gzip Settings
##
gzip on;
# gzip_vary on;
# gzip_proxied any;
# gzip_comp_level 6;
# gzip_buffers 16 8k;
# gzip_http_version 1.1;
# gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/$
##
# Virtual Host Configs
##
server_names_hash_bucket_size 64;
include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}
#mail {
# # See sample authentication script at:
# # http://wiki.nginx.org/ImapAuthenticateWithApachePhpScript
#
# # auth_http localhost/auth.php;
# # pop3_capabilities "TOP" "USER";
# # imap_capabilities "IMAP4rev1" "UIDPLUS";
#
# server {
# listen localhost:110;
# protocol pop3;
# proxy on;
# }
#
# server {
# listen localhost:143;
# protocol imap;
# proxy on;
# }
#}
My example.com file is (where 'example.com' is my site address):
server {
listen 80;
listen [::]:80;
root /var/www/example.com/html;
index index.html index.htm index.nginx-debian.html;
server_name example.com www.example.com;
location / {
proxy_pass http://localhost:3000;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
}
I recommend using PM2 to start instance of your node app in production https://github.com/Unitech/pm2
Try following NGINX configurations
upstream prod_nodejs_upstream {
server 127.0.0.1:3000;
keepalive 64;
}
server {
listen 80;
server_name example.com;
root /home/www/example;
location / {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header X-NginX-Proxy true;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_max_temp_file_size 0;
proxy_pass http://prod_nodejs_upstream/;
proxy_redirect off;
proxy_read_timeout 240s;
}
}
Once these changes applied you must restart NGINX using commands sudo nginx -t and then sudo systemctl restart nginx
Please update configuration with as below and share output of file so that time taken by upstream can be measured
upstream prod_nodejs_upstream {
server 127.0.0.1:3000;
keepalive 64;
}
server {
listen 80;
server_name example.com;
root /home/www/example;
location / {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header X-NginX-Proxy true;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_max_temp_file_size 0;
proxy_pass http://prod_nodejs_upstream/;
proxy_redirect off;
proxy_read_timeout 240s;
}
log_format apm '"$time_local" client=$remote_addr '
'method=$request_method request="$request" '
'request_length=$request_length '
'status=$status bytes_sent=$bytes_sent '
'body_bytes_sent=$body_bytes_sent '
'referer=$http_referer '
'user_agent="$http_user_agent" '
'upstream_addr=$upstream_addr '
'upstream_status=$upstream_status '
'request_time=$request_time '
'upstream_response_time=$upstream_response_time '
'upstream_connect_time=$upstream_connect_time '
'upstream_header_time=$upstream_header_time';
}
How do I use environment variables set in the OS inside nginx configs?
For example environment variables set are ENVIRON=dev, APP_NAME=test
Here's my Dockerfile:
FROM openresty/openresty:alpine
RUN set -ex && \
rm /etc/nginx/conf.d/default.conf
ADD nginx.conf /etc/nginx/
ADD custom.conf /etc/nginx/conf.d/
Here's my main nginx.conf
user nginx;
worker_processes auto;
pcre_jit on;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
env ENVIRON;
env APP_NAME;
set_by_lua $environ 'return os.getenv("ENVIRON")';
set_by_lua $appname 'return os.getenv("APP_NAME")';
http {
server_tokens off;
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
keepalive_timeout 65;
include /etc/nginx/conf.d/*.conf;
}
Here's my custom.conf from /etc/nging/conf.d/
upstream app.$environ-$appname {
server $environ-$appname:80;
}
server {
listen 80;
server_name $hostname;
error_log /dev/stdout info;
access_log /dev/stdout;
location / {
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
add_header X-Info proxied;
proxy_pass http://app.$environ-$appname;
}
}
Thanks!
That thing you are trying to do, I dont think you can do that.
From the documentation on the server directive of the upstream module:
Defines the address and other parameters of a server. The address can
be specified as a domain name or IP address, with an optional port, or
as a UNIX-domain socket path specified after the “unix:” prefix
You can't create upstream servers on the fly using variables.
If you want to route requests like this either use the map directive if the different locations are known at runtime, or set up an auth_request directive which points at an application on your server which can return the correct variables for each request in real time. Then have Nginx store that response in a variable using the auth_request_set directive.
Im working on a centos 6.7 machine and I’m trying to configure nginx to serve a node.js application. I feel like I’m really close but I’m missing something. So heres my nginx.conf and below that is my server.conf thats in my sites-enabled directory.
When I go to the public IP address it gives me a 502 bad gateway error. But if I curl the private IP with the correct port on my centos machine I can see the node application running. What am I missing here? is it a firewall issue or maybe something else?
nginx.conf
user nginx;
worker_processes 1;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
#tcp_nopush on;
keepalive_timeout 65;
#gzip on;
#include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}
sites-enabled/server.conf
server {
listen 80;
#server_name localhost;
location / {
proxy_pass http://192.xxx.x.xx:8000; // private IP
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
}
UPDATE:
I figured this out! heres the server block that worked for me
server {
listen 80 default_server;
listen [::]:80 default_server;
#server_name _;
root /usr/share/nginx/html;
include /etc/nginx/default.d/*.conf;
location / {
proxy_pass http://127.0.0.1:9000;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
error_page 404 /404.html;
location = /40x.html {
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
}
}
I want to write a comment but stack overflow does not let me do it.
I am 99% sure that Node.js website does NOT need to work with nginx or apache.
If the script setup correctly, the Node.js Application should listen to the port by itself.
Since you did not really say much of your structure, I guess you can just try to access through the public IP with the port of Node.js.
I run my Express.js app on port 8000 and I want to work with localhost rather than localhost:8000 in my dev environment.
I'm on Mac OSX El Capitan and using nginx 1.8.0
I get 502 Bad Gateway and ERR_CONNECTION_TIMED_OUT error.
I stop nginx via sudo pkill nginx and start with sudo nginx. I also use pm2 and start it via sudo pm2 start bin/www --watch, there's no problem here. I can access my app at localhost:8000
I installed nginx via homebrew.
/etc/hosts
##
# Host Database
#
# localhost is used to configure the loopback interface
# when the system is booting. Do not change this entry.
##
127.0.0.1 localhost
255.255.255.255 broadcasthost
::1 localhost
Here's my nginx.conf in /usr/local/etc/nginx
user myusernameishere staff;
worker_processes 1;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
include /usr/local/etc/nginx/sites-enabled/*;
}
sites-enabled/default.conf
server{
listen 80;
server_name 127.0.0.1 localhost;
location / {
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host;
proxy_pass http://127.0.0.1:8000;
}
}
/usr/local/etc/nginx/nginx.conf.default
#user nobody;
worker_processes 1;
#error_log logs/error.log;
#error_log logs/error.log notice;
#error_log logs/error.log info;
#pid logs/nginx.pid;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
#log_format main '$remote_addr - $remote_user [$time_local] "$request" '
# '$status $body_bytes_sent "$http_referer" '
# '"$http_user_agent" "$http_x_forwarded_for"';
#access_log logs/access.log main;
sendfile on;
#tcp_nopush on;
#keepalive_timeout 0;
keepalive_timeout 65;
#gzip on;
server {
listen 8080;
server_name localhost;
#charset koi8-r;
#access_log logs/host.access.log main;
location / {
root html;
index index.html index.htm;
}
#error_page 404 /404.html;
# redirect server error pages to the static page /50x.html
#
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}
# proxy the PHP scripts to Apache listening on 127.0.0.1:80
#
#location ~ \.php$ {
# proxy_pass http://127.0.0.1;
#}
# pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
#
#location ~ \.php$ {
# root html;
# fastcgi_pass 127.0.0.1:9000;
# fastcgi_index index.php;
# fastcgi_param SCRIPT_FILENAME /scripts$fastcgi_script_name;
# include fastcgi_params;
#}
# deny access to .htaccess files, if Apache's document root
# concurs with nginx's one
#
#location ~ /\.ht {
# deny all;
#}
}
# another virtual host using mix of IP-, name-, and port-based configuration
#
#server {
# listen 8000;
# listen somename:8080;
# server_name somename alias another.alias;
# location / {
# root html;
# index index.html index.htm;
# }
#}
# HTTPS server
#
#server {
# listen 443 ssl;
# server_name localhost;
# ssl_certificate cert.pem;
# ssl_certificate_key cert.key;
# ssl_session_cache shared:SSL:1m;
# ssl_session_timeout 5m;
# ssl_ciphers HIGH:!aNULL:!MD5;
# ssl_prefer_server_ciphers on;
# location / {
# root html;
# index index.html index.htm;
# }
#}
include servers/*;
}
Upgrade is a header from HTTP v1.1, you may need to include proxy_http_version 1.1; when using it.
Try this for your sites_enabled/default.conf:
upstream backend {
server localhost:8000;
}
server{
listen 80;
server_name 127.0.0.1 localhost;
location / {
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host;
proxy_pass http://backend;
}
}