I have two Nginx servers acting as reverse proxies for nodejs servers running on ports 5000 and 5001.
The one that is running on port 5000 is for normal form upload
The other one that is running on port 5001 is for uploading images
On the client side, what I've done is after filling out the form (title, description, and image) by the user, the image is uploaded to the image server first and the imageURL, title, and description are uploaded to the normal web server then.
The Problem
When the client fills out the form and clicks on upload if the image upload works then upload to the normal server fails or if normal server upload works then upload to the image server fails.
The error is the following one: (This could for either of them)
Access to XMLHttpRequest at 'https://myserver.com/imagev2api/profile-upload-single' from origin 'https://blogs.vercel.app' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.
Note: I've used app.use(cors()) on both servers (image and normal server)
Here's both nginx server configurations
Image Server
upstream imageserver.com {
server 127.0.0.1:5001;
keepalive 600;
}
server {
server_name imageserver.com;
error_log /var/www/log/imagserver.com.error;
access_log /var/www/log/imagserver.com.access;
location / {
proxy_pass http://imageserver.com;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
# fastcgi_split_path_info ^(.+\.php)(/.+)$;
}
listen 443 ssl http2; # managed by Certbot
ssl_certificate /etc/letsencrypt/live/linoxcloud.com/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/linoxcloud.com/privkey.pem; # managed by Certbot
ssl_protocols TLSv1.2 TLSv1.3 SSLv2 SSLv3;
ssl_session_cache shared:SSL:5m;
ssl_session_timeout 10m;
ssl_session_tickets off;
}
server {
if ($host = imageserver.com) {
return 301 https://$host$request_uri;
} # managed by Certbot
listen 80;
server_name imageserver.com;
}
Normal Server
upstream normalserver.com {
server 127.0.0.1:5000;
keepalive 600;
}
server {
server_name normalserver.com;
error_log /var/www/log/normalserver.com.error;
access_log /var/www/log/normalserver.com.access;
location / {
proxy_pass http://normalserver.com;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
listen 443 ssl http2; # managed by Certbot
ssl_certificate ...; # managed by Certbot
ssl_certificate_key ...; # managed by Certbot
ssl_protocols TLSv1.2 TLSv1.3 SSLv2 SSLv3;
ssl_session_cache shared:SSL:5m;
ssl_session_timeout 10m;
ssl_session_tickets off;
}
server {
if ($host = normalserver.com) {
return 301 https://$host$request_uri;
} # managed by Certbot
listen 80;
server_name normalserver.com;
}
I've been trying to overcome this problem for some period of time by trying literally everything.
Reference: Two NGINX servers one passing CORS issue (but this doesn't provide any insights into what the problem and solution is)
Any possible fixes, please?
You have to combine these reverse proxies in one configuration file. There was already a similar thread here: https://serverfault.com/questions/242679/how-to-run-multiple-nginx-instances-on-different-port
Hope it helps.
The problem in my case is that I'm running my NODEJS instances/servers using "pm2" and they are not working simultaneously.
Similar issue: https://github.com/Unitech/pm2/issues/4352
Elaborating on what happened was if two requests are made simultaneously one pm2 process successfully executes but meanwhile the server crashes/restarts after that execution which is making the other server throw a 502 Bad Gateway error. (unreachable as though the server is not running)
For now, I'm running one server on "pm2"
and the other one uses "forever"
Note: This issue has nothing to do with Nginx (since it can handle any number of websites with different domain names on a single port 80)
This problem happened quite recently maybe it's some "pm2" bug
In simple words, when the two requests hit individual pm2 processes, one executes, and the pm2 processes kind of restart again making the second request obsolete.
We have a typical nginx+nodejs setup with node v4.2.2 & nginx v1.9.7.2
Backend service app running on express/nodejs is deployed in clustered mode using recluster module with child processes (equal to #cores) forked that listen on same port. nginx is used as reverse proxy to backend.
Each such instance/box (4 core 8 gb) of nginx+nodejs is serving around 100 tps at load with 90th percentile of latency around 120ms.
Problem is that we get infrequent (1-2 times every 5 min) 504s on nginx access log and related error log shows
(110: Connection timed out) while connecting to upstream
As I understand, it would happen when nginx timedout establishing a connection with nodejs server but all the matrices show nodejs is healthy. Also, all the requests below and above this error have normal latency and just one off request gets stuck. There is no corresponding log entry at nodejs server means request never reached node server.
Relevant nginx config below.
worker_processes auto;
worker_rlimit_nofile 40000;
events {
worker_connections 2000;
use epoll;
}
http {
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 60;
}
upstream node {
server 127.0.0.1:3000;
keepalive 256;
}
server {
listen 80;
server_name abc.com
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_read_timeout 60;
location / {
proxy_pass http://node;
}
}
Output to ss -s is
Total: 747 (kernel 0)
TCP: 7400 (estab 481, closed 6890, orphaned 0, synrecv 0, timewait
6890/0), ports 0
It shows high timewait sockets but we don't see any errors in syslog means operating system's limits are not reached.
I have tried tuning network and nginx primarily taking clues from blog post with no luck.
Need help in forum to get right direction to debug this. Let me know what more info I can provide.
I am using nginx and node server to serve update requests. I get a gateway timeout when I request an update on large data. I saw this error from the nginx error logs :
2016/04/07 00:46:04 [error] 28599#0: *1 upstream prematurely closed
connection while reading response header from upstream, client:
10.0.2.77, server: gis.oneconcern.com, request: "GET /update_mbtiles/atlas19891018000415 HTTP/1.1", upstream:
"http://127.0.0.1:7777/update_mbtiles/atlas19891018000415", host:
"gis.oneconcern.com"
I googled for the error and tried everything I could, but I still get the error.
My nginx conf has these proxy settings:
##
# Proxy settings
##
proxy_connect_timeout 1000;
proxy_send_timeout 1000;
proxy_read_timeout 1000;
send_timeout 1000;
This is how my server is configured
server {
listen 80;
server_name gis.oneconcern.com;
access_log /home/ubuntu/Tilelive-Server/logs/nginx_access.log;
error_log /home/ubuntu/Tilelive-Server/logs/nginx_error.log;
large_client_header_buffers 8 32k;
location / {
proxy_pass http://127.0.0.1:7777;
proxy_redirect off;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $http_host;
proxy_cache_bypass $http_upgrade;
}
location /faults {
proxy_pass http://127.0.0.1:8888;
proxy_http_version 1.1;
proxy_buffers 8 64k;
proxy_buffer_size 128k;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
}
I am using a nodejs backend to serve the requests on an aws server. The gateway error shows up only when the update takes a long time (about 3-4 minutes). I do not get any error for smaller updates. Any help will be highly appreciated.
Node js code :
app.get("/update_mbtiles/:earthquake", function(req, res){
var earthquake = req.params.earthquake
var command = spawn(__dirname + '/update_mbtiles.sh', [ earthquake, pg_details ]);
//var output = [];
command.stdout.on('data', function(chunk) {
// logger.info(chunk.toString());
// output.push(chunk.toString());
});
command.stderr.on('data', function(chunk) {
// logger.error(chunk.toString());
// output.push(chunk.toString());
});
command.on('close', function(code) {
if (code === 0) {
logger.info("updating mbtiles successful for " + earthquake);
tilelive_reload_and_switch_source(earthquake);
res.send("Completed updating!");
}
else {
logger.error("Error occured while updating " + earthquake);
res.status(500);
res.send("Error occured while updating " + earthquake);
}
});
});
function tilelive_reload_and_switch_source(earthquake_unique_id) {
tilelive.load('mbtiles:///'+__dirname+'/mbtiles/tipp_out_'+ earthquake_unique_id + '.mbtiles', function(err, source) {
if (err) {
logger.error(err.message);
throw err;
}
sources.set(earthquake_unique_id, source);
logger.info('Updated source! New tiles!');
});
}
Thank you.
I solved this by setting a higher timeout value for the proxy:
location / {
proxy_read_timeout 300s;
proxy_connect_timeout 75s;
proxy_pass http://localhost:3000;
}
Documentation: https://nginx.org/en/docs/http/ngx_http_proxy_module.html
I think that error from Nginx is indicating that the connection was closed by your nodejs server (i.e., "upstream"). How is nodejs configured?
I had the same error for quite a while, and here what fixed it for me.
I simply declared in service that i use what follows:
Description= Your node service description
After=network.target
[Service]
Type=forking
PIDFile=/tmp/node_pid_name.pid
Restart=on-failure
KillSignal=SIGQUIT
WorkingDirectory=/path/to/node/app/root/directory
ExecStart=/path/to/node /path/to/server.js
[Install]
WantedBy=multi-user.target
What should catch your attention here is "After=network.target".
I spent days and days looking for fixes on nginx side, while the problem was just that.
To be sure, stop running the node service you have, launch the ExecStart command directly and try to reproduce the bug. If it doesn't pop, it just means that your service has a problem. At least this is how i found my answer.
For everybody else, good luck!
I stumbled upon *145660 upstream prematurely closed connection while reading upstream Nginx error log entry when trying to download a 2GB file from the server Nginx was a proxy for. The message indicates that the "upstream" closed connection, but in fact it was related to proxy_max_temp_file_size setting:
Syntax: proxy_max_temp_file_size size;
Default: proxy_max_temp_file_size 1024m;
Context: http, server, location
When buffering of responses from the proxied server is enabled, and the whole response does not fit into the buffers set by the proxy_buffer_size and proxy_buffers directives, a part of the response can be saved to a temporary file. This directive sets the maximum size of the temporary file. The size of data written to the temporary file at a time is set by the proxy_temp_file_write_size directive.
The zero value disables buffering of responses to temporary files.
This restriction does not apply to responses that will be cached or stored on disk.
The symptoms:
download was being forcibly stopped at around 1GB,
Nginx claimed that upstream closed connection, but without proxy server was returning the full content.
The solution:
increased proxy_max_temp_file_size for proxied location to 4096m and it started sending full content.
I was finding this error in the logs of my AWS Elastic Beanstalk instance when trying to post about half a million rows to my api.
I followed all the advice here to no avail.
What did finally work was increasing the size of my EC2 instance from 1 core and 1GB RAM to 4 core and 8 GB RAM.
You can increase the timeout in node like so.
app.post('/slow/request', function(req, res) {
req.connection.setTimeout(100000); //100 seconds
...
}
I don't think this is your case, but I'll post it if it helps anyone. I had the same issue and the problem was that Node didn't respond at all (I had a condition that when failed didn't do anything - so no response) - So if increasing all your timeouts didn't solve it, make sure all scenarios get a response.
I ran into this issue as well and found this post. Ultimately none of these answers solved my problem, instead I had to put in a rewrite rule to strip out the location /rt as the backend my developers made was not expecting any additional paths:
┌─(william#wkstn18)──(Thu, 05 Nov 20)─┐
└─(~)──(16:13)─>wscat -c ws://WebsocketServerHostname/rt
error: Unexpected server response: 502
Testing with wscat repeatedly gave a 502 response. Nginx error logs provided the same upstream error as above, but notice the upstream string shows the GET Request is attempting to access localhost:12775/rt and not localhost:12775:
2020/11/05 22:13:32 [error] 10175#10175: *7 upstream prematurely closed
connection while reading response header from upstream, client: WANIP,
server: WebsocketServerHostname, request: "GET /rt/socket.io/?transport=websocket
HTTP/1.1", upstream: "http://127.0.0.1:12775/rt/socket.io/?transport=websocket",
host: "WebsocketServerHostname"
Since the devs had not coded their websocket (listening on 12775) to expect /rt/socket.io but instead just /socket.io/ (NOTE: /socket.io/ appears to just be a way to specify websocket transport discussed here). Because of this, rather than ask them to rewrite their socket code I just put in a rewrite rule to translate WebsocketServerHostname/rt to WebsocketServerHostname:12775 as below:
upstream websocket-rt {
ip_hash;
server 127.0.0.1:12775;
}
server {
listen 80;
server_name WebsocketServerHostname;
location /rt {
proxy_http_version 1.1;
#rewrite /rt/ out of all requests and proxy_pass to 12775
rewrite /rt/(.*) /$1 break;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host;
proxy_pass http://websocket-rt;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
}
}
I meet the same problem and no one of the solutions detailed here worked for me ...
First of all I had an error 413 Entity too large so I updated my nginx.conf as following :
http {
# Increase request size
client_max_body_size 10m;
##
# Basic Settings
##
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
# server_tokens off;
# server_names_hash_bucket_size 64;
# server_name_in_redirect off;
include /etc/nginx/mime.types;
default_type application/octet-stream;
##
# SSL Settings
##
ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
ssl_prefer_server_ciphers on;
##
# Logging Settings
##
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
##
# Gzip Settings
##
gzip on;
# gzip_vary on;
# gzip_proxied any;
# gzip_comp_level 6;
# gzip_buffers 16 8k;
# gzip_http_version 1.1;
# gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;
##
# Virtual Host Configs
##
include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
##
# Proxy settings
##
proxy_connect_timeout 1000;
proxy_send_timeout 1000;
proxy_read_timeout 1000;
send_timeout 1000;
}
So I only updated the http part, and now I meet the error 502 Bad Gateway and when I display /var/log/nginx/error.log I got the famous "upstream prematurely closed connection while reading response header from upstream"
What is really mysterious for me is that the request works when I run it with virtualenv on my server and send the request to the : IP:8000/nameOfTheRequest
Thanks for reading
I got the same error, here is how I resolved it:
Downloaded logs from AWS.
Reviewed Nginx logs, no additional details as above.
Reviewed node.js logs, AccessDenied AWS SDK permissions error.
Checked the S3 bucket that AWS was trying to read from.
Added additional bucket with read permission to correct server role.
Even though I was processing large files there were no other errors or settings I had to change once I corrected the missing S3 access.
Problem
The upstream server is timing out and I don't what is happening.
Where to Look first before increasing read or write timeout if your server is connecting to a database
Server is connecting to a database and that connection is working just fine and within sane response time, and its not the one causing this delay in server response time.
make sure that connection state is not causing a cascading failure on your upstream
Then you can move to look at the read and write timeout configurations of the server and proxy.
This error can also occur when your code is getting into a loop. So investigate if you have any (indirectly) self-referencing code that's causing this.
I'm writing web socket project, everything is working like expected(locally), I using:
NGINX as a WebSockets Proxy
NODEJS as a backend server
WS as websocket module: ws
NGINX configuration:
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}
upstream backend_cluster {
server 127.0.0.1:5050;
}
# Only retry if there was a communication error, not a timeout.
proxy_next_upstream error;
server {
access_log /code/logs/access.log;
error_log /code/logs/error.log info;
listen 80;
listen 443 ssl;
server_name mydomain;
root html;
ssl_certificate /code/certs/sslCert.crt;
ssl_certificate_key /code/certs/sslKey.key;
ssl_protocols SSLv3 TLSv1 TLSv1.1 TLSv1.2; # basically same as apache [all -SSLv2]
ssl_ciphers HIGH:MEDIUM:!aNULL:!MD5;
location /websocket/ws {
proxy_pass http://backend_cluster;
proxy_http_version 1.1;
proxy_redirect off ;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
}
}
Like I mentioned this is working just fine locally and in one machine in development environments, the issue I'm worry about is when we will go to production, in production environments will have more that one nodejs server.
In production the configuration for nginx will be something like:
upstream backend_cluster {
server domain1:5050;
server domain2:5050;
}
So I don't know how NGINX solves the issue for stickiness, meaning how I know that after the 'HANDSHAKE/upgrade' is done in one server, how it will know to continue working with the same server, is there a way to tell NGINX to stick to the same server?
I hope I make my self clear.
Thanks in advanced
Use this configuration:
upstream backend_cluster {
ip_hash;
server domain1:5050;
server domain2:5050;
}
clody69's answer is pretty standard. However I prefer using the following configuration for 2 reasons :
Users connecting from the same public IP should be connecting to 2 different servers if needed. ip_hash enforces 1 server per public IP.
If user 1 is maxing out server 1's performance I want him/her to be able to use the application smoothly if he/she opens another tab. ip_hash doesn't allow that.
upstream backend_cluster {
hash $content_type;
server domain1:5050;
server domain2:5050;
}
I have nginx setup as a reverse proxy with the following config.
worker_processes 1;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
keepalive_timeout 65;
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}
server {
listen 8080;
server_name localhost;
add_header X-Cache-Status $upstream_cache_status;
location /node/ {
proxy_http_version 1.1;
proxy_pass http://127.0.0.1:8000/;
proxy_redirect off;
proxy_set_header Connection $connection_upgrade;
proxy_set_header Upgrade $http_upgrade;
}
}
}
I'm connecting to it with the following code.
socket = io.connect('http://localhost:8080', {
'resource': 'node/socket.io'
});
Once connected, everything works normally, except that messages above around 15kb are never received by Node. Nginx doesn't log anything to the access or error logs.
I'm pretty sure it's an nginx configuration issue, but I haven't found any settings that affect this behavior. I've tried client_body_buffer_size size, proxy_buffers, proxy_buffer_size, tcp_nodelay, and large_client_header_buffers.
How can I get nginx to stop dropping these messages?
I guess the problem did have to do with the proxy buffers, I just hadn't set them high enough. The following is working now for messages around 5mb.
proxy_buffers 8 2m;
proxy_buffer_size 10m;
proxy_busy_buffers_size 10m;