nodejs nginx 504 gateway timeout reverese proxy - node.js

I am trying to run a NodeJS application on port 2000 of an ubuntu server.
the application runs using pm2.
As far as I can tell the application is running fine as i can use the command:
curl http://localhost:2000
And I get the web page back almost instantly.
The problem I have is that I am trying to use nginx as the reverse proxy and for SSL. From an SSL perspective this works great as trying to browse to the website 'http://dreamingoftech.uk' I get redirected to the HTTPS version and I get the green padlock. Happy days.
However unfortunately I don't get the webpage but a 504 error.
The strange thing is, if I run:
time -p GET -H 'Host: dreamingoftech.uk' http://127.0.0.1:2000
I do eventually get the webpage returned (although i have to admit that I don't really know what this command is doing)
I am using the basic nginx.conf file except with these values added:
proxy_connect_timeout 600;
proxy_send_timeout 600;
proxy_read_timeout 600;
send_timeout 600;
and my sites-available/default file is as follows:
#HTTP
server {
listen 80;
listen [::]:80 default_server ipv6only=on;
return 301 https://$host$request_uri;
}
# HTTPS — proxy all requests to the Node app
server {
# Enable HTTP/2
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name dreamingoftech.uk;
# Use the Let’s Encrypt certificates
ssl_certificate /etc/letsencrypt/live/dreamingoftech.uk/fullchain.pe$
ssl_certificate_key /etc/letsencrypt/live/dreamingoftech.uk/privkey.$
# Include the SSL configuration from cipherli.st
include snippets/ssl-params.conf;
location / {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-NginX-Proxy true;
proxy_set_header X-Forwarded-Proto https;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_pass http://127.0.0.1:2000;
proxy_ssl_session_reuse off;
proxy_set_header Host $http_host;
proxy_cache_bypass $http_upgrade;
proxy_redirect off;
proxy_http_version 1.1;
}
In the access logs I get this error, although I am not sure if this is a red herring:
"GET / HTTP/2.0" 504 665 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.84 Safari/537.36"
My question really is could anyone advise me on the correct setup to get my website to load in a timely manner. I can't understand why a higher timeout is needed if the server is capable of delivering the page almost instantly (although again this could be my naivity).
Thanks in advance
Steve

Related

Chrome net::ERR_HTTP2_PROTOCOL_ERROR 200 after a reconnect

I'm using Node server with an express app which handles a Server Sent Events stream. This is proxied via NginX with http2 enabled. The SSE events are consumed via EventSource in a React app. I'm sending a heartbeat message every 10 seconds to keep the connection alive.
This all works great until there is some form of network interruption such as putting my laptop to sleep then re-awaking it.
Then from that point on the stream will error every 40 or so seconds with the net::ERR_HTTP2_PROTOCOL_ERROR 200 error and then reconnect instead of just reconnecting once with a steady stream.
Firefox works correctly. It doesn't error and reconnects only once.
If I configure Node to serve http2 directly instead of via NGinx as a test (via spdy library) then all works as expected so I don't think this is a Node issue and I must be missing something with my Nginx configuration and Chrome.
Nginx config as follows (location /stream is the SSE proxy)
server {
listen 28443 ssl http2;
listen [::]:28443 ssl http2;
server_name example.com;
ssl on;
ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;
http2_max_field_size 16k;
http2_max_header_size 128k;
root /var/www/example.com;
index index.html index.htm;
location / {
client_max_body_size 100M;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_pass http://localhost:28080;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'Upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
location /stream {
proxy_http_version 1.1;
# proxy_request_buffering off;
# proxy_buffering off;
# proxy_cache off;
# chunked_transfer_encoding off;
proxy_set_header Connection '';
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_pass http://localhost:28080/stream;
}
}
I've tried various combinations with proxy_buffers off and keepalive settings ect.. which seems to only affect the time between errors i.e. 5 minutes instead of 40 seconds.
Not sure if you figure this out. I got the same issue recently, by increasing the size:
http2_max_field_size 64k;
http2_max_header_size 512k;
The chrome's net::ERR_HTTP2_PROTOCOL_ERROR has gone.
Also, not sure if it applied to you. If I use firefox, I can actually visit my site correctly.
http2_max_field_size and http2_max_header_size directives are obsolete since version 1.19.7
Instead, something like the following could be used:
large_client_header_buffers 4 64k;
Now token is named
In my environment just set to:
large_client_header_buffers 10 512k;
And error is gone
Source:
http://nginx.org/en/docs/http/ngx_http_core_module.html#large_client_header_buffers

Cors + Nginx + Nodejs (VUE Frontend + Backend) doesn't work

I have my backend Nodejs running on port 23456
I have my frontend VUE running on port 8080
When i start them and when i visit my domain eg. test.dev the frontend is visible however I'm not able to login.. and it feels like it's not triggering to DB at all.
The backend is starting fine, the fronend is starting fine, it just feels that they don't talk to each other since they are different ports.
For days I have been reading about this and this seems to be a CORS issue and i have tried to find right configt but since I'm noob in this nothing works.
I'm currently running NGINX and this is how my file looks like right now (etc/nginx/sites-available/test.dev.conf:
server {
listen 80;
server_name test.dev www.test.dev;
return 301 https://test.dev$request_uri;
}
server {
# Enable HTTP/2
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name test.dev www.test.dev;
# Use the Let's Encrypt certificates
ssl_certificate /etc/letsencrypt/live/test.dev/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/test.dev/privkey.pem;
# Include the SSL configuration from cipherli.st
include snippets/ssl-params.conf;
location /api {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-NginX-Proxy true;
proxy_pass http://localhost:23456;
proxy_ssl_session_reuse off;
proxy_set_header Host $http_host;
proxy_cache_bypass $http_upgrade;
proxy_redirect off;
}
location / {
add_header 'Access-Control-Allow-Origin' 'http://localhost:8080' always;
add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS, PUT, DELETE' always;
add_header 'Access-Control-Allow-Headers' 'X-Requested-With,Accept,Content-Type, Origin' always;
proxy_pass http://localhost:8080;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
}
I can visit test.dev and I can see the frontend, however clicking on login doesn't work.
It feels like it's not connecting at all to the backend which is another port and I don't know how to get this to work..
This is the error in the console I'm getting:
Any idea how my .conf file should look like?
Thanks in advance.
I have managed to find the solution somehow.
So what i want to share is that the code above posted is correct.
Now, the solution was that I changed in the frontend VUE so that API was read from:
http://localhost:23456/api/v1
to
https://test.dev/api/v1
That's it. The login worked just find and everything works fine...
Thanks for all help!

502 Bad Gateway for NodeJS server managed by PM2 inside a lxc container

I have a digital ocean droplet running Ubuntu 18.04 and inside is is an lxc container. I have two applications in that container.
The first application (a client) lives at /var/www/html and the second one is the NodeJS application that lives at /var/www/my-site/. The Node application inside the container is managed by pm2 and everything seems to be working fine thus far because when I type in curl http://localhost:3000 at the container terminal, I get back the desired output.
Inside the main droplet (not the container) under /etc/nginx/sites-available, I have the following two server blocks - default and my-site.
The first app works fine when I try to access it through the browser via my domain but the NodeJS application returns a 502 Bad Gateway when I try to access it through sub.mydomain.com. pm2 start inside the container tells me that the node application status is online.
Here is my default server block file. This works. When I visit mydomain.com, my site shows up fine.
# HTTP — redirect all traffic to HTTPS
server {
listen 80;
listen [::]:80 default_server ipv6only=on;
return 301 https://$host$request_uri;
}
server {
# Enable HTTP/2
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name mydomain.com;
# Use the Let’s Encrypt certificates
ssl_certificate /etc/letsencrypt/live/mydomain.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/mydomain.com/privkey.pem;
# Include the SSL configuration from cipherli.st
include snippets/ssl-params.conf;
root /var/www/html;
index index.html index.htm index.nginx-debian.html;
location / {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_pass http://container_ip_address /;
}
}
Now here is the other server block - my-site.
# Upstream config
upstream site_upstream {
server 127.0.0.1:3000;
keepalive 64;
}
server {
# Enable HTTP/2
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name sub.mydomain.com www.sub.mydomain.com;
root /var/www/my-site;
# Use the Let’s Encrypt certificates
ssl_certificate /etc/letsencrypt/live/mydomain.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/mydomain.com/privkey.pem;
# Include the SSL configuration from cipherli.st
include snippets/ssl-params.conf;
location / {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-NginX-Proxy true;
proxy_pass http://site_upstream;
proxy_ssl_session_reuse off;
proxy_set_header Host $http_host;
proxy_cache_bypass $http_upgrade;
proxy_redirect off;
}
}
I have set the A Record for my subdomain on my domain's DNS settings, to my droplet's IP address and I have also created a symbolic link to /etc/nginx/sites-enabled for the my-site server block.
I have scoured the internet for a solution to this problem but nothing seems to be working. What am I missing?
Your help would be greatly appreciated. Thanks.
The problem here was that requests to the sub domain were not being directed to the lxc container.
I solved this by adding the following inside the my-site server block.
location / {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_pass http://container_ip/;
}
After that I added an asterisk to the next location block.
location /* {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-NginX-Proxy true;
proxy_pass http://site_upstream;
proxy_ssl_session_reuse off;
proxy_set_header Host $http_host;
proxy_cache_bypass $http_upgrade;
proxy_redirect off;
}
Another way of getting around this issue was by including the sub-domain in the server_name directive for the default server block. This worked but the only problem was that nginx would complain that it had to ignore the server I had set up in the my-site server block when you ran nginx -t, otherwise, it worked just fine.

nodejs nginx 502 gateway error

I am trying to use a nodejs app behind an nginx reverse proxy to handle the ssl
I have my app running on localhost:2000. I can confirm this as working with a curl command.
This is my nginx setup:
# the IP(s) on which your node server is running. I chose port 3000.
upstream dreamingoftech.uk {
server 127.0.0.1:2000;
keepalive 16;
}
# the nginx server instance
server {
listen 0.0.0.0:80;
server_name dreamingoftech.uk;
return 301 https://$host$request_uri;
}
#HTTPS
server {
listen 443 ssl http2;
server_name dreamingoftech.uk;
access_log /var/log/nginx/dreamingoftech.log;
error_log /var/log/nginx/dreamingoftech.error.log debug;
ssl_certificate /etc/letsencrypt/live/dreamingoftech.uk/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/dreamingoftech.uk/privkey.pem;
include snippets/ssl-params.conf;
# pass the request to the node.js server with the correct headers and much more can be added, see nginx config options
location / {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Proto https;
proxy_set_header X-NginX-Proxy true;
proxy_pass http://dreamingoftech.uk/;
proxy_redirect off;
#proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "";
proxy_ssl_session_reuse off;
proxy_cache_bypass $http_upgrade;
}
}
if I now curl https://dreamingoftech.uk, it takes a while but I do get the webpage delivered. albeit with the message:
curl: (18) transfer closed with 1 bytes remaining to read
However when viewed from a browser I get a 502 gateway error.
I have checked the error log and this is the result: ERROR LOG
I can't understand why the reverse proxy is adding such a time delay into the process. Any ideas would be greatly appreciated.
PS: in the upstream config I have tried localhost instead of 127.0.0.1 to no avail
I have almost the same configuration. Can you try the following
You can redirect all http to https
server {
listen 80;
return 301 https://$host$request_uri;
}
or for a specific site like this
server {
server_name dreamingoftech.uk;
return 301 https://dreamingoftech.uk$request_uri;
}
but choose only one for your case
and then you make sure you node server is running on http mode and not https.
Also you mentioned that you run node on port 3000, then use port 3000 and not 2000 as I can see in your config.
After you confirm the above redirect all packets into localhost like this
server {
listen 443;
server_name dreamingoftech.uk;
ssl_certificate /etc/letsencrypt/live/dreamingoftech.uk/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/dreamingoftech.uk/privkey.pem;
ssl on;
ssl_session_cache builtin:1000 shared:SSL:10m;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers HIGH:!aNULL:!eNULL:!EXPORT:!CAMELLIA:!DES:!MD5:!PSK:!RC4;
ssl_prefer_server_ciphers on;
location / {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# Fix the “It appears that your reverse proxy set up is broken" error.
proxy_pass http://localhost:3000;
proxy_read_timeout 90s;
proxy_redirect http://localhost:3000 https://dreamingoftech.uk;
}
}
Create a file and sum the above code put it in sites-available with a name like dreamingoftech.uk and the use ln -s to create a softlink into sites-enabled. go to your nginx.conf and make sure you include folder sites-enabled
Then must restart nginx to check if it works
#Stamos Thanks for your reply. I tried that but unfortunately it didn't work. I decided to try the most basic node app I could still using the basic modules I am using.
I tried this and it worked straight away.
The problem is with my app therefore. I will spend time rebuilding and testing step by step until I find the issue,
Thanks for your time!

Socket.io disconnects on http request

I'm running nginx for my node.js application on a AWS EC2 instance. I want to use websockets (socket.io) and normal http request/response. My problem is, whenever I have an active socket connection from my mobile device to the server and try to make a normal http request, the mobile device's socket.io error function is called with the message "502 Bad Gateway".
Only socket works. Only normal http request works as well.
I figured out, that this problem occurred after I setup nginx to use https only.
Here is my nginx config in /sites-enabled /sites-available:
server {
listen 80;
listen 443 ssl;
server_name example.com www.example.com;
ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;
include /etc/letsencrypt/options-ssl-nginx.conf;
if ($scheme != "https") {
rewrite ^ https://$host$request_uri? permanent;
}
location / {
proxy_pass http://127.0.0.1:3000;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_http_version 1.1;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host;
}
}
Nginx error log:
[error] 14117#14117: *50 upstream prematurely closed connection while reading response header from upstream, client: 78.94.9.226, server: example.com, request: "GET /socket.io/?transport=polling&b64=1&sid=rgXMQhL6mbSET8ktAAAA HTTP/1.1", upstream: "http://127.0.0.1:3000/socket.io/?transport=polling&b64=1&sid=rgXMQhL6mbSET8ktAAAA", host: "example.com"
iOS error log:
LOG SocketIOClient: Handling event: error with data: ["Got unknown error from server <html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>nginx/1.10.3 (Ubuntu)</center>\r\n</body>\r\n</html>\r\n"]
If you need any more information, let me know!
I fixed the problem by myself. It was a really dumbass problem. I created a file inside my node.js server folder called access.log and told the morgan logger to write into the file. The thing I forgot was, that I'm using PM2 to restart the server whenever there is a change in code inside the server folder. So PM2 restarted the server every time I made a http request and the socket disconnected.
Change your nginx config into two blocks
server {
listen 80;
listen 443 ssl;
server_name example.com www.example.com;
ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;
include /etc/letsencrypt/options-ssl-nginx.conf;
if ($scheme != "https") {
rewrite ^ https://$host$request_uri? permanent;
}
location /socket.io {
proxy_pass http://127.0.0.1:3000;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_http_version 1.1;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host;
}
location / {
proxy_pass http://127.0.0.1:3000;
proxy_http_version 1.1;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host;
}
}
You want to only upgrade connection for socket.io and not other urls

Resources