Chrome net::ERR_HTTP2_PROTOCOL_ERROR 200 after a reconnect - node.js

I'm using Node server with an express app which handles a Server Sent Events stream. This is proxied via NginX with http2 enabled. The SSE events are consumed via EventSource in a React app. I'm sending a heartbeat message every 10 seconds to keep the connection alive.
This all works great until there is some form of network interruption such as putting my laptop to sleep then re-awaking it.
Then from that point on the stream will error every 40 or so seconds with the net::ERR_HTTP2_PROTOCOL_ERROR 200 error and then reconnect instead of just reconnecting once with a steady stream.
Firefox works correctly. It doesn't error and reconnects only once.
If I configure Node to serve http2 directly instead of via NGinx as a test (via spdy library) then all works as expected so I don't think this is a Node issue and I must be missing something with my Nginx configuration and Chrome.
Nginx config as follows (location /stream is the SSE proxy)
server {
listen 28443 ssl http2;
listen [::]:28443 ssl http2;
server_name example.com;
ssl on;
ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;
http2_max_field_size 16k;
http2_max_header_size 128k;
root /var/www/example.com;
index index.html index.htm;
location / {
client_max_body_size 100M;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_pass http://localhost:28080;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'Upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
location /stream {
proxy_http_version 1.1;
# proxy_request_buffering off;
# proxy_buffering off;
# proxy_cache off;
# chunked_transfer_encoding off;
proxy_set_header Connection '';
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_pass http://localhost:28080/stream;
}
}
I've tried various combinations with proxy_buffers off and keepalive settings ect.. which seems to only affect the time between errors i.e. 5 minutes instead of 40 seconds.

Not sure if you figure this out. I got the same issue recently, by increasing the size:
http2_max_field_size 64k;
http2_max_header_size 512k;
The chrome's net::ERR_HTTP2_PROTOCOL_ERROR has gone.
Also, not sure if it applied to you. If I use firefox, I can actually visit my site correctly.

http2_max_field_size and http2_max_header_size directives are obsolete since version 1.19.7
Instead, something like the following could be used:
large_client_header_buffers 4 64k;

Now token is named
In my environment just set to:
large_client_header_buffers 10 512k;
And error is gone
Source:
http://nginx.org/en/docs/http/ngx_http_core_module.html#large_client_header_buffers

Related

NodeJS + Nginx: busboy behind an nginx reverse proxy

If I spin up a nodejs server and use busboy then I am able to upload large files (10+ gbs) but when i use the same nodejs code and use nginx as a reverse proxy then nginx throws "413 request entity too large"
Has anyone encountered such issue? How do we solve this? I know i can set a "client_max_body_size" variable but this would mean there will still be a hard limit of the file.
My nginx config looks like the following:
server {
listen 80;
server_name *.example.local;
location /api {
proxy_pass http://host.docker.internal:5000;
proxy_pass_header Accept;
proxy_pass_header Server;
keepalive_requests 1000;
proxy_redirect off;
proxy_buffering off;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Ssl on;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $host;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_connect_timeout 600;
proxy_send_timeout 600;
proxy_read_timeout 600;
send_timeout 600;
client_max_body_size 100M;
}
}
client_max_body_size size;
Sets the maximum allowed size of the client request body. If the size
in a request exceeds the configured value, the 413 (Request Entity Too
Large) error is returned to the client. Please be aware that browsers
cannot correctly display this error. Setting size to 0 disables
checking of client request body size.
Source
In your nginx configuration you have client_max_body_size=100M. You need to adjust it according to your needs like 15G etc. This property of configuration defines the max size of body payload a client can send to Nginx. If payload size is greater than this, a 413 http status is sent to client. Setting it to 0 (as suggested by Taxel) will disable the payload size check but this will expose the server to outside attack where a malicious user keeps your server busy by sending random big files which can deteriorate server performance.
Answering my own question:
In order to disable request/proxy buffering in my nginx reverse proxy, I had to add the following directives in the nginx config file
proxy_buffering off;
proxy_request_buffering off;
So, now my nginx config file looks like the following:
server {
listen 80;
server_name *.example.local;
location /api {
proxy_pass http://host.docker.internal:5000;
proxy_pass_header Accept;
proxy_pass_header Server;
keepalive_requests 1000;
proxy_redirect off;
proxy_buffering off;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Ssl on;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $host;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_connect_timeout 600;
proxy_send_timeout 600;
proxy_read_timeout 600;
send_timeout 600;
client_max_body_size 0;
proxy_buffering off; # <-----
proxy_request_buffering off; # <-----
}
}

issues configuring nginx with node.js app

Trying to get nginx to proxy my node.js app and use a domain with it. I'm going to have many domains mapped to the server so i'm using separate .conf files for each server block. The issue i'm having right now is that I can only seem to get the default nginx page to show up when i go to the domain. I'll try to explain the current setup as clearly as possible, and if you need any more information please let me know.
nginx.conf changes
I set the root path to where my apps files are, root /var/www; so for example, an app would be deployed to the folder /var/www/example.com.
server block config
I created a new file for the server block /etc/nginx/conf.d/example_com.conf which contains
server
{
listen 80;
listen [::]:80;
server_name example.com www.example.com;
location /var/www
{
proxy_pass http://localhost:3103;
include /etc/nginx/proxy_params;
}
}
please note that going to my http://myip:3103 renders the app as it should and the file /etc/nginx/proxy_params contains
proxy_buffers 16 32k;
proxy_buffer_size 64k;
proxy_busy_buffers_size 128k;
proxy_cache_bypass $http_pragma $http_authorization;
proxy_connect_timeout 59s;
proxy_hide_header X-Powered-By;
proxy_http_version 1.1;
proxy_ignore_headers Cache-Control Expires;
proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504 http_404;
proxy_no_cache $http_pragma $http_authorization;
proxy_pass_header Set-Cookie;
proxy_read_timeout 600;
proxy_redirect off;
proxy_send_timeout 600;
proxy_temp_file_write_size 64k;
proxy_set_header Accept-Encoding '';
proxy_set_header Cookie $http_cookie;
proxy_set_header Host $host;
proxy_set_header Proxy '';
proxy_set_header Referer $http_referer;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Original-Request $request_uri;
Is there anything I am doing wrong here? Do you need any more info? Please let me know! nginx is pretty new for me and i feel like i'm super close i'm just note understanding something. Thanks!
Your configuration provides requests processing like this:
requests to http://[www.]example.com/var/www[*] will be proxy_passed to
you app
all other requests will be processed like static/direct
requests in default nginx directory
If you haven't static files and all request has to processing by app, then you should fix your configuration like this:
location /
{
proxy_pass http://localhost:3103;
include /etc/nginx/proxy_params;
}
If you have static files that can be served by nginx, then you should to complicate you configuration a bit like here or here.
Here is documentation for understanding how to nginx works.

Binary stream upload through NGINX blocks all other POST requests to Node.js app

I'm running an application which accepts large file uploads, but also should allow the user to make other POST requests to the same app while the upload is it progress.
The app is running on node.js, with the upload being handled by formidable and s3stream to directly stream the binary content to S3 without using disk space on the server.
It seems like this is an NGINX configuration issue, as everything else works just fine locally.
My current NGINX config looks like this:
server {
listen 443 ssl http2;
server_name upload.app.io;
underscores_in_headers on;
location / {
proxy_request_buffering off;
proxy_pass http://localhost:3000;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
ssl_certificate /etc/nginx/ssl/ca-bundle.crt;
ssl_certificate_key /etc/nginx/ssl/server.key;
}
So why is it that NGINX blocks any parallel POST requests while the upload is in progress, and how do I overcome this problem?
To clarify, NGINX does not throw an error, but rather hangs the requests until it times out.
Greatly appreciate any help with this.
I've since figured out one approach to handling the problem. It may not be the best practice but it works for what I need.
I'm now using two separate location blocks to handle the requests. One for file uploads, and another one for everything else. This way, I'm able to using the /upload block without buffering directly streamed to S3 via the app, and have the / block handle all other requests normally with buffers.
Here's the config:
server {
listen 443 ssl http2;
server_name upload.app.io;
underscores_in_headers on;
location /upload {
client_max_body_size 0;
proxy_request_buffering off;
proxy_pass http://localhost:3000;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
location / {
proxy_pass http://localhost:3000;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
ssl_certificate /etc/nginx/ssl/ca-bundle.crt;
ssl_certificate_key /etc/nginx/ssl/server.key;
}
client_max_body_size 10m;enter link description here

socket io with node js is not connecting through nginx proxy_pass

The issue is when i use nginx the socket is not establishing.Any one help me in steps to follow to integrate Socket oi with nginx.
i tried this
location /field {
# the following is required for WebSockets
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header X-NginX-Proxy true;
# supposedly prevents 502 bad gateway error;
# ultimately not necessary in my case
proxy_buffers 8 32k;
proxy_buffer_size 64k;
# the following is required
proxy_pass
proxy_redirect off;
# the following is required as well for WebSockets
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
tcp_nodelay on; # not necessary
}
First of all check your nginx version. According to that page websockets supported after v1.3.13;
http://nginx.com/blog/nginx-nodejs-websockets-socketio/
Then compare your nginx conf with the below configuration as stated in the nginx blog;
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}
upstream websocket {
server 192.168.100.10:8010;
}
server {
listen 8020;
location / {
proxy_pass http://websocket;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
}
}
http://nginx.com/blog/websocket-nginx/
Also check your firewall configuration for the port you chose for socket.io server. (I've read that some ISPs are blocking websocket connection for ports other than 80 & 443, please also check if server receive packets using tcpdump etc.)
If everyhing is okay up until now, check your nginx error logs (/var/log/nginx/error.log) to see if there are any socket.io related error messages. You can paste it here for further analysis.
Then if there is no socket error in nginx logs start your node app with DEBUG mode on as below;
DEBUG=* node yourfile.js
And check if any socket connection message is printed to console. You can also paste it for further analysis.

nginx to cdn or node.js

I have a simple example running ngnix that proxies to my node.js app running on localhost:3001. Now I want to add some optimizations and the problem is I'm not sure I completely understand the way ngnix config files work.
What I want to do is to serve index.html, about.html and main.js from the CDN via a proxy-forward through ngnix. I imagine I need to add something like a rewrite just for those two files (and an entire images and css directory eventually)
So user goes to mydomain.com .. ngnix kicks in and delivers index.html from cdn.mydomain.com/index.html.
Here is what I have now:
===================
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
client_max_body_size 10m;
client_body_buffer_size 128k;
proxy_connect_timeout 600;
proxy_send_timeout 600;
proxy_read_timeout 600;
proxy_buffer_size 4k;
proxy_buffers 4 32k;
proxy_busy_buffers_size 64k;
proxy_temp_file_write_size 64k;
send_timeout 600;
proxy_buffering off;
####
# the IP(s) on which your node server is running i choose the port 3001
upstream app_yourdomian {
server 127.0.0.1:3001;
}
# the nginx server instance
server {
listen 0.0.0.0:80;
server_name ec2-75-101-203-200.compute-1.amazonaws.com ec2-75-101-203-200.compute-1.amazonaws;
access_log /var/log/nginx/yourdomain.log;
# pass the request to the node.js server with the correct headers and much more can be added, see nginx config options
location / {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header X-NginX-Proxy true;
proxy_pass http://localhost:3001;
proxy_redirect off;
}
}
============================
If you really need to proxy (not redirect to) index, about and main.js, then it would be something like having three more simple locations for each of the above
location = /index.html {
proxy_pass ...
}
You might also want to take a look at http://wiki.nginx.org/HttpCoreModule#location
For locations without regex the most specific match is used.
Feel free to ask more in the mailing list http://mailman.nginx.org/mailman/listinfo/nginx

Resources