EPIPE Error with ExpressJS, nginx proxy server - node.js

I am running multiple ExpressJS Node apps through an Nginx proxy server, and am getting an EPIPE Error thrown whenever my users try to download a file. This does not happen on my local setup (which is identical to the server's except for the proxy server), so I figure it has something to do with my Nginx configuration.
Here are my Nginx configs:
/etc/nginx/nginx.conf
user www www;
worker_processes 1;
error_log /home/alex/logs/error.log;
pid /var/run/nginx.pid;
worker_rlimit_nofile 8192;
events {
worker_connections 4096; ## Default: 1024
}
http {
include mime.types;
index index.html index.htm index.php;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] $status '
'"$request" $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /home/alex/logs/access.log main;
sendfile on;
tcp_nopush on;
gzip on;
server_names_hash_bucket_size 128; # this seems to be required for some vhosts
include /etc/nginx/conf.d/*.conf;
}
/etc/nginx/conf.d/default.conf
server {
listen 80;
server_name example.com;
# log access and stuff
access_log /home/alex/logs/example-site.log main;
# Load configuration files for the default server block.
include /etc/nginx/default.d/*.conf;
# Proxy to the NodeJS server
location / {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_pass http://127.0.0.1:8999;
}
# redirect server error pages to their HTML
include /etc/nginx/errpages.conf;
}
The ExpressJS server is sending the download using the following code:
app.get('/citrite/p/:patch', function(req, res)
{
if(set.citFiles.indexOf(req.params.patch) == -1)
{
res.send(mbuild.get404());
}
else
{
track.incrViewcount(req.params.patch, 'citrite');
res.download(set.citDir + '/' + req.params.patch, files.doneSaving);
}
});
That code and everything else works fine on my local git repo, but when I push from there and pull on the server-side, the site kicks and screams - it times out on the user's end and gives me an EPIPE error in the console. I am running Node.js version 4.2.1 and ExpressJS version 4.13.3.

I figured out what the problem was: apparently, having sendfile set to on is what was causing the downloads to stall, and turning that off (specifically, removing the directive in the config) fixed the issue. Not exactly sure why this would interfere, but getting rid of the setting cleared things up.

Related

Is it dangerous opening port 3000 of the server?

I want to deploy my Angular + NodeJS application. My NodeJS application runs on http://localhost:3000 on the server. And my Angular application tries to send it's requests to the server with this prefix address: http://server.ip.address:3000. I opened the port 3000 of the server with the following commands to help my program works and it works fine by now.
irewall-cmd --zone=public --add-port=3000/tcp --permanent
firewall-cmd --reload
But I am not sure if I did a good job or not?
My Angular app runs on nginx and my NodeJS app runs on PM2. I also tried to setting a reverse proxy as you can see below inside etc/nginx/nginx.conf, but it didn't work and just opening port 3000 worked for me!
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;
# Load dynamic modules. See /usr/share/doc/nginx/README.dynamic.
include /usr/share/nginx/modules/*.conf;
events {
worker_connections 1024;
}
http {
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
# Load modular configuration files from the /etc/nginx/conf.d directory.
# See http://nginx.org/en/docs/ngx_core_module.html#include
# for more information.
include /etc/nginx/conf.d/*.conf;
server {
listen 80 default_server;
listen [::]:80 default_server;
server_name _;
root /demo/stock-front9/dist/strategy;
# Load configuration files for the default server block.
include /etc/nginx/default.d/*.conf;
location / {
#proxy_pass http://localhost:3000;
#proxy_http_version 1.1;
# First attempt to serve request as file, then
# as directory, then redirect to index(angular) if no file found.
try_files $uri $uri/ /index.html;
}
error_page 404 /404.html;
location = /40x.html {
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
}
}
What is the best way to deploy Angular + NodeJS application and how can I do it?
You can deploy the app by just assigning port to process.env.PORT, and put the whole angular build in a public/src folder and give the public folder path in node server file.
You can take reference here https://github.com/Ris-gupta/chat-application
There's no best way but there some best practices. Opening port directly on a server is not good solution. I would suggest you to use docker and publish your application inside container with NGINX. Also you can deploy your backend server in same way.

Should I open port 3000 of my server to serve NodeJS application?

I have an Angular + Node.JS app. When I was running the program locally I defined a baseurl = http://localhost:3000/ in my Angular app and used this prefix for accessing to my NodeJS backend in my program defined links, but now when I wanted to deploy my app on a remote server, I changed the baseurldefinition to the baseurl = http://111.222.333.444:3000/(111.222.333.444 is my server ip address for example), but it doesn't work!
How should I connect my Angular app to the NodeServer on a remote server?
EDIT: This is my /etc/nginx/nginx.conf file content:
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;
# Load dynamic modules. See /usr/share/doc/nginx/README.dynamic.
include /usr/share/nginx/modules/*.conf;
events {
worker_connections 1024;
}
http {
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
# Load modular configuration files from the /etc/nginx/conf.d directory.
# See http://nginx.org/en/docs/ngx_core_module.html#include
# for more information.
include /etc/nginx/conf.d/*.conf;
server {
listen 80 default_server;
listen [::]:80 default_server;
server_name _;
root /demo/stock-front9/dist/strategy;
# Load configuration files for the default server block.
include /etc/nginx/default.d/*.conf;
location / {
#proxy_pass http://localhost:3000;
#proxy_http_version 1.1;
# First attempt to serve request as file, then
# as directory, then redirect to index(angular) if no file found.
try_files $uri $uri/ /index.html;
}
error_page 404 /404.html;
location = /40x.html {
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
}
}
I would not, I think is better to run the Node app with a tool like PM2 and then place a reverse proxy using Nginx in front of it, PM2 will act as orchestrator over your service while Nginx will provide access only through standard web ports (80, 443).
And in the case of Angular, when compiling, it should generate a static web app which you can serve using the same Nginx reverse proxy, doing it like so you'll save yourself the effort of configuring things like CORS, API routes and so forth, everything will go through Nginx.
Update on an example of Nginx config file
server {
listen 80;
server_name example.org;
location /api {
proxy_pass http://localhost:3000;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_redirect off;
proxy_http_version 1.1;
}
location / {
root /path/to/angular/compiled/app;
index index.html;
}
}
And then the angular app should point to the same host.
Good luck and cheers :)
You can still run your angular app locally. And for backend server, you can use proxy.
Please take a look at this.
https://github.com/angular/angular-cli/blob/master/docs/documentation/stories/proxy.md#using-corporate-proxy

Nginx on ec2 instance does not serve static files

What was working
I have MERN stack dashboard up and running on AWS EC2 instance.
Ec2 instance is at ec2...aws.com
React app is served on port 5004.
Node app is running on port 5003.
What was changed
The problem started when I set up the domain at GoDaddy to forward (with masking)
from somebusiness.com to ec2.....aws.com.
First, I set up NGINX reverse proxy to serve / as my react app (frontend) and /api as my node app (backend)
(you will find nginx code down below later)
http://somebusiness.com/ <- opens up React app and is working correctly
http://somebusiness.com/api/heartbeat <- not working as I expected (heartbeat is just endpoint to check if the app is alive) but for some reason, does not return application/JSON but rather a text/HTML webpage with the correct 'real URL' inside a frame of some kind:
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>
  <title>Somebusiness Dashboard</title>
  <meta name="description" content="">
  <meta name="keywords" content="">
</head>
<frameset rows="100%,*" border="0">
  <frame src="http://ec2-...aws.com/api/heartbeat" frameborder="0" />
</frameset>
</html>
The actual URL in the frame is the one that is working as expected:
http://ec2-...aws.com/api/heartbeat <- is working correctly
I think GoDaddy has something to do with this because on GoDaddy I can specify title, description, and keywords for this forwarding. So as backend, I'm forced to use http://ec2-...aws.com/api/ as my backend URL, which is OK for now (it is my secondary problem at the moment since it's just convenient to use the domain name for backend as well)
Main Problem
if i send a request to
http://ec2.....aws.com:5003/uploads/avatars/user1.jpg
the image is loaded OK.
So if i send a request to
http://ec2.....aws.com/api/uploads/avatars/user1.jpg
the image is NOT loaded.
So to recap: http://ec2-...aws.com/api/ is working OK for routes and requests, but not for serving static files. That leads me to believe my nginx setup is wrong, and I've spent countless hours trying all kinds of different setups. Here I present where i left it:
Source & code for help
Since I'm using Amazon Linux 2, I've used its tools to install nginx1 and set-up reverse proxy.
When i used NGINX on my other VPS, there was a bit different structure to it (sites-enabled & sites-available folders). But on this nginx there is only nginx.conf file where I did the setup. (I guess it doesn't make difference, just wanted to note that)
# For more information on configuration, see:
# * Official English Documentation: http://nginx.org/en/docs/
# * Official Russian Documentation: http://nginx.org/ru/docs/
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;
# Load dynamic modules. See /usr/share/doc/nginx/README.dynamic.
include /usr/share/nginx/modules/*.conf;
events {
worker_connections 1024;
}
http {
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
# Load modular configuration files from the /etc/nginx/conf.d directory.
# See http://nginx.org/en/docs/ngx_core_module.html#include
# for more information.
include /etc/nginx/conf.d/*.conf;
server {
listen 80 default_server;
listen [::]:80 default_server;
server_name eyeratereviews.com;
root /home/ec2-user/somebusiness-web-backend;
# Load configuration files for the default server block.
include /etc/nginx/default.d/*.conf;
include /etc/nginx/mime.types;
location / {
proxy_pass http://127.0.0.1:5004;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
location /api {
root /home/ec2-user/somebusiness-web-backend/uploads/avatars;
default_type application/json;
proxy_pass http://127.0.0.1:5003;
# Following is necessary for Websocket support
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
error_page 404 /404.html;
location = /40x.html {
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
}
}
}
My node.js project folder structure is like this
...
controllers/
models/
static/
- pictures/
-- logo.png
uploads/
- avatars/
-- user1.jpg
-- user2.jpg
-- ...
Your existing code does not attempt to remove the /api from the front of the URI before passing it upstream. Add a trailing / to both the location and proxy_pass values. For example:
location /api/ {
proxy_pass http://127.0.0.1:5003/;
...
}
See this document for details.
Alternatively, use a rewrite...break to modify the URI instead.
For example:
location /api {
rewrite ^/api(.*)$ $1 break;
proxy_pass http://127.0.0.1:5003;
...
}
See this document for details.

Upstream Node server closing connection to nginx

I'm using nginx as a proxy for a Node server that's rate-limiting requests. The rate is one request every 30 seconds; most requests return a response fine, but if a request is kept open for an extended period of time, I get this:
upstream prematurely closed connection while reading response header from upstream
I cannot figure out what might be causing this. Below is my nginx configuration:
# For more information on configuration, see:
# * Official English Documentation: http://nginx.org/en/docs/
# * Official Russian Documentation: http://nginx.org/ru/docs/
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;
# Load dynamic modules. See /usr/share/nginx/README.dynamic.
# include /usr/share/nginx/modules/*.conf;
events {
worker_connections 1024;
}
http {
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
# Load modular configuration files from the /etc/nginx/conf.d directory.
# See http://nginx.org/en/docs/ngx_core_module.html#include
# for more information.
include /etc/nginx/conf.d/*.conf;
server {
listen 80 default_server;
listen [::]:80 default_server;
server_name _;
root /srv/www/main/htdocs;
# Load configuration files for the default server block.
include /etc/nginx/default.d/*.conf;
location /vcheck {
proxy_pass http://127.0.0.1:8080$is_args$query_string;
# proxy_buffer_size 128k;
# proxy_buffers 4 256k;
# proxy_busy_buffers_size 256k;
# proxy_http_version 1.1;
# proxy_set_header Upgrade $http_upgrade;
# proxy_set_header Connection 'upgrade';
# proxy_set_header Host $host;
# proxy_cache_bypass $http_upgrade;
# proxy_redirect off;
proxy_read_timeout 600s;
}
location ~ \.php$ {
include fastcgi.conf;
fastcgi_split_path_info ^(.+\.php)(/.+)$;
fastcgi_pass unix:/var/run/php-fpm/php-fpm.sock;
fastcgi_index routes.php$is_args$query_string;
}
location / {
if (-f $request_filename) {
expires max;
break;
}
if ($request_filename !~ "\.(js|htc|ico|gif|jpg|png|css)$") {
rewrite ^(.*) /routes.php last;
}
}
}
}
Is there a reason why Node could be closing the connection early?
EDIT: I'm using Node's built-in HTTP server.
Seems like You've to extend response timeout of nodejs application.
So if it's expressjs app so I can guess You try this one:
install: npm i --save connect-timeout
use:
var timeout = require('connect-timeout');
app.use(timeout('60s'));
But I recommend to not to keep connection waiting and fix issue in nodejs app, find why it's halting so long.
Seems like nodejs app has issues that cannot respond and request is getting lost keeping nginx waiting.

Nginx caching when serving static content

I'm serving a directory statically with Nginx, using the following site config, and works well:
server {
listen 80;
server_name spa.me.com;
access_log /var/log/nginx/spa-access.log;
error_log /var/log/nginx/spa-error.log;
location / {
autoindex on;
root /home/ubuntu/spa/dist/;
}
}
Now I want to add caching feature to this server. So I've read some posts and:
Declare a proxy_cache_path.
Use it from my server.
Add a X-Cached header to tell the client the cache status.
So now my config is as follows:
proxy_cache_path /var/cache/nginx/spa levels=1:2 keys_zone=spa:10m inactive=15m;
server {
listen 80;
server_name spa.mediasmart.mobi;
access_log /var/log/nginx/spa-access.log;
error_log /var/log/nginx/spa-error.log;
location / {
proxy_cache spa;
add_header X-Cached $upstream_cache_status;
autoindex on;
root /home/ubuntu/spa/dist/;
}
}
For some reason, the X-Cache header is never sent, so I'm not able to know (using curl, for instance) if a resource is recovered from the cache or not. Furthermore, the cache directory is empty, so it seems no caching feature has been really enabled.
I've also tried to add the $upstream_cache_status to my logs file, creating a custom format and using it:
...
log_format spa_log_format '$remote_addr - $upstream_cache_status [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent"';
...
server {
...
access_log /var/log/nginx/spa-access.log spa_log_format;
...
}
But $upstream_cache_status never has a value. Always an empty string is printed on its place in the log.
What I'm doing wrong? Thanks

Resources