Over the last few days, I was trying to understand the Istio retry policy. Then, I found a field named "retry-on". The default value of this field is below (I use version 1.14.3 Istio).
RetryOn: "connect-failure,refused-stream,unavailable,cancelled,retriable-status-codes"
link to the source code
I want to know which case is included in "connect-failure". From the document, They explain like this.
connect-failure
Envoy will attempt a retry if a request is failed because of a connection failure to the upstream server (connect timeout, etc.). (Included in 5xx)
NOTE: A connection failure/timeout is a the TCP level, not the request level. This does not include upstream request timeouts specified via x-envoy-upstream-rq-timeout-ms or via route configuration or via virtual host retry policy.
link to doc
So, I think it will retry if any error occurs in the TCP protocol at the transport layer. I tried to prove that by creating 2 pods in the Kubernetes cluster. First is Nginx forwarding every HTTP request to the Second. The Second is the NodeJS TCP server that will break the TCP connection if you send an HTTP request with "/error" path. I show it below.
Nginx
user nginx;
error_log /var/log/nginx/error.log warn;
events {
worker_connections 65535;
use epoll;
multi_accept on;
}
http {
log_format main escape=json
'{'
'"clientIP":"$remote_addr",'
'"time-local":"$time_local",'
'"server-port":"$server_port",'
'"message":"$request",'
'"statusCode":"$status",'
'"dataLength":"$body_bytes_sent",'
'"referer":"$http_referer",'
'"userAgent":"$http_user_agent",'
'"xForwardedFor":"$http_x_forwarded_for",'
'"upstream-response-time":"$upstream_response_time",'
'"correlation-id":"$http_x_correlation_id",'
'"user-tier":"$http_x_neo_user_tier",'
'"session-id":"$http_x_session_id"'
'}';
access_log /var/log/nginx/access.log main;
client_max_body_size 100m;
client_header_timeout 5m; # default 60s
client_body_timeout 5m; # default 60s
send_timeout 5m; # default 60s
proxy_connect_timeout 5m;
proxy_send_timeout 5m;
proxy_read_timeout 5m;
server {
listen 8080;
location / {
proxy_pass http://ice-node-service.neo-platform.svc.cluster.local:8080;
}
}
}
NodeJS
var net = require('net');
var server = net.createServer();
server.listen(8080, '127.0.0.1');
server.addListener('close', () => {
console.log('close');
})
server.addListener('connection', socket => {
console.log('connect');
socket.addListener('data', data => {
try {
const [method, path] = data.toString().split("\n")[0].split(" ")
console.log(method, path);
if (path === "/error") {
socket.destroy(new Error("force error"))
} else {
socket.write(respond())
socket.end()
}
} catch (e) {
console.log(e);
}
})
})
server.addListener('error', err => {
console.log('error', err);
})
server.addListener('listening', () => {
console.log('listening');
})
function respond() {
const body = `<html><body>Hello</body></html>`
return `HTTP/1.1 200 OK
Date: ${new Date().toGMTString()}
Server: Apache
Last-Modified: Tue, 01 Dec 2009 20:18:22 GMT
ETag: "51142bc1-7449-479b075b2891b"
Accept-Ranges: bytes
Content-Length: ${body.length + 2}
Content-Type: text/html
${body}\r\n`
}
So, I tried to send a request through Nginx to the NodeJS server on "/error" path. I expected Istio would resend the request if the TCP connection is broken. But, It wasn't retried. So, I want to know why it wasn't.
I'm trying to deploy a MERN app to a DigitalOcean droplet using Nginx (I've used a connection string to connect to a MongoDB atlas instance). The frontend static files seem to successfully work, but I face two issues when it comes to the backend (a Nodejs/express app).
First, I get a connect() failed (111: connection refused) for every request from the frontend to the backend.
2021/07/12 23:54:32 [error] 5101#5101: *394 connect() failed (111: Connection refused) while
connecting to upstream, client: "My Ip", server: "example.com", request: "GET
/api/images HTTP/1.1", upstream: "http://127.0.0.1:5000/images", host: "example.com",
referrer: "https://example.com/"
Second, I face a 404 Not Found nginx/1.14.0 (Ubuntu) error while trying to check the routes through the Postman.
Not sure both of these are the same problem.
Here's my Nginx configuration in /etc/nginx/sites-available/default:
server {
root var/www/html;
index index.html index.htm index.nginx-debian.html;
server_name example.com www.example.com;
location /api/ {
proxy_pass http://localhost:5000/;
}
location / {
root /home/${my_non_root_user}/frontend/deploy;
try_files $uri /index.html;
}
listen [::]:443 SSL ipv6only=on; #managed by Certbot
listen 443 SSL; #managed by Certbot
ssl_certificate /etc/letsencrypt/live/${example.com}/fullchain.pem; #managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/${example.com}/privkey.pem; #managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; #managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; #managed by Certbot
}
server {
if ($host = www.example.com) {
return 301 https://$host$request_uri;
} #managed by Certbot
if ($host = example.com) {
return 301 https://$host$request_uri;
} #managed by Certbot
listen 80 default_server;
listen [::]:80 default_server;
server_name example.com www.example.com;
return 404; #managed by Certbot
}
This is how my backend server.js looks like:
import express from 'express'
import path from 'path'
import dotenv from 'dotenv'
import connectDB from './config/db.js'
import memberRoutes from './routes/memberRoutes.js'
import messageRoutes from './routes/messageRoutes.js'
import fileRoutes from './routes/fileRoutes.js'
dotenv.config()
connectDB()
const app = express()
app.use(express.json())
// routes
app.use('/api/members', memberRoutes)
app.use('/api/messages', messageRoutes)
app.use('/api/files', fileRoutes)
const __dirname = path.resolve()
app.use('/files', express.static(path.join(__dirname, '/files')))
const PORT = process.env.PORT || 5000
app.listen(
PORT,
console.log(`Server running in ${process.env.NODE_ENV} mode on port
${PORT}`)
)
When I make a request in Postman, to {{URL}}/messages (No difference it is a GET, or POST), where {{URL}} is THE_IP_TO_DROPLET, I get this output:
<HTML>
<head>
<title>404 Not Found</title>
</head>
<body bgcolor="white">
<center>
<h1>404 Not Found</h1>
</center>
<hr>
<center>nginx/1.14.0 (Ubuntu)</center>
</body>
</html>
Do you have any suggestion? Totally stuck in this for the whole weekend!
Thank you.
I have a server running Ubuntu 20.04, set up with an SSH Key, and I have installed Nano using Docker -
docker run --restart=unless-stopped -d -p 7075:7075/udp -p 7075:7075 -p 127.0.0.1:7076:7076 -p 127.0.0.1:7078:7078 -v /root/nano/:/root --name nano nanocurrency/nano:latest
On the VPS GUI ports 22, 443, and 7075 are open for inbound traffic.
I have a working WebSocket running on localhost:7078 which returns a stream of data to the terminal.
Nginx version 1.18.0 is installed and when I run systemctl status nginx it claims to be active with no errors, but when I try to connect from my laptop nothing happens.
Here's my Nginx config -
server {
listen 443;
# host name to respond to
server_name 000.000.00.000;
location / {
# switch off logging
access_log off;
# redirect all HTTP traffic to localhost:7080
proxy_pass http://localhost:7080;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# WebSocket support (nginx 1.4)
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
}
And node.js Websocket -
const WS = require('ws');
const ReconnectingWebSocket = require('reconnecting-websocket');
// Create a reconnecting WebSocket.
// In this example, we wait a maximum of 2 seconds before retrying.
const ws = new ReconnectingWebSocket('ws://localhost:7078', [], {
WebSocket: WS,
connectionTimeout: 1000,
maxRetries: 100000,
maxReconnectionDelay: 2000
});
// As soon as we connect, subscribe to block confirmations
ws.onopen = () => {
const subscription = {
"action": "subscribe",
"topic": "confirmation"
}
ws.send(JSON.stringify(subscription));
};
// The node sent us a confirmation
ws.onmessage = msg => {
console.log(msg.data);
if (msg.data.topic === "confirmation") {
console.log ('Confirmed', data.message.hash)
}
};
I am trying to proxy pass a node.js-socket.io app with nginx.
The client is an html file with some javascript in it;
<html>
<head>
<script src="socket.io.js"></script>
<script>
var socket = io('http://localhost:80');
socket.on('welcome', function(data){
console.log('Server says:' + data);
socket.emit('client-response', 'thank you!');
});
</script>
</head>
<body>
Socket.io
</body>
</html>
And the server block that supposed to proxy pass in nginx.conf file is this;
server {
listen 80;
listen [::]:80;
#root /usr/share/nginx/html;
include /etc/nginx/default.d/*.conf;
location / {
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
proxy_pass "http://localhost:2156";
}
error_page 404 /404.html;
location = /40x.html {
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
}
}
I have my node.js app up and running in port "2156".
When I test this, the client tries to reach the socket.io on the port 80 and fails with a 404 error (Because nginx was supposed the pass the proxy to the port 2156 but it didn't).
What am I missing here?
Edit: I've changed the client to connect at "http://localhost/socket.io/" and rewrote the nginx.conf like this:
server {
listen 80 default_server;
listen [::]:80 default_server;
root /usr/share/nginx/html;
location /socket.io/ {
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
proxy_set_header Host $host;
proxy_pass http://localhost:2156/socket.io/;
}
error_page 404 /404.html;
location = /40x.html {
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
}
}
And it worked.
I have some issues while trying to publish a nuxtjs site. Usually, I was using the generate command, but for this one I need to go full SSR, so I'm going for nuxt start.
But after building and starting the app, it's a mess. The build goes perfectly in the console, and the application start. The problem is when I try to access the site, it loads partially, but I got all these errors in the browser:
manifest.3a7efd91c5f63f114507.js
Failed to load resource: the server responded with a status of 404 ()
vendor.7519259bf7bdf608079e.js
Failed to load resource: the server responded with a status of 404 ()
app.a5cb9356f53306f973dc.js
Failed to load resource: the server responded with a status of 404 ()
default.1f3ad14df16ee86595af.js
Failed to load resource: the server responded with a status of 404 ()
index.260dc65b69022a31ad58.js
Failed to load resource: the server responded with a status of 404 ()
/_nuxt/pages/spot/_slug.e57cc2e78d8e0b160fe7.js
Failed to load resource: the server responded with a status of 404 ()
manifest.3a7efd91c5f63f114507.js
Failed to load resource: the server responded with a status of 404 ()
default.1f3ad14df16ee86595af.js
Failed to load resource: the server responded with a status of 404 ()
index.260dc65b69022a31ad58.js
Failed to load resource: the server responded with a status of 404 ()
vendor.7519259bf7bdf608079e.js
Failed to load resource: the server responded with a status of 404 ()
app.a5cb9356f53306f973dc.js
Failed to load resource: the server responded with a status of 404 ()
Nothing seems wrong during the build. When I use nuxt start, I get this:
$ nuxt start
nuxt:axios BaseURL: http://localhost:3042/api (Browser: /api) +0ms
OPEN http://localhost:3042
Here's my server conf file:
# Site global
server {
listen 443 ssl http2;
server_name www.mywebsite.com;
access_log off;
location = /robots.txt { access_log off; log_not_found off; }
location = /favicon.ico { access_log off; log_not_found off; }
location / {
proxy_pass http://127.0.0.1:3042/;
include /etc/nginx/conf.d/proxy.conf;
root /var/www/mywebsite/site;
add_header Access-Control-Allow-Origin *;
}
location ~* ^.+.(jpg|jpeg|gif|css|png|js|ico|txt|srt|swf)$ {
root /var/www/mywebsite/site/;
expires 30d;
}
ssl_certificate /etc/letsencrypt/live/www.mywebsite.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/www.mywebsite.com/privkey.pem;
}
# Redirection
server {
listen 80;
server_name mywebsite.com www.mywebsite.com;
location / {
return 301 https://www.mywebsite.com$request_uri;
}
}
And my nuxt config file:
const pkg = require('./package')
module.exports = {
mode: 'universal',
loading: { color: '#bb2b4d' },
router: {
linkActiveClass: '-active',
base: '/'
},
css: ['#/assets/icons/css/icons.css', '#/assets/snickles/snickles.css'],
plugins: ['~plugins/vue-filters.js', '~plugins/vue-modal.js'],
minify: {
removeEmptyAttributes: false,
collapseWhitespace: true,
conservativeCollapse: true,
collapseBooleanAttributes: true,
removeTagWhitespace: false,
removeStyleLinkTypeAttributes: true
},
modules: [
'#nuxtjs/axios'
],
axios: {
},
env: {
api: {
spots: `https://rest.mywebsite.com/spots`
}
},
proxy: {
},
build: {
extend(config, ctx) {
// Run ESLint on save
if (ctx.isDev && ctx.isClient) {
config.module.rules.push({
enforce: 'pre',
test: /\.(js|vue)$/,
loader: 'eslint-loader',
exclude: /(node_modules)/
})
}
}
},
postcss: [require('autoprefixer')],
vendor: ['moment', 'vue-js-modal']
}
Did I forget anything?
The most strange part is that it works perfectly well when I do the same on my own pc and not on my server. I checked the npm and node versions, they are the same (latest to date). Also, if testing with a demo template from NuxtJS, it works perfectly with the exact same server configuration.
By the way, the server is a debian 8, with all packages up to date.
Thanks in advance for any hint.
Edit: If of any use, the error log:
2018/02/14 19:12:54 [error] 12981#12981: *239930 open() "/var/www/mywebsite/site/_nuxt/pages/spot/_slug.e57cc2e78d8e0b160fe7.js" failed (2: No such file or directory), client: xxx.xxx.xxx.xxx., server: www.mywebsite.com, request: "GET /_nuxt/pages/spot/_slug.e57cc2e78d8e0b160fe7.js HTTP/2.0", host: "www.mywebsite.com", referrer: "https://www.mywebsite.com/"
2018/02/14 19:12:57 [error] 12981#12981: *239930 open() "/var/www/mywebsite/site/_nuxt/manifest.3a7efd91c5f63f114507.js" failed (2: No such file or directory), client: xxx.xxx.xxx.xxx, server: www.mywebsite.com, request: "GET /_nuxt/manifest.3a7efd91c5f63f114507.js HTTP/2.0", host: "www.mywebsite.com", referrer: "https://www.mywebsite.com/"
2018/02/14 19:12:57 [error] 12981#12981: *239930 open() "/var/www/mywebsite/site/_nuxt/vendor.7519259bf7bdf608079e.js" failed (2: No such file or directory), client: xxx.xxx.xxx.xxx, server: www.mywebsite.com, request: "GET /_nuxt/vendor.7519259bf7bdf608079e.js HTTP/2.0", host: "www.mywebsite.com", referrer: "https://www.mywebsite.com/"
2018/02/14 19:12:57 [error] 12981#12981: *239930 open() "/var/www/mywebsite/site/_nuxt/app.a5cb9356f53306f973dc.js" failed (2: No such file or directory), client: xxx.xxx.xxx.xxx, server: www.mywebsite.com, request: "GET /_nuxt/app.a5cb9356f53306f973dc.js HTTP/2.0", host: "www.mywebsite.com", referrer: "https://www.mywebsite.com/"
Again, it’s working perfectly fine with other nuxt projects, with a similar configuration. Indeed it can’t find these files in this folder, as they’re not in it — which is perfectly normal. It’s up to the app to get the routes to these files, which it usually does pretty well, with the same directory output (as I said, it’s supposed not to be in a _nuxt folder).
Thanks.
This was tagged with nginx, so, here's the nginx way of solving the problem.
After the troubleshooting through the comments, you report receiving the following in your error_log:
2018/02/14 19:12:57 [error] 12981#12981: *239930 open() "/var/www/mywebsite/site/_nuxt/manifest.3a7efd91c5f63f114507.js" failed (2: No such file or directory), client: xxx.xxx.xxx.xxx, server: www.mywebsite.com, request: "GET /_nuxt/manifest.3a7efd91c5f63f114507.js HTTP/2.0", host: "www.mywebsite.com", referrer: "https://www.mywebsite.com/"
Subsequently, running find / -type f -name manifest.3a7efd91c5f63f114507.js, or similar, results in the corresponding file being located in /var/www/mywebsite/site/.nuxt/dist.
As such, your nginx configuration is wrong, because you make it look for these things in the incorrect folder — your config has root /var/www/mywebsite/site/; instead.
The proper way would may be to use a prefix-based location together with the alias directive:
location /_nuxt/ {
alias /var/www/mywebsite/site/.nuxt/dist/;
}
However, if /_nuxt/ has stuff that may have to be proxy_pass'ed to the upstream, and you want to continue using the pcre-based location you already have had in your original config, then an alternative solution like below is also an option (otherwise, you obviously would have to remove it as redundant, to make sure that the prior prefix-based location works, see http://nginx.org/r/location):
location ~* ^.+.(jpg|jpeg|gif|css|png|js|ico|txt|srt|swf)$ {
rewrite ^/_nuxt(/.*) $1 break;
root /var/www/mywebsite/site/.nuxt/dist/;
}