Nginx 502 Bad Gateway when Jenkins pipeline is running a docker for React app - node.js

I have an Ubuntu 18.04 server running in a Droplet (DigitalOcean) secured with SSL and using an Nginx reverse proxy. Also Jenkins in running in my server (not in any docker) and configured to be accessed under the domain I created for it: jenkins.testdomain.com (all these steps following DO docs)
So the goal is to manage the deployment of a Node.js-React application to my testdomain.com later, by now, I just want to create the dist folder generated, after the 'npm build', within the /var/lib/jenkins/workspace/ , just that.
By now, I'm able to access my jenkins.testdomain.com site alright, trigger the pipeline to start the process after pushing to my repo, and start to run the stages; but it's here when start to fail nginx, when the pipeline reaches the Deliver phase (let's read 'npm build' phase), sometimes in the Build phase ('npm install').
It's at this point, reading the Jenkins console output where I see when it gets stuck and eventually shows a 502 Bad Gateway error. I will need to run the command systemctl restart Jenkins in my server console, to have access again. After restarting, the pipeline resume the work and seems to get the job done :/
In the /var/log/nginx/error.log for nginx I can read:
*1 connect() failed (111: Connection refused) while connecting to upstream, client: 85.146.85.194, server: jenkins.testdomain.com,
request: "GET /job/Basic%20NodeJS-React%20app/8/console HTTP/1.1",
upstream: "https:
//127.0.0.1:8080/job/Basic%20NodeJS-React%20app/8/console", host:
"jenkins.testdomain.com", referrer: "https:
//jenkins.testdomain.com/job/Basic%20NodeJS-React%20app/8/"
*1 SSL_do_handshake() failed (SSL: error:1408F10B:SSL routines:ssl3_get_record:wrong version number) while SSL handshaking
to upstream, client: 85.146.85.194, server: jenkins.testdomain.com,
request: "GET /favicon.ico HTTP/1.1", upstream: "https:
//127.0.0.1:8080/favicon.ico", host: "jenkins.testdomain.com",
referrer: "https:
//jenkins.testdomain.com/job/Basic%20NodeJS-React%20app/8/console" ...
In the Jenkinsfile of my node-js-react app (from jenkins repo), the agent looks like this:
pipeline {
agent {
docker {
image 'node:6-alpine'
args '-p 3000:80'
}
}
environment {
CI = 'true'
}
stages {
// Build, Test, and Deliver stages
}
}
And my jenkins.testdomain.com configuration (/etc/nginx/sites-available/jenkins.testdomain.com) is like this (pass tests from nginx -t):
server {
listen 80;
root /var/www/jenkins.testdomain.com/html;
server_name jenkins.testdomain.com www.jenkins.testdomain.com;
location / {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# Fix the "It appears that your reverse proxy set up is broken" error.
proxy_pass http://localhost:8080;
# High timeout for testing
proxy_connect_timeout 1200s;
proxy_send_timeout 1200s;
proxy_read_timeout 1200s;
proxy_redirect http://localhost:8080 https://jenkins.testdomain.com;
# Required for new HTTP-based CLI
proxy_http_version 1.1;
proxy_request_buffering off;
# Required for HTTP-based CLI to work over SSL
proxy_buffering off;
}
# Certbot auto-generated lines...
}
Any help would be very welcomed 3 days struggling with this and playing around with the different proxy_ directives from nginx and so...
Thanks in advance!

OK just add an update that some days after my latest post, I realized that the main and only reason the server was going down was a lack of resources in the droplet.
So I was using a droplet with 1GB of RAM, 25GB HD, etc.. (the most basic one), so I chose to update it to use at least 2GB of RAM and indeed, that made it work as I was expecting. Everything until now works fine and that issue didn’t happen again.
Hope it helps if someone experiences the same issue.

Related

Running .NET Core 5.0 on Debian Nginx

I wanted to move a project I have working on Windows to run on a Unix machine.
The machine running on Debian 9 with Nginx.
This project runs absolutely fine on Windows with IIS.
I've followed all the instructions on here created a service for this to run on the start of the machine and Nginx configuration to proxy the connection from the port I want to use to port 5000.
When I start the application running Dotnet Myddl.dll it starts and says it is only listening on port 5000.
Then when I try to access it, I can see a warning.
warn: Microsoft.AspNetCore.HttpsPolicy.HttpsRedirectionMiddleware[3]
Failed to determine the HTTPS port for redirect.
I know it is related to my app redirecting to HTTPS and not knowing where to redirect it, but how do I resolve this?
My service
[Unit]
Description=Myapp API
[Service]
WorkingDirectory=/var/www/myapp/publish
ExecStart=/usr/bin/dotnet /var/www/myapp/publish/myapp.dll
Restart=always
# Restart service after 10 seconds if the dotnet service crashes:
RestartSec=10
KillSignal=SIGINT
SyslogIdentifier=dotnet-example
User=www-data
Environment=ASPNETCORE_ENVIRONMENT=Production
Environment=DOTNET_PRINT_TELEMETRY_MESSAGE=false
[Install]
WantedBy=multi-user.target
server {
listen 6969;
server_name mysite.net *.mysite.net;
location / {
proxy_pass http://localhost:5000;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection keep-alive;
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
}
EDIT:
I've been trying to resolve this and still can't. When I start the app on the unix machine I get the following
root#myhost:/var/www/myapp/publish# dotnet Myapp.dll info: Microsoft.Hosting.Lifetime[0] Now listening on: localhost:5000 info: Microsoft.Hosting.Lifetime[0] Application started. Press Ctrl+C to shut down. info: Microsoft.Hosting.Lifetime[0] Hosting environment: Production info: Microsoft.Hosting.Lifetime[0] Content root path: /var/www/myapp/publish
Obviously it is missing https option, and I can't figure why.
EDIT2:
I've published the app as self contained for Linux-x64, and now I do not get the warning saying that it couldn't determine the https port, on my browser I get redirected to https://mydomain:5001 when I access it on http://mydomain:6969
Still I do not get the app listening on https on Unix, just on Windows.
EDIT3:
Noticed that if I go to one of my endpoints e.g. http://IP:6969/api/users I get a 500 response.
EDIT4:
When I was loading my application locally, I was getting straight through to the swagger page, such as /swagger/index.html, for some reason my API when complied for Linux does not accept this URL and returns me a 404, but if I get to one of my endpoints e.g. /api/users, it does return me the data I was expecting for.
Default https port should be 5001.
You can set https port following offical docs.
Or disable relatied middlware. Use nginx for https termination if needed.
I was trying to Migrate an API, and as on Windows it was returning me to my swagger page located on /swagger/index.html I was expecting the same to happen, which for some reason, doesn't.
So if I access one of my endpoints (e.g. /api/users) it does work fine.

How to setup nginx with docker compose

I have nginx config and docker compose but I am not able to link them together properly.
default.conf:
worker_processes auto;
events { worker_connections 1024; }
http {
upstream docker_nginx {
server nginx:6000;
}
server {
location / {
proxy_set_header X-Forwarded-For $remote_addr;
proxy_pass http://localhost:4000;
proxy_set_header Host $host;
}
location /jwt/user/ {
proxy_set_header X-Forwarded-For $remote_addr;
pproxy_pass http://localhost:4000;
proxy_set_header Host $host;
}
}
}
Docker compose:
version: '2'
services:
reverseproxy:
image: nginx
container_name: docker_nginx
ports:
- 6000:6000
volumes:
- ./services:/default.conf
restart: always
jwt:
build: ./jwt
image: jwt:1.0
ports:
- 4000:4000
restart: always
volumes:
- ./jwt
I want to listen on port 6000 and route request to a service on port 4000. I tried running http://nginx:6000/ and http://localhost:6000/ but I get this:
This site can’t be reached. The web page at http://localhost:6000/ might be temporarily down or it may have moved permanently to a new web address.
ERR_UNSAFE_PORT
Let me give you my repo, you can check my docker-compose.yml, it uses several services and are proxied using Nginx and are worked together.
On this link you’ll find a default.conf file and it contains several Nginx upstream configuration please take a look at how I used docker-compose service references there.
Inside the client/ directory, I also have another Nginx as a web server of a react.js project.
On server/ directory, it has a Node.js API, It connects to Redis and Postgres SQL database also built from docker-compose.yml.
I think this use case can be useful for you and other users!
There are few things which are misconfigured.
config file is mapped to wrong location
It should be ./services/default.conf:/etc/nginx/nginx.conf
You've to explicitly tell nginx (in conf file) to listen to 6000 port or otherwise map your 6000 port to 80 port inside nginx (default for nginx) "6000:80"
Yes, and as suggested in other answer, instead of proxying to localhost:4000 you have to proxy it to jwt:4000.
For more on nginx in docker see it's doc https://hub.docker.com/_/nginx/
Should - ./services:/default.conf be something like - ./default.conf:/etc/nginx/sites-enables/default.conf?
Also try replacing proxy_pass http://localhost:4000; with proxy_pass http://jwt:4000; and you probably don't need the upstream section.
The "proxy_pass" is the address of the proxied service from within the network created by docker compose - and each service's domain is the same as its name in docker-compose.yml
I have never used upstream, but if you want to use it, I think you have to put this
upstream docker-jwt {
server jwt:4000;
}
# and
proxy_pass http://docker-jwt;
but as I put above, I would skip that and just put http://jwt:4000 as the proxied address.
You also seem to have a typo here: pproxy_pass http://localhost:4000;
But if you leave the typo, and look at the docker compose logs, you will hopefully see an error from nginx, which means that you successfully mapped the config file

Nginx - Load balancing returns 404 on Windows

I have installed Nginx for Windows (64-bit) from here because the official binaries are 32-bit. The aim is to use Nginx for load balancing NodeJS applications. I am following instructions from here where, the link to sample basic configuration file also exists.
The following configuration file works successfully on Linux where nginx was installed via Ubuntu PPA. The servers are themselves started via pm2.
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}
upstream top_servers {
# Use IP Hash for session persistence
ip_hash;
# Least connected algorithm
least_conn;
# List of Node.JS Application Servers
server 127.0.0.1:3001;
server 127.0.0.1:3002;
server 127.0.0.1:3003;
server 127.0.0.1:3004;
}
server {
listen 80;
server_name ip.address;
location /topserver/ {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $http_host;
proxy_pass http://top_servers;
proxy_set_header X-Request-Id $request_id;
}
}
However, this file does not work with Windows. I am getting an error as 'No such file or directory' under the html folder of Nginx installation on Windows. I haven't done any such setting for Linux.
Can you please help me convert the above configuration file for Windows?
NOTE I don't have a choice - Windows is a must for this project.
So, I over-wrote the contents of conf/nginx.conf with the contents shown above. First, I got an error as "map" directive is not allowed here. Then, after removing this directive, I got another error as "upstream" directive is not allowed here". I think, the binaries I am using does not support load balancing.

Jenkins service stops completely during a node js build, getting a 502 bad gateway with an nginx setup on Ubuntu

I have install Jenkins on Ubuntu VPS and I am doing a nodejs build, which, at some point triggers a 502 bad gateway in nginx and the Jenkis service stops.
The build seems to stop at random times, sometimes when installing the npm models, other times when using gulp and so on.
I have to restart it manually (sudo service jenkins restart).
Here is the console output from a build:
+ jspm -v
0.16.15
Running against local jspm install.
[workspace] $ /bin/sh -xe /tmp/hudson1116016846111302145.sh
+ jspm install
Looking up npm:babel-core
Looking up npm:babel-runtime
Looking up npm:core-js
Updating registry cache...
ok Up to date - babel as npm:babel-core#^5.8.24 (5.8.34)
Looking up github:jspm/nodelibs-fs
Looking up github:jspm/nodelibs-path
Looking up github:jspm/nodelibs-process
Looking up github:systemjs/plugin-json
Looking up npm:process
Looking up npm:path-browserify
Looking up github:jspm/nodelibs-assert
Looking up npm:assert
Looking up npm:util
Looking up npm:inherits
Looking up github:jspm/nodelibs-util
ok Up to date - core-js as npm:core-js#^1.1.4 (1.2.6)
ok Up to date - babel-runtime as npm:babel-runtime#^5.8.24 (5.8.34)
ok Install tree has no forks.
ok Install complete.
[workspace] $ /bin/sh -xe /tmp/hudson4025483240884121474.sh
+ npm run build
> dxx-app#1.0.0 build /var/lib/jenkins/jobs/dxx-app/workspace
> gulp build
[15:31:11] Using gulpfile ~/jobs/dxx-app/workspace/gulpfile.js
[15:31:11] Starting 'build'...
[15:31:11] Starting 'clean'...
[15:31:11] Finished 'clean' after 19 ms
[15:31:11] Starting 'jspm-build'...
[15:31:11] Starting 'copy-assets'...
[15:31:11] Starting 'copy-fonts'...
[15:31:11] Starting 'less'...
[15:31:11] Finished 'copy-assets' after 226 ms
[15:31:12] Finished 'less' after 191 ms
[15:31:12] Finished 'copy-fonts' after 213 ms
502 Bad Gateway
nginx/1.4.6 (Ubuntu)
The proble is, when Jenkis fails, it does not record anything in my build logs, so I do not know what is happening.
I have seen online you should increase the timeout from the nginx.conf, so I have added the following to the config file:
http {
##
# Increased timeout to fix 502 bad gateway error
##
fastcgi_buffers 800 10006k;
fastcgi_buffer_size 1000k;
fastcgi_connect_timeout 999999;
fastcgi_send_timeout 999999;
fastcgi_read_timeout 999999;
Also, in nginx, sites-available, I have the following configuration:
server {
listen 80;
server_name jenkins.domain.com;
ignore_invalid_headers off;
location / {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_pass http://127.0.0.1:8080;
proxy_connect_timeout 999999999999;
proxy_send_timeout 999999999999;
proxy_read_timeout 999999999999;
}
}
Do you have any ideas?
As explained in this thread you can increase the memory in the /etc/default/jenkins where you can adjust the following flag
JAVA_ARGS="-Xmx256m" # Choose your desired memory instead of 256
I had the same issue in Ubuntu server.
I updated the JAVA_ARGS
nano /etc/default/jenkins
JAVA_ARGS="-Xmx256m"
it worked for me.
I experienced the same issue on Ubuntu server. I had to change JAVA_ARGS value in
/etc/default/jenkins
to:
JAVA_ARGS="-Xmx128m"
before the react build process stopped throwing 502 gateway error message. This is on Jenkins 2.263.4

Setup (https) SSL on localhost for meteor development

How do you create a self signed SSL certificate to use on local server on mac 10.9?
I require my localhost serving as https://localhost
I am using the linkedin API. The feature which requires the ssl on local host is explained here.
https://developer.linkedin.com/documents/exchange-jsapi-tokens-rest-api-oauth-tokens
In brief, linkedin will send the client a bearer token after the client authorises my app to access their data. The built in javascript library by linkedin will automatically send this cookie to my server / backend. This json file info is used for user authentication.
However, linkedin will not send the private cookie if the server is not https.
Quick and easy solution that works in dev/prod mode, using http-proxy ontop of your app.
1) Add in the tarang:ssl package
meteor add tarang:ssl
2) Add your certificate and key to a directory in your app /private, e.g /private/key.pem and /private/cert.pem
Then in your /server code
Meteor.startup(function() {
SSLProxy({
port: 6000, //or 443 (normal port/requires sudo)
ssl : {
key: Assets.getText("key.pem"),
cert: Assets.getText("cert.pem"),
//Optional CA
//Assets.getText("ca.pem")
}
});
});
Then fire up your app and load up https://localhost:6000. Be sure not to mix up your ports with https and http as they are served seperately.
With this I'm assuming you know how to create your own self signed certificate, there are loads of resources on how to do this. Just in case here are some links.
http://www.akadia.com/services/ssh_test_certificate.html
https://devcenter.heroku.com/articles/ssl-certificate-self
An alternative to self signed certs: it may be better to use an official certificate for your apps domain and use /etc/hosts to create a loopback on your local computer too. This is because its tedious to have to switch certs between dev and prod.
Or you could just use ngrok to port forward :)
1) start your server (i.e. at localhost:3000)
2) start ngrok from command line: ./ngrok http 3000
that should give you http and https urls to access from any device
Other solution is to use NGINX. Following steps are tested on Mac El Capitan, assuming your local website runs on port 3000 :
1. Add a host to your local machine :
Edit your host file : vi /etc/hosts
Add a line for your local dev domain : 127.0.0.1 dev.yourdomain.com
Flush your cache dscacheutil -flushcache
Now you should be able to reach your local website with http://dev.yourdomain.com:3000
2. Create a self signed SSL like explained here : http://mac-blog.org.ua/self-signed-ssl-for-nginx/
3. Install nginx and configure it to map https traffic to your local website:
brew install nginx
sudo nginx
Now you should be able to reach http://localhost:8080 and get an Nginx message.
This is the default conf so now you have to set the https conf :
Edit your conf file :
vi /usr/local/etc/nginx/nginx.conf
Uncomment the HTTPS server section and change following lines :
server_name dev.yourdomain.com;
Put your certificates you just created :
ssl_certificate /path-to-your-keys/nginx.pem;
ssl_certificate_key /path-to-your-keys/nginx.key;
Change the location section with this one:
location / {
proxy_pass http://localhost:3000;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Client-Verify SUCCESS;
proxy_set_header X-Client-DN $ssl_client_s_dn;
proxy_set_header X-SSL-Subject $ssl_client_s_dn;
proxy_set_header X-SSL-Issuer $ssl_client_i_dn;
proxy_read_timeout 1800;
proxy_connect_timeout 1800;
}
Restart nginx :
sudo nginx -s stop
sudo nginx
And now you should be able to access https://dev.yourdomain.com

Resources