502 error and unhealthy health status in the target group when using AWS ELB and ACM for https - node.js

I'm using AWS ELB and ACM to use HTTPS on Node.js but, I have been facing 502 error.
The health status of the target group for the HTTPS is "unhealthy" which is making me understand I'm doing something wrong around ELB.
The following is what I did.
[ELB]
VPC: Same VPC with the EC2 instance
Subnet #1: 10.0.1.0/24: Same subnet with the EC2 instance
Subnet #2: 10.0.3.0/24: New subnet which was created for this test
Security group: All traffic for inbound & outbound are opened (for this test purpose)
Listener_a(http:80):
Rule1:
(If) Host is [example].com OR www.[example].com
(Then) Redirect to https://www.[example].com:443/#{path}?#{query}
(path and query are untouched from the default placeholder)
Status code: HTTP_301
Rule last: untouched from the default
Listener_b(https:443):
Rule1:
(If) Host is [example].com
(Then) Redirect to https://www.[example].com:443/#{path}?#{query}
(path and query are untouched from the default placeholder)
Status code: HTTP_301
Rule last: untouched from the default
[Target groups of ELB]
Target group #1:
[Target type] [Protocol version] [Instance ID] [Name] [Port] [Zone] [Health status]
Instance HTTP1 [Instance ID of the EC2] testname1 80 us-east-2b healthy
Target group #2:
[Target type] [Protocol version] [Instance ID] [Name] [Port] [Zone] [Health status] [Health status details]
Instance HTTP1 [Instance ID of the EC2] testname2 443 us-east-2b unealthy "Health checks failed"
[Summary of ELB log]
type: h2
target:port: Private IPv4 address of the EC2 instance
request_processing_time: -1
target_processing_time: -1
response_processing_time: -1
elb_status_code: 502
target_status_code: -
request: GET https://www.[example].com:443/HTTP/2.0
[Route 53]
[Record name] [Type] [Routing Policy] [Differentiator] [Value/Route traffic to]
[example].com A Simple - www.[example].com.
www.[example].com A Simple - dualstack.[DNS name of the ELB].
[CNAME name of *.[example].com from ACM] CNAME Simple - [CNAME value from ACM]
[CNAME name of www.[example].com from ACM] CNAME Simple - [CNAME value from ACM]
[example].com NS Simple - [4 Name Servers added by Route 53]
[example].com SOA Simple - [Value added by Route 53]
[ACM]
[Domain] [Status]
*.[example].com Success
[example].com Success
www.[example].com Success
[EC2]
VPC: Same VPC with the ELB (10.0.0.0/16)
Subnet #1: Same subnet with the one of the subnet assinged to ELB (10.0.1.0/24)
Public IPv4 address: [ElasticIP assigned]
Security group: All traffic for inbound & outbound are opened (for this test purpose)
[Routing table (same for both subnet)]
[Destination] [Target]
10.0.0.0/16 local
0.0.0.0/0 [IGW]
[ACLs]
All are allowed for both Inbound and Outbound(for this test purpose).
[iptables I ran on EC2]
# iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-ports 8000
# iptables -t nat -A PREROUTING -p tcp --dport 443 -j REDIRECT --to-ports 8443
[Node.js code on EC2 (index.js)]
const fs = require('fs');
const http = require('http');
const https = require('https');
const express = require('express');
const app = express();
const path = require('path');
app.get('/', (req, res) => {
res.send("Hello World!");
});
const httpServer = http.createServer(app);
const httpsServer = https.createServer(app);
httpServer.listen(8000, () => {
console.log("App is listening on port 8000");
});
httpsServer.listen(8443, () => {
console.log("App is listening on port 8443");
});
[Summary of results accessing from browser]
https://www.[example.com] => "502 Bad Gateway"
http://[example].com => Browser redirect it to https://www.[example].com and returns "502 Bad Gateway"
[my Elastic IP] => Can see the web page w/o error
[Public IPv4 DNS of the EC2 instance] => Can see the web page w/o error
[DNS name] => Can see the web page w/o error
[Summary of results accessing from EC2 command line with curl]
http://www.[example].com => 301 Moved Permanently
https://www.[example].com => 502 Bad Gateway
I tried to figure out what is wrong based on the following documents but, so far, no luck.
https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-access-logs.html#enable-access-logging
https://docs.aws.amazon.com/elasticloadbalancing/latest/application/target-group-health-checks.html
https://aws.amazon.com/premiumsupport/knowledge-center/elb-fix-failing-health-checks-alb/
https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-troubleshooting.html
https://aws.amazon.com/premiumsupport/knowledge-center/troubleshoot-unhealthy-checks-ecs/
If anyone could provide me with your insight about what I am missing/doing wrong, it would be great. Please let me know if there is any other information needed.
Thank you!
[Additional edits (8/21/2022)]
As it looked similar to what was discussed on this thread (https://stackoverflow.com/questions/60738575/target-group-443-gives-health-checks-failed-with-these-codes-502?rq=1), I just tried changing the Health check protocol for the target group of HTTPS to use HTTP; however, the results (unhealthy) were the same.

Thanks to the comment from Mark, I updated it as follows and everything is working perfectly, now.
[Target groups of ELB]
Delete the target group for HTTPS and just keep the one for HTTP only.
[ELB]
Update the listener for HTTPS to forward to the target group for HTTP (it was originally forwarding to the target group for HTTPS)
After the update, both listeners for HTTP and HTTPS are forwarded to the same target group for HTTP. It seems like this part was the key.
[Node.js code]
Removed the listeners for https (443) as it was not needed and just keep the one for http (80) only.
[iptables]
Stop running iptables for 8443 as it was not needed and just keep the one for 8000 only.
This thread about running Node.js with port 80 was also helpful. ([https://stackoverflow.com/questions/16573668/best-practices-when-running-node-js-with-port-80-ubuntu-linode][1]).

Related

How to publicly access an AWS EC2 instance that contains an SSL redirect?

I have a Node/Express server running on an EC2 instance that is maintained by Elastic Beanstalk (single instance no load balancer).
I am struggling with implementing a correct SSL redirect that works with EC2.
The SSL redirect is as follows:
const httpsOptions = {
key: fs.readFileSync('/local/ssl/key.pem'),
cert: fs.readFileSync('/local/ssl/cert.pem'),
passphrase: '*****'
}
http.createServer(function (req, res) {
let httpsHost = req.headers.host.replace('8081', '8443');
res.writeHead(301, { "Location": "https://" + httpsHost + req.url });
res.end();
}).listen(8081);
https.createServer(httpsOptions, app).listen(8443)
This works great locally, but when I deploy to my EC2 instance and try to access the site, I get:
Here is a picture of my inbound rules as well:
Do I have the ports configured incorrectly? I cannot use port 80 and port 443 because it requires root permissions, and I won't run my server with root permissions.
When I try to access your host on port 443 I get a response, which means the host is accessible. When I try to access your server on port 8443, I receive a "Connection timed out", what could possibly be caused by wrong firewall (ie. security group) settings.
Your server instance listens to port 8443 and 8081, but in your security group rules the ports 443 and 80 are configured. If you cannot use 443/80 as you wrote, please edit the inbound rules to allow traffic from ports 8443/8081 instead.
This was solved through port forwarding.
Check current IP-tables:
sudo iptables -t nat -L
Remove last entry:
sudo iptables -t nat -D PREROUTING 1
Set up redirection as necessary, in my case:
sudo iptables -t nat -A PREROUTING -p tcp --dport 443 -j REDIRECT --to-ports 8443
sudo iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-ports 8081
This sends all 443 traffic to 8443, and all 80 traffic to 8081.

Nginx is refusing to connect on AWS EC2

I'm trying to use nginx to setup a simple node.js server, I'm running the server in background on port 4000, my nginx config file is
server {
listen 80;
listen [::]:80;
server_name 52.53.196.173;
location / {
include /etc/nginx/proxy_params;
proxy_pass http://127.0.0.1:4000;
}
}
I saved it in /etc/nginx/sites-available and also symlinked it to sites-enabled, the nginx.conf file has the include line already to load files from sites-enabled, then i restarted the service using
sudo service nginx restart
I tried going to 52.53.196.173 and it refuses to connect, however going to 52.53.196.173:4000 with port 4000 it is working, but I'm trying to make it listen on port 80 with nginx, i tried putting my .ml domain as server_name and no luck, and i have the IP 52.53.196.173 as the A record in the domain dns settings, and I'm doing this on an AWS EC2 Instance Ubuntu Server 16.04, i even tried the full ec2 public dns url no luck, any ideas?
Edit: I solved it by moving the file directly in sites-enabled instead of a symlink
There is few possible things. First of all you need to verify that nginx server is running & listening on port 80. you can check the listening ports using the following command.
netstat -tunlp
Then you need to check your server firewall & also the selinux policies. ( OR disable selinux for test )
Then you need to verify that AWS security group configured to access the http/https connections on port 80.
PS : Outputs from the following command & configurations will be helpful for troubleshooting.
netstat -tunlp
sestatus
iptables -L
* AWS Security Group Rules
* Nginx configurations ( including main configuration if changed )
P.S : OP fixed the problem by moving the config file directly into site-enabled directory. maybe, reefer the comments for more info if you are having the same issue.
Most probably port 80 might not be open in your security group or nginx is not running to accept the connections. Please post the nginx status and check the security group
check belows:
in security group, add Http (80) and Https (443) in inbound section with 0.0.0.0 ip as follow:
for 80 :
for 443 :
in Network ACL, allow inbound on http and https. outbound set custom TCP role as follow:
inbound roles:
outbound roles:
assign a elastic ip on ec2 instance, listen to this ip for public.

nodejs timed out on all ports when hosting on godaddy server

I've trying to run my nodejs/expressjs application on my godaddy server, but any port I use times out. I've tried using the application on my local device and it works fine. I have a snippet of my connection below.
var app = express();
app.listen(8080, function() {
console.log("Listening on port " + 8080);
});
When I run the program through ssh, I get no errors
node index.js
Listening on port 8080
But when I go to the corresponding location in my browser, I get:
xxx took too long to respond.
ERR_CONNECTION_TIMED_OUT
I'm pretty sure it has to do with running on the godaddy server. If anyone has experience using this service with nodejs, is there a specific port I should be using, or is there any other setup I should do?
Do you have a VPS with GoDaddy right? So I assume you have also root access.
SSH into your GoDaddy server as root and check if the node.js app actually listens on that port:
netstat -tunlp | grep 8080
If you see any result there for the node.js app and that port then the port is open.
By default, there should be a firewall on your server which might block most of the ports and allows only the necessary incoming traffic.
You can check if there is any rule for that port by issuing the command bellow:
iptables -nvL | grep 8080
If any result is returned, then you have to add an iptables rule to allow access to that port. There are multiple methods to do that:
permit full access from your IP access to the server
permit your ip to access port 8080 on the godaddy server
permit outside world to access port 8080 on your server
You could read any iptables guy, it's pretty easy to add/edit/delete firewall rules. Most of the cPanel/WHM servers come with CSF Firewall (which is based on iptables and perl scripts).
In order to allow an ip address to your firewall (if you have CSF Firewall installed) you have to issue the following command:
csf -a ip-address
I hope that helps!

Network access denied to external node.js socket

I have two servers running on Amazon's EC2. One is a standard web server running LAMP (this is behind an elastic load balancer), the other a Node.js server with Express and socket.io installed. On that Node server I have my server.js file (this file is automatically loaded on server start).
var app = require('express')();
var server = require('http').Server(app);
var io = require('socket.io')(server);
server.listen(8080);
io.on('connection', function (socket) {
socket.on('join', function() {
console.log('joined');
}
}
On the LAMP server, clients connect to this external server by:
<script src="https://cdn.socket.io/socket.io-1.3.4.js"></script>
var socket = io.connect('http://URL_HERE:8080');
socket.on('connect', function() {
socket.emit('join', room_id);
});
The problem, though, is that console spits out ERR_NETWORK_ACCESS_DENIED continuously, suggesting that access to that port is blocked. However, the inbound rules for the Node server are as follows:
Custom TCP Rule -- TCP -- 8080 -- 0.0.0.0/0 (Anywhere)
HTTP -- TCP -- 80 -- My IP
HTTPS -- TCP -- 443 -- My IP
I have tried all sorts by more or less completely opening the inbound ports, but to no avail. It may be worth nothing as well that the only way for people to access the client script (the LAMP server) is through a load balancer - you cannot directly access the LAMP server. The load balancers inbound rules are as follows:
Custom TCP Rule -- TCP -- 8080 -- 0.0.0.0/0 (Anywhere)
HTTP -- TCP -- 80 -- 0.0.0.0/0 (Anywhere)
HTTPS -- TCP -- 443 -- 0.0.0.0/0 (Anywhere)
And the LAMP server's rules:
Custom TCP Rule -- TCP -- 8080 -- sg-elb_id
HTTP -- TCP -- 80 -- sg-elb_id
HTTPS -- TCP -- 443 -- sg-elb_id
... other irrelevant rules (SHH, MYSQL, ...)
Has anyone any idea why I'd be getting an access denied error? The Node server isn't behind the load balancer - it's in effect completely external to the process. All of the security groups are not restricted in regards to outbound rules.
Turns out the reason was it couldn't find the globally installed express or socket.io modules. Doing cd /var/www/html and then sudo npm link socket.io sudo npm link express resolved the issue.

Cannot connect to EC2 Instance through HTTP

I'm having trouble accessing my website through HTTP with an EC2 instance. I've tried changing the security setting by allowing HTTP on port 80, but it still doesn't work. SSH however is working fine. What could be the issue?
This is embarrassing, but the reason I couldn't connect to my EC2 instance was because my node js app never actually started a server for me to listen on port 80.
Adding this simple snippet of code
var server = app.listen(8080, function () {
var host = server.address().address
var port = server.address().port
console.log('Example app listening, host, port)
})
Along with this shell command for redirecting fixed the issue.
iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to 8080
Could check ip-tables if it is blocking 80, check the security group if the TCP for PORT 80 is open to your IP address or ( 0.0.0.0/0 ).
Given that you are able to SSH but not http, check if you can replicate the same settings in the SG which has the config for SSH 22 - also kindly look if the web-server ( or process ) is up and running in PORT 80.
First, login that instance, and run the command to confirm http service is running.
telnet INSTANCE_public_DNS_name 80
Then run the same command for your remote machine to confirm if there are any firewall issues.
If there is problem to telnet, then click the instance name, and go to description --> Security groups --> view rules.
You should see the port opened. If not, create a new security group or edit exist security groups, and assign to that instance.

Resources