I'm building an app on my own and I'd like for it to handle 10K concurrent connections (tested via local machine and locust script)
Hosted on two ubuntu 14.04 servers with nginx reverse proxy and nodeJS app server.
Currently I get to around 3.3K concurrent users before suffering a spike of 500 connection dropped errors.
I achieved load balancing across port connections by running the app on two separate ports and using an upstream directive to spread requests over ports.
However, this has not shown any demonstrated improvement in my numbers.
Question:
I know that there is a lot of information missing here (I imagine how much bandwidth each user requires). How do I go about gathering the right information to decide this?
What other option can I consider/learn/implement to generate the biggest gain in possible concurrent users?
Thanks a lot!
If you see a lot of "Too many open files" errors in your nginx log, it is because you've reach the limit of concurrent open sockets currently set on the server.
You might have to increase the Usage Limit (ULimit) for the nginx user and probably the node app user too. This is the first issue I hit each time I load test an nginx+nodejs. (https://www.cyberciti.biz/faq/linux-unix-nginx-too-many-open-files/)
ULimit
The ulimit command gives you the limits of the current user so you have to run it while connected on the nginx user
# su in the user nginx
su - nginx
ulimit -Hn
ulimit -Sn
# or
su nginx --shell /bin/bash --command "ulimit -Hn"
su nginx --shell /bin/bash --command "ulimit -Sn"
We usually change it in /etc/security/limits.conf
sudo vi /etc/security/limits.conf
# Add the following lines
nginx soft nofile 30000
nginx hard nofile 30000
# Save
# Reload (I think rebooting here is the best way)
sysctl -p
nginx.conf
After that, you have to set the limit at the nginx software level in the nginx config file
sudo vi /etc/nginx/nginx.conf
# Add this line at the root of the config
worker_rlimit_nofile 30000;
# Reload (This will fail if you have SELinux enabled)
sudo nginx -s reload
SELinux
To allow the nginx app to set its own limit you can set a selinux bool:
setsebool httpd_setrlimit on
sudo nginx -s reload
Related
I have an app that has been successfully running on EC2 for a few years. The system is Ubuntu 18.04.2 LTS (GNU/Linux 4.15.0-1032-aws x86_64).
It's a small and simple app with low traffic. I had never made any changes to the server itself until today. I wanted to deal with the X packages can be updated. message, so I ran:
sudo apt-get update
sudo apt-get upgrade
sudo apt-get dist-upgrade
Then I ran sudo reboot. Once rebooted, the app runs perfectly. I can access it as normal via the public URL and look at things, including db (postgresql directly on the server) data with no issues or surprises.
But, when I tried to ssh into the machine again, I couldn't. I do ssh -i "key.pem" -vvv ubuntu#<IP> and get:
debug1: Connecting to <IP> [<IP>] port 22.
debug1: connect to address <IP> port 22: Operation timed out
ssh: connect to host <IP> port 22: Operation timed out
No changes were made to security groups. Also, it's such a small project, that I never setup EC2 Instance Connect or anything like that.
I had the thought of launching a new EC2 and just switching the EBS volumes, thinking EBS would bring the app and data, while the instance itself would have configs and permissions.
I do not understand much about this (clearly), and was surprised to learn that the EBS volume itself seems to be the problem and hold all the cards.
I can switch EBS volumes back and forth between the two EC2 instances. At any given time, whichever one has the newest (and therefore blank) EBS volume attached at /dev/sda1 allows SSH but surely does not run the app. And, vice-versa: Whichever EC2 instance has the original EBS volume runs the app perfectly but keeps me locked out of ssh.
In this scenario, the question is: How can I make one of the EC2 instances bypass this EBS issue and make its own decision about allowing me to connect with ssh?
Or: What is the obvious and/or silly thing I'm missing here?
PS: I do have elastic IP going for all of this, so it doesn't seem like DNS would be the source of the problem.
With John Rotenstein's help, I was able to resolve this.
Here are the core steps:
Phase 1 - Attach and mount additional volume
Per John's comment, it's possible to boot the instance from the "good" volume and then attach and mount the "bad" volume after. This allowed me to explore files and look for issues.
AWS panel
Attach volume to EC2 instance as root by using /dev/sda1 for name
Start the EC2 instance
Attach the other volume after instance has booted
Terminal
SSH into the server
See root volume information:
~$ df -hT /dev/xvda1
Check for mounted volumes:
~$ lsblk
See additional volume information:
~$ df -hT /dev/xvdf1
Switch to root user:
~$ sudo su -
Make a directory to be the mount path:
~$ mkdir /addvol
Mount the additional volume to the path:
~$ mount /dev/xvdf1 /addvol
Check additional volume contents:
~$ ls -la /addvol/home/ubuntu
Now I could see and navigate the additional volume's contents, finding config files, looking at authorized_keys, file permissions, etc.
This article from AWS helped a lot to get me here.
After finally getting to this point, I could not find any problems with the keys, or permissions, etc. John pointed me to this article dealing with Ubuntu's firewall things.
Phase 2 - Dealing with the firewall
I ran some commands from the article and tried to understand how they worked.
Once I grasped it a little, I decided to use an existing reboot script I have on the volume to ensure the firewall was ok with SSH connections.
I updated my existing custom reboot script, adding the following lines:
sudo ufw allow ssh
sudo ufw allow 22
sudo ufw disable
sudo ufw --force enable
Basically it calls to allow for ssh twice, once by name and then by port. I'm a newbie on this stuff and just went for the overkill.
Then it disables and enables the firewall to ensure it runs with these news things configured.
Because sudo ufw enable requires an interaction, I chose to use sudo ufw --force enable.
Phase 3 - Testing and using it!
After the script update, I exited the server.
AWS panel:
Stop the EC2 instance
Detach one volume from the instance
Detach the other volume from the instance
Reattach the "bad" volume, this time as root
Start the EC2 instance
Terminal:
SSH into the instance - Voila!
NOTE: Before truly working 100%, my computer complained about the known_hosts thing. The server key must have changed on the update/upgrade and/or after all of the volume changes. I don't think having to confirm hosts is a big deal, so I just usually clear all of the contents in my local .ssh/known_hosts file. If you prefer to be specific, you can find the server's information on there specifically and delete only the relevant lines.
I have installed nginx on ec2, and when I star it, I see it runs using the ec2-user.
My impression is that it is not advisable to run the webserver with ec2-user but another user.
When I look at the default configuation at /etc/nginx/nginx.conf i see user nginx; showing that a user nginx could be used.
Question now is, how do I use the nginx user to run nginx. I currently can control nginx by using sudo. If I do not, I get the following error:
systemctl stop nginx
Failed to stop nginx.service: The name org.freedesktop.PolicyKit1 was not provided by any .service files
See system logs and 'systemctl status nginx.service' for details.
How do I get to be able to control nginx like this, but with the ngin user?
when I try to switch to the user, I get the following message:
sudo -iu nginx
This account is currently not available.
But when I try to add the nginx user, I get the following message:
sudo useradd nginx
useradd: user 'nginx' already exists
SO I am not sure the correct way of dealing with this.
I have a node application which runs fine if I manually putty into the gcloud computeVM and run it.
Here are the complications (all realted to unix) :
1.) I have a domain name. So I added the dns zone record to point to the above VM.
2.) For the compute VM to respond, there should be process listening on 80
3.) If we follow the https://cloud.google.com/nodejs/getting-started/run-on-compute-engine#download_app , it specifies to run the app on 8080.
4.) For ports < 1024, it requires root privileges to open up ports.
5.) So from npm start, I changed the start up script to use "sudo npm start"
6.) Then it gave the following error : my-app-instance supervisord: nodeapp sudo: no tty present and no askpass program specified
7.) If I have to "sudo visudo" everytime and add the "username ALL = NOPASSWD:" everytime I restart the instance after deployment , its something which I would least prefer.
I have included the relevant portion of the stratup-script for more info :
# Install app dependencies
cd /myrepo/opt/app/servers
sudo npm install
# Create a nodeapp user. The application will run as this user.
useradd -m -d /home/nodeapp nodeapp
chown -R nodeapp:nodeapp /myrepo/opt/app/servers
# Configure supervisor to run the node app.
cat >/etc/supervisor/conf.d/node-app.conf << EOF
[program:nodeapp]
directory=/myrepo/opt/app/servers
command=sudo npm start
autostart=true
autorestart=true
user=nodeapp
environment=HOME="/home/nodeapp",USER="nodeapp",NODE_ENV="production"
stdout_logfile=syslog
stderr_logfile=syslog
EOF
A.) My requirement is simple : My google domain points to the above compute VM now. whenever the user types www.domainname.com, it should take him to the website without any port numbers in the url. How to open port 80 with a simple modification of start-up script(preferred) ?
B.) And also if I have to go with deploy.sh specified in the tutorial, will it get executed automatically ? Or if I have to execute it automatically , whats the procedure.
Note : I am not unix expert. Any help would be appreciated.
Look into using a reverse proxy. This allows you to run your app without root privileges on a port like 8080, and have a privileged HTTP server (like Apache or Nginx) running on port 80 and proxying traffic to your app. This is common practice, and much more secure than running your app with root privileges.
my group and I are running a server that is based upon Django and uses mod_wsgi to run an Apache server. We will not be working on this project after it is over, so I am attempting to set up cronjob similar functionality to check if the apache server has shut down(system restart or power failure), and if it has, will restart the server for me. I've found documentation on how to check if an apache server is down and restart the server if it is, but our server uses https and thus our start command is pretty verbose.
Can I simply use the functionality provided in these examples:
https://askubuntu.com/questions/277389/cron-job-to-restart-apache
https://www.digitalocean.com/community/tutorials/how-to-use-a-simple-bash-script-to-restart-server-programs
Or do I need a much more complicated process to make this happen?
The command we use to initially start the server is
python manage.py runmodwsgi --host 0.0.0.0 --port 8001 --https-port 8000 --ssl-certificate (certificate Location) --server-name (Domain Name)
I'm pretty new to Linux and using both Mod-wsgi as well as Apache so any help is greatly appreciated.
I suppose it is not good way to resolve this problem.
I recommend you use monit (https://mmonit.com/). It is cool program for checking services.
apt-get install monit
Apache restart configuration directives:
check process httpd with pidfile /var/run/httpd.pid
group apache
start program = "/etc/init.d/httpd start"
stop program = "/etc/init.d/httpd stop"
if failed host 127.0.0.1 port 80
protocol http then restart
if 5 restarts within 5 cycles then timeout
You are better off using the --setup-only option to mod_wsgi-express or the Django integration for it, to generate the configuration but not run it. Then as others have mentioned, integrate it into the system service manager.
The two commands for starting and stopping the Apache/mod_wsgi instance would be apachectl start and apachectl stop, where apachectl is that which was generated when running with the additional --setup-only option.
When running it as a system service, also make sure you use the --server-root option to specify a more persistent location for the generated configuration. Do not use the default under /tmp if running for anything but temporary development sessions as some Linux systems will remove files under /tmp causing things to start failing after a while.
Also, since under a service manager it would generally be starting as root, particularly if listening on port 80 is a requirement, ensure you use the --user and --group options to specify what user/group your Python web application should run as.
Read:
https://pypi.python.org/pypi/mod_wsgi
for more details of the --setup-only option and start-server commands for generating the configuration. Because you are using the Django integration, you will need to use the --setup-only option.
For more informed helped, bring your issue to the mod_wsgi mailing list. The mod_wsgi-express way of running Apache/mod_wsgi is new enough that unlikely that anyone here is really going to know much about it.
There is no need to do this at all. There is no reason to start up Apache manually; once it's installed as a system service, Ubuntu will start it up automatically on restart or crash.
You should reflect on why you feel the need to do this for Apache specifically, and not any of the other system services you depend on, such as the database.
Is there a brief guide to explain how to start up a application when the instance starts up and running? If it were one of the services installed through yum then I guess I can use /sbin/chkconfig to add it to the service. (To make it sure, is it correct?)
However, I just want to run the program which was not installed through yum. To run node.js program, I will have to run script sudo node app.js at home directory whenever the system boots up.
I am not used to Amazon Linux AMI so I am having little trouble finding a 'right' way to run some script automatically on every boot.
Is there an elegant way to do this?
One way is to create an upstart job. That way your app will start once Linux loads, will restart automatically if it crashes, and you can start / stop / restart it by sudo start yourapp / sudo stop yourapp / sudo restart yourapp.
Here are beginning steps:
1) Install upstart utility (may be pre-installed if you use a standard Amazon Linux AMI):
sudo yum install upstart
For Ubuntu:
sudo apt-get install upstart
2) Create upstart script for your node app:
in /etc/init add file yourappname.conf with the following lines of code:
#!upstart
description "your app name"
start on started mountall
stop on shutdown
# Automatically Respawn:
respawn
respawn limit 99 5
env NODE_ENV=development
# Warning: this runs node as root user, which is a security risk
# in many scenarios, but upstart-ing a process as a non-root user
# is outside the scope of this question
exec node /path_to_your_app/app.js >> /var/log/yourappname.log 2>&1
3) start your app by sudo start yourappname
You can use forever-service for provisioning node script as a service and automatically starting during boots. Following commands will do the needful,
npm install -g forever-service
forever-service install test
This will provision app.js in the current directory as a service via forever. The service will automatically restart every time system is restarted. Also when stopped it will attempt a graceful stop. This script provisions the logrotate script as well.
Github url: https://github.com/zapty/forever-service
As of now forever-service supports Amazon Linux, CentOS, Redhat support for other Linux distro, Mac and Windows are in works..
NOTE: I am the author of forever-service.
Quick solution for you would be to start your app from /etc/rc.local ; just add your command there.
But if you want to go the elegant way, you'll have to package your application in a rpm file,
have a startup script that goes in /etc/rc.d so that you can use chkconfig on your app, then install the rpm on your instance.
Maybe this or this help. (or just google for "creating rpm packages")
My Amazon Linux instance runs on Ubuntu, and I used systemd to set it up.
First you need to create a <servicename>.service file. (in my case cloudyleela.service)
sudo nano /lib/systemd/system/cloudyleela.service
Type the following in this file:
[Unit]
Description=cloudy leela
Documentation=http://documentation.domain.com
After=network.target
[Service]
Type=simple
TimeoutSec=0
User=ubuntu
ExecStart=/usr/bin/node /home/ubuntu/server.js
Restart=on-failure
[Install]
WantedBy=multi-user.target
In this application the node application is started. You will need a full path here. I configured that the application should simply restart if something goes wrong. The instances that Amazon uses have no passwords for their users by default.
Reload the file from disk, and then you can start your service. You need to enable it to make it active as a service, which automatically launches at startup.
ubuntu#ip-172-31-21-195:~$ sudo systemctl daemon-reload
ubuntu#ip-172-31-21-195:~$ sudo systemctl start cloudyleela
ubuntu#ip-172-31-21-195:~$ sudo systemctl enable cloudyleela
Created symlink /etc/systemd/system/multi-user.target.wants/cloudyleela.service → /lib/systemd/system/cloudyleela.service.
ubuntu#ip-172-31-21-195:~$
A great systemd for node.js tutorial is available here.
If you run a webserver:
You probably will have some issues running your webserver on port 80. And the easiest solution, is actually to run your webserver on a different port (e.g. 4200) and then to redirect that port to port 80. You can accomplish this with the following command:
sudo iptables -t nat -A PREROUTING -i -p tcp --dport 80 -j REDIRECT --to-port 4200
Unfortunately, this is not persistent, so you have to repeat it whenever your server restarts. A better approach is to also include this command in our service script:
ExecStartPre to add the port forwarding
ExecStopPost to remove the port forwarding
PermissionStartOnly to do this with sudo power
So, something like this:
[Service]
...
PermissionsStartOnly=true
ExecStartPre=/sbin/iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-port 4200
ExecStopPost=/sbin/iptables -t nat -D PREROUTING -p tcp --dport 80 -j REDIRECT --to-port 4200
Don't forget to reload and restart your service:
[ec2-user#ip-172-31-39-212 system]$ sudo systemctl daemon-reload
[ec2-user#ip-172-31-39-212 system]$ sudo systemctl stop cloudyleela
[ec2-user#ip-172-31-39-212 system]$ sudo systemctl start cloudyleela
[ec2-user#ip-172-31-39-212 system]$
For microservices (update on Dec 2020)
The previously mentioned solution gives a lot of flexibility, but it does take some time to set it up. And for each additional application, you need to go through this entire process again. By the time you'll be installing your 5th node application, you'll certainly start wondering: "there has to be a shortcut".
The advantage of PM2 is that it's just 1 service to install. Next it's PM2 which manages the actual applications.
Even the initial setup of PM2 is easy, because it automatically installs the pm2 service for you.
npm install pm2 -g
And adding new services is even easier:
pm2 start index.js --name "foo"`.
When everything's up and running, you can save your setup, to have it automatically start on reboot.
pm2 save
If you want an overview of all your running node applications,
you can run pm2 list
And PM2 also offers an online (webbased) dashboard to monitor your application remotely. You may need a license to access some of the dashboard functionality though (which is a bit over-priced imho).
You can create a script that can start and stop your app and place it in /etc/init.d; make the script adhere to chkconfig's conventions (below), and then use chkconfig to set it to start when other services are started.
You can pick an existing script from /etc/init.d to use as an example; this article describes the requirements, which are basically:
An executable script that identifies the shell needed (i.e., #!/bin/bash)
A comment of the form # chkconfig: where is often 345, startprio indicates where in the order of services to start, and stopprio is where in the order of services to stop. I generally pick a similar service that already exists and use that as a guide for these values (i.e., if you have a web-related service, start at the same levels as httpd, with similar start and stop priorities).
Once your script is set up, you can use
chkconfig --add yourscript
chkconfig yourscript on
and you should be good to go. (Some distros may require you to manually symlink to the script to /etc/init.d/rc.d, but I believe your AWS distro will do that for you when you enable the script.
Use Elastic Beanstalk :) Provides support for auto-scaling, SSL termination, blue/green deployments, etc
If you want the salty sysadmin way for a RedHat based linux distro (Amazon Linux is a flavor of RedHat), learn systemd, as mentioned by #bvdb in the answer above:
https://en.wikipedia.org/wiki/Systemd
Set everything up as described on an EC2 instance, snapshot a custom AMI, and use this custom AMI as your base for EC2 instances hosting your apps. This way you don't have to go through all that setup multiple times. You'll probably want to get acquainted with load balancers too, if you are running in a production environment with uptime requirements.
Or, yes, as mentioned by #bvdb, you could also use pm2 to interface with systemd. Though I don't think pm2 helps with running your app across multiple EC2 instances, which is definitely recommended for production environments with uptime requirements.
All of which is a very steep learning curve. Since the OP seemed to be new to all this, Elastic Beanstalk, Google App Engine, and others are a great way to get code running in the cloud without all that.
These days I dev in TypeScript, deploying to serverless function execution in the cloud for most things, and don't have to think about package installs or app startup at all.
You can use screen. Run crontab -e and add this line:
#reboot screen -d -m bash -c "cd /home/user/yourapp/; node app"
Have been using forever on AWS and it does a good job. Install using
[sudo] npm install forever -g
To add an application use
forever start path_to_application
and to stop the application use
forever stop path_to_application
This is a useful article that helped me with setting it up.