I have already taken a look at How to install mongodb in Elastic Beanstalk? dated 2014, which no longer works. as well as https://docs.mongodb.org/ecosystem/platforms/amazon-ec2/#manually-deploy-mongodb-on-ec2
I have set up a new elastic beanstalk environment running on node.js with 1 ec2 micro instance '64bit Amazon Linux 2016.03 v2.1.0 running Node.js'
I have already tried using ssh to connect into my instance and install the mongodb packages using yum command:
$ sudo yum install -y mongodb-org-server mongodb-org-shell mongodb-org-tools
and received this call back:
Loaded plugins: priorities, update-motd, upgrade-helper
No package mongodb-org-server available.
No package mongodb-org-shell available.
No package mongodb-org-tools available.
Error: Nothing to do
When I first ssh 'd into my instance, I received this error warning:
This EC2 instance is managed by AWS Elastic Beanstalk. Changes made via SSH
WILL BE LOST if the instance is replaced by auto-scaling. For more information
on customizing your Elastic Beanstalk environment, see our documentation here:
http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/customize-containers-ec2.html
Currently my environment is set up as a single instance environment, to save on costs. However, in the future I will upgrade to an auto-scaling environment.
Because of this, I am asking is it recommendable to make any changes via ssh in ec2, or should I only be using EB CLI?
I have both EC2 and EB CLI installed locally, however I have never used EB CLI before. If I should be using EB, does anyone have a recommended way to install mongodb?
In case anyone is looking for an answer, here is the advice I received from aws business support.
All code deployed to Elastic Beanstalk needs to be "stateless" I.E. Never make changes directly to a running beanstalk instance using SSH or FTP.... As this will cause inconsistencies and or data lose!
- Elastic Beanstalk is not designed for application that are not stateless.
The environment is designed to scale up and down pending on your Network / CPU load and build new instances from a base AMI. If an instance has issues or the underlying hardware, Elastic Beanstalk will terminate these running instances and replace with new instances. Hence, why no code modification must be applied or done "directly" to an existing instance as new instances will not be aware of these direct changes. ALL changes / code needs to be either uploaded to Elastic Beanstalk console or the CLI tools and the pushed to all the running instances.
More information on Elastic Beanstalk design concepts can be read at the following link
http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/concepts.concepts.design.html
Suggested Solution:
With the above in mind, if using MongoDB to store application data our recommendation would be to DE-couple the MongoDB environment from your Node.js application.
I.E Create a MongoDB Server outside of Elastic Beanstalk, example launching MongoDB directly on a EC2 instance and have your Elastic Beanstalk Node.js application connect to MongoDB Server using connection settings in your app.
-Creating MongoDB
Below is some example links that may be of use for your scenario for creating a MongoDB Server.
Deploy MongoDB on EC2,
https://docs.mongodb.org/ecosystem/platforms/amazon-ec2/
MongoDB node client
https://docs.mongodb.org/getting-started/node/client/
MongoDB on the AWS Cloud quick start guide
http://docs.aws.amazon.com/quickstart/latest/mongodb/architecture.html
-Adding environment variables to Elastic Beanstalk to reference your MongoDB server
Once you have created your MongoDB Server you can pass the needed connection settings to your Elastic Beanstalk environment using environment variables.
Example using .ebextensions .config which you can add Mongo URL / ports / users etc..
option_settings:
- option_name: MONGO_DB_URL
value: "Your MongoDB EC2 internal IP address"
Information on how to use environment properties and read them from within your application can be seen below.
http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create_deploy_nodejs.container.html#create_deploy_nodejs_custom_container-envprop
And information using .ebextensions .config can be found at the following link
http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/ebextensions.html
Alternatively you can also set environment variable using the cli or via the AWS Console
eb cli set environment variables can be read per the below link.
http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/eb3-setenv.html
Using AWS Console
To set system properties (AWS Management Console)
Open the Elastic Beanstalk console.
Navigate to the management console for your environment.
Choose Configuration.
In the Software Configuration section, choose Edit.
Under Environment Properties, create your name / values ...
Accessing Environment Configuration Settings
Inside the Node.js environment running in AWS Elastic Beanstalk, you can access the environment variables using process.env.ENV_VARIABLE similar to the following example.
process.env.MONGO_DB_URL
process.env.PARAM2
http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create_deploy_nodejs.container.html#create_deploy_nodejs_custom_container-envprop
Summary:
In summary I would recommend the following steps to integrate MongoDB with Elastic Beanstalk environments.
Step 1) Create a MongoDB Server outside of Elastic Beanstalk
Step 2) Create your Node.js application in Elastic Beanstalk that connect to your MongoDB server
3 options:
1) SSH into an eb instance and install the mongo CLI manually:
sudo yum-config-manager --add-repo https://repo.mongodb.org/yum/amazon/2013.03/mongodb-org/4.0/x86_64/
sudo yum install --nogpgcheck -y mongodb-org-shell
Disadvantage is that if EB scales down its number of instances and the instance you are currently on gets terminated, you get kicked out of the SSH session:
The system is going down for halt NOW!
Connection to 1.2.3.4 closed by remote host.
Connection to 1.2.3.4 closed.
ERROR: CommandError - An error occurred while running: ssh.
You then need to start all over again: connect to instance, install mongo CLI...
2) Pre-install mongo CLI on instances by using a .config file:
container_commands:
01-mongocli:
command: "sudo yum-config-manager --add-repo https://repo.mongodb.org/yum/amazon/2013.03/mongodb-org/4.0/x86_64/;sudo yum install --nogpgcheck -y mongodb-org-shell"
ignoreErrors: true //use this the ensure instance deployment even if mongo CLI installation fails
Again, if the instance gets terminated by auto-scaler you'd have to connect again, but you don't have to the install mongo CLI manually.
3) Create a separate instance that hosts your mongo CLI, as described in #amyloula's answer. If your mongodb is within an VPC you need to create that separate instance also within the VPC. You will then need to create a Gateway to access the instance publicly, as you cannot connect directly to an instance in a VPC.
Related
I'm pretty new to DevOps and I'm trying to set up my Node.js app on a AWS server instance. Steps I've taken:
Set up Elastic IP
Launched EC2 instance with Ubuntu server
Connected IP to instance
Allowed incoming connections on port 3000
SSH'd into the server with a .pem file
Now I'm at the point where I need to get my files uploaded to the server. I've used FileZilla (and like it) in the past to upload files but the initial part was already set up. When I set up the site on FileZilla there is no /var/www folder on the remote site.
Don't know how to connect these dots.
Also not sure what I need to run once I successfully upload the files. I imagine npm install when I'm ssh'd into the server? Most of the tutorials out there only go through the basic instance setup.
Thanks!
You don't need to have /var/www. Also, it's better that you use a version control and a remote repository like Github and then SSH to your EC2 and then clone your repository there.
Then cd into your repo and run npm install and then start your app.
And check.
Once you connect to the EC2 instance then clone your code in there. It not mandatory to be in /var/www/html but, it's best practice to keep it there. Once you clone npm install into your project home directory so all the required packages get installed. Then for running your node application in production you have to run it on service as pm2, supervisor, forever, passenger, etc. You can use any of these services and configured appropriately to run your application on desired port. As with pm2, you can follow this guide, install pm2 Then you can run with the following command w.r.t. your environment, like I want to run my application on port 5555 for production
$ PORT=5555 pm2 start app.js --name API --env production -f
Check the status using pm2 list Now, your application is running on http://server-ip:5555/ But, you won't be typing port number every-time. So, you need to configure the web server in front of your application like apache or nginx which will forward all request to your application running port. You could find the best guide to their home page. Then your application is available at http://server-ip/ You can follow this for single configuration of multiple node apps
Hope this helps.
What are the steps for deploying a Node js app, with express.js framework and MongoDB as database on AWS?
This is my first such activity and I am not able to find any reliable source over the net.
Deploy nodejs-express-mongoDB backend on AWS EC2 Ubuntu(16.04)
Back-end is a private repo on Gitlab.
Steps:
1: Create EC2 Ubuntu server on AWS,
create EC2 instance
2: Connect personal computer with AWS server - use to control cloud Ubuntu server on personal computer,
Connecting Linux Instance from Windows Using PuTTY
3: Clone repo from Gitlab to Ubuntu server,
Configure ssh key on linux server
Clone gitlab repo to linux server
4: Install Nodejs and MongoDB on Ubuntu server,
Install Nodejs
Install MongoDB
Install MongoDB
5: Launch MongoDB and run Nodejs to start node server,
6: Test back-end server using Postman or Browser.
Other useful linkYouTube
AWS has a full documentation for this on their website. These documentations are extremely thorough and you should read these first.
MongoDB on AWS
Node.js on AWS
Express.js on AWS
Here's a tutorial from 2015.
I've been looking at various methods to run commands upon creation of EC2 instances using Elastic Beanstalk on AWS. I've been given different methods to do this through AWS Tech Support, including life cycle hooks, custom AMI's, and .ebextensions. I've been having issues getting the first 2 methods (life cycle hooks and custom AMIs) to work with EB.
I'm currently using .ebextensions to run commands upon deploy, but not sure if there's a way to run a set of commands upon creation only instead of every time I deploy code. For instance, I have a file .ebextensions/03-commands.config that contains the following code:
container_commands:
01_npm_install:
command: "npm install -g -f npm#latest"
However, I only want this code to run upon instance creation, not every time I deploy, as it currently does. Does anyone know a way to accomplish this?
Thanks in advance!
I would recommend creating an idempotent script in your application that leaves a marker file on the instance in some location say /var/myapp/marker using something like mkdir -p /var/myapp-state/; touch /var/myapp-state/marker on successful execution. That way in your script you can check that if the marker file exists you can make your script a no-op.
Then you can call your script from container commands but it will be a no-op everytime because on first successful execution it will create the marker file and subsequent executions will be no-ops.
Create a custom AMI. This way you can setup your instances whoever you want and they will launch faster
http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features.customenv.html
As I see from you question, you're using: container_commands, that is means you are using Elastic Beanstalk with Docker. Right? In this case I would recommended to read: http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create_deploy_docker.html.
The idea is following, that you can create own Dockerfile, where you can specify all commands that you need to build a docker container, for example to install all dependencies.
I would recommended to use .ebextensions for the Elastic Beanstalk customization and configuration, for example to specify ELB or RDS configuration. In the Dockerfile make sense to specify all commands, that you need to build a container for your application, that includes setup of the web server, dependencies etc.
With this approach, Elastic Beanstalk will build a container, that each time when you do deploy, it execute a docker instance with deployed source code.
There is a simple option leader_only: true you need to use as per current AWS Elasticbeanstalk configurations, You simply need to add this under
container_commands:
01_npm_install:
command: "npm install -g -f npm#latest"
leader_only: true
This is the link as per AWS
AWS Elasticbeanstalk container command option
I have today a strange issue for Amazon Elastick BeanStalk: for my instances I cannot upload application:
XXX#-Vostro-2520:~/git_projects/ProjectBlog (test-env)$ git aws.push --environment Project-Blog-test
Updating the AWS Elastic Beanstalk environment Project-Blog-test...
Error: Failed to create the AWS Elastic Beanstalk application version
I did test it for 2 instances: PHP and RoR apps.
I solved this on my own:)
Amazon EB allows only for 500 Application Versions. So I logged in to console -> Elastick BeanStalk -> Action -> View Application Versions
On these page I selected and deleted some of olded files uploaded to Amazon EB. After it I was able to upload new versions with EB scripts.
Heads up that the error may also appear if you have invalid characters in the commit description. Not sure what these are to AWS, but I got the same error with the characters "http://".
I'm interested in hosting nodejs applications in a cloud and I'm looking for a free cloud hosting for my purpose. I've found that Amazon has one but I have the following question: Are there any tutorials around how I can set up and run nodejs application in Amazon EC2?
EDIT: Can you provide any good hostings for nodejs (except heroku)?
I've been using Node.js with Amazon EC2 for a while and was quite happy with both of them. For the moment AWS seems to be the cheapest and the most robust cloud provider, so picking up Amazon wouldn't be a mistake. There's nothing special about running Node.js in the cloud - you work with it like if it were your own PC. Below are some general steps to follow for the simplest Node.js application running on EC2 Ubuntu server:
Create Amazon EC2 account.
From AWS console start t1.micro instance with any Ubuntu AMI (example).
Login via SSH to your instance.
Install node.js: sudo apt-get install nodejs
Create new file test_server.js with the following content:
require("http").createServer(function(request, response){
response.writeHeader(200, {"Content-Type": "text/plain"});
response.write("Hello World!");
response.end();
}).listen(8080);
Start the server: node test_server.js
Check it's working from another console: curl http://localhost:8080
Check out these tutorials (updated for 2021)
How to Deploy a Node.js Application On AWS EC2 Server
How to Deploy a Node.js application in AWS EC2
How To Deploy Your Node.js App On AWS With NGINX And SSL
Based on this tutorial, here's an updated step by step:
1) Make an account on Amazon Web Services.
2) Create an EC2 instance; I chose Ubuntu micro.
3) Configure Security Group (name it "Node") and add ports:
HTTP (80), HTTPS (443), and a custom TCP port for your Node app (e.g. 3000)
4) Launch the instance and save the pem file (private key), e.g. "node.pem".
5) On Windows - install Cygwin + OpenSSH package. it is also recommended to install WinScp to have "explorer like" access to the linux.
6) Open Cygwin Terminal as Administrator, and set correct permissions to "node.pem" file:
chown :Users node.pem
chmod 400 node.pem
7) Find your EC2 instance public DNS name in the EC2 dasboard, and connect to it with SSH:
ssh -i node.pem ubuntu#{your EC2 public DNS name}
8) Update Ubuntu and install NodeJS:
sudo apt-get update
curl -sL https://deb.nodesource.com/setup_7.x | sudo -E bash -
sudo apt-get install -y nodejs
sudo apt-get install -y build-essential
9) Copy your NodeJS application into the EC2 instance (via Cygwin, or Winscp).
10) Install all of your Node app required modules:
cd /home/ubuntu/My_Node_App
npm install --save
11) Re-route ports with IPtables so that your app can be accessed on default http port 80:
sudo iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-ports 3000
To view the iptables routing entries, run:
sudo iptables -t nat -L
If you need to remove routing entry (first line), run:
sudo iptables -t nat -D PREROUTING 1
12) Run your app as a background process:
sudo nohup node app.js &
To kill your app process:
ps -ef | grep app.js
sudo kill {proccess id number}
My blog post on how to deploy Node-based apps on EC2: http://devblog.daniel.gs/2014/01/deploying-node-apps-on-aws-ec2-with.html
Explaining:
Deploying Node apps from your github repo (private+public)
Automating the deployment process using scripts
Reverse proxy using Nginx
and using Forever utility.
Hope this helps.
There are quite some hosting solutions for Node.js available, here are a couple of these:
Joyent
Joyent is the corporate sponsor and trademark owner of Node.js and provides an appealing alternative to Amazon EC2 for many things, not the least Node.js hosting of course, see the Joyent's Node.js Development Environment (please check the Node.js™ Development SmartMachine Terms of Service though).
Apparently they are just restructuring this development offering though:
For the past year, Joyent Cloud has provided a free development
sandbox for users of Node.js. Over time, the community has made it
clear that they want more tools and more capacity. To this end, we are
excited to announce a partnership with Nodejitsu to provide both of
these in a world-class Node.js development environment with
Nodejitsu's development and management tools running on Joyent Cloud's
Infrastructure-as-a-Service platform. The new service will launch very
shortly.
Accordingly, it is not entirely clear yet how the pricing options for a production hosting of a Node.js solution will end up, but given Joyent's competitive pricing, I'd expect a similar option at least.
Cloud Foundry
The Cloud Foundry Open Platform as a Service Project support Node.js as well, amongst many other frameworks (which makes the platform so exciting), The platform is getting quite some traction recently and is meanwhile used by several solution Platform as a service (PaaS) providers as their backend accordingly - amongst these are (in no particular order and not necessarily complete):
AppFog - Simple PaaS for Java, Node, .Net, Ruby, PHP, MySQL, Mongo, PostgreSQL, and more...
Freedom to move between IaaS at will with the easiest pricing in the cloud.
Cloud Foundry (VMware) (corporate sponsor of Cloud Foundry) - Deploy and scale applications in seconds, without locking yourself into a single cloud.
Iron Foundry - Iron Foundry is an open source project that extends Cloud Foundry™ to the .NET ecosystem by providing services, installers, and developer tools.
Most of these are in beta still and the pricing isn't settled yet, but given the competition I'd expect quite some interesting options here over time.
The easiest way to run node.js for free on EC2 is IMHO on Heroku.
check out this complete tutorial here.
This tutorial shows how to install Node.js on EC2 and configure HTTP ports and nginx for port forwarding as well as using supervisor to run the Node.js forever as it normally stops on closing your SSH console session.
I just went through the Heroku sign-up and application tutorial. Could not have been easier. What a delightful experience...
...right up to the point where you can't have a MongoDB instance as a free gear. The minimum cost (other than a free trial month) is $18/month per GB of storage.
Honestly, the better choice then is Openshift. It's got three free gears which is enough for a lot of beginner stuff like what I'm doing. Both Heroku and Openshift are within Amazon's space but their customer interface is different. I thought Heroku's was easier for beginners to get started but as I mentioned, there's no free lunch on the database side of things.