Recently I have been considering using MongoDB Atlas for a deployment which is fairly heavy in the DB department and also on the Node.JS server side.
Whereas I am quite familiar with the normal NAT Gateway--> VPC layout on AWS, in all my research on Atlas I have been left terribly confused about some really important specifics.
I am incredibly confused as to how to implement a NodeJS server with MongoDB Atlas. If I set up my own NAT Gateway & VPC on AWS with all the servers, replicas, shards etc. then I know exactly where my entry point is, where my NodeJS server is on that zone and how to replicate etc. But if use this scenario then I lose all the comfort and time-saving which Atlas affords me, not to mention the headaches of scalability etc.
However, in all the Atlas documentation, I can only find how it sets up Atlas on a VPC. I cannot figure out, (maybe my brain cells are not working), where in all this configuration the Node server would go? Or can one set up a NAT Gateway and VPC and then place an Atlas controlled MongoDB setup behind it?
The architecture with Atlas has me confused. Where do you set up your servers so they are also secure? Do I need to create a second NAT Gateway and VPC just for my Node server app? If so do I require VPC peering?
If this is dumb, please excuse the question. But before I begin a setup which will cost the company a good deal of money, I need to be clear on exactly what the layout is.
Thanking all who can handle this question in advance as any and all pointers here would be greatly appreciated.
MongoDB Atlas
You can create a VPC Peering Connection from MongoDB Atlas VPC to a VPC in your AWS account where you have your NodeJS EC2 instances running. Check the documentation Introducing VPC Peering for MongoDB Atlas for more details.
Your Custom MongoDB Cluster
Setting up MongoDB with best practices will require creating public and private subnets in the VPC while placing the database cluster in the private subnet. It will also involve having a NAT server in the public subnet for the MongoDB cluster to forward egress traffic keeping it secure within the VPC.
Reference: AWS Quick Start for MongoDB
However this can be challenging if you are new to AWS. To simplify the work use a AWS Quick Start for AWS Quick Start for MongoDB which will create the required architecture with best practices.
After provisioning the MongoDB Cluster, you can provision a NodeJS Web Server using one of the following options,
Using Elastic Beanstalk to place the NodeJS Servers with AutoScaling and Load Balancing with Best Practices.
Setup an Application Load Balancer, AutoScaling Group and NodeJS EC2 instance Launch Configuration inside the VPC to manually.
Create a EC2 template (Newly introduced) with required configuration.
Manually setup a server inside the VPC.
Note: If you place your NodeJS Web Servers in Private Subnet, make sure you create a public load balancer and configure the subdomains where NodeJS We Servers are allowed for load balancer as well so that it can route traffic to any subnet where the automated provisioning places the NodeJS Web Servers.
Related
I'm new to Kubernetes and Rancher, but have a cluster setup and a workload deployed. I'm looking at setting up an ingress, but am confused by what my DNS should look like.
I'll keep it simple: I have a domain (example.com) and I want to be able to configure the DNS so that it's routed through to the correct IP in my 3 node cluster, then to the right ingress and load balancer, eventually to the workload.
I'm not interested in this xip.io stuff as I need something real-world, not a sandbox, and there's no documentation on the Rancher site that points to what I should do.
Should I run my own DNS via Kubernetes? I'm using DigitalOcean droplets and haven't found any way to get Rancher to setup DNS records for me (as it purports to do for other cloud providers).
It's really frustrating as it's basically the first and only thing you need to do... "expose an application to the outside world", and this is somehow not trivial.
Would love any help, or for someone to explain to me how fundamentally dumb I am and wha tI'm missing!
Thanks.
You aren't dumb, man. This stuff gets complicated. Are you using AWS or GKE? Most methods of deploying kubernetes will deploy an internal DNS resolver by default for intra-cluster communication. These URLs are only useful inside the cluster. They take the form of <service-name>.<namespace>.svc.cluster.local and have no meaning to the outside world.
However, exposing a service to the outside world is a different story. On AWS you may do this by setting the service's ServiceType to LoadBalancer, where kubernetes will automatically spin up an AWS LoadBalancer, and along with it a public domain name, and configure it to point to the service inside the cluster. From here, you can then configure any domain name that you own to point to that loadbalancer.
I have a NodeJs web application running on amazon EC2 server.
Now from this node app in EC2, I have to access a database system (SqlServer) which is in the customer's in house network which can be accessed only with a VPN. What are the possible ways to do this?
Note:
- In house db cannot be exposed to public
There are a three options:
1) Expose your database publicly, and connect from your app using a secure protocol (i.e. ssl). This is probably a horrible idea, but is possible.
2) Set up a VPN between AWS and the data center where the database lives. This is a quick, easy way to set up a hybrid architecture.
3) Set up Direct Connect between AWS and the data center. This can reduce latency, provide network sovereignty, and depending on the amount of traffic between the app and the db may actually be cheaper than option 2.
You can setup a VPN between the VPC and the customers network.
ref : https://aws.amazon.com/premiumsupport/knowledge-center/create-connection-vpc/
Some of my data is in Mongo replicas that are hosted in docker containers running in kubernetes cluster. I need to access this data from the AWS lambda that is running in the same VPC and subnet (as the kubernetes minions with mongo db). lambda as well as the kubernetes minions (hosting mongo containers) are run under the same security group. I am trying to connect using url "mongodb://mongo-rs-1-svc,mongo-rs-2-svc,mongo-rs-3-svc/res?replicaSet=mongo_rs" where mongo-rs-x-svc are three kubernetes services that enables access to the appropriate replicas. When I try to connect using this url, it fails to resolve the mongo replica url (e.g. mongo-rs-2-svc). Same URL works fine for my web service that is running in its own docker container in the same kubernetes cluster.
Here is the error I get from mongo client that I use...
{\"name\":\"MongoError\",\"message\":\"failed to connect to server [mongo-rs-1-svc:27017] on first connect [MongoError: getaddrinfo ENOTFOUND mongo-rs-1-svc mongo-rs-1-svc:27017]\"}". I tried replacing mongo-rs-x-svc to their internal ip addresses in the url. In this case the above name resolution error disappeared but got another error - {\"name\":\"MongoError\",\"message\":\"failed to connect to server [10.0.170.237:27017] on first connect [MongoError: connection 5 to 10.0.170.237:27017 timed out]\"}
What should I be doing to enable this access successfully?
I understand that I can use the webservice to access this data as intermediary but since my lambda is in VPC, I have to deploy NAT gateways and that would increase the cost. Is there a way to access the webservice using the internal endpoint instead of public url? May be that is another way to get data.
If any of you have a solution for this scenario, please share. I went through many threads that showed up as similar questions or in search results but neither had a solution for this case.
This is a common confusion with Kubernetes. The Service object in Kubernetes is only accessible from inside Kubernetes by default (i.e. when type: ClusterIP is set). If you want to be able to access it from outside the cluster you need to edit the service so that it is type: NodePort or type: LoadBalancer.
I'm not entirely sure, but it sounds like your network setup would allow you to use type: NodePort for your Service in Kubernetes. That will open a high-numbered port (e.g. 32XXX) on each of the Nodes in your cluster that forwards to your Mongo Pod(s). DNS resolution for the service names (e.g. mongo-rs-1-svc) will only work inside the Kubernetes cluster, but by using NodePort I think you should be able to address them as mongodb://ec2-instance-1-ip:32XXX,ec2-instance-2-ip:32XXX,....
Coreyphobrien's answer is correct. Subsequently you were asking for how to keep the exposure private. For that I want to add some information:
You need to make the Lambdas part of your VPC that your cluster is in. For this you use the --vpc-config parameter when creating the lambdas or updating. This will create a virtual network interface in the VPC that allows the Lambda access. For Details see this.
After that you should be able to set the AWS security group for your instances so that the NodePort will only be accessible from another security group that is used for your Lambdas network interface.
This blog discusses an example in more detail.
I am trying to connect my app, running on one EC2 instance, to MongoDB, running on another EC2 instance. I'm pretty sure the problem is in the security settings, but I'm not quite sure how to handle that.
First off, my app's instance is in an autoscaling group that sits behind an ELB. The inbound security settings for the instance and ELB allow access to port 80 from anywhere, as well as all traffic from its own security group.
The EC2 instance that runs Mongo is able to take connections if the security group for that instance accepts all inbound traffic from anywhere. Any other configuration that I've tried causes the app to say that it cannot make a connection with the remote address. I've set rules to accept inbound traffic from all security groups that I have, but it only seems to work when I allow all traffic from anywhere.
Also, my db instance is set up with an elastic ip. Should I have this instance behind an ELB as well?
So my questions are these:
1) How can I securely make connections to my EC2 instance running mongo?
2) In terms of architecture, does it make sense to run my database this way, or should I have this behind a load balancer as well?
This issue is tripping me up a lot more than I thought it would, so any help would be appreciated.
NOTE
I have also set the bind_ip=0.0.0.0 in /etc/mongo.conf
Your issue is that you are using the public elastic IP to connect to your database server from your other servers. This means that the connection is going out to the internet and back into your VPC, which presents the following issues:
Security issues due to the data transmission not being contained within your VPC
Network latency issues
Your database server's security group can't identify the security group of the inbound connections
Get rid of the elastic IP on the MongoDB server, there is no need for it unless you plan to connect to it from outside your VPC. Modify your servers to use the private internal IP address assigned to your database server when creating connections to it. Finally, lock your security group back down to only allow access to the DB from your other security group(s).
Optional: Create a private hosted zone in Route53, with an A record pointing to your database server's private IP address, then use that hostname instead of the internal IP address.
I am setting up a basic dev, qa and production environment for an app in EC2. A dsn string to connect the web app to the database looks something like
jdbc:postgresql://64.210.255.235:5432/mydb
But here is where things break down, to be secure, I need to add the elastic ip addresses of the web heads to the database's security group, in addition to elastic ips to connect to the database via dsn. However, EC2 apparently lets you only have 5 elastic ips? Help, what do I do? And please explain to me like a 6th grader, I am a developer with almost no system administration or AWS experience. For an answer I need concrete steps to make this work, not high level observations.
first of all, good handle. second, what Im trying to do conceptually is super simple. I have 3 environments, 6 servers. I want to have stable jdbc dsns that I dont have to change when the EC2 DHCP updates. I need my web heads to talk to my db boxes and be secure.
2 options:
use VPC, so that you can have persistent privat IP for your DBs.
Ask amazon to increase your EIP limit by filling up this form.