I am trying to host a simple Node.js app connected to MongoDB, I look on the web, I've found 2 different methods and some guides to do so:
Different pods:
Example from the official Kubernetes documentation.
Example from the official Google Cloud Platform documentation.
Same pod:
Example from the official IBM documentation.
This confused me a bit, which is the best practise to achieve that?
You should put them in different pods.
Being in different pods you can then scale up the number of replicas of the web application part to as many instances as you need and you will still have the one instance of MongoDB.
If you had them in the same pod, you wouldn't be able to scale up to multiple instances as each replica would have its own MongoDB instance and thus separate data. If doing data updates to MongoDB from the web application, that would cause all sorts of problems as subsequent requests could hit a difference instance and so see different data.
Related
I am looking to use GCP for a micro-services application. After comparing AWS and GCP I have decided to go with Google because one major requirement for the project is to schedule tasks to run in the future (Cloud Tasks) which AWS does not seem to offer an equivalent of.
I am planning on containerizing my services and deploying to GCP using Cloud Run with a Redis cluster running as well for caching.
I understand that you cannot have multiple Firestore instances running in one project. Does this mean that all if my services will be using the same database?
I was looking to follow a model (possible on AWS) where each service had its own database instance that it reached out to.
Is this pattern possible on GCP?
Firestore indeed is for the moment limited to a single database instance per project. For performance that is usually not a problem, but for isolation such as your use-case, that can indeed be a reason to look elsewhere.
Firebase's original Realtime Database does allow multiple instances per project, and recently added a REST API for provisioning database instances. Since each Google Cloud Project can be toggled to also be a Firebase project, you could consider that.
Does this mean that all if my services will be using the same database?
I don't know all details of your case. Do you think that you can deploy a "microservice" per project? Not ideal, especially if they are to communicate using PubSub, but may be an option. In that case every "microservice" may get its own Firestore if that is a requirement.
I don't think one should consider GCP project as some kind of "hard boundaries". For me they are just another level of granularity - in addition to folders, etc.
There might be some benefits for "one microservice - one project" appraoch as well. For example, less dependent lifecycles, better (more accurate) security, may be simpler development workflows...
I am developing an iOS app to show news from a MongoDB database. The app has around 50,000 active users, so it's quite heavy on the server. I am trying to rethink how the API should be built. I have just learned a little about AWS API Gateway, Google Cloud Functions, Firebase, etc.
If I simply need a few functions to extract list of news, list of users, etc., what would be the best way to build this API as of 2017? I have always thought I should simply create a Node.js server with some endpoints. But now it seems it's more performant to create separate endpoints with, for instance, AWS API Gateway which each points to an AWS Lambda function.
But what is really the most scalable option?
Cloudfront (With Caching) --> API Gateway --> Lambda would be a scalable solution. Since you have not chosen DynamoDB, you need to manage mongoose for your storage and avaibility.
To make it bit more advanced, you can use Lambda Edge with Cloudfront to make it active/active across more than one region. So if One region goes down, app will still be available.
When you say "Performant", what does that mean in your scenario?
If you need a few functions that extract a list of news, a list of users and etc, it doesn't sound like performance would be an issue.
If you're wondering if the AWS Serverless Stack can handle thousands or millions of requests, the answer is yes, API Gateway + Lambda can handle requests at any scale.
If you only need to query data by its hash keys, DynamoDB would fit very well! Also, there's an auto scale feature. But, if you need to perform scans or some complex queries, that would be a little bit expensive and some times slow. For this scenario you can stream your DynamoDB data to Elasticsearch or a data warehouse solution.
I'm not sure if combining Lambda with MongoDB would be that great. Let's say you choose a hosted MongoDB service like Mongo Atlas, you'll need to add more complexity to your system like VPC Peering, and also, manage/optmize lambda connections to MongoDB (Optimizing AWS Lambda Performance with MongoDB Atlas). What if you're handling a million request calls, I guess a lot of Lambda Functions will start running in parallel, how many connections would be opened in your MongoDB?
Besides the AWS Serverless Stack, there's the Elastic BeansTalk that can easily scale your system.
All the solutions have pros and cons, you can scale with both, API Gateway + Lambda or Elastic Beanstalk(or some other vendor solution). I guess "scalable" is a built in feature that cloud vendors are offering these days, and "performance" is just one of the topics that should be analyzed when designing your infrastructure.
In order to use MongoDB on my node.js AWS EC2 instance do I simply install MongoDB and create a database within the instance via the command line after logging in via SSH?
In other words do I simply create a DB in the EC2 instance for my web app just as I would locally on my machine?
just from long (and at the beginning painful) experience with EC2 & MongoDB..here are some gotcha's.
I am assuming you are just starting off from your question - so I am going to assume a minimal setup:
If you install MongoDB on a server with access to the Internet, then make sure you also apply MongoDB roles to your DB. Do not, I repeat, do not leave it open to the world. Admin and read/write roles are critical here, and MongoDB docs will help you. BTW, even if it is totally secure behind a firewall and other such things, always use roles. They are your last line of defense.
Study and understand exactly how security groups work, in order to limit Inbound and Outbound.
If you can, use the Elastic IP. It will save you many headaches if you move servers, not the least of which is that your IP will not change.
When you gear up to a front facing Internet server, and Mongo behind it, be it with Sharding, Clusters etc. read up on the NAT gateway. It is confusing at first, but the NAT Gateway (not the NAT instance), is a decent setup in one configuration or another.
Take snapshots or complete images of your environment when you change it. This is great for backup, and also when you move to a more robust server, it will save you a great deal of work.
If you have not already, try using MongoBooster or RoboMongo. They will help you immensely with your Mongo work.
Good luck and enjoy!
The actual AWS implementation of MongoDB is DocumentDB, which from what i can tell is built on the opensource version of MongoDB version 3.6, so newer MongoDb features are/might/will not be supported.
An interesting article comparing DocumentDb with MongoDb Atlas(mongoDm cloud solution):
https://medium.com/#michaelrbock/nosql-showdown-mongodb-atlas-vs-aws-documentdb-5dfb00317ca2
In the end if you really want MongoDB on AWS my opinion is you should just install it on a EC2 machine, I've done it via DocumentDB and some mongodb commands don't work, or chose AWS own NOSQL solution DynamicDB instead, DocumentDB just seems to be up there for competition with MongoDB Atlas cloud solution or just for having some dedicated MongoDB for companies that use it and want to move to AWS.
You have different alternatives. To answer your question: yes, you can do it that way. But, there is also an official guide by Amazon to set up a MongoDB cluster on AWS.
Also, if you only need a NoSQL database, you should also check DynamoDB, developed by Amazon. That would eliminate the need of an EC2 instance for the database. For more info, check the official docs.
I chose to use docker with node-js and mongoDB to make a game, and set all of my game sections (battles, chat, resources, etc) into different servers so each section will have its own process, because for example my resource server will be running a lot as it will do all the calculation of the resources of each user every second and will handle requests from other servers, like if the user have enough resources to build a building or how much resources the player will lose if attacked etc.
For my alpha and beta versions i'm planning on running one server, that will run all the sections of my game, but the way i'm doing it now, every section is having its own mongodb instance, so i have resourcesDB, mainDB (users info, login info and stuff), all the instances won't have many collections, for example resourcesDB have only 2 collections, resource collections, which each user will have 1 document with stuff related to resources, and logs collection that will store all user's usage (building upgrades, battles lost etc.)
I have read about multi instances on the same servers, and they all said that its not the best way to do it, as it will have an impact on my performance, in the future i am planning on separate all of my sections into different servers so each section will have its own server and DB server if needed.
Should I maybe build 1 instance and separate it on the database level? that means that I will have to connect to the same instance from multiple servers, resource server will update all the user's resources count every second, combine that with battle server that will update the state of troops in DB and more for each war that is going on, i will also have chat database that will be updating with each private message or chat message, will it cause any issues? or it will cause problems only if I connect to a single database and separate it on the collections level?
Is there a reason why i should not continue with creating multiple instances? i read that there might be a problem with config files if using multiple instances on same server, but i assume docker handle that part.
UPDATE - i have found an excellent answer here: https://dba.stackexchange.com/questions/156811/mongodb-in-micro-services-structure/156984#156984
Using microservice architecture for your app is good idea but I don't see a good reason to separate the database. MongoDB can handle it for you (Sharding). Also MongoDB not block write to collection A when you write to collection B. if you will separate your app to multiple database it will be hard to do backups and to maintain the app.
You can read here about concurrency options for MongoDB and see how MongoDB handle collection level locking.
https://docs.mongodb.com/manual/faq/concurrency/
I'm currently writing an application which has sockets engine running on one node.js instance and restful API for various resources on other. I'm wondering what could be the best way (performance wise) to communicate between these to instances if they would be located on a different servers. I don't want to duplicate data access logic on different instances and I want to keep everything (what's related to data access) in one place.
I thought about using simple HTTP API where sockets engine would make request to resources instance and get what it asked for but I'm not sure if I won't hit any performance bumps. In fact I could actually connect straight to mongodb and get the data that I want from there without making a request to other service but I feel like it would be nice to isolate data so that could be only used by resources instance and that would be the only application that uses mongodb instance.
Is there any other better ways to achieve something similar?