I am dividing the load on my Database and want to retrieve data from ES and write data to MongoDB. Can I sync them real time? I have checked the Transporter library but I want to do it for realtime.
There are several ways to achieve that :
Using your own application server. Whenever you are inserting a new
document in the mongo, put it in the ES as well at the same time.
That way you will maintain the consistency with minimum latency.
Use logstash. It has near realtime pipelining capabilities.
You can use elasticsearch mongodb river. Its a plugin used for data synchronization between mongo and elasticsearch.
Related
I am setting up a website with MERN stack. in the backend I will be constantly fetching data from api/socket and saving that to the Mongodb database. Now the frontend React I want to real-time show/update data with Socket.
I am a bit worried about the amount of request/connections the server side and Mongodb can handle.
Its not clear how to setup a proper system to handle millions of users.
If someone can give me some info on what, how and/or where to search on how to setup a stable system.
Any info is welcome, thank you.
This is a suggestion for the MongoDB portion.
You may want to try out their official course on MongoDB to understand how their transaction works. MongoBD University, for hands-on try their "MongoDB for JavaScript Developers" course.
Here are some things to look out for:
Connection Pooling
Understand how transactions work
Indexes, especially on how to create a good one like following the ESR
MongoDB Clusters, Write into primary and read from secondary
I will update the list if I have time and remember some of the stuff.
I have two Node.js applications each running Mongoose on a different machine. There is a single MongoDB database running on the first, and the second connects to it and adds documents periodically. I'm trying to add a hook to the creation of these documents so the server running the database is aware that other server has added data. I tried using the Schema.post() method, but it doesn't seem to work since there are two separate instances of Mongoose. Is this true or am I just implementing it incorrectly? I can get the hook to fire if the document is created on the same server, but not the other.
So my thought is to add the hook to MongoDB directly, instead of Mongoose, but I'm not sure how to go about doing that. Am I on the right track?
That is true, Schema.post() only work in the same process. You either need to use a library that tails MongoDB's oplog (like mongo-oplog), implement it yourself using a message queue (or pub/sub) that all instances are connected to (like Redis, RabbitMQ, etc) or use a database that supports this natively. PostgreSQL supports this with its NOTIFY feature for example.
Intoduction
My current project has a mix of common RESTful API concepts and modern realtime websocket/long poling. I'm using mongoDB to store persistant data such as users, products, and aggregated social content. The social content is basically links to tumblr posts, twitter tweets, and facebook posts which are compiled into what I call a "shout".
Implementation
What I'm trying to accomplish is rating "shouts" based on how many likes or follows the post has out of the combined total from all social medias used. I want the data to change on the frontend as the backend updates. The back-end calls all the social medias based on checking an expiration date on the data. The server will check for new data on event that a request was made for the data. A request is made for the data every time a client connects, or everytime someone posts a new shout through my app. If there is not activity in a given duration of time, the shout is archived and updated every so often with scheduled jobs. I use socket.io to send realtime updates.
What I'm Using Redis For
The reason I need Redis is to message all my servers when one of them starts requesting data from the social media sources so I don't run into the issue where all of my servers are essentially doing the same thing when the task only needs to be done once. I also need to message my other services once a change is made. For these implementations I'm currently using Redis pub/sub. Since I'm currently using Redis, I also store session tokens in redis, and use it as a cache.
What I'm Using Mongo For
I use MongoDB to persist data, and I've setup indexing to tune performance specifically for my application.
The Problem
My problem is I feel like my stack is too big. I feel like using redis and mongo can be over kill. Should I cut out redis and use an MQ system, and store my sessions and cache in mongo and just index them for fast lookups? If so what MQ system would be suitable for my application?
Should I cut out mongodb and use all redis? Would this be cost effective for relatively large sums of data? As I would be storing hundreds of thousands(maybe more) shouts(essentially just URIs), thousands of users, and hundreds of thousands of products.
What is the benefit of using Redis as socket.io memory store, does it need additional resources. I'm using MongoDB as the database, can i use MongoDB as memory store for Socket.io, or do i replace MongoDB with Redis as database? What would be more efficient for building a real-time web app and providing maximum concurrent connections?
can i use MongoDB as memory store for Socket.io
Yes, you can try mong.socket.io
do i replace MongoDB with Redis as database?
Redis and MongoDB are different kind of databases, while mongodb is document oriented redis is key/value oriented (we can even say that redis is a data-structure server).
What would be more efficient for building a real-time web app and providing maximum concurrent connections?
Redis will be definitely faster than mongo on that matter, it supports pub/sub out of the box (while mong.socket.io uses a collection to simulate pub/sub) but you must know that all your data stored in redis must live in memory (here the only data that will be stored in redis will be additionnal socket.io informations).
I have a node.js app that polls every x seconds a mongodb database to show data variations.
I think there must be a better way to do it, using all the capabilities of node.js environment ... which is the most efficient way to trigger mongodb data variations and showing them using node?
You can use a tailable cursor
http://docs.mongodb.org/manual/tutorial/create-tailable-cursor/