I am a bit of a bind when it comes to this topic. Here's my situation, I have a discord-like style of the app where you join chat rooms. I now have the issue of how to handle and use a local DB in conjunction with the remote DB. As I'd only load the newest X messages of the current DB and then if the user scrolls get the other X and so on. But where does a local DB come into the picture? For example, I stored in the DB messages from 10 AM - 12 AM, should I get the oldest message from the local DB and skip the timeframe in the remote DB when requesting the data? What if someone deleted/ edited a message? From my current standpoint, it really seems that just having a remote DB is the way to go
Using Nodejs Typescript and MongoDB for the server and the client is a Kotlin multiplatform app and I have access to a SQL DB on the client if necessary.
Also anyhow I approach this, since this is a chatting App and I already have an established Websocket connection, would loading messages and receiving them via the Websocket make sense, or should I make a REST endpoint?
Your localdb is effectively a cache, and you should treat it as such. Now that you have a cache, you have another problem, cache invalidation. Worse, since each client has it's own, you have a distributed cache invalidation issue.
You can probably sidestep a lot of this by making message immutable, and having a REPLACES_ID and an IS_DELETED field which allows an "edit" to really be an add which then from a UI standpoint replaces another existing message.. This will at least give you something fairly computable from a cursor standpoint.
Ideally your ID field is an increasing numeric sequence that allows simple queries like "Give me X messages from this room where ID is greater than [[The last id in your cache]]" If no data is returned, your local DB has all the data, otherwise take data from the query and populate your local database to bring these messages in.
Scrolling is more complicated, but in general can be ignored, nobody scrolls (statistically irrelevant number of people go to page 2). I'd recommend if someone scrolls just wipe the local database past the first page and go back to the remote server for X records BEFORE your current page's first record.
I use socket on connection event. new players are created and seen, multiplayer array of objects in console, exists. However, not every event is seen properly (for example, 1. newest connections only see them self, while older see everyone on game. 2. I want also to show all players movements , real time. Dont know how node can handle that). For those issues in brackets, do I need to use Mongo DB or index DB to handle all data real time ?
You will need a database in order for your game to work in a distributed way i.e. so you can scale to more than more server / node process. If you are currently storing all connection/player data in memory, then this won't be accessible by other processes.
With regard to newest connections only see them self, while older see everyone on game I'd need to know more about how/where you store these connections.
For the second point, I want also to show all players movements , real time, I'll need more detail on how you are sending these movements from the client to server, & then broadcasting.
This is my first time developing a web application that requires chat functionality. There are multiple rooms and there are no restrictions as to how many people can join a room. The part that I can't get my head around is the actual approach to storing the messages. The question is more so in two parts...firstly am I correct to assume that unless all the messages are stored in a database, a newly connected user cannot retain all the previous messages? Secondly is it not recommended to save every message to the database as they are sent ? How else could I keep track if they aren't saved in real time ?
Thanks for any advice, appreciate it
If you want to store the data in memory and persist at a time interval, you can use redis for this. Also if all the data are required to store for future reference, you can implement scheduler like resque to transfer the data from redis to your db and free redis for application state data (i.e. more recent data).
Hope this will help you...
Thanks
Persistence is a business requirement, and yes a new user cannot see previous messages unless you have some sort of persistence storage. You can of course store messages permanently, and many apps out there do that. You can also store them temporarily and let the messages expire and be erased after a certain time period. All of this is easily accomplished using common tools such as MongoDB and Redis. If you do not need to persist messages, then you'll need to use web sockets to send messages between clients. Probably the most famous Node library for that is Socket.io
Intoduction
My current project has a mix of common RESTful API concepts and modern realtime websocket/long poling. I'm using mongoDB to store persistant data such as users, products, and aggregated social content. The social content is basically links to tumblr posts, twitter tweets, and facebook posts which are compiled into what I call a "shout".
Implementation
What I'm trying to accomplish is rating "shouts" based on how many likes or follows the post has out of the combined total from all social medias used. I want the data to change on the frontend as the backend updates. The back-end calls all the social medias based on checking an expiration date on the data. The server will check for new data on event that a request was made for the data. A request is made for the data every time a client connects, or everytime someone posts a new shout through my app. If there is not activity in a given duration of time, the shout is archived and updated every so often with scheduled jobs. I use socket.io to send realtime updates.
What I'm Using Redis For
The reason I need Redis is to message all my servers when one of them starts requesting data from the social media sources so I don't run into the issue where all of my servers are essentially doing the same thing when the task only needs to be done once. I also need to message my other services once a change is made. For these implementations I'm currently using Redis pub/sub. Since I'm currently using Redis, I also store session tokens in redis, and use it as a cache.
What I'm Using Mongo For
I use MongoDB to persist data, and I've setup indexing to tune performance specifically for my application.
The Problem
My problem is I feel like my stack is too big. I feel like using redis and mongo can be over kill. Should I cut out redis and use an MQ system, and store my sessions and cache in mongo and just index them for fast lookups? If so what MQ system would be suitable for my application?
Should I cut out mongodb and use all redis? Would this be cost effective for relatively large sums of data? As I would be storing hundreds of thousands(maybe more) shouts(essentially just URIs), thousands of users, and hundreds of thousands of products.
I have a good old-style LAMP webapp. A week ago I needed to add a push notification mechanism to it.
Therefore, what I did was to add node.js+socket.io on the server and poll the MySQL database every 10 seconds using node.js to check whether there were new items: if so, I would have sent them to the client(s) with socket.io.
I was pretty happy with the result, even if that is not a proper realtime notification (as there is a lag of up to 10 secs).
Now, I am about to build a new webapp which will need push notifications, too. I am wondering whether to go with the same approach as the first one (that I believe is more stable and mature) or to go totally Node.js, without PHP and Apache. As for the database, I have already decided to go for MongoDB.
Finally, my question is: if I go for Node.js+Socket.io+MongoDB will I get a truly near-real-time webapp? I mean, as soon as a new record is inserted into MongoDB, will there be some sort of event triggered that I can catch via node.js, do some checking on it and, if relevant, send the notification to the client? Or will there be anyway some sort of polling on the db server-side and lag, as with my first LAMP webapp?
A related question: can you build a realtime webapp on MySQL without doing any polling as I did with my first app. Or do you need MongoDB (or Redis)?
I hope this question is not too silly - sorry, I am just starting with Node.js and co.
Thanks.
I understand your problem because I switched to node.js from php/apache/mysql too.
Generally node.js is stable, modules and your scripts are the main reasons for errors
Real-time has nothing to do with database, it's all about client and server, you can query as many data as you want in your requests and push it to the other client.
Choosing node.js is very wise but it's harder to implement.
When you insert a new record to your db, the event is the request itself, you will make a push event along with the database query something like:
// Please note this is not real code, just an example of the idea
app.get('/query', function(request, response){
// Query your database
db.query('SELECT * FROM users', function(rows){
// Push notification to dan
socket.emit('database_query_executed', 'to_dan', rows);
// End request
response.end('success');
})
})
Of course you can use MySQL! And any database you want, as I said real-time has nothing to do with databases because the database is in the middle of the process and it's totally optional.
If you want to use node.js for push notifications and php/apache for mysql then you will need to create 2 requests for each server something like:
// this is javascript
ajax('http://node.yoursite.com/push', node_options)
ajax('http://php.yoursite.com/mysql_query', php_options)
or if you want just one request, or you want to use a form, you can call your php and inside php you can create an http or net request to node.js from php, something like:
// this is php
new HttpRequest('http://node.youtsite.com/push', HttpRequest::METH_GET);
Using:
A regular MongoDB Collection as the Store,
A MongoDB Capped Collection with Tailable Cursors as the Queue,
A Node worker with Socket.IO watching the Queue as the Worker,
A Node server to serve the page with the Socket.IO client, and to receive POSTed data (or however else the data gets added) as the Server
It goes like:
The new data gets sent to the Server,
The Server puts the data in the Store,
The Server adds the data's ObjectID to the Queue,
The Queue will send the newly arrived ObjectID to the open Tailable Cursor on the Worker,
The Worker goes and gets the actual data in the ObjectID from the Store,
The Worker emits the data through the socket,
The client receives the data from the socket.
This is 'push' from the initial addition of the data all the way to receipt at the client - no polling, so as real-time as you can get given the processing time at each step.
Re: triggers in MongoDB - please see this answer: https://stackoverflow.com/a/12405093/1651408
There are much more convenient triggers in MySQL, but to call Node.js from them would require a bit of work with MySQL UDFs (user-defined functions), for instance pushing data through a Unix socket. Please note that this is necessary only when other applications (besides your Node.js process) are updating the database, and be sure to choose InnoDB as storage in this case (row- vs. table-level locking).
Can see no big problem with your technology choice of sockets.io, even if client-side web sockets aren't supported, you'll fall back (gracefully, I hope) to polling.
Finally, your question is not silly at all, since push technology is definitely superior to the flood of polling requests - it scales better. EDIT: However, would not describe either technology as real-time.
Another EDIT: for a quite well-known and successful setup of this kind please read this: http://blog.fogcreek.com/the-trello-tech-stack/
Have you discovered Chole? It works separately from your web sever and interfaces with it by using HTTP POSTs. That way you can code your web app any which way you want.
Actually Using Push Technology like Socket.IO helps you to use
the server's resource efficiently and also helps you to leverage old browsers to modern browsers making websocket or websocket-like connection.
10 sec polling is a HTTP request which is expensive especially when a lot of users present.
Unlike polling technology, push technology is relatively cheap. Users' client is opening a dedicated socket(ie. websocket) to listen to the server's push notification.
And usually your client-side JavaScript do some actions when the push notification is received.
Using your LAMP stack and Socket.IO with different port (other than 80) will be good enough to implement what you need.
But using Node.js + MongoDB + Socket.IO actually helps you to manage your server's resource much efficiently.
Because those three have non-blocking nature.
If you understand non-blocking concept correctly and implement your app appropriately,
your identical app, an app with same feature but with different language and different database, would be able to handle a lot more requests than general LAMP stack.
Above picture is a famous chart of comparing Non-blocking vs Thread way to handle concurrency
Apache(Thread) vs Nginx(Non-blocking)
MySQL is a great database. I believe you won't need join and transactions for realtime notification.
MongoDB does not have those two features unless you implement similar features by yourself.
Because of not having those two and some characteristics of its own, MongoDB can store and fetch data much faster than traditional SQL databases.
Switching from MySQL to MongoDB will decrease the time taking to insert and fetch data.
with JS you can open a socket to your server (not old browser), the server will have a ah-hoc program (on an ad-hoc port, so you need the permission to open door and run program on your server) that will send data (almost) realtime from and to the client, and without the HTTP's protocol overhead.old browser will just fall-back to polling mechanism.
I can't see other way to do this (probably there are already "coocked" framework that do this)