I'm trying to create a Flash app with some real time functionality, and would like to use Redis' pubsub functionality which is a perfect fit for what I need.
I know that connecting to a data store directly from client is almost always bad. What are the security implications of this (since I'm not an expert on Redis), and are there ways to work around them? From what I read, there is a possible exploit of doing config sets and changing the rdb file location and be able to arbitrary overwrite files. Is there anything else? (If I don't use that particular redis instance for anything at all, i.e. no data being stored)
I understand the alternative is to write some custom socket server program and have it act as the mediating layer for connecting to redis and issuing commands -- that's the work I'd like to avoid having to write, if possible.
** Edit **
Just learned about the rename-command configuration to disable commands. If I disable every single command on the redis instance and leave only SUBSCRIBE and PUBLISH open, would this be good enough to run on production?
I think it would be a bad idea to connect directly your client to Redis. Redis offers an authentication system for a unique user only. It expects this user to be your server app.
From my point of view, directly exposing Redis is always a bad idea. It would allow anybody to access all of your data. This is confirmed by the Redis doc.
So you won't avoid adding or developing the server side of your app.
Related
I have an app that receives data from several sources in realtime using logins and passwords. After data is recieved it's stored in memory store and replaced after new data is available. Also I use sessions with mongo-db to auth user requests. Problem is I can't scale this app using pm2, since I can use only one connection to my datasource for one login/password pair.
Is there a way to use different login/password for each cluster or get cluster ID inside app?
Are memory values/sessions shared between clusters or is it separated? Thank you.
So if I understood this question, you have a node.js app, that connects to a 3rd party using HTTP or another protocol, and since you only have a single credential, you cannot connect to said 3rd party using more than one instance. To answer your question, yes it is possibly to set up your clusters to use a unique use/pw combination, the tricky part would be how to assign these credentials to each cluster (assuming you don't want to hard code it). You'd have to do this assignment when the servers start up, and perhaps use a a data store to hold these credentials and introduce some sort of locking mechanism for each credential (so that each credential is unique to a particular instance).
If I was in your shoes, however, what I would do is create a new server, whose sole job would be to get this "realtime data", and store it somewhere available to the cluster, such as redis or some persistent store. The server would then be a standalone server, just getting this data. You can also attach a RESTful API to it, so that if your other servers need to communicate with it, they can do so via HTTP, or a message queue (again, Redis would work fine there as well.
'Realtime' is vague; are you using WebSockets? If HTTP requests are being made often enough, also could be considered 'realtime'.
Possibly your problem is like something we encountered scaling SocketStream (websockets) apps, where the persistent connection requires same requests routed to the same process. (there are other network topologies / architectures which don't require this but that's another topic)
You'll need to use fork mode 1 process only and a solution to make sessions sticky e.g.:
https://www.npmjs.com/package/sticky-session
I have some example code but need to find it (over a year since deployed it)
Basically you wind up just using pm2 for 'always-on' feature; sticky-session module handles the node clusterisation stuff.
I may post example later.
I've been doing a lot of research lately and it appears to me that going stateless serverside brings benefits to both performance & scalability.
I am although trying to figure out how to achieve session-less-ness on Node.JS. It seems to me that basically all I have to do is assign a token to a logged in user, so I would have something like this in my DB:
{ user:'foo#example.com', pass:'123456', token:'long_id_here' }
so that the token can be send with every HTTP request like this:
/set/:key/:val/:token
to be checked against aforementioned DB object. Is this what it is actually meant to be a session-less web service?
If this is the right way, then I do not understand things like token expiry, and other security issues. I would like to be pointed out to NPM package of some sort?
On a side note, is it best for a token, to use a hash of the user+password, or to assign a different one at every login?
The reason to go sessionless is that most default session implementations use an in-memory store. That means that the session information is stored in memory local to that instance. Most websites these days are scaling out as traffic increases. This means they add more servers and balance the load between the servers. The problem with in-memory session stores is your user can log into Server 1, but if their next request is routed to Server 2, they don't have a session created yet and will appear to be logged off.
You don't necessarily need to go sessionless to scale out with node or any other server side language. You just need to use a session that isn't in local memory that would be accessible to all nodes. If you're using something like Express or Connect, you can easily use a session implementation like connect-redis which will enable you to have a fast session store which is accessible to all of your node instances so it doesn't matter which one is hit.
I have a good old-style LAMP webapp. A week ago I needed to add a push notification mechanism to it.
Therefore, what I did was to add node.js+socket.io on the server and poll the MySQL database every 10 seconds using node.js to check whether there were new items: if so, I would have sent them to the client(s) with socket.io.
I was pretty happy with the result, even if that is not a proper realtime notification (as there is a lag of up to 10 secs).
Now, I am about to build a new webapp which will need push notifications, too. I am wondering whether to go with the same approach as the first one (that I believe is more stable and mature) or to go totally Node.js, without PHP and Apache. As for the database, I have already decided to go for MongoDB.
Finally, my question is: if I go for Node.js+Socket.io+MongoDB will I get a truly near-real-time webapp? I mean, as soon as a new record is inserted into MongoDB, will there be some sort of event triggered that I can catch via node.js, do some checking on it and, if relevant, send the notification to the client? Or will there be anyway some sort of polling on the db server-side and lag, as with my first LAMP webapp?
A related question: can you build a realtime webapp on MySQL without doing any polling as I did with my first app. Or do you need MongoDB (or Redis)?
I hope this question is not too silly - sorry, I am just starting with Node.js and co.
Thanks.
I understand your problem because I switched to node.js from php/apache/mysql too.
Generally node.js is stable, modules and your scripts are the main reasons for errors
Real-time has nothing to do with database, it's all about client and server, you can query as many data as you want in your requests and push it to the other client.
Choosing node.js is very wise but it's harder to implement.
When you insert a new record to your db, the event is the request itself, you will make a push event along with the database query something like:
// Please note this is not real code, just an example of the idea
app.get('/query', function(request, response){
// Query your database
db.query('SELECT * FROM users', function(rows){
// Push notification to dan
socket.emit('database_query_executed', 'to_dan', rows);
// End request
response.end('success');
})
})
Of course you can use MySQL! And any database you want, as I said real-time has nothing to do with databases because the database is in the middle of the process and it's totally optional.
If you want to use node.js for push notifications and php/apache for mysql then you will need to create 2 requests for each server something like:
// this is javascript
ajax('http://node.yoursite.com/push', node_options)
ajax('http://php.yoursite.com/mysql_query', php_options)
or if you want just one request, or you want to use a form, you can call your php and inside php you can create an http or net request to node.js from php, something like:
// this is php
new HttpRequest('http://node.youtsite.com/push', HttpRequest::METH_GET);
Using:
A regular MongoDB Collection as the Store,
A MongoDB Capped Collection with Tailable Cursors as the Queue,
A Node worker with Socket.IO watching the Queue as the Worker,
A Node server to serve the page with the Socket.IO client, and to receive POSTed data (or however else the data gets added) as the Server
It goes like:
The new data gets sent to the Server,
The Server puts the data in the Store,
The Server adds the data's ObjectID to the Queue,
The Queue will send the newly arrived ObjectID to the open Tailable Cursor on the Worker,
The Worker goes and gets the actual data in the ObjectID from the Store,
The Worker emits the data through the socket,
The client receives the data from the socket.
This is 'push' from the initial addition of the data all the way to receipt at the client - no polling, so as real-time as you can get given the processing time at each step.
Re: triggers in MongoDB - please see this answer: https://stackoverflow.com/a/12405093/1651408
There are much more convenient triggers in MySQL, but to call Node.js from them would require a bit of work with MySQL UDFs (user-defined functions), for instance pushing data through a Unix socket. Please note that this is necessary only when other applications (besides your Node.js process) are updating the database, and be sure to choose InnoDB as storage in this case (row- vs. table-level locking).
Can see no big problem with your technology choice of sockets.io, even if client-side web sockets aren't supported, you'll fall back (gracefully, I hope) to polling.
Finally, your question is not silly at all, since push technology is definitely superior to the flood of polling requests - it scales better. EDIT: However, would not describe either technology as real-time.
Another EDIT: for a quite well-known and successful setup of this kind please read this: http://blog.fogcreek.com/the-trello-tech-stack/
Have you discovered Chole? It works separately from your web sever and interfaces with it by using HTTP POSTs. That way you can code your web app any which way you want.
Actually Using Push Technology like Socket.IO helps you to use
the server's resource efficiently and also helps you to leverage old browsers to modern browsers making websocket or websocket-like connection.
10 sec polling is a HTTP request which is expensive especially when a lot of users present.
Unlike polling technology, push technology is relatively cheap. Users' client is opening a dedicated socket(ie. websocket) to listen to the server's push notification.
And usually your client-side JavaScript do some actions when the push notification is received.
Using your LAMP stack and Socket.IO with different port (other than 80) will be good enough to implement what you need.
But using Node.js + MongoDB + Socket.IO actually helps you to manage your server's resource much efficiently.
Because those three have non-blocking nature.
If you understand non-blocking concept correctly and implement your app appropriately,
your identical app, an app with same feature but with different language and different database, would be able to handle a lot more requests than general LAMP stack.
Above picture is a famous chart of comparing Non-blocking vs Thread way to handle concurrency
Apache(Thread) vs Nginx(Non-blocking)
MySQL is a great database. I believe you won't need join and transactions for realtime notification.
MongoDB does not have those two features unless you implement similar features by yourself.
Because of not having those two and some characteristics of its own, MongoDB can store and fetch data much faster than traditional SQL databases.
Switching from MySQL to MongoDB will decrease the time taking to insert and fetch data.
with JS you can open a socket to your server (not old browser), the server will have a ah-hoc program (on an ad-hoc port, so you need the permission to open door and run program on your server) that will send data (almost) realtime from and to the client, and without the HTTP's protocol overhead.old browser will just fall-back to polling mechanism.
I can't see other way to do this (probably there are already "coocked" framework that do this)
I am creating a TCP based game server for iOS, it involves registration and login.
Users will be stored as a collection in MongoDB.
when login is done, I generate a unique session id - How ?
I wanted to know what all data remains with node server and what can be stored in db.
data like session tokens, or collection of sockets if I am maintaining a persistent connection etc.
Node.JS does not have any sessions by default. In fact, there is a plug-in for managing sessions with MongoDB.
It's not clear that you really need sessions however. If you're opening a direct socket with socket.io, that is a defacto session.
Node.js itself does not manage sessions for you. It simply exposes an API to underlying unix facilities for Socket communication. HTTP in it self is a stateless protocol and does not have sessions either. SSH on the other hand is a stateful protocol, but I do not think either one would be good for you.
Creating a uniuqe ID is really simple, all you need to do is hash some data about the user. Their SHA(IP address + time + username). See: http://nodejs.org/api/crypto.html
One approach a lot of applications take is to create their own protocol and send messages using that. You will have to handle a lot of cases with that. And I myself have never dealt with mobile where you have serious connectivity challenges and caching requirements that are not a big problem on desktops.
To solve these problem, founder of Scribd started a company called Parse which should make it much easier for your to do things. Have a look at their website: https://parse.com/.
If you want to do some authentication however, have a look at Everyauth, it provides a lot of that for you. You can find it here: https://github.com/bnoguchi/everyauth/.
Brief Description:
Well, since many days I've been looking for an answer to this question but there seems to be answers for 'How to create a Push Notification Server' and like questions. I am using node.js and it's quite easy to 'create' a push notification server using sock.js (I've heard socket.io isn't good as compared to sock.js). No problem till here. But what I want is how to model such a server.
Details:
OK, so, let's say I've an application where there's a chat service (just an example this is, actual thing is big as you might have guessed). A person sends a message in a room and all the people in the room get notified. But what I want is a 'stateful' chat - that is, I want to store the messages in a data store. Here's where the trouble comes. Storing the message in the database and later telling everyone that "Hey, there's a message for you". This seems easy when we need the real-time activity for just one part of the app. What to do when the whole app is based on real-time communication? Besides this, I also want to have a RESTful api.
My solution (with which I am not really happy)
What I thought of doing was this: (on the server side of course)
Data Store
||
Data Layer (which talks to data store)
||
------------------
| |
Real-Time Server Restful server
And here, the Real-time server listens to interesting events that the data-layer publishes. Whenever something interesting happens, the server notifies the client. But which client? - This is the problem with my method
Hope you can be of help. :)
UPDATE:
I think I forgot to emphasize an important part of my question. How to implement a pub-sub system? (NOTE: I don't want the actual code, I'll manage that myself; just how to go about doing it is where I need some help). The problem is that I get quite boggled when writing the code - what to do how (my confusion is quite apparent from this question itself). Could please provide some references to read or some advice as to how to begin with this thing?
I am not sure if I understood you correctly; but I will summarize how I read it:
We have a real-time chat server that uses socket connections to publish new messages to all connected clients.
We have a database where we want to keep chat logs.
We have also a restful interface to access the realtime server to get current chats in a lazier manner.
And you want to architect your system this way:
In the above diagram, the components I circled with purple curve wants to be updated like all other clients. Am I right? I don't know what you meant with "Data Layer" but I thought it is a daemon that will be writing to database and also interfacing database for other components.
In this architecture, everything is okay in the direction you meant. I mean DataStore is connected by servers to access data, maybe to query client credentials for authentication, maybe to read user preferences etc.
For your other expectation from these components, I mean to allow these components to be updated like connected clients, why don't you allow them to be clients, too?
Your realtime server is a server for clients; but it is also a client for data layer, or database server, if we prefer a more common naming. So we already know that there is nothing that stops a server from being a client. Then, why can't our database system and restful system also be clients? Connect them to realtime server the same way you connect browsers and other clients. Let them enjoy being one of the people. :)
I hope I did not understand everything completely wrong and this makes sense for the question.