Intro
We're developing a system to support multiple real-time messages (chat) and updates (activity notifications).
That is, user A can receive via Web Socket messages for :
receiving new chat messages
receiving updates for some activity, for example if someone like their photo.
and more.
We use one single WebSocket connection to send all these different messages to the client.
However, we also support multiple applications/clients to be open by the user at the same time.
(i.e - user A connect on their web browser, and also from their mobile app, at the same time).
Architecture
We have a "Hub" that stores a map of UserId to a list of active websocket sessions.
(user:123 -> listOf(session#1, session#2))
Each client, once websocket connection is established, has its own Consumer which subscribes to a pulsar topic "userId" (e.g - user:123 topic).
If user A connected on both mobile and web, each client has its own Consumer to topic user:A.
When user A sends a new message from session #1 to user B, the flow is :
user makes a REST POST request to send a message.
service stores a new message to DB.
service sends a Pulsar message to topic user:B and user:A.
return 200 status code + created Message response.
Problem
If user A has two sessions open (two clients/websockets), and they send a message from session #1, how can we make sure only session #2 gets the message ?
Since user A has already received the 200 response with the created message in session #1, there's no need to send the message to him again by sending a message to his Consumer.
I'm not sure if it's a Pulsar configuration, or perhaps our architecture is wrong.
how can we make sure only session #2 gets the message ?
I'm going to address this at the app level.
Prepend a unique nonce (e.g. a guid) to each message sent.
Maintain a short list of recently sent nonces,
aging them out so we never have more than, say, half a dozen.
Upon receiving a message,
check to see if we sent it.
That is, check to see if its nonce is in the list.
If so, silently discard it.
Equivalently, name each connection.
You could roll a guid just once when a new websocket is opened.
Or you could incorporate some of the websocket's addressing
bits into the name.
Prepend the connection name to each outbound message.
Discard any received message which has "sender" of "self".
With this de-dup'ing approach
there's still some wasted network bandwidth.
We can quibble about it if you wish.
When the K-th websocket is created,
we could create K topics,
each excluding a different endpoint.
Sounds like more work than it's worth!
I'm very new to node.js and sokcet.io that's why I need to ask you about the plan I have to see if it makes sense or is plain stupid. I need ongoing server/client communication for two reasons: sending real-time notifications to the user when they have one, and two, for chat between users.
Here is my plan for managing notifications:
PHP script finds out user X has a new notification.
Using Elephant.io send a message to server with user X's id as the data.
On the server side, upon receiving the message, if user X is connected emit him a message telling they have a notification.
user X's brower, Upon recieving the message, uses AJAX to poll the database and receive the text for the notification.
For chat, this is my plan (messages should be save on DB):
When user X submits a chat message to user Y, use ajax to send the text to a PHP script and save it on DB. On success, use elephant.io to send a message to user Y telling them that they have a new chat message.
User Y's browser, after receiving the server message, uses AJAX to poll a php script to receive the new text.
Do you think these plans are superior to short polling using AJAX? I appreciate any comments to improve them.
Finally,I'm curious to know how reliable these technologies (node.js, socket.io, elephant.io) are. Do they work well when the server becomes busy? How do they handle exceptions and errors ,etc.
To be able to chat with someone by sending the vLine im-type messages, you don’t need to have a mediaSession established with the remote user first, correct? All you need is the remote person’s ID and you can send them message, correct?
Or not? And if not, how do you do send an IM with vLine?
Correct. You do not need to have a mediaSession in order to send an IM. The vline.MediaSession is only used for audio and/or video sessions.
To send an IM, you just need to have the vline.Person object representing the person that you want to send the message to. You can then call postMessage or publishMessage to send the IM, with the difference between the two being that postMessage saves the IM history, while publishMessage does not.
Also, you can send IMs to a user even if that user is not online and use the Person.getUnreadCount API to keep track of the number of unread messages from a specific Person.
I have a system of personal messages between users on node.js sockets.io, is now faced with the problem that I do not understand how to make the notification about unread messages. Let's say, how to determine what a user who has received a message read this message, and how to make a notification message if the user is not in correspondence?
ps Chat History and active dialogs are stored in mysql.
Assume if the user has an open socket, and the message has been sent, then when the user clicks in a certain area of the page (perhaps the send message box?), an onClick event is fired to notify the server the user has received the message.
If the user is not on the webpage, there's not a way to contact them through sockets.io.
I'm using Php and MySQL.
I have just signup for pubnub push API and successfully made my first push notification using the PHP Push API provided by Pubnub. I'm new into this Realtime technology so I have come up with some questions which I find confusing to understand myself. I have googled many times and searched all over stackoverflow. I didn't get any relevant suggestions or questions elsewhere, so I'm writing down my question here seeking your advice and expertise help.
Pubnub says it's not a good thing to create more than two channels per client. So, in my application, I have a requirement to create more than two channels to listen to notifications happening everywhere on my website but I will go with two channels for every logged in users as Pubnub suggested.
Logged in users listens to Channel1-Public
Logged in users listens to private UsersOwnDynamic-Channel to receive notifications related and only meant for him.
FYI: This link in PubNub says about creating LongChannel names to avoid Channel Snooping
My questions are given below:
A. Do I always have to create a new private Dynamic channel name everytime I logged into the website. If so, how would other users know how to send a notification to my private Channel.Or, do I just need to have only one static channel name stored in database table, so that other authenticated users will query the table and get my private channel name to sent me notifications. If this is the case, don't you think if a hacker get hold of some private channel name of certain users, they will be able to listen to that channel?
B.I'm using PHP and MySQL, so I still cannot think out a way or come up with a solution to send message to private channels of another user.
Lets take an example for a simple friend request system.
- UserA sends a friend request to UserB.
- UserB is listening to his own dynamic private channel name called DynamicPrivateChannelB
(how will UserA find the channel name for UserB which is private? Im thinking the only way for this is that the private channel of UserB should be stored in a database table for every loggedin users to query. Am I thinking the right way ? )
<?php
//first way. How can i possibly achieve this.
$sqlquery = "sent friend request from userA to userB";
require('Pubnub.php');
$pubnub = new Pubnub( 'pubkey', 'subkey' );
$pubnub->publish( array(
'channel' => 'how do i find the private channel name for userB to sent this notification?',
'message' => array('friend_request' => 'A friend request') )
);
//2nd way ? Is this the right way ?
$sqlquery = "sent friend request from userA to userB";
$privatechannelofuserB = "get the channel name of userB from the db table";
require('Pubnub.php');
$pubnub = new Pubnub( 'pubkey', 'subkey' );
$pubnub->publish( array(
'channel' => '$privatechannelofuserB',
'message' => array('friend_request' => 'A friend request') )
);
?>
C. If we are always generating dynamic private channel names, storing in database table, updating whenever new dynamic channel names are generated. I think it will cause a problem because some message won't get delivered as new dynamic private channel names replaces old ones.
D. So, I have many notifications to sent to a single channel like New Friends request, New Private message replies, New gifts requests and many others like that. How do I sent all these data to a channel and how to find out and parse the incoming new notifications data. I know JSON is the format to send but im not sure about the format of sending.
According to this link, a single Pubnub channel can contain only up to 100 messages. Does it mean that if 200 messages comes at once to a single channel the first 100 gets delivered and the remaining is on queue ? How about if 10,000 messages comes at once to a single channel ? Do all the remaining messages stays in queue ? if so, how does it gets delivered to the subscriber in realtime ?
Let me give another simple scenario that I'm trying to achieve.
UserA is authenticated and logged in to the website.
UserA generates his own Dynamic channel name, UserAx732dsw3efsdfsdfsdf
UserA starts listening to his newly created channel UserAx732dsw3efsdfsdfsdf (Now, userA should start receiving messages from others)
- UserB sends a private message to userA. (Now, only userA should get notified on his private channel about the new private
message, how can userB or the System find out the channel
name UserAx732dsw3efsdfsdfsdf because, this is a private channel
generated dynamically by userA , neither the System or
userB has accessed to it. This same thing is happening for userB as well, if userB should be notified by any other entity or by the system again, there should be a way to find out the dynamic
channel name of userB .
Another issue is this scenario is that, if a user is dynamically generating channel name everytime he/she logged in to the website. What will happen to all the messages that was sent to the dynamic channel ? Do pubnub saves all the created channel names on their server? Is there any way that a system or a user can find out whether a channel name is still being own and atleast one user is listening to the channel ?.
I'm curious to know this because of the following concepts that I have:
UserA creates dynamicChannelA when he/she logged in to the website at 1AM.
UserA starts getting lots of notification pushout to his dynamic channel dynamicChannelA
Now, UserA logs out from the website at 1:30 AM, what will happen to the many other users who are still pushing out notification
to his dynamicChannelA because by the next time when UserA
logs into the website, UserA will be listening to different
dynamic channel name.UserA won't be listening to his previous
channel dynamicChannelA .
I'm thinking to use the method of retrieving channel name of a particular user from the database table. Is there any method or way to prevent unauthorised subscription of the channel? Because anybody can subscribe to the channel name if they have the subscribe key and channel name no matter how long the channel name is. I'm just curious because all subscription is happening in the client side and the subscription key and channel names are visible.
There's no one single way of tackling the issues that you've run into. Our customers have used a wide variety of design patterns to deal with them. I myself have run into this type of thing building PubNub apps and I'll help you as much as I can.
Pubnub says its not a good thing to create more than two channels per client. So, in my application, i have a requirement to create more than two channels to listen to notifications happening everywhere on my website but i will go with two channels for every logged in users as Pubnub suggested.
Logged in users listens to Channel1-Public
Logged in users listens to private UsersOwnDynamic-Channel to receive notifications related and only meant for him.
This is a good way to do it, and the way many of our massive scale customers do it. A global channel and a private, user-only channel.
A.
Do i always have to create a new private Dynamic channel name everytime i logged into the website.
Not necessarily, though this is a good way. You can use PUBNUB.uuid() in JavaScript on the client-side to do this. Or, generate it server-side using PHP and render it to the client. Maybe you could set it as a cookie so the client always has access to it.
If so, how would other users know how to send a notification to my private Channel.
They could get the id's from the PHP server; either via the global channel or the user's own private channel, which they are listening to.
Or, do i just need to have only one static channel name stored in database table, so that other authenticated users will query the table and get my private channel name to sent me notifications.
You could do it this way too. You might have the global channel the users can send to be different then the global channel they are listening to. Only the server has that subscribe key. Thus, authenticated users send a message to the server that tells it "I need the appropriate user keys", then the server does a query and sends a message back on that users private channel.
If this is the case, don't you think if a hacker get hold of some private channel name of certain users, they will be able to listen to that channel?
If you withhold the subscribe key on the global send channel, only the server can see the chatter on that channel.
B.
Im using PHP and mysql, so i still cannot think out a way or come up with a solution to send message to private channels of another user.
Lets take an example for a simple friend request system.
- UserA sends a friend request to UserB.
- UserB is listening to his own dynamic private channel name called DynamicPrivateChannelB
(how will UserA find the channel name for UserB which is private? Im thinking the only way for this is that the private channel of UserB should be stored in a database table for every loggedin users to query. Am i thinking the right way ? )
This is very similar to your previous question. There's no one way to do it, but the design pattern I outlined above should work. To recap this design pattern:
Server side
Listens on Global-user-send-channel for user messages. The server is the only entity that has this subscribe key
Can query db to get user-ids, and then send to the various ids at will
Can also send on the Global-user-receive-channel, which all clients are listening on. The server is the only entity that has this publish key.
Client side
Listens on Global-user-receive-channel. This is how it gets mass server broadcasts. Cannot send on this channel (only has subscribe key)
Sends server messages on Global-user-send-channel. Cannot recieve on this channel (only has publish key)
Listens on private user channel. This is how user gets private messages. It can also use this for client-to-client communication.
Prevent abuse by appending all private messages with a private, per-user key stored on the server, and provided during the initial page-load. That way, a client knows if a message claiming to be from the server is legit.
C.
If we are always generating dynamic private channel names, storing in database table, updating whenever new dynamic channel names are generated. I think it will cause a problem because some message won't get delivered as new dynamic private channel names replaces old ones.
If you're careful about when you generate new channel names, this shouldn't be a problem. Keep in mind, the client can always say on the Global-user-send-channel 'hey, I'm here! this is my id. keep me updated'. I usually design my apps to have the clients automatically shout this out every 30 seconds or so.
D.
So, i have many notifications to sent to a single channel like New Friends request, New Private message replies, New gifts requests and many others like that. How do i sent all these data to a channel and how to find out and parse the incoming new notifications data. I know JSON is the format to send but im not sure about the format of sending.
JSON is good for sending and recieving. The way I do it is to have a property called "name" which defines what type of message it is. For example:
{
"id" : "blah_blah_unique_id", // sender_client_id
"name" : "friend_request", // type of message
"data" : { // the data itself
"requested_friend_id" : "blah_blah_some_other_unique_id"
}
}
You can actually use whatever format you want, but we'll wrap it in JSON (usually that means just wrapped in quotes) when it gets pushed through PubNub.
Hope this helps!
New Questions
According to this link, a single Pubnub channel can contain only upto 100 messages. Does it mean that if 200 messages comes at once to a single channel the first 100 gets delivered and the remaining is on queue ? How about if 10,000 messages comes at once to a single channel ? Do all the remaining messages stays in queue ? if so, how does it gets delivered to the subscriber in realtime ?
The 100 message limit is in regards to PubNub.history. If someone is subscribed and 200 messages come in, they will receive all 200 messages.
(Now, only userA should get notified on his private channel about the new private message, how can userB or the System find out the channel name UserAx732dsw3efsdfsdfsdf because, this is a private channel generated dynamically by userA , neither the System or userB has accessed to it. This same thing is happening for userB as well, if userB should be notified by any other entity or by the system again, there should be a way to find out the dynamic channel name of userB .
There's no one-size-fits-all solution to this question, but what I'd do is have the server generate that unique id on page load and render it to the client on your initial HTTP request.
Another issue is this scenario is that, if a user is dynamically generating channel name everytime he/she logged in to the website. What will happen to all the messages that was sent to the dynamic channel ? Do pubnub saves all the created channel names on their server?
You don't have to dynamically generate every time. You could.... but you also set a cookie with that unique id, or pull it from the database and render it to the client on page load (that's what I'd do). We don't save channel names.
Is there any way that a system or a user can find out whether a channel name is still being own and atleast one user is listening to the channel ?.
Not out of the box, but you could implement this easily. Just have your server send out a ping and set up your clients to always respond to pings if they're listening.
Now, UserA logs out from the website at 1:30 AM, what will happen to the many other users who are still pushing out notification to his dynamicChannelA because by the next time when UserA logs into the website, UserA will be listening to different dynamic channel name.UserA won't be listening to his previous channel dynamicChannelA .
The way you can prevent this is periodic (every 30 seconds?) pings from the server, who can keep track if users are still there. In the coming months we'll be launching a presence API to do this automatically, btw.
Im thinking to use the method of retrieving channel name of a particular user from the database table. Is there any method or way to prevent unauthorised subscription of the channel? Because anybody can subscribe to the channel name if they have the subscribe key and channel name no matter how long the channel name is. Im just curious because all subscription is happening in the client side and the subscription key and channel names are visible
The main way is strategically witholding publish/subscribe keys. You're right that anyone with the proper details can listen in - this is a big problem of client-only systems. For the time being, you'll have to come up with creative ways to get around it.