Live Notification - DB Polling - Best Practice - node.js

I'm providing my users with live notifications.
I'm debating two options and can't decide which is the best way to go when polling the DB.
(The notifications are transmitted using WebSockets.)
Option 1 (current):
I hold a list of all the logged in users.
Every 1000 ms, I check the db for new notifications and if there is any, I send a message via WS to the appropriate user.
Pros:
This task is rather not expensive on resources
Cons:
In off-times, where's there's only a new notification every 1 minute, I poll the DB 60 times for no reason.
In a sense, it's not real-time because it takes 1 full second for new notifications to update. Had it been a chat service, 1 second is a very long time to wait.
Option 2:
Create a hook that whenever a new notification is saved (or deleted), the db get polled.
Pros:
The db does not get polled when there are no new notifications
Actual real-time response.
Cons:
In rush-hour, when there might be as many as 10 new notifications generated every second, the db will be polled very often, potentially blocking its response time for other elements of the site.
In case a user was not logged in when their notification was generated, the notification update will be lost (since I only poll the db for logged in users), unless I also perform a count whenever a user logs in to retrieve their notifications for when they were offline. So now, not only do I poll the DB when ever my notification hook is triggered, but also I poll the db every time a user logs-in. If I have notifications generated every second, and 10 log-ins every second, I will end up polling my DB 20 times a second, which is very expensive for this task.
Which option would you choose? 1, 2? or neither? Why?
Here is the code I am currently using (option 1):
var activeSockets = [] //whenever a user logs in or out, the array gets updated to only contain the logged-in users in any given moment
var count = function () {
process.nextTick(function () {
var ids = Object.keys(activeSockets) //the ids of all the logged in users
//every user document has a field called newNotification that updates whenever a new notification is available. 0=no new notifications, >0=there are new notifications
User.find({_id:{$in:ids}}).select({newNotifications:1}).lean().exec(function (err,users) {
for(var i=0;i<users.length;i++) {
var ws = activeSockets[String(users[i]._id)]
if(ws!=undefined) {
if (ws.readyState === ws.OPEN) {
//send the ws message only if it wasn't sent before.
if(ws.notifCount!=users[i].newNotifications) {
ws.send(JSON.stringify({notifications:users[i].newNotifications}));
activeSockets[String(users[i]._id)].notifCount = users[i].newNotifications
}
}
else {
//if the user logged out while I was polling, remove them from the active users array
delete activeSockets[String(users[i]._id)]
}
}
}
setTimeout(function () {
count()
},1000)
})
})
}
The implementation of Option 2 would be just as simple. Instead of calling
count()
using
setTimeout()
I only call it in my "new notification", "delete notification", and "log-in" hooks.
Code:
var activeSockets = [] //whenever a user logs in or out, the array gets updated to only contain the logged-in users in any given moment
var count = function () {
process.nextTick(function () {
var ids = Object.keys(activeSockets) //the ids of all the logged in users
//every user document has a field called newNotification that updates whenever a new notification is available. 0=no new notifications, >0=there are new notifications
User.find({_id:{$in:ids}}).select({newNotifications:1}).lean().exec(function (err,users) {
for(var i=0;i<users.length;i++) {
var ws = activeSockets[String(users[i]._id)]
if(ws!=undefined) {
if (ws.readyState === ws.OPEN) {
//send the ws message only if it wasn't sent before.
if(ws.notifCount!=users[i].newNotifications) {
ws.send(JSON.stringify({notifications:users[i].newNotifications}));
activeSockets[String(users[i]._id)].notifCount = users[i].newNotifications
}
}
else {
//if the user logged out while I was polling, remove them from the active users array
delete activeSockets[String(users[i]._id)]
}
}
}
//setTimeout was removed
})
})
}
Hooks:
hooks = {
notifications : {
add: function () {
count()
//and other actions
},
remove: function () {
count()
//and other actions
}
},
users: {
logIn: function () {
count()
//and other actions
}
}
}
So, Which option would you choose? 1, 2? or neither? Why?

Related

Better approach to store FCM tokens array in a collection/doc and update it on token refresh

I am building an application where people can host events. It allows for users to follow the events and all the event followers can chat in a group. I am storing the FCM token for push notifications in a user collection in Firestore. But when I send the notification my current logic is not quite optimal and it takes several minutes for the notification to send to each user as I am first getting all user's tokens for each user's document and then combing those tokens in a list and send push notification to each token using a loop in my cloud function. I think it takes time for sending the messages and the group chat does not seem to be real-time because of that computation.
What I thought is to store the FCM tokens of each user inside every event he follows. So that when there is a chat message the list of tokens is fetched from the specific event and they are sent a multicast notification. But here again, I am not sure if it is a good approach because I will need to keep track of refreshed tokens and update the tokens in each document where it exists.
Let's say a user has joined 50 events and on app launch, his FCM token got refreshed now I will have to run a cloud function that will see if the token is renewed it should loop through all of those 50 events and update his token.
I want to know what can be the best approach for this use case.
Below is the code that I am using:
exports.sendNotification = functions.firestore
.document("event/{eventid}/{chat}/{chatid}")
.onCreate((snap, context) => {
processData(snap, context);
return null;
});
async function processData(snap, context) {
const doc = snap.data();
const eventID = context.params.eventid;
const senderID = doc.sender;
var senderName = doc.senderName;
await admin
.firestore()
.collection("event")
.doc(eventID)
.get()
.then(value => {
var joinedBy = value.data()["joinedBy"];
joinedBy.forEach(item => {
getTokens(uid, senderID, doc, eventName, senderName);
});
});
}
async function getTokens(uid, senderID, doc, eventName, senderName) {
await admin
.firestore()
.collection("people")
.doc(uid)
.get()
.then(value => {
var token = value.data()["pushToken"];
if (uid != senderID) {
if (token) {
const payload = {
notification: {
title: `${eventName} : ${senderName}`,
body: doc.text,
badge: "1",
sound: "default"
},
};
sendMessage(token, payload);
} else {
console.log("Can not find pushToken target user");
}
} else {
console.log("uid and sender id is same so notification not sent");
}
});
}
One thing to consider in your current approach is whether you really first need to read all tokens for the message and only then starting calling FCM. Might things get faster if you call FCM right away on each set of tokens you read, and thus perform some of the reads and FCM calls in parallel?
Another consideration is that you say that FCM's built-in topics work really fast for you. Internally FCM's topic subsystem associates a bulk of device IDs with the topic, and then reads the tokens for the topic as needed.
Is this something you can mimic for your app? For example, can you store the tokens in larger groups inside fewer documents, thus reducing the number of read operations you need to do.

is it okay if I intentionally make my Google Cloud Functions has multiple errors?

I have collection and sub-collection like this
users/{userID}/followers/{followerID}
everytime a follower document is deleted in followers sub-collection, then it will trigger this firestore trigger below to decrease the numberOfFollowers field in user document. this is triggered when a user click unfollow button
exports.onDeleteFollower = functions
.firestore.document("users/{userID}/followers/{followerID}")
.onDelete((snapshot, context) => {
// normally triggered after a user push unfollow button
// then update the user document
const userID = context.params.userID;
const updatedData = {
numberOfFollowers: admin.firestore.FieldValue.increment(-1),
};
return db.doc(`users/${userID}`).update(updatedData);
});
now I have a case like this ....
if a user deletes their account, then I will delete the user document ( users/{userID} ), but if I delete a user document, it will not automatically delete all documents inside its sub-collection, right
so after I delete the user document, I have another function to delete all documents inside the followers sub-collection.
but the problem is, the onDeleteFollower triggers function above will be executed multiple times, and it will throw error multiple times, because the user document has been deleted ( the function above will be used to a update a field in deleted user doc)
I will have this error in functions emulator
⚠ functions: Error: 5 NOT_FOUND: no entity to update: app: "myApp"
path <
Element {
type: "users"
name: "u1-HuWQ5hoCQnOAwh0zRQM0nOe96K03"
}
>
⚠ Your function was killed because it raised an unhandled error.
I actually can write a logic to check if a user document still exist or not. if exist then update numberOfFollowers field
but deleting a user document is very rare if compared to a user click the unfollow button, I think it is not very efficient.
I have a plan like this, I will intentionally let the errors happened. say a user has 1000 followers, then it will trigger the onDeleteFollower function above, then I will have 1000 function errors
my question is .....
is it okay if I have multiple errors in a short time like that? will Google Cloud Function terminates my function, or .... I don't know, I am worried something bad will happen that I don't know
as far as I know, cloud functions will automatically run the function again after it is killed, will my function always ready again after an error like that?
I can't let the follower update the organizer (user) document directly from the client app, because it is not safe. creating security rules to facilitate this is complicated and it seems error prone
Have you considered instead of setting/removing users/{userID}/followers/{followerID} directly, that you create a "follow request" system?
"users/{userID}/followRequests/{requestID}": { // requestID would be auto-generated
user: "{followerID}",
follow: true // true = add user as follower, false = remove user as follower
}
This then allows you to use a single onCreate trigger to update your followers list eliminating the need for your current onCreate and onDelete triggers on users/{userID}/followers/{followerID}. From this function you can implement restrictions on following other users like follow limits or denying follow requests for blocked users.
export const newFollowRequest = functions.firestore
.document('users/{userId}/followRequests/{requestId}')
.onCreate(async (snap, context) => {
const request = snap.data();
const followingUserId = request.user;
const followedUserId = context.params.userId;
const db = admin.firestore();
const userDocRef = db.doc(`users/${followedUserId}`);
const followerDocRef = userDocRef.child(`followers/${followingUserId}`);
// /users/${followingUserId}/following/${followedUserId} ?
try {
if (request.follow) {
// Example restriction: Is the user who is attempting to follow
// blocked by followedUserId?
// await assertUserIDHasNotBlockedUserID(followedUserId, followingUserId);
// following
db.update(userDocRef, {
numberOfFollowers: admin.firestore.FieldValue.increment(1),
});
db.set(followerDocRef, {
/* ... */
});
} else {
// unfollowing
db.update(userDocRef, {
numberOfFollowers: admin.firestore.FieldValue.increment(-1),
});
db.delete(followerDocRef);
}
// delete this request when successful
db.delete(snap.ref);
// commit database changes
await db.commit();
console.log(`#${followingUserId} ${request.follow ? "followed" : "unfollowed"} #${followedUserId} successfully`);
} catch (err) {
// something went wrong, update this document with a failure reason (to show on the client);
let failureReason = undefined;
switch (err.message) {
case "other user is blocked":
failureReason = "You are blocked by #otherUser";
break;
case "user is blocked":
failureReason = "You have blocked #otherUser";
break;
}
return db.ref(snap.ref)
.update({
failureReason: failureReason || "Unknown server error";
})
.then(() => {
if (failureReason) {
console.log("REQUEST REJECTED: " + failureReason);
} else {
console.error("UNEXPECTED ERROR:", err)
}
},
(err) => {
console.error("UNEXPECTED FIRESTORE ERROR:", err);
});
}
});

Get user join / leave events retroactively from Channels

I'm trying to do some analytics on average response time from some of our users on Twilio Chat.
I'm iterating through my channels, and I'm able to pull the info about messages, so I can compare times a message went un-responded to. However, I can't determine which users were in the channel at that time.
Is there anything on the channel that would give me historic member data? Who was in the channel? The channel.messages().list() method is only giving me the text of the messages sent to the channel and who it was by, but the user who may have been in a channel to respond changes throughout a channel's life time.
This is on the backend using the node.js SDK. note: This isn't a complete implementation for what I'm trying to do, but taking it in steps to get access to the information I'd need to do this. Once I have these messages and know which users are supposed to be in a channel at a given time, I can do the analytics to see how long it took for the users I am looking for to respond.
var fs = require('fs');
const Twilio = require('twilio');
const client = new Twilio(env.TWILIO_ACCOUNT_SID, env.TWILIO_AUTH);
const service = client.chat.services(env.TWILIO_IPM_SERVICE_SID);
async function getChatMessages() {
const fileName = 'fileName.csv';
const getLine = message => {
return `${message.channelSid},${message.sid},${message.dateCreated},${message.from},${message.type},${message.body}\n`;
}
const writeToFile = message => { fs.appendFileSync(fileName, getLine(message)); };
const headerLine = `channelSid,messageSid,dateCreated,author,type,body`;
fs.writeFileSync(fileName, headerLine);
await service.channels.each(
async (channel, done) => {
i++;
let channelSid = channel.sid;
if( channel.messagesCount == 0 ) return;
try {
await channel.messages().list({limit:1000, order:"asc"}).then(
messages => {
messages.forEach( writeToFile );
}
);
} catch(e) {
console.log(`There was an error getting messages for ${channelSid}: ${e.toString()}`);
}
if( i >= max ) done();
}
);
}
I'm beginning to be resigned to the fact that maybe this would only have been possible to track had I set up the proper event listeners / webhooks to begin with. If that's the case, I'd like to know so I can get that set up. But, if there's any endpoint I can reach out to and see who was joining / leaving a channel, that would be ideal for now.
The answer is that unfortunately you can not get this data retroactively. Twilio offers a webhooks API for chat which you can use to track this data yourself as it happens, but if you don't capture the events, you do not get access to them again.

socket.io emit to specific user and give them x amount of time to respond

I have an online chat. It uses rooms. If a user sends message and the other user is not online, it should increment the "missed messages" counter. I tried to create a timeout with setTimeout and if they emit an event it clears that timeout.
However chatOnline doesn't fire as I expected to, which leads to it always reporting the user is offline and incrementing the counter for missed_texts column in rethinkdb (which is not shown because it isn't relevant).
How can I retrieve if the user is online from socket.io? My goal is to avoid having to store presence info in the database, which could get out of control quickly.
Code I tried:
socket.on('chatSend',function(data){
if(socket.client.user.room_id !== null){
//were storing some crap in the socket object for easy retrieval.
data.user_id = socket.client.user.id;
data.room_id = socket.client.user.room_id;
data.timestamp = ~~(new Date() / 1000);
//insert message into chat table
r.table('chat').insert(data).run().then(function(res){
//retrive generated record from table
r.table('chat').get(res.generated_keys[0]).run().then(function(data2){
io.sockets.in(data2.room_id).emit('chatNew',data);//emit to all users in room
log('chatSend');
//attempt to see if the other user is online
getOtherUser(data2.room_id,socket.client.user.id,function(tid){
log('other user id: %d',tid);
//all users automatically join a room matching their user id when connecting.
//unsure how to see if the user is online. this doesnt work.
//this is what i need help with. retrieving the other users socket resource if they are online,
//and if they are not then return null or false, etc so i can work with that.
var othersocket = io.sockets.in(tid);
//if timeout completes before they respond, they are not online.
var tmptime = setTimeout(function(){
log('other user not online.');
othersocket.removeListener('chatOnline',tmpfunc);
missedTexts(data2.room_id,tid,'INCR');
},5000);
var tmpfunc = function(){
clearTimeout(tmptime);
//remove the listener
othersocket.removeListener('chatOnline',tmpfunc);
};
//emit chatOnline to other user socket
//when they respond, cleartimeout, resulting in counter not being incremented.
othersocket.on('chatOnline',tmpfunc);
othersocket.emit('chatOnline');
});
});
});
}
});

How to do an Idle status for a Pusher user?

Typical chat app. Using the presence channel to tell who is online, but looking for an elegant way to mark a User in the presence channel with an idle flag.
The full solution to this is probably reasonably complicated and it also depends on the runtime - I'm going to assume web web client.
Ultimately you need a way of doing two things:
to detect a user as being "idle"
to signal all other users about that user being idel
To detect a user is idle
window.onblur so you know your app window is no longer focused
mousemove tracking to see if the user is actually doing anything within your application.
In order to achieve this you probably just want a timeout and only if that timeout triggers do you send an event to indicate the user is idle:
var timeout = null;
function sendUserIdle() {
// see part 2
}
function resetIdleTracking() {
if( timeout !== null ) {
// reset
clearTimeout( timeout );
}
timeout = setTimeout( sendUserIdle, 3*60*1000 ); // 3 minutes
}
window.addEventListener( 'mousemove', resetIdleTracking );
Signal other users about idle users
A missing feature of Pusher presence channels IMO is the ability to update presence information. So, you need another way of achieving this. I think you have two solutions:
Enable client events on the presence channel and trigger an event from the idle user when your code detects the user becoming idle.
Send a message to the server from the idle client. The server then triggers a message telling the users that the user is idle.
See: accessing channel members.
1. Using client events
function sendUserIdle() {
var channel = pusher.channel( 'presence-<your-channel>' );
// client events have to have a 'client-' prefix
channel.trigger( 'client-user-idle', channel.members.me );
}
2. Sending to the server
function sendUserIdle() {
makeAjaxRequest( '/idle-notification-endpoint', channel.members.me );
}
Note: you can serialise channel.members.me using JSON.stringify( channel.members.me )
On the server (in pseudo code):
userInfo = getUserInfoFromRequest();
pusher.trigger( 'presence-<your-channel>', 'user-idle', userInfo );
Showing a client is idle
Upon receipt of the event you would update the list of users UI accordingly (mark that user as idle).
channel.bind( 'user-idle', function( user ) {
var uniqueUserId = user.id;
// update UI
}

Resources