I am using node.js (server framework) and mongoose.js (mongo based schema modeling) as the backend for an iOS app and I am using Mocha (test framwork) to make sure everything works.
What I really want to know, and can find no documentation on, is how to test on the server if the push notifications are being appropriately sent. I am using apnagent and at the moment I can see that push notifications are being sent correctly by manually checking my device but I am having difficulty finding an automated way to test that they are working correctly.
That may be enough of a description to answer at a high level what needs to be done. But in case it is not here is the actual code:
Mongoose Model fires off a push notification upon creation:
#this code is called after this model is saved in mongodb
eventModel.post 'save', (doc) ->
#push the message
sendMessageToDevice = (event, token) ->
message =
event_body:
eventId: event._id
lat: event.lngLat[1]
lng: event.lngLat[0]
agent.createMessage()
.device(token)
.alert('New Event! ' + event.description)
.set(message)
.send()
#cycle through the users to push to
#get all the unique device tokens in the database for APN
users.getAllUniqueDeviceTokens (error, devices) ->
if error then return util.handleError error
console.log "Sending push notices to all devices (%d):", devices.length
console.log devices
for token in devices
sendMessageToDevice doc, token
#send some verification here that the code ran correctly???
Then in my Mocha test file I have:
it 'should receive push notification from fort creation', (done) ->
#some logic here to verify that push notifications were sent
done()
In many situations, while writing tests, it is either impossible or simply too dangerous to verify that an action has really taken place (i.e. a push notification has been delivered). Imagine writing a unit test for the rm command where you would like to ensure that doing rm -rf / succeeds. Obviously, you cannot let this action take place and verify that your root partition is indeed empty!
What you can do, however (and should do, really), is verify that whatever commands, routines or other actions necessary to accomplish the task are being invoked correctly, without actually allowing them to take place.
In your particular situation, you do not need to verify that your push notification has been delivered because your application is not responsible for the notification's delivery. However, you can test that the push notification is being correctly delivered to the push server.
So, instead of testing for successful delivery, you test
Whether the outgoing request is properly formatted (i.e. JSON is valid)
Whether it contains the data you expect it to contain (i.e. a field in JSON is present and contains expected data)
Whether the authentication token required by the server is included
Whether the target server is correct (i.e. you are indeed sending the data to xxx.apple.com and not to localhost)
Ideally, these test requests will not even reach the target server - doing so would mean you are relying on two factors that are not always perfectly stable:
network connectivity
target server availability and proper functionality
In the past, I dealt with this so that I first manually issued a correct request, captured the response and then mocked the whole communication in the unit test (using i.e. nock. That way, I am completely in control of the whole communication.
As far as I know, there's no way to check if an APNS request has reached its destination or not. Apple tends to have this "everything's fine, and if it's not, then it should be your fault" policy with us developers. If things haven't changed since I started coding, you make an APNS request by sending raw data (JSON payload, you probably know the whole format) through the 2195 port, and you get absolutely no response for that.
Only thing I can think of, if you have a physical iOS device (an iPod, an iPhone or an iPad), you can "automate" a test by launching a PUSH request with a hardcoded token, corresponding to your device and a test app, and if you receive the notification then it works.
Oh, and if it doesn't work, please make sure you have all required ports open if you're operating behind a firewall. It's the first big stone I stepped into when I first dove into this ;) (related: https://support.apple.com/en-us/HT203609)
I would use a request mocking framework like nock to intercept the request to APN. The urls seem to be located in the code here.
Related
Background
I have a monolith Node.js + PostgreSQL app that, besides other things, needs to provide real-time in-app notifications to end users.
It is currently implemented in the following way:
there's a db table notifications which has state (pending/sent), userid (id of the notification receiver), isRead (did a user read the notification), type and body - notification data.
once specific resources get created or specific events occur, a various number of users should receive in-app notifications. When a notification is created, it gets persisted to the db and gets sent to the user using WebSockets. Notifications can also get created by a cron job.
when a user receives N number of notifications of the same type, they get collapsed into one single notification. This is done via db trigger by deleting repeated notifications and inserting a new one.
usually it works fine. But when the number of receivers exceeds several thousands, the app lags or other requests get blocked or not all notifications get sent via WebSockets.
Examples of notifications
Article published
A user is awarded with points
A user logged in multiple times but didn't perform some action
One user sends a friend request to another
One user sent a message to another
if a user receives 3+ Article published notifications, they get collapsed into the N articles published notification (N gets updated if new same notifications get received).
What I currently have doesn't seem to work very well. For example, for the Article created event the api endpoint that handles the creation, also handles notifications send-outs (which is maybe not a good approach - it creates ~5-6k notifications and sends them to users via websockets).
Question
How to correctly design such functionality?
Should I stay with a node.js + db approach or add a queuing service? Redis Pub/Sub? RabbitMQ?
We deploy to the k8s cluster, so adding another service is not a problem. More important question - is it really needed in my case?
I would love some general advice or resources to read on this topic.
I've read several articles on messaging/queuing/notifications system design but still don't quite get if this fits my case.
Should the queue store the notifications or should they be in the db? What's the correct way to notify thousands of users in real-time (websockets? SSE?)?
Also, the more I read about queues and message brokers, the more it feels like I'm overcomplicating things and getting more confused.
Consider using the Temporal open source project. It would allow modeling each user lifecycle as a separate program. The Temporal makes the code fully fault tolerant and preserves its full state (including local variables and blocking await calls) across process restarts.
My application stack is ios(front-end) and node.js(back-end). I have to send notification to devices. In my node.js part im using apns module to send notification, its working fine......
Now i have to send Mass notification like at a time consider i have 10,000 devices to notified, the logic what im following is
I'm looping through 10,000 devices and calling apns provider.
1.Why this for loop approach
I have to store each notification details in my mongodb collection, so i followed this approach.
The problem is the notification is received by some devices and that too very late(next day).
I read the link also
https://www.raywenderlich.com/156966/push-notifications-tutorial-getting-started
saying apns will reject.
Is the above approach is correct also any way to make all notification deliverer.
Please share your ideas. Thanks in advance.
If you need to process each individual notification before/after it is sent I would instead recommend a design change from a loop and have you look at job queue instead.
With this design pattern, instead of your only step being to loop over notifications and send via APN, you push these notification into a queue/messaging system and have workers which pull from the queue and process (send via APN and write to mongo) the notifications. The nice part of this design is that as your application grows you can add on more workers to handle the increased load without rewriting your application/architecture. Once you have it built it may look something like this:
I personally use RabbitMQ for my job queue, but that decision is something you need to research on your own. For example if you don't want to manage the messaging system you could look into something like AWS Simple Queue Service.
I think looping through 10,000 devices ids and calling APNS provider is not the right way forward. The documentations strictly says here node-apn readme file to reuse apn.Provider rather than recreate it every time to achieve the best possible performance.
If you send notification using arrays of device ids rather than just a device id then you will get a response from the APNS mentioning all the details for each device.
Thoughts on how you guys would/do handle this... If you need to send a webhook when something happens. Say, when a customer registers.. How would you guarantee that a webhook has been sent for it? You couldn't just have the code send the webhook after the registration happens, as what if the server crashes right after registration but before/during the webhook request.. If you were using MongoDB, I guess you could listen to the oplog and see what a registration record is inserted, but then what if your oplog monitoring script crashes.. I thought about storing locally a file that records the last successfully processed timestamp but then you're really limiting yourself to processing the oplog one at a time, which on an active DB could really get behind.
I thought about using a queuing/task system but then what if the request to store the task in the queue crashes..
Maybe store a field on the registration DB record that says if the webhook has been sent yet, send it once the registration happens and have a script that checks every once in awhile for webhooks that failed to send within a certain amount of time?
Just trying to think of the best way to make sure a webhook is actually sent for a particular event.
I'm working on a chat application and using socket.io / node for that. Basically I came up with the following strategies:
Send message from the client which is received by the socket server which then sends it to the receiving client. On the background I store that to the message on the DB to be retrieved later if the user wishes to seee his old conversations.
The pros of this approach is that the user gets the message almost instantly since we don't wait for the DB operation to complete, but the con is that if the DB operation failed and exactly that time the client refreshed its page to fetch the message, it won't get that.
Send message form the client to the server, the server then stores it on the DB first and then only sends it to the receiving client.
The pros is that we make sure that the message will be received to the client only if its stored in the DB. The con is that it will be no way close to real time since we'll be doing a DB operation in between slowing down the message passing.
Send message to the client which then is stored on a cache layer(redis for example) and then instantly broadcast it to the receiving client. On background keep fetching records from redis and updating DB. If the client refreshes the page, we first look into the DB and then the redis layer.
The pros is that we make the communication faster and also make sure messages are presented correctly on demand. The con is that this is quite complex as compared to above implementations, and I'm wondering if there's any easier way to achieve this?
My question is whats the way to go if you're building a serious chat application that ensures both - faster communication and data persistence. What are some strategies that app like facebook, whatsapp etc. use for the same? I'm not looking for exact example, but a few pointers will help.
Thanks.
I would go for the option number 2. I've been doing myself Chat apps in node and I found out that this is the best option. Saving in a database takes few milliseconds, which includes the 0.x milliseconds to write in the databse and the few milliseconds of latency in communication ( https://blog.serverdensity.com/mongodb-benchmarks/ ).
SO I would consider this approach realtime. The good thing with this is that if it fails, you can display a message to the sender that it failed, for whatever reason.
Facebook, whatsapp and many other big messaging apps are based on XMPP (jabber) which is a very, very big protocol for instant messaging and everything is very well documented on how to do things but it is based in XML, so you still have to parse everything etc but luckily there are very good libraries to handle with xmpp. So if you want to go the common way, using XMPP you can, but most of the big players in this area are not following anymore all the standards, since does not have all the features we are used to use today.
I would go with doing my own version, actually, I already something made (similar to slack), if you want I could give you access to it in private.
So to end this, number 2 is the way to go (for me). XMPP is cool but brings also a lot of complexity.
I've written a series of tests in mocha for a node.js client/server application that uses UDP to send messages between the client and server.
My tests are mainly on the 'client side', treating the server as a block box and validating the responses from the server. The problem is that some of the "conversations" span multiple message send and receive events, sometimes going up to the dozens of seconds. It seems bizarre to have a 1,000 line test script with only one huge call to it at the top - I want to run multiple tests during the "conversation". I want more granularity in which specific parts of the test fail if it does fail (e.g. the first 2 responses from the server were fine but the 3rd was malformed), but the initial tests pass.
I've looked in to nesting the calls to it (doesn't seem to work) and most recently separating the calls to it in separate steps, with each step representing the sending or receipt of one message at the client side.
This approach doesn't seem to work because mocha terminates the node application after the very first step, never waiting for the socket to receive more responses from the server and complete the rest of the steps.
How can I create "evented" it calls in mocha?
The first would be called at the start of the test, then each successive it would only be called after receiving a response from the server. I'm looking for any solution that gives me the granularity on tests without having to write enormous functions in my it call that include dozens of messages between client and server. It's also not ok to attempt validating responses from the server outside of the context of a conversation where many messages have been sent and received before hand, because those messages determine the validity of the server response.
See a sample implementation that I created at https://gist.github.com/4490219. You can see the result is that the first test passes but the second is never executed even though the socket is clearly still open and waiting for requests from the server.
(P.S. apologies for the text formatting of the gist - I couldn't seem to select Javascript as a language type when creating it)
(P.P.S. I really don't want to have to use a series of setTimeout calls at the start of each step to make mocha think it has to wait).
The solution is to use a combination of nested its (which actually can be achieved if you use a describe between each nested it) and using a registration it with long timeout who's done is only called when a response arrives from the server.
For future googlers I created another gist to show the solution for this problem.
https://gist.github.com/Trindaz/4490646