Imagine I have a (ReST) API which provides access to versions of a resource. In order to achieve a lower latency I want push out notifications of a new resource as it becomes available. One way to do this is to use webhooks.
Webhooks seem to be typically viewed as long lived (days, weeks...) or semi-permanent resources.
Nowadays we can now upgrade connections to websockets for relatively short-lived low latency sessions.
I think there is still a middle ground where clients create transient webhooks to receive real-time notifications.
For a semi-permanent resource it makes sense for the subscription to be managed by the client.
For a transient webhook we need the server to manage the life-time of web-hooks in case a client forgets to delete them itself.
I haven't seen any discussion of this kind of webhook online. Is transient webhook the correct term?
Are there any best practices for when to automatically delete them?
If the client forgets to or cannot send the DELETE when should the server delete the resource?
Should the reply to the original POST include a time-to-live?
Should it post a probe heart-beat periodically and keep the hook if there is no reply after N attempts?
When server decides that /foobar/webhooks/ can be deleted should it become a 410 GONE or a 404?
It seems like there is good scope here for some standardisation to avoid all the potential pitfalls.
I would accept an answer (comments also welcome) that improves on my own and links to one or more well documented approaches to this or describes some good patterns.
My guess would be something like:
The webhooks for a ReST resource /foobar reside under /foobar/webhooks/
A specific web-hook would reside under /foobar/webhooks/
A client sends an HTTP POST to /foobar/webhooks to create a subscription.
The client optionally includes an indication of how long it expects to need it.
The server replies with a 202 ACCEPT and the location of the webhook /foobar/webhooks/
That webhook should provide the a time-to-live indication.
When it is done the client sends an HTTP DELETE to /foobar/webhooks/
If the client wants the hook to last longer it should tell the server by posting to /foobar/webhooks/ with a message asking it to keep
the hook alive longer.
You also need to be mindful of security, which is discussed well here.
Another resource is:
https://realtimeapi.io/hub/rest-hooks/
Related
I need to interface a secure system that is situated in a protected LAN environment that absolutely must not expose any inbound tcp connections. Thus serving up an http server to provide a means to a webhook service to call as its callback is not an option.
I need to use a third party SMS provider to send and receive text messages. Receiving text messages is the problem.
So I've never really understand how so many services require their customers to provide a webhook. It's such a headache. Companies provide nice things like SDK. For example, when I want to send SMS messages using Twilio of Telnyx for example, I take their jar files, and basically just call a few methods and the SMS is sent without much fanfare. But the fun starts when I want process incoming SMS messages with those (and other) providers. They demand you host a webhook. So there are webhook providers that remove the pain of providing ways to fend of ddos attacks and other heinous scripting kiddies and whatnot, but then those webhook providers still require me to host an http server for it to drop its events to. Why can't I just keep a tcp connection alive to such a provider, and read from a connection that won't send data back until there is something to report, with a periodic heartbeat? And why can't companies like Twilio and Telnyx and others provide all this, without webhooks? It ought to be as simple as providing an event handler interface that simply gets called when needed. The customer shouldn't have to worry about anything more than just dropping in a library and providing a callback method, imo! It's completely feasible, yet it's not provided! Why?
So, because those companies are forcing me to provide a webhook, is there a webhook provider that I can keep a connection open to, where read request trickle out content as needed? A way where I do not have to host any http server?
Given an event driven micro service architecture with asynchronous messaging, what solutions are there to implementing a 'synchronous' REST API wrapper such that requests to the REST interface wait for a response event to be published before sending a response to the client?
Example: POST /api/articles
Internally this would send a CreateArticleEvent in the services layer, eventually expecting an ArticleCreatedEvent in response containing the ID of the persisted article.
Only then would the REST interface response to the end client with this ID.
Dealing with multiple simultaneous requests - is keeping an in-memory map of inflight requests in the REST api layer keyed by some correlating identifier conceptually a workable approach?
How can we deal with timing out requests after a certain period?
Generally you don't need to maintain a map of in-flight requests, because this is basically done for you by node.js's http library.
Just use express as it's intended, and this is probably something you never really have to worry about, as long as you avoid any global state.
If you have a weirder pattern in mind to build, and not sure how to solve it. It might help to share a simple example. Chances are that it's not hard to rebuild and avoid global state.
With express, have you tried middleware? You can chain a series of callback functions with a certain timeout after the article is created.
I assume you are in the context of Event Sourcing and microservices? If so I recommend that you don't publish a CreateArticleEvent to the event store, and instead directly create the article in the database and then publish the ArticleCreatedEvent to the Event store.
Why you ask? Generally this pattern is created to orchestrate different microservices. In the example show in the link above, it was used to orchestrate how the Customer service should react when an Order is created. Note the past tense. The Order Service created the order, and Customer Service reacts to it.
In your case it is easier (and probably better) to just insert the order into the database (by calling the ArticleService directly) and responding with the article ID. Then just publish the ArctileCreatedEvent to your event store, to trigger other microservices that may want to listen to it (like, for example, trigger a notification to the editor for review).
Event Sourcing is a good pattern, but we don't need to apply it to everything.
I am currently working on a web application using the MEAN stack. It has a social aspect to it so I want to be able to push notifications to users.
The way I do it now is when something happens that should be a notification it gets stored in a mongo database with an unread flag. Each client will send a get request to the server every 30 second and will receive every notification marked as unread, and will then mark it as read.
I want to switch to using a message queue and sockets so less network resources will be used, and also provide the user with a real-time experience. I've thought about using redis and its pubsub structure but I can't seem to figure out how to do this securely. If I push out notifications to the affected users, won't it be easy for someone malicious to subscribe to somebody else's channel and receive notifications not meant for them? Am I missing something or is it just the wrong approach for such a system?
Edit: Figure I update with the solution I went with if anyone else reading this is having the same problem.
Instead of using rabbitmq, as the answer suggested, I figured that a much more easy and elegant solution is to just use socket.io. When new sockets connects to the server I save a mapping from the userID to the socketId in a redis in-memory DB. (After I've validated their token) That way, if I need to push a notification to a user I just look up the socketId in the redis DB, and then send it to the correct socket.
This way I don't need any security beyond that as socketIDs are unguessable, and the message is only sent across the single socket that belongs to the given user.
This way it will only get sent through the connection of the given socket, as socketIDs are only used server side to keep track of all the connection. This means no one else can "listen" using someone else's socketID.
you can use RabbitMQ for this. Also authentication is there. Please go through following link and try.
https://www.rabbitmq.com/access-control.html
also, you can apply authentication in existing structure using subscription auth tokens with all subscribed users only.
even redis has its security with topics. Please have a look in link below
https://redis.io/topics/security
Let's say I want to write a mobile chat application (just as an example).
How to receive only the messages meant for one client and don't let other clients receive messages which where not meant for them!?
Create a temp queue only known to the client? - Secure enough?
Encrypting the message with clients public key? - Own PKI needed!
Restrict access to queues based on some credentials the client sends with every request? - Every request needs to be authenticated!
...?
If a client sends a message to the outgoing queue, how to prevent other clients from reading the message directly out of the queue!?
Restricting access to write-only? - Don't know if this is possible...
Encrypting the message? - Own PKI needed!
...?
I hope my question/problem is clear and I'm really looking forward to hear your ideas and best practices!
Thanks in advance!
//edit: So using a temp queue for every client with encrypted messages might be a good choice. Or do you have any other ideas???
If you use RabbitMQ AMQP broker, then you can use Validated User-ID extension power, but you have to create separate users for each client.
Using per-client queue maybe a good choice, but you have to realize that it "security through obscurity" and it smells. But as you suggested, message encryption may fix that.
You can play with Access Control but you may find better to have some server application to handle complex user management things and use it api from clients for better user policies management.
I am working on a Chatting application (needs to connect to a server) on iPhone. The sending packet from iPhone shouldn't be a problem.
But I would like to know whether it is possible for iPhone to establish a incoming socket connection to server continuously or forever under mobile environment.
OR What do I need to do to give the connection alive ? Need to send something over it to keep it alive ?
Thanks.
Not sure why you want to have chatting app to have persisted connection... I'd better use SMS like model. Anyways, Cocoa NSStream is based on NSSocket and allows a lot of functionality. Take a look at it.
Response to the question. Here is in a nutshell, what I would do:
Get an authentication token from the server.
this will also take care of user presence if necessary but now we are talking about the state; once presence is known, the server may send out notifications to clients that are active and have a user on their contact list.
Get user's contact list and contact presence state.
When a message send, handle it according to addressee state, i.e. if online, communicate back to the other user, if offline, queue for later delivery or reject.
Once token expires, reject communication with appropriate error and make the client to request a new token.
Communication from server to client, can be based on pull or push model. In first case, client periodically makes a request and fetches all messages. This may sound not good but in reality, how often users compose and send messages? Several times a minute? That's not too much. So fetching may happen every 5-10 seconds.
For push model, client must be able to listen and accept connections.
Finally, check out SIP, session initiation protocol. No need to use full version of it though. Just basic stuff.
This is very rough and perhaps simplified. I don't know the target complexity of your chatting system. For example, the simplest thing can also be that server just enables client to client communication by distributing their end points and clients take care of everything themselves.
Good luck!
Super out of date response, but maybe it will help the next person.
I would use xmppframework and a jabber server.