I have a scenario in which a FIX server will send to multiple clients. I have found examples in which this is done by sending to each session round-robin fashion however, is there any facility in FIX that allows what might be called multicasting? I.e. all clients subscribe, for example, to a shared session.
Many thanks.
Related
I am building a webapplication using nodejs as the server and Docker Swarm to handle replication and load balancing.
Right now, I need to handle real-time data updates between clients and the replicated servers, so i thought of using Socket.IO to handle the connections. All the requests pass via an NGINX server that redirect it to the manager node of the swarm, and its him that handles the balancing.
Since the topology of the network can change rapidly based on the load of the network, i am reticent of letting NGINX handle the balancing and applying sticky sessions... (maybe am wrong)
For my understanding with this setup, if a client connects to my server, the load balancer of docker will send the request to one of my N replicated servers, and this an only this server will know that the client connected.
So, its possible that if some traditional HTTP-Request updates my data on another replica, the information will not be sent because of the lack of existence of this connection in the given server.
Is there a way of handling situations like this? I thought of including a Message queue between servers to send the data to all of them and then the one containing the connection will send the data, but is that the recommended way of doing it?
Thank you very much
I investigated a bit further since the time of the question. I'll post what I found in case it helps somebody with a similar issue.
One option i found is to use a MessageQueue or something similar to broadcast the messages to all the replicas, then each one filters only the messages that can send because the replica itself has knowledge of the TCP connections available in that replica.
But i think that would put excessive stress over the replicas because all of them are receiving all of the messages, so one solution would be to create a queue or a service that links the id of the given connection to the replica, and forward the messages only to those replicas interested.
I think it can be easily done with topics or making a queue for each tcp connection with some id as a identifier, and then pushing to the corresponding queue.
If anyone sees any problem or wants to add something, it will be very much appreciated!
I need some help when choosing for message broker(RaabitMQ, Redis, etc) or other right tools for this situation.
I am upgrading my game server. It is written by Node.js. it consist of several process, i.e. GameRoom, Lobby, Chat, etc. When a user make request, the message will be routed to relevant process to process it. I do this by routing by my code and each process communicate with each other by node-ipc. However, this is not too efficient and is not scalable. Also, some process has very high work load(Lobby as many requests are related to it), we create several process of Lobby and route message randomly to different process of Lobby. I think message broker can help in this case and also I can even scale up by putting different process in different physical servers. I would like to know which message broker is suitable for this? Can a sender send a message to a queue which multiple consumers compete for a message and only one consumer consume it and reply the message to the sender? Thanks.
I'm not going to be able to talk about Kafka from experience, but any message-queue solution, as will RabbitMQ and ActiveMQ will do what you need.
I assume you're planning a flow like so:
REST_API -> queue -> Workers ----> data persistance <--------+
| |
+------> NotificationManager ----> user
The NotificationManager could be a service that lets the user know via Websockets or any other async communication method.
Some solutions will be better put together and take more weight off your shoulders. Solutions that are not just message-queues but are also task-queues will have ways with getting responses from workers.
Machinery, a project that's been getting my attention lately does all of those , whilst using MongoDB and RabbitMQ itself.
I have one or more daemon app running and to communicate with it I have a client app. The client app is something simple executed on the command line. Chances are only one will be up at a given moment. When I do a command such as daemon update-config the client does mq_open and sends the command. Some commands like list I'd want results. It appears that if I run mq_send in my daemon after I receive I may receive the message within the daemon app.
What's the best way to send the reply to the client w/o accidentally processing it in the daemon? After a quick lookup there didn't appear to be an obvious solution so I do sleep(1) which seems to solve my problem completely even though it's a 'hack'. Whats the best solution? is sleep the most understandable and straightforward solution? I don't feel like generating random/unique values, passing it in and opening another mq to send it. The sleep for a second feels like the best solution but I wonder what your solutions may be.
When using messaging systems, you can do RPC calls even if it is not the best paradigm to use messaging in general. The general approach to RPC with messaging is:
have distinct queues for requests and for replies (the latter ones can be ephemeral queues, created for each request, or persistent queues);
give to each message a unique ID, that will be used in the replies to identify which message it was replying to. (it's called correlation_id in AMQP for example).
I do guess that you can use the same approach with Posix queues as well.
I am building an app with several Node.js instances as a Backend (http server, socket server and several a pool of domain servers). Now I am trying to cover several communication and configuration aspects and am wondering if redis makes an appropriate solution.
So, I would use it for the following purposes:
Implementation of a shared run-time lookup table. It's a table of several hundreds of relativelly simple records, accessed and manipulated by 2 node-instances.
Implementation of message queues. Each domain server receives commands from the http server and should execute them sequentially. Domain server should be able to listen on a redis-event, and execute each new command upon its arival
socket sever also has a regis message queue and listen to its event, in order to push notification to connected clients
Is redis "too heavy" for such a purpose?
Does it offer all needed functionality?
I can definitelly implement a look-up in a file and/or memory and a queue using sockets. However, it might make a code cleaner and a solution more robust with redis.
Redis is definitely not a heavy solution, on the contrary.
It's small, insanely fast (when using pipelining), easy to deploy. I consider it as a light solution, a kind of swiss knife that may solves many problems.
Redis based message queues are OK if you don't expect any guarantee on the message delivery. That is to say Redis based queues can't assure you the client has received the message. If it's a problem for your application you should consider using an heavier solution, like 0mq or Rabbitmq.
We have the following scenario that we would like to solve using Apache Camel:
An asynchronous request arrives to an AMQP endpoint configured in Camel. This message contains a header property for a reply-to that should be used for the response. Camel must pass this message to another service using JMS and then route the response back to the reply-to queue from the AMQP request. This seems like a textbook example for using the InOut functionality in Camel but we have one problem: The reply from JMS service could take a long time, in some cases several days.
As I understand it, if we are using InOut it would mean that we would lock a thread to the the long running service. If we are unlucky, we could get several long running calls simultaneously and in the worst case scenario it could be that all threads are busy waiting for replies thus clogging the system.
What strategy should I use for solving the problem described above? At the moment, I have created to separate routes: One that listens to the AMQP endpoint and forwards the message to the JMS endpoint. The other route listens to the replyto-queue for the jms system and would be responsible for sending the reply back to the AMQP reply-to. The problem I have right now is how I should store the AMQP reply-to between these two routes and I am not sure this is a good solution overall for this problem.
Any tips or ideas on how to solve this problem would be greatly appreciated.
If you have to wait for more than a minute for reply, it's probably a good thing to treat the reply as async. and create separate request and response routes.
Since you mention several days, you might even want to survive an application restart (or even backup-restore) to correlate the response. In such cases, you need to store correlation information in a persistent store such as a database or a JMS queue using message properties - and selectors to retrieve the correlation information back.
I've used both queues and databases for long time request/reply correlation information with success.
It's always a good practice to be able to fail over/restart the server or the application at any time knowing that any ongoing processing will take up where it left off without errors.
There is a cost in complexity and performance, but robustness is often perferred to performance.