assumptions:
There are microservices behind an api-gateway, they communicate through HTTP synchronously. obviously, each one of those microservices is a web server. now I want my microservice to play as a Kafka producer and "consumer" too. more clearly, my microservice produces events and listens to some topics for other events.
problem:
It seems traditionally a Kafka consumer use an infinite loop to poll the messages and stay alive (send heart-beats) my process thread is busy by serving host and cant be locked by an infinite loop. is there any way to somehow "listen" to the topic without block the thread just like pulsar listener or rabbit? I prefer not to separate my web server and consumer-processor due to the technical complexity I have to handle in development. may I use Kafka streams in this case? Is there something wrong with my assumptions?
You could just put your Consumer in its own Thread.
Assuming Java since you mentioned Kafka Streams, then if you're using Quarkus, Micronaut, Spring-Kafka, Reactor, Vertx, or similar Java web framework that also have Kafka connectors, that's what they do. If you're not using a web framework beyond basic Jersey, you probably should.
Even with Clojure, just abstract the Thread away like https://github.com/gklijs/bkes-demo/blob/main/topology/src/nl/openweb/topology/client.clj or any of the several Clojure clients about now. If you go wild you can use core.async.
Related
I need some help when choosing for message broker(RaabitMQ, Redis, etc) or other right tools for this situation.
I am upgrading my game server. It is written by Node.js. it consist of several process, i.e. GameRoom, Lobby, Chat, etc. When a user make request, the message will be routed to relevant process to process it. I do this by routing by my code and each process communicate with each other by node-ipc. However, this is not too efficient and is not scalable. Also, some process has very high work load(Lobby as many requests are related to it), we create several process of Lobby and route message randomly to different process of Lobby. I think message broker can help in this case and also I can even scale up by putting different process in different physical servers. I would like to know which message broker is suitable for this? Can a sender send a message to a queue which multiple consumers compete for a message and only one consumer consume it and reply the message to the sender? Thanks.
I'm not going to be able to talk about Kafka from experience, but any message-queue solution, as will RabbitMQ and ActiveMQ will do what you need.
I assume you're planning a flow like so:
REST_API -> queue -> Workers ----> data persistance <--------+
| |
+------> NotificationManager ----> user
The NotificationManager could be a service that lets the user know via Websockets or any other async communication method.
Some solutions will be better put together and take more weight off your shoulders. Solutions that are not just message-queues but are also task-queues will have ways with getting responses from workers.
Machinery, a project that's been getting my attention lately does all of those , whilst using MongoDB and RabbitMQ itself.
I'm trying to get my head around MassTransit in combination with RabbitMQ.
The basic concepts are working in a test project, but what I need is the following:
My system will have one or more servers that react to real life events (telephony). These events wil, by means of MassTransit and RabbitMQ, translate into messages that will be picked up by one or more receivers via a separate server, set up as RabbitMQ host. So far so good.
However, I cannot assume that I always have a connection between the publisher and the host machines. Just assume that the publishing server will continue to consume the real life events, but now cannot publish it's messages.
So, the question is: Does MassTransit have some kind of mechanism to store messages locally some way until the connection is re-established?
Or should I install RabbitMQ on every publishing server as well, in order to create a local exchange? Then I have to make the exchanges synchronize themselves after a reconnect.
Probably you have to implement a store and forward policy. Instead of publishing directly your message through MassTransit and RabbitMQ, you can store the message in a persistence repository (a local database) and delegate to some other process the notification through Masstransit of the messages stored before. This approach is often referred as "Client High Availability". This does not substitute the standard HA (High Availability) on server like the one implemented by RabbitMQ. But it's a good approach to use in a distributed system (like the one you described) because it could help you a lot in scenarios of server failure (e.g. an issue on RabbitMQ server that causes some loss of messages that you still have inside the store of some client and therefore you can make it process again).
I am building a web app which has two parts. In one part it uses a real time connection between the server and the client and in the other part it does some cpu intensive task to provide relevant data.
Implementing the real time communication in nodejs and the cpu intensive part in python/java. What is the best way the nodejs server can participate in a duplex communication with the other server ?
For a basic solution you can use Socket.IO if you are already using it and know how it works, it will get the job done since it allows for communication between a client and server where the client can be a different server in a different language.
If you want a more robust solution with additional options and controls or which can handle higher traffic throughput (though this shouldn't be an issue if you are ultimately just sending it through the relatively slow internet) you can look at something like ØMQ (ZeroMQ). It is a messaging queue which gives you more control and lots of different communications methods beyond just request-response.
When you set either up I would recommend using your CPU intensive server as the stable end(server) and your web server(s) as your client. Assuming that you are using a single server for your CPU intensive tasks and you are running several NodeJS server instances to take advantage of multi-cores for your web server. This simplifies your communication since you want to have a single point to connect to.
If you foresee needing multiple CPU servers you will want to setup a routing server that can route between multiple web servers and multiple CPU servers and in this case I would recommend the extra work of learning ØMQ.
You can use http.request method provided to make curl request within node's code.
http.request method is also used for implementing Authentication api.
You can put your callback in the success of request and when you get the response data in node, you can send it back to user.
While in backgrount java/python server can utilize node's request for CPU intensive task.
I maintain a node.js application that intercommunicates among 34 tasks spread across 2 servers.
In your case, for communication between the web server and the app server you might consider mqtt.
I use mqtt for this kind of communication. There are mqtt clients for most languages, including node/javascript, python and java. In my case I publish json messages using mqtt 'topics' and any task that has registered to subscribe to a 'topic' receives it's data when published. If you google "pub sub", "mqtt" and "mosquitto" you'll find lots of references and examples. Mosquitto (now an Eclipse project) is only one of a number of mqtt brokers that are available. Another very good broker that is written in Java is called hivemq.
This is a very simple, reliable solution that scales well. In my case literally millions of messages reliably pass through mqtt every day.
You must be looking for socketio
Socket.IO enables real-time bidirectional event-based communication.
It works on every platform, browser or device, focusing equally on reliability and speed.
Sockets have traditionally been the solution around which most
realtime systems are architected, providing a bi-directional
communication channel between a client and a server.
We are looking to build a facade in nodejs that will accept requests from a client and then farm out the requests to a number of services using request/reply pattern to a number of different backend services. We want these requests held on individual queues in the event that one of the backend services is down. From initially reading of the ZeroMQ docs, it appears each queue is bound to its own port. When sending a message to a socket, there doesn't appear to be a way of naming a queue/topic to send to.
Is there a one-one mapping between ports and queues?
Thanks, Tom
ZeroMQ doesn't have the concept of "queues" or "topics". Your application consists of tasks, connected across some protocol, e.g. tcp://, and sending each other messages in various patterns. In your example one task will bind to an address:port and the workers will connect to it. The sender then sends requests to its socket, which deals them out to workers.
The best way to learn ZeroMQ is to work through at least the first couple of chapters of the Guide, before you design your own application. Many of the existing messaging concepts you're familiar with disappear into simpler patterns with ZeroMQ.
assume we have an application which uses winsock to implement tcp communication.
for each socket we create a thread and block-receiving on it.
when data arrives, we would like to notify other threads (listening threads).
i was wondering what is the best way to implement this:
move away from this design and use a non-blocking socket, then the listening thread will have to iterate constantly and call a non-blocking receive, thus making it thread safe (no extra threads for the sockets)
use asynchronous procedure calls to notify listening threads - which again will have to alert-wait for apc to queue for them.
implement some thread safe message queue, where each socket thread will post messages to it, and the listener, again, will go over it every interval and pull data from it.
also, i read about WSAAsyncSelect, but i saw that this is used to send messages to a window. isnt there something similar for other threads? (well i guess apcs are...)
Thanks!
Use I/O completion ports. See the CreateIoCompletionPort() and the GetQueuedCompletionStatus() functions of the Win32 API (under File Management functions). In this instance, the socket descriptors are used in place of file handles.
You'll always be better off abstracting the mechanics of socket API (listening, accepting, reading & writing) in a separate layer from the application logic. Have an object that captures the state of a connection, which is created during an incoming connection and you can maintain buffers in this object for the incoming and outgoing traffic. This will allow your network interface layer to be independent of the application code. This will also make the code cleaner by separating the application functionality from the underlying communication mechanism.
Blocking or non-blocking socket decision depends on the level of scalability that your applications needs to achieve. If your application needs to support hundreds of incoming connections, adopting a thread-per-socket approach is not going to be very wise. You'll be better off going for an Io ports based implementation, which will make your app immensely scaleable at added code complexity. However, if you only foresee a few 10s of connections at any point in time, you can go for an asynchronous sockets model using Win32 events or messages. Win32 events based approach doesn't scale very well beyond a certain limit as you would have to manage multiple threads if the number of concurrent sockets exceed 63 (as WaitForMultipleObjects can only support a max of 64 sockets). Windows message based mechanism doesn't have this limitation though. OHOH, Win32 event based approach does not require a GUI window to work.
Check out WSAEventSelect along with WSAAsyncSelect API documentation in MSDN.
You might want to take a look at boost::asio package as well. It provides a neat (though a little complex) C++ abstraction over sockets API.