For communication between the various services used pubsub. If you create multiple workers
, they both accept data both to process them. What are some methods
that have worked in the same message, only one of the workers
.
PS Perhaps there are some layers, so-called message brokers
Related
I want to implement retry logic while consuming from Kafka topic using KafkaJS, so basically, I will have 2 topics main-topic and retry-topic and I will
read from -> main-topic
if processing fails | -> retry topic
so is it a bad practice to use one consumer for listening from both topics(both main and retry), as kafka allows to listen from multiple topics using same consumer.
It's not a bad practice at all.
The only problem you may run into using one consumer is that the topics may need differ configurations (connection settings, deserializer, etc). In that case, you can create two separate Consumer instances rather than one subscribing to both.
We have a distributed application using various Docker containers, communicating internally via RabbitMq (using Rabbot). This is all working well, including using RabbitMq as a container independent timer using message timeouts and deadlettering. Each of the containers guarantees (configures) the full topology on startup. Whichever is first will actually set it up.
However, we now need to seed our timers exactly once, when the rabbitMQ topology is first setup. Is there any way to seed a rabbitMQ queue with a message on creation, or detect if the queue already existed on the rabbitMq servers or that the configuration call resulted in creating the queue?
We have a Azure service fabric micro-service which listen to multiple azure service bus topics(Topic A, Topic B).
Topic A has more then 10 times message traffic then topic B. and to handle the scale-ability of service we will create the multiple instance of service.
My first question is, In most of the services instance will not get the message in Topic B, As Topic B has less traffic, So will it be waste of resources ?
2 Is it better to create different micro-services for Topic A and Topic B listeners, and create 10x instance of micro-service which listen to topic A and x instance of topic B listener service ?
Is create a message listener in azure service bus, keep on pulling message every time ? means continuously looking/ checking for message, message is there or not.
Thanks Guys for your supports.
If one service receives messages from 2 topics, there's little waste of resources. Listening for messages is not a very resource intensive process.
This depends on your application requirements.
This depends on whether you are using SBMP / SOAP (default) or AMQP as the communication protocol. AMQP is connection based. SBMP does (long) polling.
Microservices advocates the idea of loosely coupled services, where each micro-service will handle his own domain.
Following the microservices approach, if you understand that you had to create two different topics to publish your messages, probably it is because they have different scopes\domain, needing their own micro-service.
In your description it is hard to identify if the domain of TopicA and TopicB are related, so we can not offer a good suggestion.
In any case, if one service listen for both topics, let's assume TopicA handles 1000 messages and TopicB handles 100 per second.
In case you have to publish a new version of your application to handle changes on TopicB messages, you would have to stop the handling of TopicA, that was not necessary. So you are coupling the services, that to begin with should be two independent services, or both topics should be handle as a single one.
Regarding your questions:
1 My first question is, In most of the services instance will not get
the message in Topic B, As Topic B has less traffic, So will it be
waste of resources ?
Waste of resources is relative how you design your application, it might be if your service listen the queue\topic and handle it at the same time, and uses too much memory to keep running all the time. In this scenario, would be case to split them and make a Queue\Topic Listener and other Message Handler that will receive the message to process, and if it keep too long without processing messages you shut it down, leaving just the listener. You could also use actors instead of a service.
2 Is it better to create different micro-services for Topic A and
Topic B listeners, and create 10x instance of micro-service which
listen to topic A and x instance of topic B listener service ?
Yes for the services, regarding the the number of instances, it should be driven by the size of the queue, otherwise you would have too much listeners and also wasting resources, if you follow the approach of splitting the services, you would need one listener receiving the messages from the queue\topic and it would delivery the messages to multiple messages handlers(service instances\actors) and the queue\topic listener control the number of running instances at same time.
3 Is create a message listener in azure service bus, keep on pulling
message every time ? means continuously looking/ checking for message,
message is there or not.
Is not the only approach, but it's correct.
My requirement is to load balance 2 MQTT nodes running on different VMs and then having consumers to these MQTT brokers on both nodes. The job of the consumers will be to subscribe on one topic and after receiving the data, publish it to Kafka. Problem I see if that since both MQTT consumers are subscribed on the same topic, they will receive the same message and both will insert it into Kafka thereby creating duplicates. is there anyway to avoid writing duplicates into Kafka?
I have tried Mosquitto and Mosca brokers but they do not support clustering. So subscribed clients were not getting messages if they got subscribed to a different node then the node where message was published. Both nodes are behind HAProxy.
I am currently using emqtt broker which supports clustering and the load balancing issue gets solved by that but it seems it does not support shared subscriptions across cluster nodes.
A feature like the Kafka consumer group is what is required I believe. Any ideas?
Have you tried HiveMQ?
It offers so called shared subscriptions.
If shared subscriptions are used, all clients which share the same subscription will receive messages in an alternating fashion.
I'm having an application where I map devices from the physical world to Reliable Actors in Azure Fabric. Each time I receive a message from a device, I want to push a message to an event hub.
What I'm doing right now is creating/using/closing the EventHubClient object for each message.
This is very inefficient (it takes about 1500ms) but it solves an issue I had in the past where I was keeping the EventHubClient in memory. When I have a lot of devices, the underlying virtual machine can quickly run out of network connections.
I'm thinking about creating a new actor that would be responsible for pushing data to the EventHub (by keeping the EventHubClient alive). Because of the turned based concurrency model of Reliable Actors, I'm not sure it's a good idea. If I get 10 000 devices pushing data "at the same time", each of their actors will block to push the message to the new actor that pushes message to the EventHub.
What is the recommended approach for this scenario ?
Thanks,
One approach would be to create a stateless service that is responsible for pushing messages to the EventHub. Each time an Actor receives a message from the device (by the way, how are they communicating with actors?) the Actor calls the stateless service. The stateless service in turn would be responsible for creating, maintining and disposing of one EventHubClient per service. Reliable Service would not introduce the same 'overhead' when it comes to handling incoming messages as a Reliable Actor would. If it is important for your application that the messages reach the EventHub in strictly the same order that they were produced in then you would have to do this with a Stateful Service and a Reliable Queue. (Note, this there is on the other hand no guarantee that Actors would be able to finish handling incoming messages in the same order as they are produced)
You could then fine tune-tune the solution by experimenting with the instance count (https://learn.microsoft.com/en-us/azure/service-fabric/service-fabric-availability-services) to make sure you have enough instances to handle the throughput of incoming messages. How many instances are roughly determined by the number of nodes and cores per node, although other factors may also affect.
Devices communicate with your Actors, the Actors in turn communicate with the Service (may be Stateless or Stateful if you want to queue message, see below), each Service manages an EventHubClient that can push messages to the EventHub.
If your cluster is unable to support an instance count for this service that is high enough (a little simplified: more instances = higher throughput), then you may need to create it as a Stateful Service instead and put messages in a Reliable Queue in the Service and then have the the RunAsync for the Service processing the queue in order. This could take the pressure of peaks in performance.
The Service Fabric Azure-Samples WordCount shows how you work with different Partitions to make the messages from Actors target different instances (or really partitions).
A general tip would be to not try to use Actors for everything (but for the right things they are great and reduces complexity a lot), the Reliable Services model support a lot more scenarios and requirements and could really complement your Actors (rather than trying to make Actors do something they are not really designed for).
You could use a pub/sub pattern here (use the BrokerService).
By decoupling event publishing from event processing, you don't need to worry about the turn based concurrency model.
Publishers:
The Actor sends out messages by simply publishing them to a BrokerService.
Subscribers
Then you use one or more Stateless Services or (different) Actors as subscribers of the events.
They would send them into EventHub in their own pace.
Event Hub Client
Using this approach you'd have full control over the EventHubClient instance counts and lifetimes.
You could increase event processing power by simply adding more subscribers.
In my opinion you should directly call from your actors the event hub in a background thread with an internal memory queue. You should aggregate messages and use SendBatch to improve performance.
The event hub is able to receive the load by himself.