Seeking an understanding of ServiceStack.Redis: IRedisClient.PublishMessage vs IMessageQueueClient.Publish - servicestack

I am having a hard time separating the IRedisClient.PublishMessage and IMessageQueueClient.Publish and realize I must be mixing something up.
ServiceStack gives us the option to listen for pub/sub broadcasts like this:
static IRedisSubscription _subscription;
static IRedisClient redisClientSub;
static int received = 0;
static void ReadFromQueue()
{
redisClientSub = redisClientManager.GetClient();
_subscription = redisClientSub.CreateSubscription();
_subscription.OnMessage = (channel, msg) =>
{
try
{
received++;
}
catch (Exception ex)
{
}
};
Task.Run(() => _subscription.SubscribeToChannels("Test"));
}
Looks nice, straightforward. But what about the producer?
When looking at the classes available, I thought that one could either user the IRedisClient.PublishMessage(string toChannel, string message) or IMessageQueueClient.Publish(string queueName, IMessage message).
redisClient.PublishMessage("Test", json);
// or:
myMessageQueueClient.Publish("Test", new Message<CoreEvent>(testReq));
In both cases, you specify the channel name yourself. This is the behaviour I am seeing:
the subscriber above only receives the message if I use IRedisClient.PublishMessage(string toChannel, string message) and never if I use IMessageQueueClient.Publish(string queueName, IMessage message)
If I publish using IRedisClient.PublishMessage, I expected the "Test" channel to be populated (if I view with a Redis browser), but it is not. I never see any trace of the queue (let's say I don't start the subscription, but producers adds messages)
If I publish using IMessageQueueClient.Publish(string queueName, IMessage message), the channel "Test" is created and the messages are persisted there, but never popped/fetched-and-deleted.
I want to understand the difference between the two. I have looked at source code and read all I can about it, but I haven't found any documentation regarding IRedisClient.PublishMessage.

Mythz answered this on ServiceStack forum, here.
He writes:
These clients should not be used interchangeably, you should only be
using ServiceStack MQ clients to send MQ Messages or the Message MQ
Message wrapper.
The redis subscription is low level API to create a Redis Pub/Sub
subscription, a more useful higher level API is the Managed Pub/Sub
Server which wraps the pub/sub subscription behind a managed thread.
Either way, MQ Server is only designed to process messages from MQ
clients, if you’re going to implement your own messaging
implementation use your own messages & redis clients not the MQ
clients or MQ Message class.
and
No IRedisClient (& ServiceStack.Redis) APIs are for Redis Server, the
PublishMessage API sends the redis PUBLISH command. IRedisSubscription
creates a Redis Pub/Sub subscription, see Redis docs to learn how
Redis Pub/Sub works. The ServiceStack.Redis library and all its APIs
are just for Redis Server, it doesn’t contain any
ServiceStack.Messaging MQ APIs.
So just use ServiceStack.Redis for your custom Redis Pub/Sub
subscription implementation, i.e. don’t use any ServiceStack.Messaging
APIs which is for ServiceStack MQ only.

Related

Spring Integration Tcp project

I have a project that part of it is using Tcp connection, the case is as per below , I will also include a screen shot.
We have two clients, client 1 and client 2 those are conveyor belts so if we receive data on client one input we should send the reply to client 2 output and vise vers, I'm sure we can do it using Spring integration Tcp and probably getways. Am I approaching correctly Tcp integration at this case?
Yet I do not have code implementation but started to put something on it.
Sounds like you implementing a chat (or similar user-to-user) communication.
No, gateways won't help you here.
You need to have a TcpReceivingChannelAdapter and TcpSendingMessageHandler connected to the same AbstractServerConnectionFactory. The TcpSendingMessageHandler is registered as a TcpSender with that connection and all the sending connections are stored in the Map<String, TcpConnection> connections. When we produce a message to this MessageHandler, it tries to consult that registry like this:
private void handleMessageAsServer(Message<?> message) {
// We don't own the connection, we are asynchronously replying
String connectionId = message.getHeaders().get(IpHeaders.CONNECTION_ID, String.class);
TcpConnection connection = null;
if (connectionId != null) {
connection = this.connections.get(connectionId);
}
if (connection != null) {
So, on the receiving side (TcpReceivingChannelAdapter and its sub-flow) you need to ensure somehow that you really set a proper IpHeaders.CONNECTION_ID header for producing so-called reply in the end to a desired client.
You probably can react for the TcpConnectionOpenEvent via #EventListener and register some business key with the connectionId for the future correlation. When you send a message, you supply that target user business key, in the TcpReceivingChannelAdapter sub-flow you take that business key and obtain a desired connectionId from you registry. And enrich it into the IpHeaders.CONNECTION_ID header for automatic logic in the TcpSendingMessageHandler.
When TcpConnectionCloseEvent happens you have to remove its respective entry from your custom registry.
Since TCP/IP comes without headers support there is no any out-of-the-box mechanism to implement such a correlation feature.
Although TcpConnectionOpenEvent might not be enough for you since there is no any business info when connection is established. Perhaps you would need to implement some hand-shake logic in the TcpReceivingChannelAdapter flow to distinguish a real message and connection metadata for registering in the custom registry.
See more info in the docs: https://docs.spring.io/spring-integration/docs/current/reference/html/ip.html#ip-correlation
It might be also better for your use-case to look into a WebSocket support: https://docs.spring.io/spring-integration/docs/current/reference/html/web-sockets.html#web-sockets

nodejs rhea npm for amqp couldn't create subscription queue on address in activemq artemis

I have an address "pubsub.foo" already configured as multicast in broker.xml.
<address name="pubsub.foo">
<multicast/>
</address>
As per the Artemis documentation:
When clients connect to an address with the multicast element, a subscription queue for the client will be automatically created for the client.
I am creating a simple utility using rhea AMQP Node.js npm to publish messages to the address.
var connection = require('rhea').connect({ port: args.port, host: args.host, username:'admin', password:'xxxx' });
var sender = connection.open_sender('pubsub.foo');
sender.on('sendable', function(context) {
var m = 'Hii test'
console.log('sent ' + m);
sender.send({body:m});
connection.close();
});
I enabled debug log and while running the client code I see the message like this.
2020-02-03 22:43:25,071 DEBUG [org.apache.activemq.artemis.core.postoffice.impl.PostOfficeImpl] Message org.apache.activemq.artemis.protocol.amqp.broker.AMQPMessage#68933e4b is not going anywhere as it didn't have a binding on address:pubsub.foo
I also tried different variations of the topic, for example, client1.pubsub.foo, pubsub.foo::client1 however no luck from the client code. Please share your thoughts. I am new to ActiveMQ Artemis.
What you're observing actually is the expected behavior.
Unfortunately, the documentation you cited isn't as clear as it could be. When it says a subscription queue will be created in response to a client connecting it really means a subscriber not a producer. That's why it creates a subscription queue. The semantics for a multicast address (and publish/subscribe in general) dictate that a message sent when there are no subscribers will be dropped. Therefore, you need to create a subscriber and then send a message.
If you want different semantics then I recommend you use anycast.

How to listen to a queue using azure service-bus with Node.js?

Background
I have several clients sending messages to an azure service bus queue. To match it, I need several machines reading from that queue and consuming the messages as they arrive, using Node.js.
Research
I have read the azure service bus queues tutorial and I am aware I can use receiveQueueMessage to read a message from the queue.
However, the tutorial does not mention how one can listen to a queue and read messages as soon as they arrive.
I know I can simply poll the queue for messages, but this spams the servers with requests for no real benefit.
After searching in SO, I found a discussion where someone had a similar issue:
Listen to Queue (Event Driven no polling) Service-Bus / Storage Queue
And I know they ended up using the C# async method ReceiveAsync, but it is not clear to me if:
That method is available for Node.js
If that method reads messages from the queue as soon as they arrive, like I need.
Problem
The documentation for Node.js is close to non-existant, with that one tutorial being the only major document I found.
Question
How can my workers be notified of an incoming message in azure bus service queues ?
Answer
According to Azure support, it is not possible to be notified when a queue receives a message. This is valid for every language.
Work arounds
There are 2 main work arounds for this issue:
Use Azure topics and subscriptions. This way you can have all clients subscribed to an event new-message and have them check the queue once they receive the notification. This has several problems though: first you have to pay yet another Azure service and second you can have multiple clients trying to read the same message.
Continuous Polling. Have the clients check the queue every X seconds. This solution is horrible, as you end up paying the network traffic you generate and you spam the service with useless requests. To help minimize this there is a concept called long polling which is so poorly documented it might as well not exist. I did find this NPM module though: https://www.npmjs.com/package/azure-awesome-queue
Alternatives
Honestly, at this point, you may be wondering why you should be using this service. I agree...
As an alternative there is RabbitMQ which is free, has a community, good documentation and a ton more features.
The downside here is that maintaining a RabbitMQ fault tolerant cluster is not exactly trivial.
Another alternative is Apache Kafka which is also very reliable.
You can receive messages from the service bus queue via subscribe method which listens to a stream of values. Example from Azure documentation below
const { delay, ServiceBusClient, ServiceBusMessage } = require("#azure/service-bus");
// connection string to your Service Bus namespace
const connectionString = "<CONNECTION STRING TO SERVICE BUS NAMESPACE>"
// name of the queue
const queueName = "<QUEUE NAME>"
async function main() {
// create a Service Bus client using the connection string to the Service Bus namespace
const sbClient = new ServiceBusClient(connectionString);
// createReceiver() can also be used to create a receiver for a subscription.
const receiver = sbClient.createReceiver(queueName);
// function to handle messages
const myMessageHandler = async (messageReceived) => {
console.log(`Received message: ${messageReceived.body}`);
};
// function to handle any errors
const myErrorHandler = async (error) => {
console.log(error);
};
// subscribe and specify the message and error handlers
receiver.subscribe({
processMessage: myMessageHandler,
processError: myErrorHandler
});
// Waiting long enough before closing the sender to send messages
await delay(20000);
await receiver.close();
await sbClient.close();
}
// call the main function
main().catch((err) => {
console.log("Error occurred: ", err);
process.exit(1);
});
source :
https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-nodejs-how-to-use-queues
I asked myslef the same question, here is what I found.
Use Google PubSub, it does exactly what you are looking for.
If you want to stay with Azure, the following ist possible:
cloud functions can be triggered from SBS messages
trigger an event-hub event with that cloud function
receive the event and fetch the message from SBS
You can make use of serverless functions which are "ServiceBusQueueTrigger",
they are invoked as soon as message arrives in queue,
Its pretty straight forward doing in nodejs, you need bindings defined in function.json which have type as
"type": "serviceBusTrigger",
This article (https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-service-bus#trigger---javascript-example) probably would help in more detail.

Domain events for local consumption in DDD

I'm familiarizing myself DDD lately and trying to get hold of the key concepts and I have a query incase of publishing domain events for the local subscribers, so can I assume that the event publisher, takes care of both publishing to remote subscribers thro AQMP while also leveraging observable to publish it towards the local subscribers, is this a scalable solution? or is there a familiar pattern to handle this?(Also advise if there is a reactive solution to the prblm, may be RxJava or the likes)
The locality of subscribers, local or remote, should be completely transparent to the domain entity raising an publishing the event.
Your thoughts are very similar to design I use. I use a quite primitive version of a local event publishers, which communicate via messaging middleware (AMQP to propagate events to all other local publishers).
I'm not doing that in Java, but in fact, for the local event "bus" I use Rx.Net (reactive extensions have an almost identical API in all languages, so RxJava would work)
a simplified version with transaction support ripped out would look like this:
public class EventHub
{
private readonly ISubject<object, object> _messages;
public EventHub()
{
var _subject = new Subject<object>();
m_messages = Subject.Synchronize(_subject);
}
public void Publish<T>(T message)
{
m_messages.OnNext(message);
}
public IObservable<T> AsObservable<T>()
{
return m_messages
.ObserveOn(ThreadPoolScheduler.Instance)
.OfType<T>()
.Synchronize()
.AsObservable();
}
}
Adding remote capability would be adding another subscriber, which would subscribe to all events and route them to other local EventHubs, and also listen to other eventhubs and publish their events on the local hub. It would not change the EventHub component.

How to create multiple threads for ServiceStack RabbitMQ consumer?

I need to integrate MQ feature in my ServiceStack application. I have registered the Message Handler in AppHost. The handler for my ServiceStack request(Post) will publish the message to the MQ broker. I have created the consumer for that message. This is all working fine.
Now what I need is to have multiple threads available to consume the message and process that. I have read that ServiceStack implementation of rabbitMQ provides the feature to specify multiple threads for an operation:
https://github.com/ServiceStack/ServiceStack/wiki/Rabbit-MQ#allocating-multiple-threads-for-specific-operations
But I am not able to specify these threads. I have registered the handler as
container.Register(c => new RabbitMqServer());
var mqServer = container.Resolve();
mqServer.RegisterHandler(ServiceController.ExecuteMessage, noOfThreads: 4);
But it gives me error as RegisterHandler does not have parameter 'noOfThreads'.
I am running the 4.0.24.0 version for ServiceStack RabbitMQ. Is there something else that I am missing here?
The noOfThreads is only available on the RabbitMqServer and not a feature of the generic IMessageService. You need to cast the IMessageService you get back from the Container.Resolve() to a RabbitMqServer.
container.Register<IMessageService>(c => new RabbitMqServer());
var mqServer = (RabbitMqServer)container.Resolve<IMessageService>();
mqServer.RegisterHandler<CallBatchMessage>(ServiceController.ExecuteMessage, noOfThreads: 4);

Resources