I need to integrate MQ feature in my ServiceStack application. I have registered the Message Handler in AppHost. The handler for my ServiceStack request(Post) will publish the message to the MQ broker. I have created the consumer for that message. This is all working fine.
Now what I need is to have multiple threads available to consume the message and process that. I have read that ServiceStack implementation of rabbitMQ provides the feature to specify multiple threads for an operation:
https://github.com/ServiceStack/ServiceStack/wiki/Rabbit-MQ#allocating-multiple-threads-for-specific-operations
But I am not able to specify these threads. I have registered the handler as
container.Register(c => new RabbitMqServer());
var mqServer = container.Resolve();
mqServer.RegisterHandler(ServiceController.ExecuteMessage, noOfThreads: 4);
But it gives me error as RegisterHandler does not have parameter 'noOfThreads'.
I am running the 4.0.24.0 version for ServiceStack RabbitMQ. Is there something else that I am missing here?
The noOfThreads is only available on the RabbitMqServer and not a feature of the generic IMessageService. You need to cast the IMessageService you get back from the Container.Resolve() to a RabbitMqServer.
container.Register<IMessageService>(c => new RabbitMqServer());
var mqServer = (RabbitMqServer)container.Resolve<IMessageService>();
mqServer.RegisterHandler<CallBatchMessage>(ServiceController.ExecuteMessage, noOfThreads: 4);
Related
How can I consume Server Sent Events with Spring Integration? I am aware Spring supports SSE with Webflux, but how to convert the incoming Flux into separate Message instances? And possibly wrap this code into some Spring-Integration-Lifecycle-aware component (MessageProducerSupport?)
WebClient client = WebClient.create("http://myhost:8080/sse");
ParameterizedTypeReference<ServerSentEvent<String>> type
= new ParameterizedTypeReference<ServerSentEvent<String>>() {};
Flux<ServerSentEvent<String>> eventStream = client.get()
.uri("/stream-sse")
.retrieve()
.bodyToFlux(type);
eventStream.subscribe(
content -> ;/* here I believe the message should be produced/sent to a channel */ );
See Spring Integration WebFlux Outbound Gateway: https://docs.spring.io/spring-integration/docs/current/reference/html/webflux.html#webflux-outbound:
The setExpectedResponseType(Class<?>) or setExpectedResponseTypeExpression(Expression) identifies the target type of the response body element conversion. If replyPayloadToFlux is set to true, the response body is converted to a Flux with the provided expectedResponseType for each element, and this Flux is sent as the payload downstream. Afterwards, you can use a splitter to iterate over this Flux in a reactive manner.
WebFlux.outboundGateway("http://myhost:8080/sse/stream-sse")
.httpMethod(HttpMethod.GET)
.replyPayloadToFlux(true)
.setExpectedResponseTypeExpression(new ParameterizedTypeReference<ServerSentEvent<String>>() {})
To make it start working just after an application is ready, yo can implement an ApplicationRunner to send a "void" message into a channel for the flow with that WebFlux.outboundGateway(). I don't think we need a special, dedicated component just for SSE requesting and producing. The combination of existing components is fully enough.
I have an Azure Message Bus Topic.
I have one "Session enabled" Azure Message Bus Consumer for this Topic.
I have around 3 Worker Services that are using the same Consumer. So the work is shared between these 3 Workers.
The Messages which are sent to the consumer need to be ordered, thats why I am using the "Session Feature" on the Consumer.
I believe that on a first Message, the Session of the Message gets bind to a Worker Service.
For certain Reasons I want to abandon not only a Message but also the session so that it can be picked up by another of the 3 Worker Services.
My questions:
Is this possible?
If yes how can I do this in the code?
Is there something like "Accept Session Or Not" Handler which kicks in when Message received?
See code below:
private void SetupServiceBusSessionProcessors2()
{
var busProcessorOptions = new ServiceBusSessionProcessorOptions();
var busProcessor = _busClient.CreateSessionProcessor("fooTopic", "fooSubscription", busProcessorOptions);
busProcessor.ProcessMessageAsync += args => ProcessSessionMessageHandler2(args);
}
private async Task ProcessSessionMessageHandler2(ProcessSessionMessageEventArgs args)
{
if (false) // Condition here which Abandons Message AND Session
{
// the following line of code seems only to abandon the Message
// but it seems like the session is locked to this service
// i want that other services which are listening via the same consumer can try to handle the session
await args.AbandonMessageAsync(args.Message);
}
}
This is possible in version 7.3.0-beta.1 using the ReleaseSession method on the event args. Note that this is a beta version so the API is subject to change before there is a stable release.
I am having a hard time separating the IRedisClient.PublishMessage and IMessageQueueClient.Publish and realize I must be mixing something up.
ServiceStack gives us the option to listen for pub/sub broadcasts like this:
static IRedisSubscription _subscription;
static IRedisClient redisClientSub;
static int received = 0;
static void ReadFromQueue()
{
redisClientSub = redisClientManager.GetClient();
_subscription = redisClientSub.CreateSubscription();
_subscription.OnMessage = (channel, msg) =>
{
try
{
received++;
}
catch (Exception ex)
{
}
};
Task.Run(() => _subscription.SubscribeToChannels("Test"));
}
Looks nice, straightforward. But what about the producer?
When looking at the classes available, I thought that one could either user the IRedisClient.PublishMessage(string toChannel, string message) or IMessageQueueClient.Publish(string queueName, IMessage message).
redisClient.PublishMessage("Test", json);
// or:
myMessageQueueClient.Publish("Test", new Message<CoreEvent>(testReq));
In both cases, you specify the channel name yourself. This is the behaviour I am seeing:
the subscriber above only receives the message if I use IRedisClient.PublishMessage(string toChannel, string message) and never if I use IMessageQueueClient.Publish(string queueName, IMessage message)
If I publish using IRedisClient.PublishMessage, I expected the "Test" channel to be populated (if I view with a Redis browser), but it is not. I never see any trace of the queue (let's say I don't start the subscription, but producers adds messages)
If I publish using IMessageQueueClient.Publish(string queueName, IMessage message), the channel "Test" is created and the messages are persisted there, but never popped/fetched-and-deleted.
I want to understand the difference between the two. I have looked at source code and read all I can about it, but I haven't found any documentation regarding IRedisClient.PublishMessage.
Mythz answered this on ServiceStack forum, here.
He writes:
These clients should not be used interchangeably, you should only be
using ServiceStack MQ clients to send MQ Messages or the Message MQ
Message wrapper.
The redis subscription is low level API to create a Redis Pub/Sub
subscription, a more useful higher level API is the Managed Pub/Sub
Server which wraps the pub/sub subscription behind a managed thread.
Either way, MQ Server is only designed to process messages from MQ
clients, if you’re going to implement your own messaging
implementation use your own messages & redis clients not the MQ
clients or MQ Message class.
and
No IRedisClient (& ServiceStack.Redis) APIs are for Redis Server, the
PublishMessage API sends the redis PUBLISH command. IRedisSubscription
creates a Redis Pub/Sub subscription, see Redis docs to learn how
Redis Pub/Sub works. The ServiceStack.Redis library and all its APIs
are just for Redis Server, it doesn’t contain any
ServiceStack.Messaging MQ APIs.
So just use ServiceStack.Redis for your custom Redis Pub/Sub
subscription implementation, i.e. don’t use any ServiceStack.Messaging
APIs which is for ServiceStack MQ only.
Background
I have several clients sending messages to an azure service bus queue. To match it, I need several machines reading from that queue and consuming the messages as they arrive, using Node.js.
Research
I have read the azure service bus queues tutorial and I am aware I can use receiveQueueMessage to read a message from the queue.
However, the tutorial does not mention how one can listen to a queue and read messages as soon as they arrive.
I know I can simply poll the queue for messages, but this spams the servers with requests for no real benefit.
After searching in SO, I found a discussion where someone had a similar issue:
Listen to Queue (Event Driven no polling) Service-Bus / Storage Queue
And I know they ended up using the C# async method ReceiveAsync, but it is not clear to me if:
That method is available for Node.js
If that method reads messages from the queue as soon as they arrive, like I need.
Problem
The documentation for Node.js is close to non-existant, with that one tutorial being the only major document I found.
Question
How can my workers be notified of an incoming message in azure bus service queues ?
Answer
According to Azure support, it is not possible to be notified when a queue receives a message. This is valid for every language.
Work arounds
There are 2 main work arounds for this issue:
Use Azure topics and subscriptions. This way you can have all clients subscribed to an event new-message and have them check the queue once they receive the notification. This has several problems though: first you have to pay yet another Azure service and second you can have multiple clients trying to read the same message.
Continuous Polling. Have the clients check the queue every X seconds. This solution is horrible, as you end up paying the network traffic you generate and you spam the service with useless requests. To help minimize this there is a concept called long polling which is so poorly documented it might as well not exist. I did find this NPM module though: https://www.npmjs.com/package/azure-awesome-queue
Alternatives
Honestly, at this point, you may be wondering why you should be using this service. I agree...
As an alternative there is RabbitMQ which is free, has a community, good documentation and a ton more features.
The downside here is that maintaining a RabbitMQ fault tolerant cluster is not exactly trivial.
Another alternative is Apache Kafka which is also very reliable.
You can receive messages from the service bus queue via subscribe method which listens to a stream of values. Example from Azure documentation below
const { delay, ServiceBusClient, ServiceBusMessage } = require("#azure/service-bus");
// connection string to your Service Bus namespace
const connectionString = "<CONNECTION STRING TO SERVICE BUS NAMESPACE>"
// name of the queue
const queueName = "<QUEUE NAME>"
async function main() {
// create a Service Bus client using the connection string to the Service Bus namespace
const sbClient = new ServiceBusClient(connectionString);
// createReceiver() can also be used to create a receiver for a subscription.
const receiver = sbClient.createReceiver(queueName);
// function to handle messages
const myMessageHandler = async (messageReceived) => {
console.log(`Received message: ${messageReceived.body}`);
};
// function to handle any errors
const myErrorHandler = async (error) => {
console.log(error);
};
// subscribe and specify the message and error handlers
receiver.subscribe({
processMessage: myMessageHandler,
processError: myErrorHandler
});
// Waiting long enough before closing the sender to send messages
await delay(20000);
await receiver.close();
await sbClient.close();
}
// call the main function
main().catch((err) => {
console.log("Error occurred: ", err);
process.exit(1);
});
source :
https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-nodejs-how-to-use-queues
I asked myslef the same question, here is what I found.
Use Google PubSub, it does exactly what you are looking for.
If you want to stay with Azure, the following ist possible:
cloud functions can be triggered from SBS messages
trigger an event-hub event with that cloud function
receive the event and fetch the message from SBS
You can make use of serverless functions which are "ServiceBusQueueTrigger",
they are invoked as soon as message arrives in queue,
Its pretty straight forward doing in nodejs, you need bindings defined in function.json which have type as
"type": "serviceBusTrigger",
This article (https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-service-bus#trigger---javascript-example) probably would help in more detail.
I have written a request-reply broker using zeromq and the C programming language. The broker routes client requests to the appropriate services, and then routes the reply back to the client. The services are written in JAVA.
Can someone please explain how to have the services communicate with the broker. I am sure that this must be a common scenario, but I don't have much experience, so can someone please help me with making my code inter-operable.
Please assume that the services will not be zeromq aware. Is node.js to be used in such a scenario? Will I have to write an http front end?
Here's one way you can do it using async PUSH/PULL sockets. I'm psuedo-coding this, so fill in the blanks yourself:
Assuming the Java services are POJO's residing in their own process, let's say we have a simple service with no zmq dependencies:
public class MyJavaService{
public Object invokeService(String params){
}
}
Now we build a Java delegate layer that pulls in messages from the broker, delegating requests to the Java service methods, and returning the response on a separate socket:
//receive on this
Socket pull = ctx.createSocket(ZMQ.PULL)
pull.connect("tcp://localhost:5555")
//respond on this
Socket push = ctx.createSocket( ZMQ.PUSH)
psuch.connect("tcp://localhost:5556")
while( true){
ZMsg msg = pull.recvMsg( pull)
//assume the msg has 2 frames,
//one for service to invoke,
//the other with arguments
String svcToInvoke = msg.popString()
String svcArgs = msg.popString()
if( "MyJavaService".equals(svcToInvoke)){
ZMsg respMsg = new ZMsg()
respMsg.push( (new MyJavaService()).invokeService( svcArgs))
respMsg.send( push)
}
}
On the broker side, just create the PUSH/PULL sockets to communicate with the Java services layer (I'm not a C++ programmer, so forgive me)
int main () {
zmq::context_t context(1);
zmq::socket_t push(context, ZMQ_PUSH);
push.bind( "tcp://localhost:5555");
// First allow 0MQ to set the identity
zmq::socket_t pull(context, ZMQ_PULL);
pull.bind( "tcp://localhost:5556");
//code here to handle request/response,
//to clients
..
}
Using PUSH/PULL works for this approach, but the ideal approach is to use ROUTER on the server, and DEALER on the client, for full asynchronous communication, example here.
Hope it helps!