How do I create a Pulsar consumer that listens to topic(s), but does not acknowledge the messages, basically only "eavesdropping" the conversation.
The Reader interface could be a good fit for this use case. In the beginning, you need to specify from which message you would like to start reading the topic. There is no need to acknowledge messages. When the reader is stopped there will be no subscription left behind. You can find an example here.
One more option is to use interceptors, but it will work only in case you have access to the consumer code.
Related
Following the example found in GitHub https://github.com/spring-cloud/spring-cloud-gcp/tree/master/spring-cloud-gcp-samples/spring-cloud-gcp-pubsub-polling-binder-sample regarding polling messages from a PubSub subscription, I was wondering...
Is it possible to make a PollableMessageSource retrieve List<Message<?>> instead of a single message per poll?
I've seen the #Poller notation only being used in Source typed objects, never in Processor or Sink. Is it possible to use in such context when for example using #StreamListener or with a functional approach?
The PollableMessageSource binding and Source stream applications are fully based on the Poller and MessageSource abstraction from Spring Integration where its contract is to produce a single message to the channel configured. The point of the messaging is really to process a single message not affecting others. The failure for one message doesn't mean to fail others in the flow.
On the other hand you probably mean GCP Pub/Sub messages to be produced as a list in the Spring message payload. That is really possible, but via some custom code from Pub/Sub consumer and MessageSource impl. Although I would think twice to expect some batched from the source. Probably you may utilize an aggregator to build some small windows if your further logic is about processing as list. But again: it is going to be a single Spring message.
May be better to start thinking about a reactive function implementation where you indeed can expect a Flux<Message<?>> as an input and Spring Cloud Stream framework will take care for you how to emit the data from Pub/Sub into the reactive stream you expect.
See more info in docs: https://docs.spring.io/spring-cloud-stream/docs/3.1.0/reference/html/spring-cloud-stream.html#_reactive_functions_support
I would like to know if I can have persistence in my Spring Integration setup when I use a aggregator, which is not backed by a MessageStore, by leveraging the persistence of AMQP (RabbitMQ) queues before and after the aggregator.
I imagine that this would use ack's: The aggregator won't ack a message before it's collected all the parts and sent out the resulting message.
Additionally I would like to know if this is ever a good idea :)
I am new working with queue's, and am trying to get a good feel for patterns to use.
My business logic for this is as follows:
I receive a messages on one queue.
Each message must result in two unrelated webservice calls (preferably in parallel).
The results of these two calls must be combined with details from the original message.
The combination must then be sent out as a new message on a queue.
Messages are important, so they must not be lost.
I was/am hoping to use only one 'persistent' system, namely RabbitMQ, and not having to add a database as well.
I've tried to keep the question specific, but any other suggestions on how to approach this are greatly appreciated :)
What you would like to do recalls me Scatter-Gather EI Pattern.
So, you get a message from the AMQP send it into the ScatterGather endpoint and wait for the aggregated reply. That's enough for to stick with the default acknowledge.
Right, the scatterChannel can be PublishSubscribeChannel with an executor to call Web Services in parallel. Anyway the gatherer process will wait for replies according the release strategy and will block the original AMQP listener do not ack the message prematurely.
I was using AzureQueue to communicate between roles. My messages like "GoToMaintenanceMode", "StopSendingEmails", "DoNotAcceptRequests" etc. But I realized that, it won't work for my scenario when I have multiple instances due to queue message will shows up only 1 instance at a time.
So my question is beside the options below is there an elegant way to handle this issue something like Role.AllInstances.Run() etc.?
the method I'm using it right now:
instance peeks the message, adds it's own instance id to the message and puts it back to the queue, and does not peek the message if it contains it's own instance id.
P.S. I do not want to implement TCP listener, asking for native solution if there is one.
You could use Windows Azure Service Bus Topics/Subscriptions instead of queues. They support multicasting (i.e. multiple receivers).
A short how-to guide can be found here
http://www.windowsazure.com/en-us/develop/net/how-to-guides/service-bus-topics/
Basically, your queue would become a Topic all your instances would become Subscriber to the Topic.
As you do not want TCP listener option or Service bus option. How about extending your same idea with multiple queues. Instance1 will read from Queue1 and Instance2 will read from Queue2 and so on. The only thing you need to handle is the number of queues and simultaneous adding of queue messages to all the queues.
I have no clue if it's better to ask this here, or over on Programmers.SE, so if I have this wrong, please migrate.
First, a bit about what I'm trying to implement. I have a node.js application that takes messages from one source (a socket.io client), and then does processing on the message, which might result in zero or more messages back out, either to the sender, or other clients within that group.
For the processing, I would like to essentially just shove the message into a queue, then it works its way through various message processors that might kick off their own items, and eventually, the bit running socket.io is informed "Hey, send this message back"
As a concrete example, say a user signs into the service, that sign in message is then placed in the queue, where the authorization processor gets it, does it's thing, then places a message back in the queue saying the client's been authorized. This goes back to the socket.io socket that is connected to the client, along with other clients that might be interested. It can also go to other subsystems that might want to do more processing on authorization (looking up user info, sending more info to the client based on their data, etc).
If I wanted strong coupling, this would be easy, but I tried that before, and it just goes to a mess of spaghetti code that's very fragile, and I would like to avoid that. Another wrench in the setup is this should be cluster-able, which is where the real problem comes in. There might be more than one, say, authorization processor running. But the authorization message should be processed only once.
So, in short, I'm looking for a pattern/technique that will allow me to, essentially, have multiple "groups" of subscribers for a message, and the message will be processed only once per group.
I thought about maybe having each instance of a processor generate a unique name that would be used as a list in Reids. This name would then be registered with some sort of dispatch handler, and placed into a set for that group of subscribers. Then when a message arrives, the dispatch pulls a random member out of that set, and places it into that list. While it seems like this would work, it seems somewhat over-complicated and fragile.
The core problem is I've never designed a system like this, so I'm not even sure the proper terms to use or look up. If anyone can point me in the right direction for this, I would be most appreciative.
I think what your describing is similar to https://www.getbridge.com/ service. I it but ended up writing my own based on zeromq, it allows you to register services, req -> <- rec and channels which are pub / sub workers.
As for the design, I used a client -> broker -> services & channels which are all plug and play using auto discovery, you have the services register their schema with the brokers who open a tcp connection so that brokers on other servers can communicate with that broker groups services. Then internal services and clients connect via unix sockets or ipc channels which ever is preferred.
I ended up wrapping around the redis publish/subscribe functions a bit to do this. Each type of message processor gets a "group name", and there can be multiple instances of the processor within that group (so multiple instances of the program can run for clustering).
When publishing a message, I generate an incremental ID, then store the message in a string key with that ID, then publish the message ID.
On the receiving end, the first thing the subscriber does is attempt to add the message ID it just got from the publisher into a set of received messages for that group with sadd. If sadd returns 0, the message has already been grabbed by another instance, and it just returns. If it returns 1, the full message is pulled out of the string key and sent to the listener.
Of course, this relies on redis being single threaded, which I imagine will continue to be the case.
What you might be looking for is an AMQP protocol implementation,where you can have queue get custom exchanges,and implement a pub-sub model.
RabbitMQ - a popular amqp protocol implementation with lots of libraries
it also has node.js library
I'm just getting started with Windows Azure Service Bus (Topics & Queues) and I'm trying to implement a Competing-Consumers messaging pattern.
Essentially, I want to have a set of message Producers and a set of message Consumers. Once a message is produced, I want the first available Consumer to process the message. No other Consumers should get the message.
Is there a way to do this in Azure?
Simple. Just make two (or more) receivers that concurrently receive from a single queue and you're done. Any retrieved message goes to exactly one of those receivers since the cursor over the mesasage log is advanced as a message is taken. Competing consumers are an inherent capability of a networked queue so there's really nothing special needed.
If you need the opposite - each message goes to each consumer - you make a subscrioption per consumer which gives you an isolated cusor over the message log that can move independent of other receivers. For kicks, you can obviously also have competing consumers on a subscription.
Clemens
Topics are a feature of brokered messaging, but are a one-to-many "publish/subscribe" pattern. Queues are one-to-one message communication. So yes, it sounds like you should simply use queues. Also see http://msdn.microsoft.com/en-us/library/hh689723(VS.103).aspx.
You probably don't want Topics then, but rather Brokered Messaging.
You can emulate Topic-like functionality in Brokered Messaging by using the message's Label and/or Content Type properties along with the PeekLock receive mode.