I have a web service that ingests objects, sends a notification over AMQP, and returns a JSON response to the requester. Each request is performed on a single thread and I am trying to implement publisher confirms and I am struggling on how I should set it up. I have it working but I don't like the way I am doing it.
The way I am doing it is:
Put some headers on the message
Have a publish-subscribe-channel with 2 subscribers
Subscriber 1) creates a blocking queue so it is ready
and sends the message over amqp
Subscriber 2) begins pulling for 5 seconds on that queue until it gets its confirm
The amqp:outbound-channel-adapter sends its publisher confirms to a service activator
The publisherConfirmReceiver receives the confirm and puts it in the blocking queue causing the subscriber 2's pulling to complete and return the result of the confirm.
This technique does work properly but I don't like making the assumption that the chain is going to receive the message before the waitForPublisherConfirm Service Activator from the publish subscribe channel. In this case order matters regarding which component receives the message first.
If the waitForPublisherConfirms service activator receives the message first it will just block the thread for 5 seconds, then allow the chain to send the message via the amqp:outbound-channel-adapter.
I tried putting the waitForPublisherConfirms after the amqp:outbound-channel-adapter but since the outbound-channel-adapter doesn't "return" anything so the service activator never gets called after it in the chain.
I feel like there should be a better way of doing this. My goal is to wait for publisher confirms (or a timeout which I cannot find support for in spring's publisher confirms) before sending a response to the requester.
Could you help me shape the solution a little better or let me know if it is OK to rely on the fact that the first subscriber to the publish-subscribe-channel will always receive a message first.
Sorry this one is so long.
Some configuration
<int:header-enricher input-channel="addHeaders" output-channel="metadataIngestNotifications">
<int:header name="routingKey" ref="routingKeyResolver" method="resolveReoutingKey"/>
<int:header name="notificationId" expression="payload.id" />
</int:header-enricher>
<int:chain input-channel="metadataIngestNotifications" output-channel="nullChannel" >
<int:service-activator id="addPublisherConfirmQueue"
requires-reply="false"
ref="publisherConfirmService"
method="addPublisherConfirmQueue" />
<int:object-to-json-transformer id="transformObjectToJson" />
<int-amqp:outbound-channel-adapter id="amqpOutboundChannelAdapter"
amqp-template="rabbitTemplate"
exchange-name="${productNotificationExchange}"
confirm-ack-channel="publisherConfirms"
confirm-nack-channel="publisherConfirms"
mapped-request-headers="*"
routing-key-expression="headers.routingKey"
confirm-correlation-expression="headers.notificationId" />
</int:chain>
<int:service-activator id="waitForPublisherConfirm"
input-channel="metadataIngestNotifications"
output-channel="publisherConfirmed"
requires-reply="true"
ref="publisherConfirmService"
method="waitForPublisherConfirm" />
<int:service-activator id="publisherConfirmReceiver"
ref="publisherConfirmService"
method="receivePublisherConfirm"
input-channel="publisherConfirms"
output-channel="nullChannel" />
Class
public class PublisherConfirmService {
private final Map<String, BlockingQueue<Boolean>> suspenders = new HashMap<>();
public Message addPublisherConfirmQueue(#Header("notificationId") String id, Message m){
LogManager.getLogger(this.getClass()).info("Adding publisher confirm queue.");
BlockingQueue<Boolean> bq = new LinkedBlockingQueue<>();
suspenders.put(id, bq);
return m;
}
public boolean waitForPublisherConfirm(#Header("notificationId") String id) {
LogManager.getLogger(this.getClass()).info("Waiting for publisher confirms for Notification: " + id);
BlockingQueue<Boolean> bq = suspenders.get(id);
try {
Boolean result = bq.poll(5, TimeUnit.SECONDS);
if(result == null){
LogManager.getLogger(this.getClass()).error("The broker took too long to return a publisher confirm. NotificationId: " + id);
return false;
}else if(!result){
LogManager.getLogger(this.getClass()).error("The publisher confirm indicated that the message was not confirmed. NotificationId: " + id);
return false;
}
} catch (InterruptedException ex) {
LogManager.getLogger(this.getClass()).error("Something went wrong polling for the publisher confirm for notificationId: " + id, ex);
return false;
}finally{
suspenders.remove(id);
}
return true;
}
public void receivePublisherConfirm(String id, #Header(AmqpHeaders.PUBLISH_CONFIRM) boolean confirmed){
LogManager.getLogger(this.getClass()).info("Received publisher confirm for Notification: " + id);
if (suspenders.containsKey(id)){
BlockingQueue<Boolean> bq = suspenders.get(id);
bq.add(confirmed);
}
}
}
How about to take a look to the Aggregator solution for the same purpose?
The <recipient-list-router> to send message to the aggregator's input-channel and the second channel for the <int-amqp:outbound-channel-adapter>.
The confirm-ack-channel must be something which brings the message to the same aggregator after some transformation, e.g. a proper extraction for the correlationKey and so on.
Related
I created a simple Azure Service bus (Queue) and a client that is sending message to service bus. Using below code to send message:
using Microsoft.Azure.ServiceBus;
using Microsoft.Extensions.Configuration;
public async Task SendMessageAsync<T>(T message, string queueName)
{
try
{
var queueClient = new QueueClient(_config.GetConnectionString("AzureServiceBus"), queueName);
string messageBody = JsonSerializer.Serialize(message);
var byteMessage = new Message(Encoding.UTF8.GetBytes(messageBody));
queueClient.SendAsync(byteMessage);
Console.WriteLine((message as Employee).FirstName);
}
catch (Exception ex)
{
var c = ex;
}
}
Sending message using:
using SenderApp;
Console.WriteLine("Hello, World!");
QueueService service = new QueueService();
for (int i = 0; i < 100; i++)
{
Employee e = new Employee();
e.FirstName = "1 " + i.ToString();
e.LastName = "2 " + i.ToString();
service.SendMessageAsync<Employee>(e, "employeequeue");
}
When I try to see active messages, There is nothing in the queue:
However I do see some traffic. But the number of message I sent (over 100) is not equal to number of incoming request show (62) at the bottom of the image. I am not sure what is happening to my messages? This defeats the purpose of the queue.
Please guide me why I am not seeing any messages. What is the best way to handle this ?
I am using following nuget packages:
<PackageReference Include="Microsoft.Azure.ServiceBus" Version="5.2.0" />
<PackageReference Include="Microsoft.Extensions.Configuration.Abstractions" Version="6.0.0" />
<PackageReference Include="Microsoft.Extensions.Hosting" Version="6.0.1" />
A message sent to an Azure Service Bus queue will be delivered to the queue unless operation is failing. In that case, an exception will be thrown. Check the following:
Exception handling doesn't swollow exceptions
Await asynchronous send operations to ensure messages are dispatched
Namespace/queue used for sending is what you use to receive
There are no competing consumers, actively receiving messages from the queue.
Validate TCP ports needed for AMQP are not blocked. If those ports are blocked, you could configure your client to use WebSockets.
So I still dont know what caused this issue. But I realized Microsoft.Azure.ServiceBus package was deprecated and later I started using Azure.Messaging.ServiceBus package to connect to service bus and things started to work.
I used following code to send message to queue:
string connectionString = "Endpoint=sb://test.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=f3f+qMYTyVwE18YNl5J6ygJFi30v6J/Smph5HZvyQyE=";
string queueName = "employeequeue";
// since ServiceBusClient implements IAsyncDisposable we create it with "await using"
await using var client = new ServiceBusClient(connectionString);
// create the sender
ServiceBusSender sender = client.CreateSender(queueName);
// create a message that we can send. UTF-8 encoding is used when providing a string.
ServiceBusMessage message = new ServiceBusMessage("Hello world! " + id);
// send the message
await sender.SendMessageAsync(message);
return "Sent";
Used following code to receive message:
string queueName = "employeequeue";
// since ServiceBusClient implements IAsyncDisposable we create it with "await using"
await using var client = new ServiceBusClient(connectionString);
// create a receiver that we can use to receive and settle the message
ServiceBusReceiver receiver = client.CreateReceiver(queueName);
// the received message is a different type as it contains some service set properties
ServiceBusReceivedMessage receivedMessage = await receiver.ReceiveMessageAsync();
string body = receivedMessage.Body.ToString();
// complete the message, thereby deleting it from the service
await receiver.CompleteMessageAsync(receivedMessage);
More info is available # https://github.com/Azure/azure-sdk-for-net/blob/Azure.Messaging.ServiceBus_7.7.0/sdk/servicebus/Azure.Messaging.ServiceBus/README.md
I am developing an API in spring-integration using DSL, this how it works
JDBC Polling Adapter initiates the flow and gets some data from tables and send it to DefaultRequestChannel, from here the message is handled/flowing thru various channels.
Now I am trying to
1. send a email, if any errors (e.g connectivity issue, bad record found while polling the data) occurred/detected in my error channel.
After sending email to my support group, I want to suspend my flow for 15 mins and then resume automatically.
I tried creating a sendEmailChannel (recipient of my errorChannel), it doesn't work for me. So just created a transformer method like below
this code is running fine, but is it a good practice?
#
#Transformer(inputChannel="errorChannel", outputChannel="suspendChannel")
public Message<?> errorChannelHandler(ErrorMessage errorMessage) throws RuntimeException, MessagingException, InterruptedException {
Exception exception = (Exception) errorMessage.getPayload();
String errorMsg = errorMessage.toString();
String subject = "API issue";
if (exception instanceof RuntimeException) {
errorMsg = "Run time exception";
subject = "Critical Alert";
}
if (exception instanceof JsonParseException) {
errorMsg = ....;
subject = .....;
}
MimeMessage message = sender.createMimeMessage();
MimeMessageHelper helper = new MimeMessageHelper(message);
helper.setFrom(senderEmail);
helper.setTo(receiverEmail);
helper.setText(errorMsg);
helper.setSubject(subject);
sender.send(message);
kafkaProducerSwitch.isKafkaDown());
return MessageBuilder.withPayload(exception.getMessage())
.build();
}
I am looking for some better way of handling the above logic.
And also any suggestions to suspend my flow for few mins.
You definitely can use a mail sending channel adapter from Spring Integration box to send those messages from the error channel: https://docs.spring.io/spring-integration/docs/5.1.5.RELEASE/reference/html/#mail-outbound. The Java DSL variant is like this:
.handle(Mail.outboundAdapter("gmail")
.port(smtpServer.getPort())
.credentials("user", "pw")
.protocol("smtp")))
The suspend can be done via CompoundTriggerAdvice extension, when you check the some AtimocBoolean bean for the state to activate one or another trigger in the beforeReceive() implementation. Such a AtimocBoolean can change its state in one more subscriber to that errorChannel because this one is a PublishSubscribeChannel by default. Don't forget to bring the state back to normal after that you return a false from the beforeReceive(). Just because that is enough to mark your system as normal at this moment since it is is going to work only after 15 mins.
I have a spring integration flow which produces messages that should be kept around waiting for an appropriate consumer to come along and consume them.
#Bean
public IntegrationFlow messagesPerCustomerFlow() {
return IntegrationFlows.
from(WebFlux.inboundChannelAdapter("/messages/{customer}")
.requestMapping(r -> r
.methods(HttpMethod.POST)
)
.requestPayloadType(JsonNode.class)
.headerExpression("customer", "#pathVariables.customer")
)
.channel(messagesPerCustomerQueue())
.get();
}
#Bean(name = PollerMetadata.DEFAULT_POLLER)
public PollerSpec poller() {
return Pollers.fixedRate(100);
}
#Bean
public QueueChannel messagesPerCustomerQueue() {
return MessageChannels.queue()
.get();
}
The messages in the queue should be delivered as server-sent events via http as shown below.
The PublisherSubscription is just a holder for the Publisher and the IntegrationFlowRegistration, the latter is used to destroy the dynamically created flow when it is no longer needed (note that the incoming message for the GET has no content, which is not handled properly ATM by the Webflux integration, hence a small workaround is necessary to get access to the path variable shoved to the customer header):
#Bean
public IntegrationFlow eventMessagesPerCustomer() {
return IntegrationFlows
.from(WebFlux.inboundGateway("/events/{customer}")
.requestMapping(m -> m.produces(TEXT_EVENT_STREAM_VALUE))
.headerExpression("customer", "#pathVariables.customer")
.payloadExpression("''") // neeeded to make handle((p,h) work
)
.log()
.handle((p, h) -> {
String customer = h.get("customer").toString();
PublisherSubscription<JsonNode> publisherSubscription =
subscribeToMessagesPerCustomer(customer);
return Flux.from(publisherSubscription.getPublisher())
.map(Message::getPayload)
.doFinally(signalType ->
publisherSubscription.unsubscribe());
})
.get();
}
The above request for server-sent events dynamically registers a flow which subscribes to the queue channel on demand with a selective consumer realized by a filter with throwExceptionOnRejection(true). Following the spec for Message Handler chain that should ensure that the message is offered to all consumers until one accepts it.
public PublisherSubscription<JsonNode> subscribeToMessagesPerCustomer(String customer) {
IntegrationFlowBuilder flow = IntegrationFlows.from(messagesPerCustomerQueue())
.filter("headers.customer=='" + customer + "'",
filterEndpointSpec -> filterEndpointSpec.throwExceptionOnRejection(true));
Publisher<Message<JsonNode>> messagePublisher = flow.toReactivePublisher();
IntegrationFlowRegistration registration = integrationFlowContext.registration(flow.get())
.register();
return new PublisherSubscription<>(messagePublisher, registration);
}
This construct works in principle, but with the following issues:
Messages sent to the queue while there are no subscribers at all lead to a MessageDeliveryException: Dispatcher has no subscribers for channel 'application.messagesPerCustomerQueue'
Messages sent to the queue while no matching subscriber is present yet lead to an AggregateMessageDeliveryException: All attempts to deliver Message to MessageHandlers failed.
What I want is that the message remains in the queue and is repeatedly offered to all subscribers until it is either consumed or expires (a proper selective consumer). How can I do that?
note that the incoming message for the GET has no content, which is not handled properly ATM by the Webflux integration
I don't understand this concern.
The WebFluxInboundEndpoint works with this algorithm:
if (isReadable(request)) {
...
else {
return (Mono<T>) Mono.just(exchange.getRequest().getQueryParams());
}
Where GET method really goes to the else branch. And the payload of the message to send is a MultiValueMap. And also we recently fixed with you the problem for the POST, which is released already as well in version 5.0.5: https://jira.spring.io/browse/INT-4462
Dispatcher has no subscribers
Can't happen on the QueueChannel in principle. There is no any dispatcher on there at all. It is just queue and sender offers message to be stored. You are missing something else to share with us. But let's call things with its own names: the messagesPerCustomerQueue is not a QueueChannel in your application.
UPDATE
Regarding:
What I want is that the message remains in the queue and is repeatedly offered to all subscribers until it is either consumed or expires (a proper selective consumer)
Only what we see is a PollableJmsChannel based on the embedded ActiveMQ to honor TTL for messages. As a consumer of this queue you should have a PublishSubscribeChannel with the setMinSubscribers(1) to make MessagingTemplate to throw a MessageDeliveryException when there is no subscribers yet. This way a JMS transaction will be rolled back and message will return to the queue for the next polling cycle.
The problem with in-memory QueueChannel that there is no transactional redelivery and message once polled from that queue is going to be lost.
Another option is similar to JMS (transactional) is a JdbcChannelMessageStore for the QueueChannel. Although this way we don't have a TTL functionality...
We have an inbound channel adapter that receives notifications of an event. The complexity of the consumer's criteria restrict our ability to use a simple routing key to distribute the messages, so the application uses a splitter to send that message to interested subscriber's queues via a direct exchange.
We want to use publisher confirms on our outbound channel adapter the ensure delivery to the client queues. We want to wait for the publisher confirm to ack the original message, and if a publisher confirm fails to be received or if the ack==false we want to nack the original message that came from the inbound channel adapter.
I assume this will be done in the confirm-callback from the Rabbit Template but I am not sure how to accomplish this. (Or if it is even possible)
<rabbit:connection-factory id="rabbitConnectionFactory"
host="${amqpHost}"
username="${amqpUsername}"
password="${amqpPassword}"
virtual-host="${amqpVirtualHost}"
publisher-confirms="true" />
<rabbit:template id="rabbitTemplate"
connection-factory="rabbitConnectionFactory"
confirm-callback="PublisherConfirms" />
<int-amqp:inbound-channel-adapter channel="notificationsFromRabbit"
queue-names="#{'${productNotificationQueue}' + '${queueSuffix}'}"
connection-factory="rabbitConnectionFactory"
mapped-request-headers="*"
message-converter="productNotificationMessageConverter" />
<int:chain input-channel="notificationsFromRabbit" output-channel="notificationsToClients">
<int:service-activator ref="NotificationRouter"
method="addRecipientsHeaders" />
<int:splitter ref="NotificationRouter"
method="groupMessages" />
<int:object-to-json-transformer />
</int:chain>
<int-amqp:outbound-channel-adapter channel="notificationsToClients"
amqp-template="rabbitTemplate"
exchange-name="${servicesClientsExchange}"
routing-key=""
mapped-request-headers="*" />
At the moment we are acking the messages in the groupMessages method by passing the Channel and Delivery tag as paramters. But, if the broker never sends a return or returns with ack=false then it is too late to nack the message from the inbound channel adapter.
Am I going to need a bean that keeps a Map<Channel, Long> of the channel and delivery tags to access in the confirm-callback or is there some other way?
Is the channel from the inbound channel adapter going to be closed by the time I receive a publisher confirm?
As long as you suspend the consumer thread until all the acks/nacks have been received, you can do what you want.
If you make notificationsFromRabbit a publish-subscribe channel you can add another subscriber (service-activator) where you suspend the thread; wait for all the acks/nacks and take the action you desire.
EDIT:
You can also use Spring Integration to manage the acks for you and it will emit them as messages from the outbound adapter (rather than using a callback yourself).
EDIT2:
You could then use the splitter's sequence size/sequence number headers in your correlation data, enabling the release of the consumer when all the acks are received.
EDIT3:
Something like this should work...
On the outbound adapter, set confirm-correlation-expression="#this" (the whole outbound message).
Class with two methods
private final Map<String, BlockingQueue<Boolean> suspenders;
public void suspend(Message<?> original) {
BlockingQueue<Boolean> bq = new LinkedBlockingQueue();
String key = someKeyFromOriginal(original);
suspenders.put(key, bq);
Boolean result = bq.poll(// some timeout);
try {
if (result == null) {
// timed out
}
else if (!result) {
// throw some exception to nack the message
}
}
finally {
suspenders.remove(key);
}
}
public void ackNack(Message<Message<?>> ackNak) {
Message<?> theOutbound = ackNak.payload;
BlockingQueue<Boolean> bq = suspenders.get(someKeyFromOriginal(theOutbound));
if (bq == null) // late ack/nack; ignore
else {
// check the ack/nack header
// if nack, bq.put(false)
// else, use another map field, to
// keep track of ack count Vs sequenceSize header in
// theOutbound; when all acks received, bq.put(true);
}
}
Suspend the consumer thread in the first method; route the acks/nacks from the outbound adapter to the second method.
Caveat: This is not tested, just off the top of my head; but it should be pretty close.
My requirements are stated below:
I have to develop a wrapper service on top a queue,so i was just going through some message Queue like (ActiveMQ,Apollo,Kafka). But decided to proceed with ActiveMQ to match our usecases.Now the requirement are as follows:
1) A restful api through which different publisher will publish to queue,based on clientId queue will be selected.
2) Consumer will consume message through restful api and will consume message in batches. say consumer as for something like give me 10 message from queue.
Now the service should provide 10 message if there is 10 message or if message number is less or zero it will send accordingly. After receiving the message the client will process with the message and send back acknowledgement through different res-full uri. upon receiving that acknowledgement,the MQService should commit or rollback message from the queue.
In order to this in the MQService layer, i have used a cached,where im keeping the JMS connection and session object till acknowledgemnt is received or ttl expire.
In-order to retrieve message in batches and send back to client, i have created a multi-threaded consumer,so that for 5 batch message request,the service layer will create 5 thread each having different connection and session object( as stated in ActiveMQ multiple consumer http://activemq.apache.org/multiple-consumers-on-a-queue.html)
Basic use-case:
MQ(BROKER)[A] --> Wrapper(MQService)[B]-->Client [C]
Note:[B] is a restfull service having JMS consumer implemented in it.It keeps the connection and session object in cache.
[C] request to [B] to give 3 message
[B] must fetch 3 message if available in queue,wrap it in batchmsgFormat and send it to [C]
[C] process the message and send acknowledgemnt suces/failed to [B] through /send-ack uri.
Upon receiving Ack from [C], [B] will commit the Jms session and close the session and connection object. Also it will evict those from the cache.
The above work-flow is working fine with single message fetching.
But the queue hungs up on JMS MesageConsumer.receive() when try to fetch message with mutilple consumer using multithreading. ...
Here the JMS Consumer code in MQService layer:
----------------------------------------------
public BatchMessageFormat getConsumeMsg(final String clientId, final Integer batchSize) throws Exception {
BatchMessageFormat batchmsgFormat = new BatchMessageFormat();
List<MessageFormat> msgdetails = new ArrayList<MessageFormat>();
List<Future<MessageFormat>> futuremsgdetails = new ArrayList<Future<MessageFormat>>();
if (batchSize != null) {
Integer msgCount = getMsgCount(clientId, batchSize);
for (int batchconnect = 0; batchconnect <msgCount; batchconnect++) {
FutureTask<MessageFormat> task = new FutureTask<MessageFormat>(new Callable<MessageFormat>() {
#Override
public MessageFormat call() throws Exception {
MessageFormat msg=consumeBatchMsg(clientId,batchSize);
return msg;
}
});
futuremsgdetails.add(task);
Thread t = new Thread(task);
t.start();
}
for(Future<MessageFormat> msg:futuremsgdetails){
msgdetails.add(msg.get());
}
batchmsgFormat.setMsgDetails(msgdetails);
return batchmsgFormat
}
Message fetching:
private MessageFormat consumeBatchMsg(String clientId, Integer batchSize) throws JMSException, IOException{
MessageFormat msgFormat= new MessageFormat();
Connection qC = ConnectionUtil.getConnection();
qC.start();
Session session = qC.createSession(true, -1);
Destination destination = createQueue(clientId, session);
MessageConsumer consumer = session.createConsumer(destination);
Message message = consumer.receive(2000);
if (message!=null || message instanceof TextMessage) {
TextMessage textMessage = (TextMessage) message;
msgFormat.setMessageID(textMessage.getJMSMessageID());
msgFormat.setMessage(textMessage.getText());
CacheObject cacheValue = new CacheObject();
cacheValue.setConnection(qC);
cacheValue.setSession(session);
cacheValue.setJmsQueue(destination);
MQCache.instance().add(textMessage.getJMSMessageID(),cacheValue);
}
consumer.close();
return msgFormat;
}
Acknowledgement and session closing:
public String getACK(String clientId,String msgId,String ack)throws JMSException{
if (MQCache.instance().get(msgId) != null) {
Connection connection = MQCache.instance().get(msgId).getConnection();
Session session = MQCache.instance().get(msgId).getSession();
Destination destination = MQCache.instance().get(msgId).getJmsQueue();
MessageConsumer consumer = session.createConsumer(destination);
if (ack.equalsIgnoreCase("SUCCESS")) {
session.commit();
} else {
session.rollback();
}
session.close();
connection.close();
MQCache.instance().evictCache(msgId);
return "Accepted";
} else {
return "Rejected";
}
}
Does anyone worked on similar scenario or can you pls throw some light? Is there any other way to implement this batch mesage fetching as well as client failure handling?
Try after setting the prefetch limit to 0 as below:
ConnectionFactory connectionFactory
= new ActiveMQConnectionFactory("tcp://localhost:61616?jms.prefetchPolicy.queuePrefetch=0");
I'll give a few pointers to help to code this logic better.
I'm assuming you are using pure JMS 1.1 as much as possible. Ensure that you have one place where you get the connection from the pool or create a connection. You need not do that inside a thread. You can do that outside. Sessions must be created inside a thread and shouldn't be shared. This will impact the logic in the function consumeBatchMsg().
Secondly, its simpler to use one thread to consume all the messages of the given batchSize. I see that you are using transacted session. So you can do one commit after getting all the messages of the batchSize.
If you really want to take the complicated route of having multiple consumers on a queue (probably little better performance), you can using CountDownLatch or CyclicBarrier of Java and set it to batchSize to trigger. Once all the threads have received the messages, it can commit and close the sessions in the respective threads. Never let the session instance go out of the context of the thread that created it.