I have a JMS Outbound Gateway which sends messages out via a request queue and receives messages in via a response queue. I would like to know what is the simplest way to apply throttling to the receive part of messages off the response queue. I have tried setting a Poller to the Outbound Gateway but, when I set it, the response messages are not consumed at all. Can a Poller be used in Outbound Gateways for the purpose of message consumption throttling? If so, how? If not, how can I best throttle message response consumption instead?
My stack is:
o.s.i:spring-integration-java-dsl:1.0.0.RC1
o.s.i:spring-integration-jms:4.0.4.RELEASE
My IntegrationgConfig.class:
#Configuration
#EnableIntegration
public class IntegrationConfig {
...
#Bean
public IntegrationFlow testFlow() {
return IntegrationFlows
.from("test.request.ch")
.handle(Jms.outboundGateway(connectionFactory)
.receiveTimeout(45000)
.requestDestination("REQUEST_QUEUE")
.replyDestination("RESPONSE_QUEUE")
.correlationKey("JMSCorrelationID"), e -> {
e.requiresReply(true);
e.poller(Pollers.fixedRate(1000).maxMessagesPerPoll(2)); // when this poller is set, response messages are not consumed at all...
})
.handle("testService",
"testMethod")
.channel("test.response.ch").get();
}
...
}
Cheers,
PM
Since you are going to fetch messages from the response queue the .poller() doesn't help you.
We need poller if our endpoint's input-channel (in your case test.request.ch) is a PollableChannel. See docs on the matter.
There .replyContainer() option on the Jms.outboundGateway for you. With that you can configure concurrentConsumers options to achieve better throughput on response queue.
Otherwise the JmsOutboundGateway creates MessageConsumer for each request message.
Related
I have to listen a queue using spring integration flow and intgeration sqs. Once message is received from queue it should trigger a integration flow. Below is the things which I am trying but everythings fine in but afater receiving test it is not triggering any Integration flow. Please let me know where I am doing wrong:
UPDATED as per comment from Artem
Adapter for SQS.
#Bean
public MessageProducerSupport sqsMessageDrivenChannelAdapter() {
SqsMessageDrivenChannelAdapter adapter = new SqsMessageDrivenChannelAdapter(amazonSQSAsync, "Main");
adapter.setOutputChannel(inputChannel());
adapter.setAutoStartup(true);
adapter.setMessageDeletionPolicy(SqsMessageDeletionPolicy.NEVER);
adapter.setMaxNumberOfMessages(10);
adapter.setVisibilityTimeout(5);
return adapter;
}
Queue configured:
#Bean
public MessageChannel inputChannel() {
return new DirectChannel();
}
Now the main integration flow trigger point:
#Bean
public IntegrationFlow inbound() {
return IntegrationFlows.from("inputChannel").transform(i -> "TEST_FLOW").get();
}
}
Appreciate any type of help.
The sqsMessageDrivenChannelAdapter() must be declared as a #Bean
The inbound() must be declared as a #Bean
This one fully does not make sense IntegrationFlows.from(MessageChannels.queue()). What is the point to start the flow from anonymous channel? Who and how is going to produce messages to that channel?
Make yourself familiar with different channels: https://docs.spring.io/spring-integration/docs/current/reference/html/core.html#channel-implementations
Pay attention that QueueChannel must be consumed via polling endpoint.
Right, there is a default poller auto-configured by Spring Boot, but it is based on a single thread in the TaskScheduler and has a polling period as 10 millis.
I wouldn't recommend to hand off SQS messages to the QueueChannel: when consumer fails, you lose the data. It is better to process those messages in the consumer thread.
Otherwise your intention is not clear in the provided code.
Can you, please, share with us what error you get or anything else?
You also can turn on DEBUG logging level for org.springframework.integration to see how your messages are processed.
We have a MQ request/reply pattern implementation
Here we use a IBM MQ Cluster host. Here the request/reply queues on both sides are linked to each other by the MQ cluster environment, as the queue managers of different systems within the cluster talks to each other.
Our Requestor code uses Spring JMS Integration - JmsOutboundGateway to send and receive message
The service provider is a Mainframe application which we have no control.
public class JmsOutboundGatewayConfig {
#Bean
public MessageChannel outboundRequestChannel() {
return new DirectChannel();
}
#Bean
public QueueChannel outboundResponseChannel() {
return new QueueChannel();
}
#Bean
#ServiceActivator(inputChannel = "outboundRequestChannel")
public JmsOutboundGateway jmsTestOutboundGateway(ConnectionFactory connectionFactory) {
JmsOutboundGateway gateway = new JmsOutboundGateway();
gateway.setConnectionFactory(connectionFactory);
gateway.setRequestDestinationName("REQUEST.ALIAS.CLUSTER.QUEUE");
gateway.setReplyDestinationName("REPLY.ALIAS.CLUSTER.QUEUE");
gateway.setReplyChannel(outboundResponseChannel());
gateway.setRequiresReply(true);
return gateway;
}
}
// Requestor - sendAndReceive code
outboundRequestChannel.send(new GenericMessage<>("payload"));
Message<?> response = outboundResponseChannel.receive(10000);
Issue:
The issue we are facing when we send message, the gateway code is also passing the replyTo = queue://REPLY.ALIAS.CLUSTER.QUEUE.
Now the mainframe program that consumes this message , it is forced to reply back to the replyTo queue. It is failing on mainframe side as this replyTo queue which we send is not part of their MQ Mgr/env.
I could not find a way to remove the replyTo when sending message. As JmsOutboundGateway set this replyTo using the "ReplyDestinationName" which I had configured.
Our requestor will need to set the "ReplyDestinationName" as we are listening to this Alias-cluster reply queue for reply back.
I looked at the Channel interceptor options, I could only Intercept the message to alter it, but no option to change the replyTo.
Is there way to alter the replyTo i.e replyTo and ReplyDestination different?
Is there anyway to remove/not-set the replyTo when sending message to request queue?
Just wondering how to get this working for such MQ cluster environment where the replyTo queue will have to kept what the mainframe consumer service want, that is different to the replyDestination queue which we use.
Considering that the replyTo is used by the mainframe service to reply back. If it is not passed the mainframe service will use its own reply queue which is linked to our reply-cluster-alias queue.
Any inputs appreciated?
Thanks
Saishm
Further clarification:
The cluster mq env we have, Our spring jms outbound gateway is writing request to - "REQUEST.ALIAS.CLUSTER.QUEUE" & listening to the reply on "REPLY.ALIAS.CLUSTER.QUEUE"
So the jmsOutboundGateway sets the replyTo=REPLY.ALIAS.CLUSTER.QUEUE
Now the mainframe service on the other side is reading the message from "REQUEST.LOCAL.QUEUE". In the cluster env the "REQUEST.ALIAS.CLUSTER.QUEUE" ands its QMGR are linked to "REQUEST.LOCAL.QUEUE" and its QMGR, this is all managed within the cluster MQ env.
The mainframe service when consuming the request, sees that the incoming message had a replyTo and tries to send the response to this replyTo.
The issue is mainframe was supposed to reply to "REPLY.LOCAL.QUEUE" which is linked to REPLY.ALIAS.CLUSTER.QUEUE
If there is no replyTo it would have send the reply to "REPLY.LOCAL.QUEUE".
Now from the jmsOutBoundGateway I dont have any options to remove replyTo when sending mEssage or edit it to "REPLY.LOCAL.QUEUE" and keep listening to the response/reply of the request on "REPLY.ALIAS.CLUSTER.QUEUE"
Doesn't look like a JmsOutboundGateway will fit your IBM MQ cluster configuration and requirements. You just cannot use that replyTo feature since we cannot bypass JMS protocol over here.
Consider to use a pair of JmsSendingMessageHandler and JmsMessageDrivenEndpoint components, respectively. Your JmsSendingMessageHandler will just send a JMS message to the REQUEST.ALIAS.CLUSTER.QUEUE and forget. Only what you need is to supply a JmsHeaders.CORRELATION_ID. The JmsHeaderMapper will populate a jmsMessage.setJMSCorrelationID() property - the same way as JmsOutboundGateway does that by default. In this case your mainframe service is free to use its own reply queue and I guess it will supply our correlationId correctly.
A JmsMessageDrivenEndpoint has to be subscribed to the REPLY.ALIAS.CLUSTER.QUEUE and you do a correlation between request and reply yourself. For example a Map of correlationId and Future to fulfill when reply comes back.
I am using
MessageProducerSupport messageProducer =
Jms.messageDriverChannelAdapter(jmsConnectionFactory, TransactedMessageListenerContainer.class)
.destination(queue)
.get();
to consume messages from ActiveMQ queue.
This is first part of my IntegrationFlow and then multiple stages occur (transform, route, handle..) within transaction
It is there to handle messages from upstream
In order to get the ACK from Spring integration pipeline I used Jms.inboundGateway(jmsConnectionFactory, TransactedMessageListenerContainer.class) which doesn't break existing flow and everything works
When I set replyTo header of upstream message, I would assume Spring Integration would send the object of the last stage of IntegrationFlow which was successful back to replyTo queue
Is my approach correct?
Is it possible to achieve such use-case?
Yes, that's correct and should work by its (Messaging Gateway) premise.
The Jms.inboundGateway() is based on the ChannelPublishingJmsMessageListener with the expectReply = true and there is a code:
private Destination getReplyDestination(javax.jms.Message request, Session session) throws JMSException {
Destination replyTo = request.getJMSReplyTo();
....
return replyTo;
}
to obtain a replyTo from the request.
Everything that works well, if your last MessageHandler in the flow is a AbstractReplyProducingMessageHandler and really returns something to be produced to the replyChannel from headers.
If you aren't sure in your case, so share the end of your flow and the place where would you like to send a reply.
I need solve this scenario. I have two amqp consumer set to fetch one message.
#Bean
public IntegrationFlow jmsPrimaryFlow() {
return IntegrationFlows.from(
Amqp.inboundGateway(
taskManager().getPrimaryMessageListenerContainer()).errorChannel(errorChannel())
)
.channel(taskChannel())
.get();
}
#Bean
public IntegrationFlow jmsSecondaryFlow() {
return IntegrationFlows.from(
Amqp.inboundGateway(
taskManager().getSecondaryMessageListenerContainer()).errorChannel(errorChannel())
.autoStartup(false)
)
.channel(taskChannel())
.get();
}
taskChannel is queuechannel but allow only one message consume at once so no parallel processing.
How can I reject one message after some timeout if another message took too long to proceed.
so this message will returned back to queue to proceed by another node? Just I mean that those two consumer prefetch two messages but only one can be processed at once so how release second prefetched message if the first one took to long to proceed.
Your question is not clear. You could set a capacity limit (say 1) on the queue channel and set a sendTimeout on the gateway. Then, if the queue is full, attempts to add messages will fail after the timeout. However, using a queue channel in this scenario is dangerous - you can lose messages if the server fails because messages are ack'd as soon as they are deposited in the queue.
If you use a RendezvousChannel instead, the producer will block waiting for the consumer to receive the message.
But bear in mind, even this single message can be lost if the server crashes after the handoff.
I have a Spring Integration flow which sends a message out via a JMS Outbound Gateway which is configured to have a receive timeout of 45 seconds. I am trying to test the receive timeout period by sending a message out in a setup where the message is never consumed on the other side (therefore a response doesn't come back). However, when I run the test, the message is placed in the outbound queue but, the Outbound Gateway's receive timeout never occurs (after 45 seconds). Any ideas what reasons there could be for this happening (not happening)?
My stack is:
o.s.i:spring-integration-java-dsl:1.0.0.M3
o.s.i:spring-integration-jms:4.0.4.RELEASE
My IntegrationgConfig.class:
#Configuration
#EnableIntegration
public class IntegrationConfig {
...
#Bean
public IntegrationFlow testFlow() {
return IntegrationFlows
.from("test.request.ch")
.handle(Jms.outboundGateway(connectionFactory)
.receiveTimeout(45000)
.requestDestination("REQUEST_QUEUE")
.replyDestination("RESPONSE_QUEUE")
.correlationKey("JMSCorrelationID"))
.handle("testService",
"testMethod")
.channel("test.response.ch").get();
}
...
}
In terms of JMS configuration, the connection factory used is a standard CachingConnectionFactory which targets an MQConnectionFactory.
Thanks in advance for any help on this.
PM
--- UPDATE ---
I have turned on debugging and I can see that when the timeout occurs the following message is logged:
AbstractReplyProducingMessageHandler - handler 'org.springframework.integration.jms.JmsOutboundGateway#0' produced no reply for request Message: [Payload byte[835]][...]
Just need to find out how to capture this event in the flow?
--- UPDATE 2 ---
The message being sent out has an ERROR_CHANNEL header set on it to which I would expect the timeout exception to be routed to - but this routing does not happen?
Is it possible that the CachingConnectionFactory is handling the exception and not passing it back to the flow?
To make it working you need to add the second Lambda to the .handle() with Jms:
.handle(Jms.outboundGateway(connectionFactory)
.receiveTimeout(45000)
.requestDestination("REQUEST_QUEUE")
.replyDestination("RESPONSE_QUEUE")
.correlationKey("JMSCorrelationID"),
e -> e.requiresReply(true))
By default AbstractReplyProducingMessageHandler doesn't require reply even if receiveTimeout is exhausted, and we see that by logs shown by you.
However, I see that we should revise all MessageHandlerSpecs, because XML support changes the requires-reply for some components to true by default.
Feel free to raise a JIRA issue on the matter and we'll address it soon, because the GA release for Java DSL is planned over a week or two: https://spring.io/blog/2014/10/31/spring-integration-java-dsl-1-0-rc1-released
Thank you for the attention to this stuff!