I want to create a simple IntegrationFlow with Spring integration, and I am having difficulties.
I want to create an integration flow that takes messages from multiple queues in Rabbit Mq and posts the messages to different Rest endpoints.
i want to know if i can parallelize this.
i have two scenarios that i want to check the feasibility :
the first one i want to create a thread for every RabbitMq Queue that
would listen and execute the flow after receiving a message :
Scenario 1
the second scenario : in this scenario i want to create a dynamic number of threads for every queue , the number of threads goes up or down depending on the number of messages.
Scenario 2
HttpHeaders headers = new HttpHeaders();
headers.setContentType(MediaType.APPLICATION_JSON);
RestTemplate restTemplate = new RestTemplate();
SimpleMessageListenerContainer container = new SimpleMessageListenerContainer(connectionFactory);
container.setQueueNames(BOUTIQUE_QUEUE_NAME);
container.setAcknowledgeMode(AcknowledgeMode.AUTO);
return IntegrationFlows.from(Amqp.inboundAdapter(container)) /* Get Message from RabbitMQ */
.handle(msg ->
{
String msgString = new String((byte[]) msg.getPayload(), StandardCharsets.UTF_8);
HttpEntity<String> requestBody = new HttpEntity<String>(msgString, headers);
restTemplate.postForObject(ENDPOINT_LOCAL_URL, requestBody, String.class);
System.out.println(msgString);
})
.get();
}
For the first scenario, simply configure a inbound adapter for each and set the output channel to a common channel for the downstream flow.
For the second scenario, simply set the concurrentConsumers and maxConcurrentConsumers on the listener container and it will scale up/down the threads as needed.
See the Spring AMQP Documentation.
Related
I'm writing a Spring Integration application which receives messages from an input system, transforms them and sends them to an output system. The connection to the output system is not always available. The messages from the input system can come at any moment. In case if they are coming when the output system is not available, they shouldn't be lost and send eventually when the output system is available. So I store the messages from the input system in a QueueChannel:
#Configuration
class TcInFlowConfiguration {
#Bean
fun tcInFlow(
#Qualifier(TC_MESSAGE_LISTENER) listener: MessageProducerSupport,
#Qualifier(TC_MESSAGE_CHANNEL) messageChannel: MessageChannel
): IntegrationFlow {
return IntegrationFlow
.from(listener)
.transform { msg: ByteArray -> RamsesTcFrame.deserialize(msg) }
.channel(messageChannel)
.get()
}
#Bean
#Qualifier(TC_MESSAGE_CHANNEL)
fun tcMessageChannel(): MessageChannel {
return MessageChannels.queue().get()
}
The app receives an API call to open/close the connection to the output system, so I create and remove the output integration flow programmatically via IntegrationFlowContext:
val outFlow = IntegrationFlow
.from(TC_MESSAGE_CHANNEL)
.handle(createMessageSender())
.get()
integrationFlowContext.registration(outFlow).register()
When the messages are polled from the queue to be processed by the outFlow, the default Spring task executor is used (I see "scheduling-1" as a thread name in logs). The problem is that I have multiple independent integration flows in the app with the multiple queue channels, so they all got mixed up by being processed by the same task executor. What I want is to process each flow in its own dedicated thread, so the flows won't block each other. How can I achieve this?
Spring Boot v3.0.2, Spring v6.0.4
I tried setting a task scheduler for my QueueChannel:
val queueChannel = MessageChannels.queue().get()
queueChannel.setTaskScheduler(taskScheduler)
It didn't have any effect, taskScheduler seems to be simply not used by QueueChannel implementation.
I tried using ExecutorChannel instead of QueueChannel which supports setting a custom Executor. Unfortunately, ExecutorChannel doesn't buffer messages in memory, so if there are no subscribers to the channel the messages are lost.
Finally, I tried defining a poller in the outFlow to poll the messages from the QueueChannel:
IntegrationFlow
.from(TC_MESSAGE_CHANNEL)
.handle(createMessageSender()) { e -> e.poller(Pollers.fixedDelay(10).taskExecutor(taskExecutor)) }
.get()
This didn't work either. After the connection to the output system is closed and the outFlow is removed, the intermediate channel created by the poller remains in Spring context. So when the new message arrives in QueueChannel it goes to that intermediate channel which is a subscribable channel without subscribers, so the message is lost.
That's correct. The QueueChannel is just a buffer for messages. It really only matters how you consume messages from there. And the common way is to use a PollingConsumer like you do with that e.poller(). It is also correct to configure a taskExecutor() if you don't like to have your messages to be consumed by a TaskScheduler's thread. Not sure what you talk about an "intermediate channel" since it doesn't look like you have one declared in your outFlow. You have that .from(TC_MESSAGE_CHANNEL) and then immediately a handle() with a proper poller. So, no any extra channel in between or after. Unless you do something else in your createMessageSender().
I would suggest do not have a dynamic flow, but rather singleton one for that output system. The QueueChannel can be configured for a persistent message store and poller can be transactional. So, if no connection to a target system, the transaction is going to be rolled back and message remains in the store: https://docs.spring.io/spring-integration/docs/current/reference/html/system-management.html#message-store.
You also can just stop() a polling consumer for that handle() when no connection, so no messages are polled from the queue at that moment.
I have to listen a queue using spring integration flow and intgeration sqs. Once message is received from queue it should trigger a integration flow. Below is the things which I am trying but everythings fine in but afater receiving test it is not triggering any Integration flow. Please let me know where I am doing wrong:
UPDATED as per comment from Artem
Adapter for SQS.
#Bean
public MessageProducerSupport sqsMessageDrivenChannelAdapter() {
SqsMessageDrivenChannelAdapter adapter = new SqsMessageDrivenChannelAdapter(amazonSQSAsync, "Main");
adapter.setOutputChannel(inputChannel());
adapter.setAutoStartup(true);
adapter.setMessageDeletionPolicy(SqsMessageDeletionPolicy.NEVER);
adapter.setMaxNumberOfMessages(10);
adapter.setVisibilityTimeout(5);
return adapter;
}
Queue configured:
#Bean
public MessageChannel inputChannel() {
return new DirectChannel();
}
Now the main integration flow trigger point:
#Bean
public IntegrationFlow inbound() {
return IntegrationFlows.from("inputChannel").transform(i -> "TEST_FLOW").get();
}
}
Appreciate any type of help.
The sqsMessageDrivenChannelAdapter() must be declared as a #Bean
The inbound() must be declared as a #Bean
This one fully does not make sense IntegrationFlows.from(MessageChannels.queue()). What is the point to start the flow from anonymous channel? Who and how is going to produce messages to that channel?
Make yourself familiar with different channels: https://docs.spring.io/spring-integration/docs/current/reference/html/core.html#channel-implementations
Pay attention that QueueChannel must be consumed via polling endpoint.
Right, there is a default poller auto-configured by Spring Boot, but it is based on a single thread in the TaskScheduler and has a polling period as 10 millis.
I wouldn't recommend to hand off SQS messages to the QueueChannel: when consumer fails, you lose the data. It is better to process those messages in the consumer thread.
Otherwise your intention is not clear in the provided code.
Can you, please, share with us what error you get or anything else?
You also can turn on DEBUG logging level for org.springframework.integration to see how your messages are processed.
We have a MQ request/reply pattern implementation
Here we use a IBM MQ Cluster host. Here the request/reply queues on both sides are linked to each other by the MQ cluster environment, as the queue managers of different systems within the cluster talks to each other.
Our Requestor code uses Spring JMS Integration - JmsOutboundGateway to send and receive message
The service provider is a Mainframe application which we have no control.
public class JmsOutboundGatewayConfig {
#Bean
public MessageChannel outboundRequestChannel() {
return new DirectChannel();
}
#Bean
public QueueChannel outboundResponseChannel() {
return new QueueChannel();
}
#Bean
#ServiceActivator(inputChannel = "outboundRequestChannel")
public JmsOutboundGateway jmsTestOutboundGateway(ConnectionFactory connectionFactory) {
JmsOutboundGateway gateway = new JmsOutboundGateway();
gateway.setConnectionFactory(connectionFactory);
gateway.setRequestDestinationName("REQUEST.ALIAS.CLUSTER.QUEUE");
gateway.setReplyDestinationName("REPLY.ALIAS.CLUSTER.QUEUE");
gateway.setReplyChannel(outboundResponseChannel());
gateway.setRequiresReply(true);
return gateway;
}
}
// Requestor - sendAndReceive code
outboundRequestChannel.send(new GenericMessage<>("payload"));
Message<?> response = outboundResponseChannel.receive(10000);
Issue:
The issue we are facing when we send message, the gateway code is also passing the replyTo = queue://REPLY.ALIAS.CLUSTER.QUEUE.
Now the mainframe program that consumes this message , it is forced to reply back to the replyTo queue. It is failing on mainframe side as this replyTo queue which we send is not part of their MQ Mgr/env.
I could not find a way to remove the replyTo when sending message. As JmsOutboundGateway set this replyTo using the "ReplyDestinationName" which I had configured.
Our requestor will need to set the "ReplyDestinationName" as we are listening to this Alias-cluster reply queue for reply back.
I looked at the Channel interceptor options, I could only Intercept the message to alter it, but no option to change the replyTo.
Is there way to alter the replyTo i.e replyTo and ReplyDestination different?
Is there anyway to remove/not-set the replyTo when sending message to request queue?
Just wondering how to get this working for such MQ cluster environment where the replyTo queue will have to kept what the mainframe consumer service want, that is different to the replyDestination queue which we use.
Considering that the replyTo is used by the mainframe service to reply back. If it is not passed the mainframe service will use its own reply queue which is linked to our reply-cluster-alias queue.
Any inputs appreciated?
Thanks
Saishm
Further clarification:
The cluster mq env we have, Our spring jms outbound gateway is writing request to - "REQUEST.ALIAS.CLUSTER.QUEUE" & listening to the reply on "REPLY.ALIAS.CLUSTER.QUEUE"
So the jmsOutboundGateway sets the replyTo=REPLY.ALIAS.CLUSTER.QUEUE
Now the mainframe service on the other side is reading the message from "REQUEST.LOCAL.QUEUE". In the cluster env the "REQUEST.ALIAS.CLUSTER.QUEUE" ands its QMGR are linked to "REQUEST.LOCAL.QUEUE" and its QMGR, this is all managed within the cluster MQ env.
The mainframe service when consuming the request, sees that the incoming message had a replyTo and tries to send the response to this replyTo.
The issue is mainframe was supposed to reply to "REPLY.LOCAL.QUEUE" which is linked to REPLY.ALIAS.CLUSTER.QUEUE
If there is no replyTo it would have send the reply to "REPLY.LOCAL.QUEUE".
Now from the jmsOutBoundGateway I dont have any options to remove replyTo when sending mEssage or edit it to "REPLY.LOCAL.QUEUE" and keep listening to the response/reply of the request on "REPLY.ALIAS.CLUSTER.QUEUE"
Doesn't look like a JmsOutboundGateway will fit your IBM MQ cluster configuration and requirements. You just cannot use that replyTo feature since we cannot bypass JMS protocol over here.
Consider to use a pair of JmsSendingMessageHandler and JmsMessageDrivenEndpoint components, respectively. Your JmsSendingMessageHandler will just send a JMS message to the REQUEST.ALIAS.CLUSTER.QUEUE and forget. Only what you need is to supply a JmsHeaders.CORRELATION_ID. The JmsHeaderMapper will populate a jmsMessage.setJMSCorrelationID() property - the same way as JmsOutboundGateway does that by default. In this case your mainframe service is free to use its own reply queue and I guess it will supply our correlationId correctly.
A JmsMessageDrivenEndpoint has to be subscribed to the REPLY.ALIAS.CLUSTER.QUEUE and you do a correlation between request and reply yourself. For example a Map of correlationId and Future to fulfill when reply comes back.
I'm trying to learn how to handle MQTT Messages in Spring-Integration.
Have created a converter, that subscribes with a single MqttPahoMessageDrivenChannelAdapter per MQTT Topic for consuming and converting the messages.
The problem is our data provider is planning to "speed-up" publishing messages on his side. So instead of having a few(<=10) topics each of which has messages with about 150 fields it is planned to publish each of those fields to the separate MQTT topic.
This means my converter would have to consume ca. 1000 mqtt topics, but I do not know whether:
Is spring-integration still a good choice for it. Cause afaik. the mentioned adapter uses the PAHO MqttClient that will consume the messages from all of the topics it is subscribed to in one single thread and creating 1000 instances of those adapters is an overkill.
If we stick further to spring-integration and use the provided components, would it be a good idea to create a single inbound adapter for all of the fields, that previously were in messages of one topic but moving the conversion away from the adapter bean to a separate bean ( that does the conversion) connected with an executer-channel to the adapter and thus executing the conversion of those fields on some threadpool in parallel.
Thanks in advance for your answers!
I think your idea makes sense.
For that purpose you need to implement a passthrough MqttMessageConverter and provide an MqttMessage as a payload and topic as a header:
public class PassThroughMqttMessageConverter implements MqttMessageConverter {
#Override
public Message<?> toMessage(String topic, MqttMessage mqttMessage) {
return MessageBuilder.withPayload(mqttMessage)
.setHeader(MqttHeaders.RECEIVED_TOPIC, topic)
.build();
}
#Override
public Object fromMessage(Message<?> message, Class<?> targetClass) {
return null;
}
#Override
public Message<?> toMessage(Object payload, MessageHeaders headers) {
return null;
}
}
So, you really will be able to perform a target conversion downstream, after a mentioned ExecutorChannel in the custom transformer.
You also may consider to implement a custom MqttPahoClientFactory (an extension of the DefaultMqttPahoClientFactory may work as well) and provide a custom ScheduledExecutorService to be injected into the MqttClient you are going create in the getClientInstance().
I'm new to Spring Integration. The situation is that I've to connect to Tcp server dynamically(i.e. the DNS will be dynamically generated at runtime based on some params). Because of this I'm using Service Activator to manually create Tcp Connections and send messages. I've overridden CachingClientConnectionFactory to make use of shared connections concept(with single-use='false'). I was listening to messages using TcpReceivingChannelAdaptor by overriding "onMessage" method. The problem is that the server either responds with a Success or failure(with Generic messages) with no CorrelationID. Is there any way to correlate the request with the response ?
I tried using TcpOutboundGateway, but with this approach also I get the same problem. I used TcpConnectionSupport to send messages :
//Sample Code.
final String correlationId = "" // Dynamic unique number
TcpOutboundGateway outboundGateway = new TcpOutboundGateway(){
public synchronized boolean onMessage(Message<?> message) {
ByteArrayToStringConverter converter = new ByteArrayToStringConverter();
String response = converter.convert((byte[]) message
.getPayload());
logger.info(correlationId);
return false;
}
};
DefaultCachingClientConnectionFactory connFactory = new DefaultCachingClientConnectionFactory();
TcpConnectionSupport con = connFactory.obtainConnection();
GenericMessage<String> msg = new GenericMessage<String>("Sample Message" + correlationId);
con.registerListener(outboundGateway);
con.send(msg);
// DefaultCachingClientConnectionFactory is the subclass of CachingClientConnectionFactory.
When I send multiple messages, every time I get the same correlation printed in the "onMessage" method.
I read here that Outbound Gateway will correlate messages. Please help me. Maybe I'm doing something wrong.
Thanks
Unless you include correlation data in the message you can't correlate a response to a request.
The gateway achieves this by only allowing one outstanding request on a socket at a time; hence the reply has to be for the request. This is not very useful at high volume with a shared connection; hence the caching client cf was introduced. The gateway keeps a map of outstanding requests based on the connection id.
The gateway, in conjunction with the caching client connection factory should do what you need. However, overriding onMessage is not a good idea, because that's where the reply correlation is done.