I am trying to learn about how to build IntegrationFlows as units, and join them up.
I set up a very simple processing integration flow:
IntegrationFlow processingFlow = f -> f
.<String>handle((p, h) -> process(p))
.log();
flowContext.registration(processingFlow)
.id("testProcessing")
.autoStartup(false)
.register();
Processing is very simple:
public String process(String process) {
return process + " has been processed";
}
Then I compose a flow from a source, using .gateway() to join the source to the processing:
MessageChannel beginningChannel = MessageChannels.direct("beginning").get();
StandardIntegrationFlow composedFlow = IntegrationFlows
.from(beginningChannel)
.gateway(processingFlow)
.log()
.get();
flowContext.registration(composedFlow)
.id("testComposed")
.autoStartup(false)
.addBean(processingFlow)
.register();
Then I start the flow and send a couple of messages:
composedFlow.start();
beginningChannel.send(MessageBuilder.withPayload(new String("first string")).build());
beginningChannel.send(MessageBuilder.withPayload(new String("second string")).build());
The logging handler confirms the handle method has been called for the first message, but the main thread then sits idle, and the second message is never processed.
Is this not the correct way to compose integration flow from building blocks? Doing so with channels requires registering the channel as a bean, and I'm trying to do all this dynamically.
It must be logAndReply() in the processingFlow. See their JavaDocs for difference. The log() in the end of flow makes it one-way. That’s why you are blocked since gateway waits for reply, but there is no one according your current flow definition. Unfortunately we can’t determine that from the framework level: there might be cases when you indeed don’t return according some routing or filtering logic. The gateway can be configured with a reply timeout. By default it is an infinite.
Related
Need to process messages sequentially, irrespective of the spring.task.scheduling.pool.size threads defined. Hence, we defined a ExecutorChannel with single thread. However, we see the messages are processed parallelly by the caller's thread. Please suggest how to process the messages sequentially without blocking the caller thread.
#Bean
public MessageChannel svcErrorChannel() {
return new ExecutorChannel(Executors.newSingleThreadExecutor());
}
return IntegrationFlows.from(svcErrorChannel())
.log(ERROR, m -> "ErrorFlow Initiated: " + m.getPayload())
Application Logs:
2023-02-04 20:21:03,407 [boundedElastic-1 ] ERROR o.s.i.h.LoggingHandler - 1c710133ada428f0 ErrorFlow Initiated: org.springframework.messaging.MessageHandlingException: xxxxxxxxxxxxxxxx
2023-02-04 20:21:03,407 [boundedElastic-2 ] ERROR o.s.i.h.LoggingHandler - 1c710133ada428f0 ErrorFlow Initiated: org.springframework.messaging.MessageHandlingException: xxxxxxxxxxxxxxxxx
The log() operator is essentially a ChannelInterceptor where that logging happens in a preSend() hook - the part of a MessageChannel on producer side. Therefore it is expected to see your Reactor threads in those logs.
If you really would like to log consumed (not produced) messages, then you need to use a .handle(new LoggingHandler()) instead of log() operator. Or you can use a .bridge() before your log().
See docs for more info: https://docs.spring.io/spring-integration/reference/html/dsl.html#java-dsl-log
All the examples I've seen of the Spring Integration DSL Scatter-Gatherer explicitly set the .applySequence(true) on the scatterer.
E.g. like this:
#Bean
public IntegrationFlow helloFlow() {
return IntegrationFlows
.from(Http.inboundChannelAdapter("hello").get())
.scatterGather(s -> s
.applySequence(true)
.recipientFlow(f -> f.handle((m, h) -> 33))
.recipientFlow(f -> f.handle((m, h) -> 444))
)
.split()
.log()
.get();
}
If I omit .applySequence(true) I get
java.lang.IllegalStateException: Null correlation not allowed. Maybe the CorrelationStrategy is failing?
Why is the sequence needed in this case ?
If it needed in so many cases, why isn't .applySequence(true) just the default with the option of explicitly setting it to false if desired for some reason ? And when would you explicitly want it to be set to false ?
The scatterer part of this component is fully based on the Recipient List Router which comes with false for that option by default. So, for consistency and runtime optimization we keep it false in scatter-gather as well: https://docs.spring.io/spring-integration/docs/current/reference/html/message-routing.html#router-implementations-recipientlistrouter. In real world it is really rear case when gathered messages come back with those generated sequence details headers. Typically the gatherer is configured to custom correlation and release strategies. It is more demo and samples feature to applySequence to be honest. Plus, don’t forget the message immutability and with option we enforce the framework to create a new message to add sequence details headers.
Error message: Caused by: java.lang.IllegalStateException: Null correlation not allowed. Maybe the CorrelationStrategy is failing?
My implementation,
#Bean
public IntegrationFlow start() {
return IntegrationFlows
.from("getOrders")
.split()
.publishSubscribeChannel(c -> c.subscribe(s -> s.channel(q -> q.queue(1))
.<Order, Message<?>>transform(p -> MessageBuilder.withPayload(new Item(p.getItems())).setHeader(ORDERID, p.getOrderId()).build())
.split(Item.class, Item::getItems)
.transform() // let's assume, an object created for each item, let's say ItemProperty to the object.
// Transform returns message; MessageBuilder.withPayload(createItemProperty(getItemName, getItemId)).build();
.aggregate() // so, here aggregate method needs to aggregate ItemProperties.
.handle() // handler gets List<ItemProperty> as an input.
))
.get();
}
Both splitters works fine. I've also tested the transformer after the second splitter, works fine. But, when it comes to aggregate it is failing. What I am missing here?
You are missing the fact that transformer is that type of endpoint which deal with the whole message as is. And if you create a message by yourself, it doesn't modify it.
So, with your MessageBuilder.withPayload(createItemProperty(getItemName, getItemId)).build(); you just miss important sequence details headers after splitter. Therefore an aggregator after that doesn't know what to do with your message since you configure it for default correlation strategies but you don't provide respective headers in the message.
Technically I don't see a reason to create a message over there manually: the simple return createItemProperty(getItemName, getItemId); should be enough for you. And the framework will take about message creation on your behalf with respective request message headers copying.
If you really still think that you need to create a message yourself in that transform, then you need to consider to copyHeaders() on that MessageBuilder from the request message to carry required sequence details headers.
We have a requirement where I need the same message(payload) to be processed in two different channels. We are under the impression that using PubliSHSubscribe Channel will help us deal with that by making a copy of message to both the channels. However we figured that each channel was getting executed one after the other and if e make any changes in the payload in one channel, its effecting the payload of other channel as well.
#Bean
public IntegrationFlow bean1() {
return IntegrationFlows
.from("Channel1")
.handle(MyMessage.class, (payload, header) -> obj1.method1(payload))
.channel(MessageChannels.publishSubscribe("subscribableChannel").get())
.get();
}
#Bean
public IntegrationFlow bean21() {
return IntegrationFlows
.from("subscribableChannel")
.handle(MyMessage.class, (payload, header) -> obj2.method2(payload,header))
.channel("nullChannel")
.get();
}
#Bean
public IntegrationFlow bean22() {
return IntegrationFlows
.from("subscribableChannel")
.handle(MyMessage.class, (payload, header) -> obj3.method3(payload))
.channel("nullChannel")
.get();
}
IN the above example, if i make changes to payload in bean21, its effecting the input payload passed to bean 22.
My requirement is to pass the same payload to bean21 and bean22 and to execute them parallely? Can you please advise how to accomplish that?
That's correct. Spring Integration is just Java and there is no any magic in copying payload between different messages. It is really just the same object in the memory. Now imagine you have a pure Java and would like to call two different method with the same Foo object. And then you modify that object in one method. What happens in another one? Right, it will see a modifications on the object.
To achieve your goal with definitely copying object to a new instance, you have to ensure it yourself. For example implement Cloneable interface on your class or provide copy constructor or any other possible solution to create a new object.
In the beginning of the flow of one of the subscriber you should perform that clone operation and you will have a new object without impacting another subscriber.
See more info in this JIRA: https://jira.spring.io/browse/INT-2979
I am trying to use AMQP-backed message channels in my Spring Integration app, but I think I am fundamentally misunderstanding something, specifically around the Message<?> interface and how instances of GenericMessage<?> are written to and read from, a RabbitMQ queue.
Given I have a Spring Integration app containing the following domain model object:
#Immutable
class Foo {
String name
long quantity
}
and I declare an AMQP backed message channel called fooChannel as follows:
#Bean
public AmqpChannelFactoryBean deliveryPlacementChannel(CachingConnectionFactory connectionFactory) {
AmqpChannelFactoryBean factoryBean = new AmqpChannelFactoryBean(true)
factoryBean.setConnectionFactory(connectionFactory)
factoryBean.setQueueName("foo")
factoryBean.beanName = 'fooChannel'
factoryBean.setPubSub(false)
factoryBean
}
When I initially tried to send a message to my fooChannel I received a java.io.NotSerializableException. I realised this to be caused by the fact that the RabbitTemplate used by my AMQP-backed fooChannel was using a org.springframework.amqp.support.converter.SimpleMessageConverter which can only work with Strings, Serializable instances, or byte arrays, of which my Foo model is none of those things.
Therefore, I thought that I should use a org.springframework.amqp.support.converter.Jackson2JsonMessageConverter to ensure my Foo model is properly converted to/from and AMQP message. However, it appears that the type of the message that is being added to the RabbitMQ queue which is backing my fooChannel is of type org.springframework.messaging.support.GenericMessage. This means that when my AMQP backed fooChannel tries to consume messages from the RabbitMQ queue it receives the following exception:
Caused by: com.fasterxml.jackson.databind.JsonMappingException: No suitable constructor found for type [simple type, class org.springframework.messaging.support.GenericMessage]: can not instantiate from JSON object (missing default constructor or creator, or perhaps need to add/enable type information?)
From looking at the GenericMessage class I see that is designed to be immutable, which clearly explains why the Jackson2JsonMessageConverter can't convert from JSON to the GenericMessage type. However, I am unsure what I should be doing in order to allow my fooChannel to be backed by AMQP and have the conversion of my Spring Integration messages containing my Foo model work correctly?
In terms of the flow of my application I have the following Transformer component which consumes Bar models from the (non-AMQP backed) barChannel and places Foo models on the fooChannel as follows:
#Transformer(inputChannel = 'barChannel', outputChannel = 'fooChannel')
public Foo transform(Bar bar) {
//transform logic removed for brevity
new Foo(name: 'Foo1', quantity: 1)
}
I then have a ServiceActivator component which I wish to have consume from my fooChannel as follows:
#ServiceActivator(inputChannel = 'fooChannel')
void consumeFoos(Foo foo){
// Do something with foo
}
I am using spring-integration-core:4.2.5.RELEASE and spring-integration-amqp:4.2.5.RELEASE.
Can anyone please advise where I am going wrong with the configuration of my Spring Integration application?
If any further information is needed to in order to better clarify my question or problem, please let me know. Thanks
Yes - amqp-backed channels are currently limited to Java serializable objects.
We should provide an option to map the Message<?> to a Spring AMQP Message (like the channel adapters do) rather than...
this.amqpTemplate.convertAndSend(this.getExchangeName(), this.getRoutingKey(), message);
...which converts the entire message.
You could use a pair of channel adapters (outbound/inbound) instead of a channel.
Since you are using Java config, you could wrap the adapter pair in a new MessageChannel implementation.
I opened a JIRA Issue for this.