Async split/aggregate gateway flows - spring-integration

I'm trying to build a recipe for asynchronous orchestration using spring integration gateways (both inbound and outbound). After seeing an example here, I tried using scatter-gather like this:
#Configuration
public class IntegrationComponents {
#Value("${rest.endpoint.base}")
private String endpointBase;
#Bean
public HttpRequestHandlingMessagingGateway inboundGateway() {
return Http.inboundGateway("/test-inbound-gateway-resource")
.requestMapping(mapping -> mapping.methods(HttpMethod.POST))
.requestTimeout(3000)
.replyTimeout(3000)
.get();
}
#Bean
public HttpRequestExecutingMessageHandler outboundGateway1() {
return Http.outboundGateway(endpointBase + "/test-resource-1")
.httpMethod(HttpMethod.POST)
.expectedResponseType(String.class)
.get();
}
#Bean
public HttpRequestExecutingMessageHandler outboundGateway2() {
return Http.outboundGateway(endpointBase + "/test-resource-2")
.httpMethod(HttpMethod.POST)
.expectedResponseType(String.class)
.get();
}
#Bean
public StandardIntegrationFlow integrationFlow() {
ExecutorService executor = Executors.newCachedThreadPool();
IntegrationFlow flow1 = IntegrationFlows.from(MessageChannels.executor(executor))
.handle(outboundGateway1())
.get();
IntegrationFlow flow2 = IntegrationFlows.from(MessageChannels.executor(executor))
.handle(outboundGateway2())
.get();
return IntegrationFlows
.from(inboundGateway())
.transform(String.class, String::toUpperCase)
.channel(MessageChannels.executor(executor))
.scatterGather(
scatterer -> scatterer
.applySequence(true)
.recipientFlow(flow1)
.recipientFlow(flow2),
gatherer -> gatherer
.outputProcessor(messageGroup -> {
List<Message<?>> list = new ArrayList<>(messageGroup.getMessages());
String payload1 = (String) list.get(0).getPayload();
String payload2 = (String) list.get(1).getPayload();
return MessageBuilder.withPayload(payload1 + "+" + payload2).build();
}))
.get();
}
}
This executes, but my payloads are swapped, because in this case outboundGateway1 takes longer to execute than outboundGateway2. Payload 2 comes first, then payload 1.
Is there a way to tell scatter-gather to define/maintain order when sending to the output processor?
On a similar note, maybe split/aggregate and/or using a router is a better pattern here? But if so, what would that look like?
I tried the following split/route/aggregate, but it failed saying "The 'currentComponent' (org.springframework.integration.router.RecipientListRouter#b016b4e) is a one-way 'MessageHandler' and it isn't appropriate to configure 'outputChannel'. This is the end of the integration flow.":
#Configuration
public class IntegrationComponents {
#Value("${rest.endpoint.base}")
private String endpointBase;
#Bean
public HttpRequestHandlingMessagingGateway inboundGateway() {
return Http.inboundGateway("/test-inbound-gateway-resource")
.requestMapping(mapping -> mapping.methods(HttpMethod.POST))
.requestTimeout(3000)
.replyTimeout(3000)
.get();
}
#Bean
public HttpRequestExecutingMessageHandler outboundGateway1() {
return Http.outboundGateway(endpointBase + "/test-resource-1")
.httpMethod(HttpMethod.POST)
.expectedResponseType(String.class)
.get();
}
#Bean
public HttpRequestExecutingMessageHandler outboundGateway2() {
return Http.outboundGateway(endpointBase + "/test-resource-2")
.httpMethod(HttpMethod.POST)
.expectedResponseType(String.class)
.get();
}
#Bean
public StandardIntegrationFlow integrationFlow() {
ExecutorService executor = Executors.newCachedThreadPool();
IntegrationFlow flow1 = IntegrationFlows.from(MessageChannels.executor(executor))
.handle(outboundGateway1())
.get();
IntegrationFlow flow2 = IntegrationFlows.from(MessageChannels.executor(executor))
.handle(outboundGateway2())
.get();
return IntegrationFlows
.from(inboundGateway())
.transform(String.class, String::toUpperCase)
.split()
.channel(MessageChannels.executor(executor))
.routeToRecipients(r -> r
.recipientFlow(flow1)
.recipientFlow(flow2))
.aggregate()
.get();
}
}

Can you not simply Collections.sort() the list in the output processor? Each message will have a IntegrationMessageHeaderAccessor.SEQUENCE_NUMBER header since you set applySequence.

Related

spring integration http inbound gateway parameters

I have a SpringBoot 2.2.6 application and I would like to set an endpoint with spring-integration therefore I have the follow configuration:
#Bean
public MessageChannel reply() {
return new DirectChannel();
}
#Bean
public IntegrationFlow inbound(TestTransformer testTransformer) {
return IntegrationFlows.from(Http.inboundGateway("/foo")
.requestMapping(m -> m.methods(HttpMethod.GET))
.replyChannel("reply")
.requestPayloadType(String.class))
.channel("httpRequest")
.get();
}
#Bean
#ServiceActivator(inputChannel = "httpRequest", outputChannel = "reply")
public Function<Message<?>, String> handler() {
return new Function<Message<?>, String>() {
public String apply(Message<?> message) throws MessagingException {
log.info("myHandler: " + message.getPayload());
log.info("myHandler: " + message.getHeaders());
return "ok";
}
};
}
Now if I call the enpoint passing params as http://localhost:8080/MyApp/foo?q=test&q1=test2 I receive the parameters in a JSON form.
Is possible to receive something like #PathVariable in the MVC for example writing:
return IntegrationFlows.from(Http.inboundGateway("/foo/{name}")
I have tried it but doesn't work and I canno't find any docs talking about that (at least with java bean configuration)
Thanks
Maybe:
#Bean
public IntegrationFlow test(TestTransformer testTransformer, Jackson2JsonObjectMapper obj) {
return IntegrationFlows.from(Http.inboundGateway("/foo/{name}")
.requestMapping(m -> m.methods(HttpMethod.GET))
.payloadExpression("#pathVariables.name")
.replyChannel("reply")
.requestPayloadType(String.class))
.transform(testTransformer)
.transform(new ObjectToJsonTransformer(obj))
.channel("httpRequest")
.get();
}
is more usefull... The docs is full of xml config and I read it but I'm not able to trasnform them all into java DSL

Why is my timeToLiveFunction never called?

I have the following flow:
#Bean
public IntegrationFlow workerInfoJmsOut(ConnectionFactory connectionFactory, WorkerProperties properties) {
return IntegrationFlows
.from(ChannelNames.WORKER_INFO)
.handle(Jms.outboundAdapter(connectionFactory)
.timeToLiveFunction(message -> properties.getHeartbeatTtl().toMillis())
.destination(JmsQueueNames.WORKER_INFO))
.get();
}
And the following gateway:
#MessagingGateway
public interface DispatcherGateway {
#Gateway(requestChannel = ChannelNames.WORKER_INFO)
void submit(WorkerMessage message);
}
Which is called every 1s:
#Scheduled(fixedDelay = 1000)
public void beat() {
dispatcherGateway.submit(WorkerInfoNotice.of(...));
}
However, the breakpoint in the timeToLiveFunction is never hit:
Did I misunderstand something or why is it never hit?
spring-integration-jms:5.3.1.RELEASE
I found out that I need to enable explicit QoS
#Bean
public IntegrationFlow workerInfoJmsOut(ConnectionFactory connectionFactory, WorkerProperties properties) {
return IntegrationFlows
.from(ChannelNames.WORKER_INFO)
.handle(Jms.outboundAdapter(connectionFactory)
.configureJmsTemplate(spec -> spec.explicitQosEnabled(true))
.timeToLiveFunction(message -> properties.getHeartbeatTtl().toMillis())
.destination(JmsQueueNames.WORKER_INFO))
.get();
}

publishSubscribeChannel unit test can't work well

my integration config class is below,when i do some unit test on them,found that:
when i send message to UserRecipientSubscribeCacheChannel,it work well;
when i send a message to an upper level of channel userReportWriteCompletedRouteChannel, it work failed,and it don't throws any exceptions yet. i can't understand it. the messages that i sent is same,of course.
because of the fail section, the next handler can't work ok.
ty!!
it work ok below, it print ===>ip location channel message:GenericMessage [payload=[MailRecipientActionDocumen...and ===>user recipient channel message:GenericMessage [payload=[UserRecipientSubscribeDataRedisStructure...
#Test
public void test_sendMessageUserRecipientSubscribeCacheChannel(){
UserRecipientSubscribeCacheChannel.send(createMessageWithIp());
}
it work fail below, it print ===>ip location channel message:GenericMessage [payload=[MailRecipientActionDocumen... only
notice that: the fail section, In front of handler has a transformer.
#Test
public void test_sendMessageToRouteChannel() {
userReportWriteCompletedRouteChannel.send(createMessageWithIp());
}
my code config below:
#Bean
public SubscribableChannel userReportWriteCompletedSubscribeChannel() {
return new DirectChannel();
}
#Bean
public QueueChannel userReportWriteCompletedRouteChannel() {
return new QueueChannel();
}
#Bean
public MessageChannel ipLocationResolveCacheChannel() {
return new DirectChannel();
}
#Bean
public MessageChannel userRecipientSubscribeCacheChannel() {
return new DirectChannel();
}
#MessagingGateway(name = "userReportWriteCompletedListener",
defaultRequestChannel = "userReportWriteCompletedRouteChannel")
public interface UserReportWriteCompletedListener {
#Gateway
void receive(List<UserMailRecipientActionDocument> docs);
}
#Bean
public IntegrationFlow bridgeFlow() {
return flow -> flow.channel("userReportWriteCompletedRouteChannel")
.bridge(bridgeSpe -> bridgeSpe
.poller(pollerFactory -> pollerFactory.fixedRate(500).maxMessagesPerPoll(1)))
.channel("userReportWriteCompletedSubscribeChannel")
;
}
#Bean
public IntegrationFlow subscribeFlow() {
return IntegrationFlows.from("userReportWriteCompletedSubscribeChannel")
.publishSubscribeChannel(publishSubscribeSpec -> publishSubscribeSpec
.subscribe(flow -> flow
.channel(IP_LOCATION_RESOLVE_CACHE_CHANNEL)
)
.subscribe(flow -> flow
.channel(USER_RECIPIENT_SUBSCRIBE_CACHE_CHANNEL)
))
.get();
}
#Bean
public RedisStoreWritingMessageHandler ipLocationResolveCacheHandler(RedisTemplate<String, ?> redisTemplate) {
final RedisStoreWritingMessageHandler ipLocationResolveCacheHandler =
new RedisStoreWritingMessageHandler(redisTemplate);
ipLocationResolveCacheHandler.setKey("IP_LOCATION_RESOLVE_CACHE");
return ipLocationResolveCacheHandler;
}
#Bean
public RedisStoreWritingMessageHandler userRecipientSubscribeCacheHandler(RedisTemplate<String, ?> redisTemplate) {
final RedisStoreWritingMessageHandler userRecipientSubscribeCacheHandler =
new RedisStoreWritingMessageHandler(redisTemplate);
userRecipientSubscribeCacheHandler.setKey("USER_RECIPIENT_SUBSCRIBE_CACHE");
return userRecipientSubscribeCacheHandler;
}
#Bean
public IpLocationResolveRedisStructureFilterAndTransformer recipientActionHasIpFilterAndTransformer() {
return new IpLocationResolveRedisStructureFilterAndTransformer();
}
#Bean
public UserRecipientSubscribeDataRedisStructureTransformer subscribeDataRedisStructureTransformer(
IpLocationClient ipLocationClient) {
return new UserRecipientSubscribeDataRedisStructureTransformer(ipLocationClient);
}
#Bean
public IntegrationFlow ipLocationResolveCacheFlow(
#Qualifier("ipLocationResolveCacheHandler") RedisStoreWritingMessageHandler writingMessageHandler) {
return flow -> flow.channel(IP_LOCATION_RESOLVE_CACHE_CHANNEL)
.handle(message -> {
System.out.println("===>ip location channel message:" + message);
})
;
}
#Bean
public IntegrationFlow userRecipientActionDataCacheFlow(
#Qualifier("userRecipientSubscribeCacheHandler") RedisStoreWritingMessageHandler messageHandler,
UserRecipientSubscribeDataRedisStructureTransformer transformer) {
return flow -> flow.channel(USER_RECIPIENT_SUBSCRIBE_CACHE_CHANNEL)
.transform(transformer)
.handle(message -> {
System.out.println("===>user recipient channel message:" + message);
})
}
i expect 2 print message info ,but print 1 only.
Today, i found that the bridge flow may had some problem, When I move the handler behind the channeluserReportWriteCompletedSubscribeChannel, it can't print any message;
when i remove channel and add handler directly, it will print message.
does i use the bridge wrong?
#Bean
public IntegrationFlow bridgeFlow() {
return flow -> flow.channel("userReportWriteCompletedRouteChannel")
.bridge(bridgeSpe -> bridgeSpe
.poller(pollerFactory -> pollerFactory.fixedRate(100).maxMessagesPerPoll(1)))
.handle(message -> {
System.out.println("===>route channel message:" + message);
}) // handle ok , will print message
.channel("userReportWriteCompletedSubscribeChannel")
// .handle(message -> {
// System.out.println("===>route channel message:" + message);
// }) // handle fail , will not printing message
;
}
test:
#Test
//invalid
public void test_sendMessageToRouteChannel() {
userReportWriteCompletedRouteChannel.send(createMessageWithIp());
}

How to configure a trigger in spring integration flow to get value from a method invoking message source?

How to configure a trigger in spring integration flow to get value from a method invoking message source and start it in another flow ?
#Bean
public IntegrationFlow integrationFlow() {
return IntegrationFlows.from(messageSource,channelSpec -> channelSpec.poller(Pollers.trigger(new SomeTrigger())).handle(...).get()
}
#Bean
public MessageSource<?> messageSource() {
MethodInvokingMessageSource source = new MethodInvokingMessageSource();
source.setObject(new Random());
source.setMethod("nextInt");
}
#Bean
public IntegrationFlow someOtherFlow() {
return IntegrationFlows.from("messageChannel")
///some logic to trigger and get the value of random int
}
The MessageSource has receive() method, so you can do just this:
#Bean
public MessageSource<?> randomIntSource() {
MethodInvokingMessageSource source = new MethodInvokingMessageSource();
source.setObject(new Random());
source.setMethodName("nextInt");
return source;
}
#Bean
public IntegrationFlow someOtherFlow() {
return IntegrationFlows.from("messageChannel")
.handle(randomIntSource(), "receive")
.handle(System.out::println)
.get();
}
pay attention to the .handle(randomIntSource(), "receive").

Breaking up DSL IntegrationFlows

I've been playing around with Spring Integration (SI) DSL. I have a Rest service with the following Async Gateway defined :
#MessagingGateway
public interface Provision {
#Async
#Gateway(requestChannel = "provision.input")
ListenableFuture<List<ResultDto>> provision(List<ItemsDto> stuff);
}
From the Line-by-line walk-through I have the follow example IntegrationFlow.
#Bean
public IntegrationFlow provision() {
return f -> f
.split(ArrayList.class, List::toArray)
.channel(c -> c.executor(Executors.newCachedThreadPool()))
.<ItemsDto, String>route(ItemsDto::getType, m -> m
.subFlowMapping("IPTV", sf -> sf
.<ItemsDto, String>route(ItemsDto::getAction, m2 -> m2
.subFlowMapping("OPEN", sf2 -> sf2
.handle((p, h) -> iptvService.open((ItemsDto) p))))
)
)
.aggregate();
}
I have several layers of routing as you can see. I need to break things up a bit. I've tried several things which don't work (here I don't get a response...the thread doesn't wait):
#Bean(name = "routerInput")
private MessageChannel routerInput() {
return MessageChannels.direct().get();
}
#Bean
public IntegrationFlow provision() {
return f -> f
.split(ArrayList.class, List::toArray)
.channel(c -> c.executor(Executors.newCachedThreadPool()))
.<ItemsDto, String>route(ItemsDto::getType, m ->
m.subFlowMapping("IPTV", sf -> sf.channel("routerInput"))
)
.aggregate();
}
#Bean
public IntegrationFlow action() {
return IntegrationFlows.from("routerInput")
.<ItemsDto, String>route(ItemsDto::getAction, m -> m
.subFlowMapping("OPEN", sf -> sf
.handle(p -> iptvService.open((ItemsDto) p.getPayload())))).get();
}
I'm obviously conceptually missing something :) Can someone perhaps assist with a "how to and why" opinion?
I have a list of items that need to be split, routed by "type", then routed by "action", and finally aggregated (containing response of handler). Each handled item needs to process in parallel.
Thanks in advance
Update:
From Artem's suggestion I removed all the async stuff. I trimmed it down to almost nothing...
#Bean(name = "routerInput")
private MessageChannel routerInput() {
return MessageChannels.direct().get();
}
#Bean
public IntegrationFlow provision() {
return f -> f
.split()
.<ItemDto, String>route(ItemDto::getType, m ->
m.subFlowMapping("IPTV", sf -> sf.channel("routerInput")))
.aggregate();
}
#Bean
public IntegrationFlow action() {
return IntegrationFlows.from("routerInput")
.<ItemDto, String>route(ItemDto::getAction, m -> m
.subFlowMapping("OPEN", sf -> sf
.handle((p, h) -> iptvService.open((ItemDto) p)))).get();
}
I got it to respond by changing
.handle(p ->
to this
.handle((p, h) ->
So it at least responds, but it does not aggregate the 3 test items that were split. Output consists of 1 item. Do I need to use a stream collect? Release policy? Shouldn't this be fine?
It's probably simpler to use channelMapping rather than subflowMapping if you want to break it apart...
#Bean
public IntegrationFlow typeRoute() {
return IntegrationFlows.from(foo())
.split()
.<ItemsDto, String>route(ItemsDto::getType, m -> m
.channelMapping("foo", "channel1")
.channelMapping("bar", "channel2"))
.get();
}
#Bean
public IntegrationFlow fooActionRoute() {
return IntegrationFlows.from(channel1())
.<ItemsDto, String>route(ItemsDto::getAction, m -> m
.channelMapping("foo", "channel3")
.channelMapping("bar", "channel4"))
.get();
}
#Bean
public IntegrationFlow barActionRoute() {
return IntegrationFlows.from(channel1())
.<ItemsDto, String>route(ItemsDto::getAction, m -> m
.channelMapping("foo", "channel5")
.channelMapping("bar", "channel6"))
.get();
}
#Bean
public IntegrationFlow fooFooHandle() {
return IntegrationFlows.from(channel3())
// handle
.channel(aggChannel())
.get();
}
Create flows for the other options and aggregate each result:
// fooBarHandle(), barFooHandle(), barBarHandle()
#Bean IntegrationFlow agg() {
return IntegrationFlows.from(aggChannel())
.aggregate()
.get();
}
The parallelism is handled by using ExecutorChannels...
#Bean
public MessageChannel channel1() {
return new ExecutorChannel(exec());
}
#Bean
public MessageChannel channel2() {
return new ExecutorChannel(exec());
}
#Bean
public MessageChannel channel3() {
return new DirectChannel();
}
#Bean
public MessageChannel channel4() {
return new DirectChannel();
}
#Bean
public MessageChannel channel5() {
return new DirectChannel();
}
#Bean
public MessageChannel channel6() {
return new DirectChannel();
}
#Bean
public MessageChannel aggChannel() {
return new DirectChannel();
}
I don't see any big problems in your configuration and it really should work.
Want I don't like there is:
If you use async Gateway (ListenableFuture<List<ResultDto>>) you don't need #Async annotation there, because it will be already by the Gateway contract.
You don't need convert List::toArray if your payload is a List already. It is just enough to use .split() without params.
That's for the design style.
I'm not sure yet what's the problem there, but would you mind to make your whole flow sync and share with us here the DEBUG for the flow and point out where you see the problem.
Moved the aggregate to the "action" Bean and it worked.
Thanks for your patience :)

Resources