we are trying to make parallel calls to different recipient using scatter-gather and it works fine. But the second recipient flow is not starting unless the first one is complete(traced in Zipkin). is there is a way to make all recipients async.. very similar to split-aggregate with executor channel.
public IntegrationFlow flow1() {
return flow -> flow
.split().channel(c -> c.executor(Executors.newCachedThreadPool()))
.scatterGather(
scatterer -> scatterer
.applySequence(true)
.recipientFlow(flow2())
.recipientFlow(flow3())
.recipientFlow(flow4())
.recipientFlow(flow5()),
gatherer -> gatherer
.outputProcessor(messageGroup -> {
Object request = gatherResponse(messageGroup);
return createResponse(request);
}))
.aggregate();
}
flow2(),flow3(),flow4() methods are methods with InterationFlow as return type.
sample code flow2() :
public IntegrationFlow flow2() {
return integrationFlowDefinition -> integrationFlowDefinition
.enrichHeaders(
h -> h.header(TransportConstants.HEADER_CONTENT_TYPE, MediaType.APPLICATION_XML_VALUE))
.transform(ele -> createRequest1(ele))
.wireTap("asyncXMLLogging")
.handle(wsGateway.applyAsHandler(endpoint1))
.transform(
ele -> response2(ele));
}
This is indeed possible with the mentioned executor channel. All you recipient flows must really start from the ExecutorChannel. In your case you have to modify all of them to something like this:
public IntegrationFlow flow2() {
return IntegrationFlows.from(MessageChannels.executor(taskExexecutor()))
.enrichHeaders(
h -> h.header(TransportConstants.HEADER_CONTENT_TYPE, MediaType.APPLICATION_XML_VALUE))
.transform(ele -> createRequest1(ele))
.wireTap("asyncXMLLogging")
.handle(wsGateway.applyAsHandler(endpoint1))
.transform(
ele -> response2(ele))
.get();
}
Pay attention to the IntegrationFlows.from(MessageChannels.executor(taskExexecutor())). That's exactly how you can make each sub-flow async.
UPDATE
For the older Spring Integration version without IntegrationFlow improvement for the sub-flows we can do like this:
public IntegrationFlow flow2() {
return integrationFlowDefinition -> integrationFlowDefinition
.channel(c -> c.executor(Executors.newCachedThreadPool()))
.enrichHeaders(
h -> h.header(TransportConstants.HEADER_CONTENT_TYPE, MediaType.APPLICATION_XML_VALUE))
.transform(ele -> createRequest1(ele))
.wireTap("asyncXMLLogging")
.handle(wsGateway.applyAsHandler(endpoint1))
.transform(
ele -> response2(ele));
}
This is similar to what you show in the comment above.
Related
I'm currently trying to write and integration flow then reads a csv file and processes it in chunks (Calls API for enrichment) then writes in back out as a new csv. I currently have an example working perfectly except that it is polling a directory. What I would like to do is be able to pass the file-path and file-name to the integration flow in the headers and then just perform the operation on that one file.
Here is my code for the polling example that works great except for the polling.
#Bean
#SuppressWarnings("unchecked")
public IntegrationFlow getUIDsFromTTDandOutputToFile() {
Gson gson = new GsonBuilder().disableHtmlEscaping().create();
return IntegrationFlows
.from(Files.inboundAdapter(new File(inputFilePath))
.filter(getFileFilters())
.preventDuplicates(true)
.autoCreateDirectory(true),
c -> c
.poller(Pollers.fixedRate(1000)
.maxMessagesPerPoll(1)
)
)
.log(Level.INFO, m -> "TTD UID 2.0 Integration Start" )
.split(Files.splitter())
.channel(c -> c.executor(Executors.newFixedThreadPool(7)))
.handle((p, h) -> new CSVUtils().csvColumnSelector((String) p, ttdColNum))
.channel("chunkingChannel")
.get();
}
#Bean
#ServiceActivator(inputChannel = "chunkingChannel")
public AggregatorFactoryBean chunker() {
log.info("Initializing Chunker");
AggregatorFactoryBean aggregator = new AggregatorFactoryBean();
aggregator.setReleaseStrategy(new MessageCountReleaseStrategy(batchSize));
aggregator.setExpireGroupsUponCompletion(true);
aggregator.setGroupTimeoutExpression(new ValueExpression<>(100L));
aggregator.setOutputChannelName("chunkingOutput");
aggregator.setProcessorBean(new DefaultAggregatingMessageGroupProcessor());
aggregator.setSendPartialResultOnExpiry(true);
aggregator.setCorrelationStrategy(new CorrelationStrategyIml());
return aggregator;
}
#Bean
public IntegrationFlow enrichFlow() {
return IntegrationFlows.from("chunkingOutput")
.handle((p, h) -> gson.toJson(new TradeDeskUIDRequestPayloadBean((Collection<String>) p)))
.enrichHeaders(eh -> eh.async(false)
.header("accept", "application/json")
.header("contentType", "application/json")
.header("Authorization", "Bearer [TOKEN]")
)
.log(Level.INFO, m -> "Sending request of size " + batchSize + " to: " + TTD_UID_IDENTITY_MAP)
.handle(Http.outboundGateway(TTD_UID_IDENTITY_MAP)
.requestFactory(
alliantPooledHttpConnection.get_httpComponentsClientHttpRequestFactory())
.httpMethod(HttpMethod.POST)
.expectedResponseType(TradeDeskUIDResponsePayloadBean.class)
.extractPayload(true)
)
.log(Level.INFO, m -> "Writing response to output file" )
.handle((p, h) -> ((TradeDeskUIDResponsePayloadBean) p).printMappedBodyAsCSV2())
.handle(Files.outboundAdapter(new File(outputFilePath))
.autoCreateDirectory(true)
.fileExistsMode(FileExistsMode.APPEND)
//.appendNewLine(true)
.fileNameGenerator(m -> m.getHeaders().getOrDefault("file_name", "outputFile") + "_out.csv")
)
.get();
}
public class CorrelationStrategyIml implements CorrelationStrategy {
#Override
public Object getCorrelationKey(Message<?> message) {
return message.getHeaders().getOrDefault("", 1);
}
}
#Component
public class CSVUtils {
#ServiceActivator
String csvColumnSelector(String inputStr, Integer colNum) {
return StringUtils.commaDelimitedListToStringArray(inputStr)[colNum];
}
}
private FileListFilter<File> getFileFilters(){
ChainFileListFilter<File> cflf = new ChainFileListFilter<>();
cflf.addFilter(new LastModifiedFileListFilter(30));
cflf.addFilter(new AcceptOnceFileListFilter<>());
cflf.addFilter(new SimplePatternFileListFilter(fileExtention));
return cflf;
}
If you know the file, then there is no reason in any special component from the framework. You just start your flow from a channel and send a message to it with File object as a payload. That message is going to be carried on to the slitter in your flow and everything is going to work OK.
If you really want to have a high-level API on the matter, you can expose a #MessagingGateway as a beginning of that flow and end-user is going to call your gateway method with desired file as an argument. The framework will create a message on your behalf and send it to the message channel in the flow for processing.
See more info in docs about gateways:
https://docs.spring.io/spring-integration/docs/current/reference/html/messaging-endpoints.html#gateway
https://docs.spring.io/spring-integration/docs/current/reference/html/dsl.html#integration-flow-as-gateway
And also a DSL definition starting from some explicit channel:
https://docs.spring.io/spring-integration/docs/current/reference/html/dsl.html#java-dsl-channels
If I want to generate some sample data for testing purposes of the Spring Integration DSL functionality, one way I have come up with so far is like this:
#Bean
public IntegrationFlow myFlow() {
return IntegrationFlows
.from(Http.inboundChannelAdapter("numbers").get())
.scatterGather(s -> s
.applySequence(true)
.recipientFlow(f -> f.handle((a, b) -> Arrays.asList(1,2,3,4,5,6,7,8,9,10)))
)
.split() // unpack the wrapped list
.split() // unpack the elements of the list
.log()
.get();
}
Is there another/better way to do the same thing ? Using the Scatter-Gather EIP seems like overkill for something so basic...
You really can inject a specific MessageChannel from your flow to some testing component to send messages directly to that channel to be processed from the expecting endpoint in the flow. It is not clear what that scatter-gather does in your flow and where is the channel you talk in the title , but another way is to use a #MessagingGateway interface .
See more in docs:
https://docs.spring.io/spring-integration/docs/current/reference/html/messaging-endpoints.html#gateway
https://docs.spring.io/spring-integration/docs/current/reference/html/dsl.html#java-dsl-channels
Here's another way to get some test data injected into a flow:
#Bean
IntegrationFlow pollingFlow() {
return IntegrationFlow
.from(() -> new GenericMessage<>(List.of(1,2,3)),
e -> e.poller(Pollers.fixedRate(Duration.ofSeconds(1))))
// do something here ever second
.get();
}
I found a simpler way of doing the same thing using .transform() instead of .scatterGather()
#Bean
public IntegrationFlow testFlow() {
return IntegrationFlows
.from(Http.inboundChannelAdapter("test").get())
.transform(t -> Arrays.asList(1,2,3,4,5,6,7,8,9,10))
.split() // unpack the wrapped list
.split() // unpack the elements of the list
.get();
}
And here's another way you could do it using .gateway()
#Bean
public IntegrationFlow testFlow() {
return IntegrationFlows
.from(Http.inboundChannelAdapter("test").get())
.gateway(f -> f.handle((a, b) -> Arrays.asList(1,2,3,4,5,6,7,8,9,10)))
.split() // unpack the elements of the list
.get();
}
And yet another:
#Bean
IntegrationFlow fromSupplier() {
return IntegrationFlow.fromSupplier(() -> List.of(1,2,3,4),
s -> s.poller(p -> p.fixedDelay(1, TimeUnit.DAYS)))
.log(Message::getPayload)
.nullChannel();
}
I'm building a micro service for multiple properties. So, each property has different configuration. To do that, I've implemented something like this;
#Autowired
IntegrationFlowContext flowContext;
#Bean
public void setFlowContext() {
List<Login> loginList = DAO.getLoginList(); // a web service
loginList.forEach(e -> {
IntegrationFlow flow = IntegrationFlows.from(() -> e, c -> c.poller(Pollers.fixedRate(e.getPeriod(), TimeUnit.SECONDS, 5)))
.channel("X_CHANNEL")
.get();
flowContext.registration(flow).register();
});
}
By this implementation, I'm getting the loginList before application started. So, after application is started, I'm not able to get loginList from web service since there is no poller config. The problem is loginList could change; new logins credentials could be added or deleted. Therefore, I want to implement something will work X time period to get loginList from web service, then, by loginList I need to register the flows that are created for each loginList. To achieve, I've implemented something like this;
#Bean
public IntegrationFlow setFlowContext() {
return IntegrationFlows
.from(this::getSpecification, p -> p.poller(Pollers.fixedRate(X))) // the specification is constant.
.transform(payload -> DAO.getLoginList(payload))
.split()
.<Login>handle((payload, header) -> {
IntegrationFlow flow = IntegrationFlows.from(() -> payload, c -> c.poller(Pollers.fixedRate(payload.getPeriod(), TimeUnit.SECONDS, 5)))
.channel("X_CHANNEL")
.get();
flowContext.registration(flow).register().start();
return null;
})
.get();
}
Basically, I've used start() method, but this is not working as aspected. See this;
flowContext.registration(flow).register().start();
Lastly, I've read the Dynamic and Runtime Integration Flows, but still couldn't implement this feature.
Dynamic flow registration cannot be used within a #Bean definition.
It is designed to be used at runtime AFTER the application context is fully initialized.
I use DSL implementation of Spring Integration.
I have code below and I can not use my custom error flow. When authenticate method throws Runtime Exception, the errorChannel starts to process. I enrich header to use my custom error flow, but not use.
// In Class - 1
#Bean
public MarshallingWebServiceInboundGateway marshallingWebServiceInboundGateway(BeanFactoryChannelResolver channelResolver, Jaxb2Marshaller marshaller) {
MarshallingWebServiceInboundGateway wsInboundGateway = new MarshallingWebServiceInboundGateway();
wsInboundGateway.setRequestChannel(channelResolver.resolveDestination("incomingRequest.input"));
wsInboundGateway.setReplyChannel(channelResolver.resolveDestination("outgoingResponse.input"));
wsInboundGateway.setErrorChannel(channelResolver.resolveDestination("errorChannel"));
wsInboundGateway.setMarshaller(marshaller);
wsInboundGateway.setUnmarshaller(marshaller);
return wsInboundGateway;
}
// In Class - 2
#Bean
public IntegrationFlow incomingRequest() {
return f -> f.<Object, Class<?>>route(t -> t.getClass(),
mapping -> mapping.subFlowMapping(payloadType1(),
sf -> sf.gateway("type1.input", ConsumerEndpointSpec::transactional))
.subFlowMapping(payloadType2(),
sf -> sf.gateway("type2.input", ConsumerEndpointSpec::transactional)),
conf -> conf.id("router:Incoming request router"));
}
// In Class - 3
#Bean
public IntegrationFlow type1() {
IntegrationFlow integrationFlow = f -> f
.enrichHeaders(h -> h.header(MessageHeaders.ERROR_CHANNEL, "error222", true))
.<Type1>handle((p, h) -> authentication.authenticate(p),
conf -> conf.id("service-activator:Authenticate"))
.transform(transformer::transformType1MsgToDataX,
conf -> conf.id("transform:Unmarshall type1 Message"))
.enrichHeaders(h -> h.headerExpression(TypeDataIntegrationMessageHeaderAccessor.MESSAGE_ID, "payload.id")
.headerExpression(TypeDataIntegrationMessageHeaderAccessor.MESSAGE_TYPE, "payload.messageType"))
.handle((GenericHandler<DataX>) repository::successResponseMessage,
conf -> conf.id("service-activator:return success"))
.channel("outgoingResponse.input")
;
return integrationFlow;
}
// In Class - 3
#Bean
public IntegrationFlow error222Flow() {
return IntegrationFlows.from("error222").handle("repository", "failureResponseMessage").get()
;
}
EDIT:
After Artem's answers, my code like below. But still, I can't access header parameter in the error flow. I get error - "No channel resolved by router 'router:error response prepare' "
// In Class - 1
#Bean
public MarshallingWebServiceInboundGateway marshallingWebServiceInboundGateway(BeanFactoryChannelResolver channelResolver, Jaxb2Marshaller marshaller) {
MarshallingWebServiceInboundGateway wsInboundGateway = new MarshallingWebServiceInboundGateway();
wsInboundGateway.setRequestChannel(channelResolver.resolveDestination("incomingRequest.input"));
wsInboundGateway.setReplyChannel(channelResolver.resolveDestination("outgoingResponse.input"));
wsInboundGateway.setErrorChannel(channelResolver.resolveDestination("errorResponse.input"));
wsInboundGateway.setMarshaller(marshaller);
wsInboundGateway.setUnmarshaller(marshaller);
return wsInboundGateway;
}
// In Class - 2
#Bean
public IntegrationFlow incomingRequest() {
return f -> f.<Object, Class<?>>route(t -> t.getClass(),
mapping -> mapping.subFlowMapping(payloadType1(),
sf -> sf.gateway("type1.input", ConsumerEndpointSpec::transactional))
.subFlowMapping(payloadType2(),
sf -> sf.gateway("type2.input", ConsumerEndpointSpec::transactional)),
conf -> conf.id("router:Incoming request router"));
}
// In Class - 2
#Bean
public IntegrationFlow errorResponse(){
return f -> f.<MessageHandlingException, Object>route(t -> t.getFailedMessage().getHeaders().get("ABCDEF"),
mapping -> mapping.subFlowMapping("ABCDEF",
sf -> sf.gateway("customError.input", ConsumerEndpointSpec::transactional)),
conf -> conf.id("router:error response prepare"));
}
// In Class - 3
#Bean
public IntegrationFlow type1() {
IntegrationFlow integrationFlow = f -> f
.enrichHeaders(h -> h.header("ABCDEF", "ABCDEF", true))
.<Type1>handle((p, h) -> authentication.authenticate(p),
conf -> conf.id("service-activator:Authenticate"))
.transform(transformer::transformType1MsgToDataX,
conf -> conf.id("transform:Unmarshall type1 Message"))
.enrichHeaders(h -> h.headerExpression(TypeDataIntegrationMessageHeaderAccessor.MESSAGE_ID, "payload.id")
.headerExpression(TypeDataIntegrationMessageHeaderAccessor.MESSAGE_TYPE, "payload.messageType"))
.handle((GenericHandler<DataX>) repository::successResponseMessage,
conf -> conf.id("service-activator:return success"))
.channel("outgoingResponse.input")
;
return integrationFlow;
}
// In Class - 3
#Bean
public IntegrationFlow customError(){
return f -> f.handle((GenericHandler<MessageHandlingException>)eventRepository::failureResponseMessage,
conf -> conf.id("service-activator:return failure"));
}
EDIT - 2:
I try Artem's test code, it works in this scenario. If I convert the type1 flow to the subflow mapping as below (I do it, because I'm doubtful my subflow code block), the error flow can't print ABCDEF parameter value.
After that, I add another header(XYZTWR) to the subflow mapping, but it can't be printed too.
#Bean
public IntegrationFlow type1() {
return f -> f.<String, String>route(t -> t.toString(), mapping -> mapping.subFlowMapping("foo",
sf -> sf.gateway("fooFlow.input", ConsumerEndpointSpec::transactional).enrichHeaders(h -> h.header("XYZTRW", "XYZTRW", true))));
}
#Bean
public IntegrationFlow fooFlow() {
return f -> f.enrichHeaders(h -> h.header("ABCDEF", "ABCDEF", true))
.handle((p, h) -> {
throw new RuntimeException("intentional");
});
}
My S.OUT is :
GenericMessage [payload=foo, headers={history=testGateway,type1.input, id=1fad7a65-4abe-c41d-0b22-36839a103269, timestamp=1503029553071}]
The errorChannel header starts work when we shift message to the different thread executor or queue channel. Otherwise standard throw and try...catch works in the same call stack.
So in your case the authentication exception is just thrown to the caller - WS Inbound Gateway. And here you have configured global error channel.
I did this testing:
#Configuration
#EnableIntegration
#IntegrationComponentScan
public static class ContextConfiguration {
#Bean
public IntegrationFlow errorResponse() {
return IntegrationFlows.from(errorChannel())
.<MessagingException, Message<?>>transform(MessagingException::getFailedMessage,
e -> e.poller(p -> p.fixedDelay(100)))
.get();
}
#Bean
public IntegrationFlow type1() {
return f -> f
.enrichHeaders(h -> h.header("ABCDEF", "ABCDEF", true))
.handle((p, h) -> { throw new RuntimeException("intentional"); });
}
#Bean
public PollableChannel errorChannel() {
return new QueueChannel();
}
}
#MessagingGateway(errorChannel = "errorChannel", defaultRequestChannel = "type1.input")
public interface TestGateway {
Message<?> sendTest(String payload);
}
...
#Autowired
private TestGateway testGateway;
#Test
public void testErrorChannel() {
Message<?> message = this.testGateway.sendTest("foo");
System.out.println(message);
}
And my SOUT shows me:
GenericMessage [payload=foo, headers={ABCDEF=ABCDEF, id=ae5d2d44-46b7-912d-17d4-bf2ee656140a, timestamp=1502999446725}]
Please, make DEBUG logging level for the org.springframework.integration category and observe in which step your message is losing desired headers.
UPDATE
OK. I see your problem. Since you use sf -> sf.gateway("fooFlow.input", ConsumerEndpointSpec::transactional), in other words you call the downstream via gateway, everything you've done there is behind the door and you can get as guilty to back in case of error only what you send there - gateway's request message. The downstream failedMessage is swallowed by default.
To fix the problem you should consider an addition errorChannel() option for that .gateway() and handle the downstream error there. Or... just don't use .gateway() in the router's subflow, but simple channel mapping.
The .transactional() can be configured also on the any .handle().
I'm trying to test some stuff with spring-integration using the DSL. This is only a test so far, the flow is simple:
create some messages
process (log) them in parallel
aggregate them
log the aggregate
Apart from the aggregator, it is working fine:
#Bean
public IntegrationFlow integrationFlow() {
return IntegrationFlows
.from(integerMessageSource(), c -> c.poller(Pollers.fixedRate(1, TimeUnit.SECONDS)))
.channel(MessageChannels.executor(Executors.newCachedThreadPool()))
.handle((GenericHandler<Integer>) (payload, headers) -> {
System.out.println("\t delaying message:" + payload + " on thread "
+ Thread.currentThread().getName());
try {
Thread.sleep(2000);
} catch (InterruptedException e) {
System.err.println(e.getMessage());
}
return payload;
})
.handle(this::logMessage)
.aggregate(a ->
a.releaseStrategy(g -> g.size()>10)
.outputProcessor(g ->
g.getMessages()
.stream()
.map(e -> e.getPayload().toString())
.collect(Collectors.joining(",")))
)
.handle(this::logMessage)
.get();
}
If I leave out the .aggregate(..), part, the sample is working.
Withe the aggregator, I get the following exception:
Caused by: org.springframework.beans.factory.BeanCreationException: The 'currentComponent' (org.faboo.test.ParallelIntegrationApplication$$Lambda$9/1341404543#6fe1b4fb) is a one-way 'MessageHandler' and it isn't appropriate to configure 'outputChannel'. This is the end of the integration flow.
As far as I understand, it complains that there is no output from the aggregator?
The full source can be found here: hithub
The problem is the handle() before the aggregator - it produces no result so there is nothing to aggregate...
.handle(this::logMessage)
.aggregate(a ->
Presumably logMessage(Message<?>) has a void return type.
If you want to log before the aggregator, use a wireTap, or change logMessage to return the Message<?> after logging.
.wireTap(sf -> sf.handle(this::logMessage))