I have setup file poller/channel adapter which polls a directory and handler integration flow using Java DSL. But I am not getting any reference how to add another directory/channel adapter and bridge to same handler. Here is my code.
#Bean
public IntegrationFlow integrationFlow(JobLaunchingGateway jobLaunchingGateway) {
return IntegrationFlows.from(Files.inboundAdapter(new File(incomingDir)).
filter(new SimplePatternFileListFilter("*.csv")).
filter(new AcceptOnceFileListFilter<>()),
c -> c.poller(Pollers.fixedRate(500).maxMessagesPerPoll(1))).
handle(fileMessageToJobRequest()).
handle(jobLaunchingGateway).
log(LoggingHandler.Level.WARN, "headers.id + ': ' + payload").
get();
}
Thanks #Artem. How about following ?
#Bean
public IntegrationFlow integrationFlowUi(JobLaunchingGateway jobLaunchingGateway) {
return IntegrationFlows.from(Files.inboundAdapter(new File(incomingDirUi)).
filter(new SimplePatternFileListFilter("*.csv")).
filter(new AcceptOnceFileListFilter<>()),
c -> c.poller(Pollers.fixedRate(500).maxMessagesPerPoll(1))).
channel("to-bridge").
handle(fileMessageToJobRequest()).
handle(jobLaunchingGateway).
log(LoggingHandler.Level.WARN, "headers.id + ': ' + payload").
get();
}
#Bean
public IntegrationFlow integrationFlowSftp(JobLaunchingGateway jobLaunchingGateway) {
return IntegrationFlows.from(Files.inboundAdapter(new File(incomingDirSftp)).
filter(new SimplePatternFileListFilter("*.csv")).
filter(new AcceptOnceFileListFilter<>()),
c -> c.poller(Pollers.fixedRate(500).maxMessagesPerPoll(1))).
channel("to-bridge").get();
}
One of the first class citizens in Spring Integration is a MessageChannel entity. You always can set explicit channels between endpoints in you IntegrationFlow definition and explicitly send messages to them.
For you “merging” use-case I would suggest to place a .channel() before .handle() and declare the second flow for the second directory, but in the end of that flow use the same .channel() to “bridge” messages from this flow to the middle of the first one.
See more information in the reference manual : https://docs.spring.io/spring-integration/docs/5.0.0.RELEASE/reference/html/java-dsl.html#java-dsl-channels
Related
I have two IntegrationFlows
both receive messages from Apache Kafka
first IntegrationFlow - in the input channel, Consumer1(concurrency=4) reads topic_1
second IntegrationFlow - in the input channel, Consumer2(concurrency=4) reads topic_2
but these two IntegrationFlows, send messages to the output channel, where one common class MyMessageHandler is specified
like this:
#Bean
public IntegrationFlow sendFromQueueFlow1(MyMessageHandler message) {
return IntegrationFlows
.from(Kafka
.messageDrivenChannelAdapter(consumerFactory1, "topic_1")
.configureListenerContainer(configureListenerContainer_priority1)
)
.handle(message)
.get();
}
#Bean
public IntegrationFlow sendFromQueueFlow2(MyMessageHandler message) {
return IntegrationFlows
.from(Kafka
.messageDrivenChannelAdapter(consumerFactory2, "topic_2")
.configureListenerContainer(configureListenerContainer_priority2)
)
.handle(message)
.get();
}
class MyMessageHandler have method send(message), this method passes messages further to another service
class MyMessageHandler {
protected void handleMessageInternal(Message<?> message)
{
String postResponse = myService.send(message); // remote service calling
msgsStatisticsService.sendMessage(message, postResponse);
// *******
}
}
inside each IntegrationFlow, 4 Consumer-threads are working (
a total of 8 threads), and they all go to one class MyMessageHandler,
into one metod send()
what problems could there be?
two IntegrationFlow, do they see each other when they pass a message to one common class??? do I need to provide thread safety in the MyMessageHandler class??? Do I need to prepend the send () method with the word synchronized???
But what if we make a third IntegrationFlow?
so that only one IntegrationFlow can pass messages through itself to the MyMessageHandler class? then would it be thread safe? example:
#Bean
public IntegrationFlow sendFromQueueFlow1() {
return IntegrationFlows
.from(Kafka
.messageDrivenChannelAdapter(consumerFactory1, "topic_1")
.configureListenerContainer(configureListenerContainer_priority1)
)
.channel(**SOME_CHANNEL**())
.get();
}
#Bean
public IntegrationFlow sendFromQueueFlow2() {
return IntegrationFlows
.from(Kafka
.messageDrivenChannelAdapter(consumerFactory2, "topic_2")
.configureListenerContainer(configureListenerContainer_priority2)
)
.channel(**SOME_CHANNEL**())
.get();
}
#Bean
public MessageChannel **SOME_CHANNEL**() {
DirectChannel channel = new DirectChannel();
return channel;
}
#Bean
public IntegrationFlow sendALLFromQueueFlow(MyMessageHandler message) {
return IntegrationFlows
.from(**SOME_CHANNEL**())
.handle(message)
.get();
}
You need to make your handler code thread-safe.
Using synchronized on the whole method you will effectively disable the concurrency.
It's better to use thread-safe techniques - no mutable fields or use limited synchronization blocks, just around critical code.
Given I have application which uses Spring Integration and I define gateway:
#Component
#MessagingGateway
public interface SmsGateway {
#Gateway(requestChannel = CHANNEL_SEND_SMS)
void sendSms(SendSmsRequest request);
}
public interface IntegrationChannels {
String CHANNEL_SEND_SMS = "channelSendSms";
}
I also attach IntegrationFlow to CHANNEL_SEND_SMS channel:
#Bean
public IntegrationFlow sendSmsFlow() {
return IntegrationFlows.from(CHANNEL_SEND_SMS)
.transform(...)
.handle(...)
.get();
}
Whenever I call sendSms gateway method from business code, sendSmsFlow is executed as expected.
When I want to attach another IntegrationFlow to the same CHANNEL_SEND_SMS channel, e.g.
#Bean
public IntegrationFlow differentFlow() {
return IntegrationFlows.from(CHANNEL_SEND_SMS)
.transform(...)
.handle(...)
.get();
}
then this differentFlow is not executed.
Why does it behave this way?
Is there any solution to make it work for both flows?
The default channel type is DirectChannel and messages are distributed to multiple subscribed channels in a round robin fashion by default.
Declare CHANNEL_SEND_SMS as a PublishSubscribeChannel if you want each flow to get every message.
This will only work with a void gateway method; if there is a return type, you will get the first (or random if there is any async downstream processing) and the others will be discarded.
I have a requirement where i need to make a rest call when it fails i have to retry 3 times and depending on the status code received i need perform different action, i couldn't find a proper spring integration dsl example. How to configure the error handler and retry
#Bean
public IntegrationFlow performCreate() {
return IntegrationFlows.from("createFlow")
.handle(Http.outboundGateway("http://localhost:8080/create")
.httpMethod(HttpMethod.GET)
.expectedResponseType(String.class)
.requestFactory(simpleClientHttpRequestFactory())
.errorHandler(??)
)
.log(LoggingHandler.Level.DEBUG, "response", m -> m.getPayload())
.log(LoggingHandler.Level.DEBUG, "response", m -> m.getHeaders())
.get();
}
private SimpleClientHttpRequestFactory simpleClientHttpRequestFactory() {
SimpleClientHttpRequestFactory simpleClientHttpRequestFactory = new SimpleClientHttpRequestFactory();
simpleClientHttpRequestFactory.setReadTimeout(5000);
simpleClientHttpRequestFactory.setConnectTimeout(5000);
return simpleClientHttpRequestFactory;
}
The Java DSL .handle() has a second argument - Consumer<GenericEndpointSpec<?>> and that one can be configured with the:
/**
* Configure a list of {#link Advice} objects to be applied, in nested order, to the
* endpoint's handler. The advice objects are applied only to the handler.
* #param advice the advice chain.
* #return the endpoint spec.
*/
public S advice(Advice... advice) {
One of those advises is in the Spring Integration box - RequestHandlerRetryAdvice: https://docs.spring.io/spring-integration/docs/5.0.4.RELEASE/reference/html/messaging-endpoints-chapter.html#retry-advice
https://docs.spring.io/spring-integration/docs/5.0.4.RELEASE/reference/html/java-dsl.html#java-dsl-endpoints
.handle(Http.outboundGateway("http://localhost:8080/create")
.httpMethod(HttpMethod.GET)
.expectedResponseType(String.class)
.requestFactory(simpleClientHttpRequestFactory()),
e -> e.advice(retryAdvice())
...
#Bean
public RequestHandlerRetryAdvice retryAdvice() {
RequestHandlerRetryAdvice requestHandlerRetryAdvice = new RequestHandlerRetryAdvice();
requestHandlerRetryAdvice.setRecoveryCallback(errorMessageSendingRecoverer());
return requestHandlerRetryAdvice;
}
#Bean
public ErrorMessageSendingRecoverer errorMessageSendingRecoverer() {
return new ErrorMessageSendingRecoverer(recoveryChannel())
}
#Bean
public MessageChannel recoveryChannel() {
return new DirectChannel();
}
#Bean
public IntegrationFlow handleRecovery() {
return IntegrationFlows.from("recoveryChannel")
.log(LoggingHandler.Level.ERROR, "error",
m -> m.getPayload())
.get();
}
Is it possible to register MessageSources at runtime with spring-integration-dsl?
In my case I want to create multiple FileReadingMessageSources (based on input from UI) and then send the payload to a specific channel/jms route (which is read from metadata or user input)
Another Question is, is it possible to dynamically register IntegrationFlows?
It's a bit tricky and requires some Spring infrastructure knowledges, but yes it is possible:
#Service
public static class MyService {
#Autowired
private AutowireCapableBeanFactory beanFactory;
#Autowired
#Qualifier("dynamicAdaptersResult")
PollableChannel dynamicAdaptersResult;
public void pollDirectories(File... directories) {
for (File directory : directories) {
StandardIntegrationFlow integrationFlow = IntegrationFlows
.from(s -> s.file(directory),
e -> e.poller(p -> p.fixedDelay(1000))
.id(directory.getName() + ".adapter"))
.transform(Transformers.fileToString(),
e -> e.id(directory.getName() + ".transformer"))
.channel(this.dynamicAdaptersResult)
.get();
this.beanFactory.initializeBean(integrationFlow, directory.getName());
this.beanFactory.getBean(directory.getName() + ".transformer", Lifecycle.class).start();
this.beanFactory.getBean(directory.getName() + ".adapter", Lifecycle.class).start();
}
}
}
Investigate this my sample, please, and let me know what isn't clear for you.
I am trying to implement the following using Spring Integration with DSL and lambda:
Given a message, send it to N consumers (via publish-subscribe). Wait for limited time and return all results that have arrived form consumers (<= N) during that interval.
Here is an example configuration I have so far:
#Configuration
#EnableIntegration
#IntegrationComponentScan
#ComponentScan
public class ExampleConfiguration {
#Bean(name = PollerMetadata.DEFAULT_POLLER)
public PollerMetadata poller() {
return Pollers.fixedRate(1000).maxMessagesPerPoll(1).get();
}
#Bean
public MessageChannel publishSubscribeChannel() {
return MessageChannels.publishSubscribe(splitterExecutorService()).applySequence(true).get();
}
#Bean
public ThreadPoolTaskExecutor splitterExecutorService() {
final ThreadPoolTaskExecutor executorService = new ThreadPoolTaskExecutor();
executorService.setCorePoolSize(3);
executorService.setMaxPoolSize(10);
return executorService;
}
#Bean
public DirectChannel errorChannel() {
return new DirectChannel();
}
#Bean
public DirectChannel requestChannel() {
return new DirectChannel();
}
#Bean
public DirectChannel channel1() {
return new DirectChannel();
}
#Bean
public DirectChannel channel2() {
return new DirectChannel();
}
#Bean
public DirectChannel collectorChannel() {
return new DirectChannel();
}
#Bean
public TransformerChannel1 transformerChannel1() {
return new TransformerChannel1();
}
#Bean
public TransformerChannel2 transformerChannel2() {
return new TransformerChannel2();
}
#Bean
public IntegrationFlow errorFlow() {
return IntegrationFlows.from(errorChannel())
.handle(m -> System.err.println("[" + Thread.currentThread().getName() + "] " + m.getPayload()))
.get();
}
#Bean
public IntegrationFlow channel1Flow() {
return IntegrationFlows.from(publishSubscribeChannel())
.transform("1: "::concat)
.transform(transformerChannel1())
.channel(collectorChannel())
.get();
}
#Bean
public IntegrationFlow channel2Flow() {
return IntegrationFlows.from(publishSubscribeChannel())
.transform("2: "::concat)
.transform(transformerChannel2())
.channel(collectorChannel())
.get();
}
#Bean
public IntegrationFlow splitterFlow() {
return IntegrationFlows.from(requestChannel())
.channel(publishSubscribeChannel())
.get();
}
#Bean
public IntegrationFlow collectorFlow() {
return IntegrationFlows.from(collectorChannel())
.resequence(r -> r.releasePartialSequences(true),
null)
.aggregate(a ->
a.sendPartialResultOnExpiry(true)
.groupTimeout(500)
, null)
.get();
}
}
TransformerChannel1 and TransformerChannel2 are sample consumers and have been implemented with just a sleep to emulate delay.
The message flow is:
splitterFlow -> channel1Flow \
-> channel2Flow / -> collectorFlow
Everything seem to work as expected, but I see warnings like:
Reply message received but the receiving thread has already received a reply
which is to be expected, given that partial result was returned.
Questions:
Overall, is this a good approach?
What is the right way to gracefully service or discard those delayed messages?
How to deal with exceptions? Ideally I'd like to send them to errorChannel, but am not sure where to specify this.
Yes, the solution looks good. I guess it fits for the Scatter-Gather pattern. The implementation is provided since version 4.1.
From other side there is on more option for the aggregator since that version, too - expire-groups-upon-timeout, which is true for the aggregator by default. With this option as false you will be able to achieve your requirement to discard all those late messages. Unfortunately DSL doesn't support it yet. Hence it won't help even if you upgrade your project to use Spring Integration 4.1.
Another option for those "Reply message received but the receiving thread has already received a reply" is on the spring.integraton.messagingTemplate.throwExceptionOnLateReply = true option using spring.integration.properties file within the META-INF of one of jar.
Anyway I think that Scatter-Gather is the best solution for you use-case.
You can find here how to configure it from JavaConfig.
UPDATE
What about exceptions and error channel?
Since you get deal already with the throwExceptionOnLateReply I guess you send a message to the requestChannel via #MessagingGateway. The last one has errorChannel option. From other side the PublishSubscribeChannel has errorHandler option, for which you can use MessagePublishingErrorHandler with your errorChannel as a default one.
BTW, don't forget that Framework provides errorChannel bean and the endpoint on it for the LoggingHandler. So, think, please, if you really need to override that stuff. The default errorChannel is PublishSubscribeChannel, hence you can simply add your own subscribers to it.