I have the following Spring Integration flow:
It gathers records from one database, converts to json and sends to another database.
The idea is to have 10 pollers (channel0 to 9). Each one is a pollingFlowChanN Bean. But I suspect they are sharing the same thread.
How to I make the polling multi-thread in this scenario?
private IntegrationFlow getChannelPoller(final int channel, final int pollSize, final long delay) {
return IntegrationFlows.from(jdbcMessageSource(channel, pollSize), c -> c.poller(Pollers.fixedDelay(delay)
.transactional(transactionManager)))
.split()
.handle(intControleToJson())
.handle(pgsqlSink)
.get();
}
#Bean
public IntegrationFlow pollingFlowChan0() {
return getChannelPoller(0, properties.getChan0PollSize(), properties.getChan0Delay());
}
#Bean
public IntegrationFlow pollingFlowChan1() {
return getChannelPoller(1, properties.getChan1PollSize(), properties.getChan1Delay());
}
....
I assume you use the latest Spring Boot, which have a TaskScheduler auto-configured with one thread: https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#features.spring-integration. That's the best guess why your tasks use the same thread.
See also answer here: Why does the SFTP Outbound Gateway not start working as soon as I start its Integration Flow?
Related
My Message handler for publishing messages to the kinesis stream is as follows
public MessageHandler kinesisMessageHandler(final AmazonKinesisAsync amazonKinesis,
#Qualifier("successChannel") MessageChannel successChannel,
#Qualifier("errorChannel") MessageChannel errorChannel) {
KinesisMessageHandler kinesisMessageHandler = new KinesisMessageHandler(amazonKinesis);
kinesisMessageHandler.setSync(false);
kinesisMessageHandler.setOutputChannel(successChannel);
kinesisMessageHandler.setFailureChannel(errorChannel);
return kinesisMessageHandler;
}
#Bean(name = "errorChannel")
public MessageChannel errorChannel() {
return MessageChannels.direct().get();
}
#Bean(name = "successChannel")
public MessageChannel successChannel() {
return MessageChannels.direct().get();
}
The setSync flag is set as false so that the messages are getting processed asynchronously.Also, I have created separate IntegrationFlow to receive and process Kinesis response from the success & error channel.
public IntegrationFlow successMessageIntegrationFlow(MessageChannel successChannel,
MessageChannel inboundKinesisMessageChannel,
MessageReceiverServiceActivator kinesisMessageReceiverServiceActivator) {
return IntegrationFlows.from(successChannel).channel(inboundKinesisMessageChannel)
.handle(kinesisMessageReceiverServiceActivator, "receiveMessage").get();
}
#Bean
public IntegrationFlow errorMessageIntegrationFlow(MessageChannel errorChannel,
MessageChannel inboundKinesisErrorChannel,
MessageReceiverServiceActivator kinesisErrorReceiverServiceActivator
) {
return IntegrationFlows.from(errorChannel).channel(inboundKinesisErrorChannel)
.handle(kinesisErrorReceiverServiceActivator, "receiveMessage").get();
}
I wanted to know if you see any issues in using Direct Channel to receive success & error responses from Kinesis and processing it using an IntegrationFlow. As far as I know, with Direct Channel a producer is a blocker during send until the consumer finishes its work and returns management to the producer caller back. Is it a correct assumption that here the producer is executed in a different set of thread pools by the AmazonKinesisAsyncClient and the producer will not wait for the IntegrationFlow to process the messages? Let me know If I need to implement it differently
Your assumption about blocking is correct: the control does not come back to the producing thread. So, if have a limited number of threads in that Kinesis client, you need to be sure that you free them as soon as possible. You might consider to have those callbacks in the queue channel instead. They are asynchronous anyway, but won’t hold Kinesis client if that.
You still have a flaw in your flows: .channel(inboundKinesisMessageChannel) . That means the same channel in the middle if two different flows . And if it is a direct one , then you end up with round robin distribution. I would just remove it altogether .
How can I setup a reactive flow using DSL for the following steps:
Receive an SQS Message using SqsMessageDrivenChannelAdapter
Validate the Json message [JsonSchemaValidator class with validate method]
Transform the json to objects
Pass the objects to a service activator (BusinessService : business logic, state machine)
Persist the Objects R2DBC outbound adapter
I was looking at this : https://github.com/spring-projects/spring-integration/blob/master/spring-integration-core/src/test/java/org/springframework/integration/dsl/reactivestreams/ReactiveStreamsTests.java
In the above example, there are dedicated flows created that return a Publisher and in the tests the Publishers are subscribed. However, my flow will be triggered when SqsMessageDrivenChannelAdapter brings in a message into a channel.
How to achieve a reactive flow configuration, for the scenario above steps 1 to 5?
Update : Sample code added
#Bean
public IntegrationFlow importFlow() {
IntegrationFlows.from(sqsInboundChannel())
.handle((payload, messageHeaders) -> jsonSchemaValidator.validate(payload.toString()))
.transform(Transformers.fromJson(Entity.class))
.handle((payload, messageHeaders) ->businessService.process((Entity) payload))
.handle(
Jpa.outboundAdapter(this.entityManagerFactory)
.entityClass(Entity)
.persistMode(PersistMode.PERSIST),
ConsumerEndpointSpec::transactional)
.get();
}
#Bean
public MessageProducer sqsMessageDrivenChannelAdapter() {
SqsMessageDrivenChannelAdapter sqsMessageDrivenChannelAdapter =
new SqsMessageDrivenChannelAdapter(asyncSqsClient, queueName);
sqsMessageDrivenChannelAdapter.setAutoStartup(true);
sqsMessageDrivenChannelAdapter.setOutputChannel(sqsInboundChannel());
return sqsMessageDrivenChannelAdapter;
}
#Bean
public MessageChannel sqsInboundChannel() {
return MessageChannels.flux().get();
}
Update 2 : Moved JPA to a diff thread using executor channel
#Bean
public IntegrationFlow importFlow() {
IntegrationFlows.from(sqsInboundChannel())
.handle((payload, messageHeaders) -> jsonSchemaValidator.validate(payload.toString()))
.transform(Transformers.fromJson(Entity.class))
.handle((payload, messageHeaders) ->businessService.process((Entity) payload))
.channel(persistChannel())
.handle(
Jpa.outboundAdapter(this.entityManagerFactory)
.entityClass(Entity)
.persistMode(PersistMode.PERSIST),
ConsumerEndpointSpec::transactional)
.get();
}
#Bean
public MessageProducer sqsMessageDrivenChannelAdapter() {
SqsMessageDrivenChannelAdapter sqsMessageDrivenChannelAdapter =
new SqsMessageDrivenChannelAdapter(asyncSqsClient, queueName);
sqsMessageDrivenChannelAdapter.setAutoStartup(true);
sqsMessageDrivenChannelAdapter.setOutputChannel(sqsInboundChannel());
return sqsMessageDrivenChannelAdapter;
}
#Bean
public MessageChannel sqsInboundChannel() {
return MessageChannels.flux().get();
}
#Bean
public MessageChannel persistChannel() {
return MessageChannels.executor(Executors.newCachedThreadPool()).get();
}
You probably need to make yourself more familiar with what we have so far for Reactive Streams in Spring Integration: https://docs.spring.io/spring-integration/docs/current/reference/html/reactive-streams.html#reactive-streams
The sample you show with that test class is fully not relevant to your use case. In that test we try to cover some API we expose in Spring Integration, kinda unit tests. It has nothing to do with the whole flow.
Your use-case is really just a full black box flow starting with SQS listener and ending in the R2DBC. Therefore there is no point in your flow to try to convert part of it into the Publisher and then bring it back to another part of the flow: you are not going to track some how and subscribe to that Publisher yourself.
You may consider to place a FluxMessageChannel in between endpoints in your flow, but it still does not make sense for your use-case. It won't be fully reactive as you expect just because a org.springframework.cloud.aws.messaging.listener.SimpleMessageListenerContainer is not blocking on the consumer thread to be ready for a back-pressure from downstream.
The only really reactive part of your flow is that R2DBC outbound channel adapter, but probably it does not bring you too much value because the source of data is not reactive.
As I said: you can try to place a channel(channels -> channels.flux()) just after an SqsMessageDrivenChannelAdapter definition to start a reactive flow from that point. At the same time you should try to set a maxNumberOfMessages to 1 to try to make it waiting for a free space in before pulling the next mesasge from SQS.
I'm developing a multi property micro service by spring integration. I'm getting each property's login credentials from database like LOGIN table. LOGIN table has these fields; LOGIN.username, LOGIN.pass and LOGIN.period(poller's period). If I want to make work the micro service with different poller configurations based on LOGIN.period field, how can I do that?
#Bean
public IntegrationFlow start() {
return IntegrationFlows
.from(() -> DAO.getLoginList()) // from a web service.
.split() // splits the each login credentials for each property.
.channel("X_CHANNEL") // subscribes to a channel todo business logic.
.get();
}
Is it possible to implement a component to make work flow in different poller configurations based on the LOGIN.period value from database?
Based on the Artem Bilan's answer, I've implement the IntegrationFlowContext and IntegrationFlow instances;
#Autowired
IntegrationFlowContext flowContext;
#Bean
public void setFlowContext() {
List<Login> loginList = DAO.getLoginList(); // a web service
loginList.forEach(e -> {
IntegrationFlow flow = IntegrationFlows.from(() -> e, c -> c.poller(Pollers.fixedRate(e.getPeriod(), TimeUnit.SECONDS, 5)))
.channel("X_CHANNEL")
.get();
flowContext.registration(flow).register();
});
}
Show, please, how you get that info from the data base.
But if your point that you may have several records in DB and you want to have several pollers for all of them, then you need to take a look dynamic flow registrations: https://docs.spring.io/spring-integration/docs/5.3.2.RELEASE/reference/html/dsl.html#java-dsl-runtime-flows
So, you read data from the DB, create IntegrationFlow in the loop for every record and configure their pollers according the data from the record.
I'm looking for a component about how to connect different integration flows by root flow. I've seen this tutorial(see 5.2. Bridge); it has one main root flow, then two different flows. I've tried this in my application, but it didn't work without putting PollerMetadata.DEFAULT_POLLER. Gives an error: no default poller is available within the context. When I add PollerMetadata.DEFAULT_POLLER, times in bridge endpointConfigurer is not working as expected. Probably, It clashes by default poller configuration.
In short, how can I connect two different flows by one main root? But, different flows has to work in different times.
I don't know that I'm using right component or not, any help will be appreciated. Thank you.
In addition, I've seen this question which is kind a similar. It could help to understand my question.
UPDATE
#Bean(name = PollerMetadata.DEFAULT_POLLER)
public PollerMetadata poller() {
return Pollers.fixedRate(60, TimeUnit.SECONDS, 10).get();
}
#Bean
public IntegrationFlow fileReader() {
return IntegrationFlows
.from(sourceDirectory())
.split()
.publishSubscribeChannel(c -> c
.subscribe("fileWriter"))
.publishSubscribeChannel(c -> c
.subscribe("anotherFileWriter"))
.get();
}
#Bean
public IntegrationFlow fileWriter() {
return IntegrationFlows
.from("fileWriter")
.bridge(e -> e.poller(10, TimeUnit.SECONDS, 5))
.handle()
.get();
}
#Bean
public IntegrationFlow anotherFileWriter() {
return IntegrationFlows
.from("anotherFileWriter")
.bridge(e -> e.poller(20, TimeUnit.SECONDS, 5))
.handle()
.get();
}
You should show your code when asking questions like this.
It's not entirely clear what you mean by
In short, how can I connect two different flows by one main root?
If they both consume from the same pollable channel each message will only go to one or the other flow.
To specifically answer your question, the example you pointed to shows how to configure a specific poller on the endpoint...
>.bridge(e -> e.poller(Pollers.fixedRate(1, TimeUnit.SECONDS, 20)))
You only need a default poller if you omit the poller on the endpoint.
I have a JdbcPollingChannelAdapter which reads data via JDBC. I want to make it poll manually (using a commandChannel). It should never poll automatically, and it should run immediately when I trigger a manual poll.
Below I am using a poller which runs every 24 hours to get the channel running at all. I cannot use a cronExpression that never fires as in Quartz: Cron expression that will never execute since Pollers.cronExpression() takes no year.
#Bean
public MessageSource<Object> jdbcMessageSource() {
return new JdbcPollingChannelAdapter(this.dataSource, "SELECT...");
}
#Bean
public IntegrationFlow jdbcFlow() {
return IntegrationFlows
.from(jdbcMessageSource(),
spec -> spec.poller(
Pollers.fixedRate(24, TimeUnit.HOURS)))
.handle(System.out::println)
.get();
}
Well, you go right way about JdbcPollingChannelAdapter and commandChannel, but you don't have configure SourcePollingChannelAdapter as you do with that IntegrationFlows.from(jdbcMessageSource().
What you need is really the jdbcMessageSource(), but to poll it manually you should configure command-based flow:
#Bean
public IntegrationFlow jdbcFlow() {
return IntegrationFlows
.from("commandChannel")
.handle(jdbcMessageSource(), "receive")
.handle(System.out::println)
.get();
}
Exactly that receive() is called from the SourcePollingChannelAdapter on the timing basis.