What is the proper way of using bridge endpointConfigurer? - spring-integration

I'm looking for a component about how to connect different integration flows by root flow. I've seen this tutorial(see 5.2. Bridge); it has one main root flow, then two different flows. I've tried this in my application, but it didn't work without putting PollerMetadata.DEFAULT_POLLER. Gives an error: no default poller is available within the context. When I add PollerMetadata.DEFAULT_POLLER, times in bridge endpointConfigurer is not working as expected. Probably, It clashes by default poller configuration.
In short, how can I connect two different flows by one main root? But, different flows has to work in different times.
I don't know that I'm using right component or not, any help will be appreciated. Thank you.
In addition, I've seen this question which is kind a similar. It could help to understand my question.
UPDATE
#Bean(name = PollerMetadata.DEFAULT_POLLER)
public PollerMetadata poller() {
return Pollers.fixedRate(60, TimeUnit.SECONDS, 10).get();
}
#Bean
public IntegrationFlow fileReader() {
return IntegrationFlows
.from(sourceDirectory())
.split()
.publishSubscribeChannel(c -> c
.subscribe("fileWriter"))
.publishSubscribeChannel(c -> c
.subscribe("anotherFileWriter"))
.get();
}
#Bean
public IntegrationFlow fileWriter() {
return IntegrationFlows
.from("fileWriter")
.bridge(e -> e.poller(10, TimeUnit.SECONDS, 5))
.handle()
.get();
}
#Bean
public IntegrationFlow anotherFileWriter() {
return IntegrationFlows
.from("anotherFileWriter")
.bridge(e -> e.poller(20, TimeUnit.SECONDS, 5))
.handle()
.get();
}

You should show your code when asking questions like this.
It's not entirely clear what you mean by
In short, how can I connect two different flows by one main root?
If they both consume from the same pollable channel each message will only go to one or the other flow.
To specifically answer your question, the example you pointed to shows how to configure a specific poller on the endpoint...
>.bridge(e -> e.poller(Pollers.fixedRate(1, TimeUnit.SECONDS, 20)))
You only need a default poller if you omit the poller on the endpoint.

Related

How to make polling multi-thread on spring integration?

I have the following Spring Integration flow:
It gathers records from one database, converts to json and sends to another database.
The idea is to have 10 pollers (channel0 to 9). Each one is a pollingFlowChanN Bean. But I suspect they are sharing the same thread.
How to I make the polling multi-thread in this scenario?
private IntegrationFlow getChannelPoller(final int channel, final int pollSize, final long delay) {
return IntegrationFlows.from(jdbcMessageSource(channel, pollSize), c -> c.poller(Pollers.fixedDelay(delay)
.transactional(transactionManager)))
.split()
.handle(intControleToJson())
.handle(pgsqlSink)
.get();
}
#Bean
public IntegrationFlow pollingFlowChan0() {
return getChannelPoller(0, properties.getChan0PollSize(), properties.getChan0Delay());
}
#Bean
public IntegrationFlow pollingFlowChan1() {
return getChannelPoller(1, properties.getChan1PollSize(), properties.getChan1Delay());
}
....
I assume you use the latest Spring Boot, which have a TaskScheduler auto-configured with one thread: https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#features.spring-integration. That's the best guess why your tasks use the same thread.
See also answer here: Why does the SFTP Outbound Gateway not start working as soon as I start its Integration Flow?

Spring Integration Flow DSL with SQS and Reactive

How can I setup a reactive flow using DSL for the following steps:
Receive an SQS Message using SqsMessageDrivenChannelAdapter
Validate the Json message [JsonSchemaValidator class with validate method]
Transform the json to objects
Pass the objects to a service activator (BusinessService : business logic, state machine)
Persist the Objects R2DBC outbound adapter
I was looking at this : https://github.com/spring-projects/spring-integration/blob/master/spring-integration-core/src/test/java/org/springframework/integration/dsl/reactivestreams/ReactiveStreamsTests.java
In the above example, there are dedicated flows created that return a Publisher and in the tests the Publishers are subscribed. However, my flow will be triggered when SqsMessageDrivenChannelAdapter brings in a message into a channel.
How to achieve a reactive flow configuration, for the scenario above steps 1 to 5?
Update : Sample code added
#Bean
public IntegrationFlow importFlow() {
IntegrationFlows.from(sqsInboundChannel())
.handle((payload, messageHeaders) -> jsonSchemaValidator.validate(payload.toString()))
.transform(Transformers.fromJson(Entity.class))
.handle((payload, messageHeaders) ->businessService.process((Entity) payload))
.handle(
Jpa.outboundAdapter(this.entityManagerFactory)
.entityClass(Entity)
.persistMode(PersistMode.PERSIST),
ConsumerEndpointSpec::transactional)
.get();
}
#Bean
public MessageProducer sqsMessageDrivenChannelAdapter() {
SqsMessageDrivenChannelAdapter sqsMessageDrivenChannelAdapter =
new SqsMessageDrivenChannelAdapter(asyncSqsClient, queueName);
sqsMessageDrivenChannelAdapter.setAutoStartup(true);
sqsMessageDrivenChannelAdapter.setOutputChannel(sqsInboundChannel());
return sqsMessageDrivenChannelAdapter;
}
#Bean
public MessageChannel sqsInboundChannel() {
return MessageChannels.flux().get();
}
Update 2 : Moved JPA to a diff thread using executor channel
#Bean
public IntegrationFlow importFlow() {
IntegrationFlows.from(sqsInboundChannel())
.handle((payload, messageHeaders) -> jsonSchemaValidator.validate(payload.toString()))
.transform(Transformers.fromJson(Entity.class))
.handle((payload, messageHeaders) ->businessService.process((Entity) payload))
.channel(persistChannel())
.handle(
Jpa.outboundAdapter(this.entityManagerFactory)
.entityClass(Entity)
.persistMode(PersistMode.PERSIST),
ConsumerEndpointSpec::transactional)
.get();
}
#Bean
public MessageProducer sqsMessageDrivenChannelAdapter() {
SqsMessageDrivenChannelAdapter sqsMessageDrivenChannelAdapter =
new SqsMessageDrivenChannelAdapter(asyncSqsClient, queueName);
sqsMessageDrivenChannelAdapter.setAutoStartup(true);
sqsMessageDrivenChannelAdapter.setOutputChannel(sqsInboundChannel());
return sqsMessageDrivenChannelAdapter;
}
#Bean
public MessageChannel sqsInboundChannel() {
return MessageChannels.flux().get();
}
#Bean
public MessageChannel persistChannel() {
return MessageChannels.executor(Executors.newCachedThreadPool()).get();
}
You probably need to make yourself more familiar with what we have so far for Reactive Streams in Spring Integration: https://docs.spring.io/spring-integration/docs/current/reference/html/reactive-streams.html#reactive-streams
The sample you show with that test class is fully not relevant to your use case. In that test we try to cover some API we expose in Spring Integration, kinda unit tests. It has nothing to do with the whole flow.
Your use-case is really just a full black box flow starting with SQS listener and ending in the R2DBC. Therefore there is no point in your flow to try to convert part of it into the Publisher and then bring it back to another part of the flow: you are not going to track some how and subscribe to that Publisher yourself.
You may consider to place a FluxMessageChannel in between endpoints in your flow, but it still does not make sense for your use-case. It won't be fully reactive as you expect just because a org.springframework.cloud.aws.messaging.listener.SimpleMessageListenerContainer is not blocking on the consumer thread to be ready for a back-pressure from downstream.
The only really reactive part of your flow is that R2DBC outbound channel adapter, but probably it does not bring you too much value because the source of data is not reactive.
As I said: you can try to place a channel(channels -> channels.flux()) just after an SqsMessageDrivenChannelAdapter definition to start a reactive flow from that point. At the same time you should try to set a maxNumberOfMessages to 1 to try to make it waiting for a free space in before pulling the next mesasge from SQS.

How to register the integration flows in runtime?

I'm building a micro service for multiple properties. So, each property has different configuration. To do that, I've implemented something like this;
#Autowired
IntegrationFlowContext flowContext;
#Bean
public void setFlowContext() {
List<Login> loginList = DAO.getLoginList(); // a web service
loginList.forEach(e -> {
IntegrationFlow flow = IntegrationFlows.from(() -> e, c -> c.poller(Pollers.fixedRate(e.getPeriod(), TimeUnit.SECONDS, 5)))
.channel("X_CHANNEL")
.get();
flowContext.registration(flow).register();
});
}
By this implementation, I'm getting the loginList before application started. So, after application is started, I'm not able to get loginList from web service since there is no poller config. The problem is loginList could change; new logins credentials could be added or deleted. Therefore, I want to implement something will work X time period to get loginList from web service, then, by loginList I need to register the flows that are created for each loginList. To achieve, I've implemented something like this;
#Bean
public IntegrationFlow setFlowContext() {
return IntegrationFlows
.from(this::getSpecification, p -> p.poller(Pollers.fixedRate(X))) // the specification is constant.
.transform(payload -> DAO.getLoginList(payload))
.split()
.<Login>handle((payload, header) -> {
IntegrationFlow flow = IntegrationFlows.from(() -> payload, c -> c.poller(Pollers.fixedRate(payload.getPeriod(), TimeUnit.SECONDS, 5)))
.channel("X_CHANNEL")
.get();
flowContext.registration(flow).register().start();
return null;
})
.get();
}
Basically, I've used start() method, but this is not working as aspected. See this;
flowContext.registration(flow).register().start();
Lastly, I've read the Dynamic and Runtime Integration Flows, but still couldn't implement this feature.
Dynamic flow registration cannot be used within a #Bean definition.
It is designed to be used at runtime AFTER the application context is fully initialized.

Is it possible to configure poller for each entity from a data source?

I'm developing a multi property micro service by spring integration. I'm getting each property's login credentials from database like LOGIN table. LOGIN table has these fields; LOGIN.username, LOGIN.pass and LOGIN.period(poller's period). If I want to make work the micro service with different poller configurations based on LOGIN.period field, how can I do that?
#Bean
public IntegrationFlow start() {
return IntegrationFlows
.from(() -> DAO.getLoginList()) // from a web service.
.split() // splits the each login credentials for each property.
.channel("X_CHANNEL") // subscribes to a channel todo business logic.
.get();
}
Is it possible to implement a component to make work flow in different poller configurations based on the LOGIN.period value from database?
Based on the Artem Bilan's answer, I've implement the IntegrationFlowContext and IntegrationFlow instances;
#Autowired
IntegrationFlowContext flowContext;
#Bean
public void setFlowContext() {
List<Login> loginList = DAO.getLoginList(); // a web service
loginList.forEach(e -> {
IntegrationFlow flow = IntegrationFlows.from(() -> e, c -> c.poller(Pollers.fixedRate(e.getPeriod(), TimeUnit.SECONDS, 5)))
.channel("X_CHANNEL")
.get();
flowContext.registration(flow).register();
});
}
Show, please, how you get that info from the data base.
But if your point that you may have several records in DB and you want to have several pollers for all of them, then you need to take a look dynamic flow registrations: https://docs.spring.io/spring-integration/docs/5.3.2.RELEASE/reference/html/dsl.html#java-dsl-runtime-flows
So, you read data from the DB, create IntegrationFlow in the loop for every record and configure their pollers according the data from the record.

How can I specify a Poller for a queue channel using java-dsl?

I want to consume messages from a Queue channel using java-dsl, but Integrations.from doesn't have a signature allowing me to specify a Poller.
How can I achieve this?
Ex.:
#Bean
IntegrationFlow flow() {
return IntegrationFlows.from(this.channel())
.handle(...)
.get();
}
#Bean
MessageChannel channel() {
return MessageChannels.queue().get();
}
Well, actually it is an endpoint responsibility to provide poller properties.
If you are familiar with an XML configuration you should remember that to poll from <queue> we should configure <poller> sub-element for <service-activator> etc.
The same approach is applied in Java DSL as well. The next endpoint definition should be with desired poller:
IntegrationFlows.from(this.channel())
.handle(..., e -> e.poller(Pollers...))
.get();
I've had trouble for some reason setting a poller on the endpoint definition as Artem described - for some reason it gets ignored. You can always set a default poller. This has worked for me:
#Bean(name = PollerMetadata.DEFAULT_POLLER)
public PollerMetadata poller() {
return Pollers.fixedRate(500).get();
}

Resources