Spring Integration: call service when flow completes - spring-integration

I have a Spring Itegration (5.0.2) flow that reads data from an HTTP endpoint and publish Kafka messages using the (split) data from the HTTP response.
I would like to execute a "final" action before the flow completes, some sort of "try, catch, finally" but I'm not sure what is the best way to achieve this.
This is my code:
#Bean
public IntegrationFlow startFlow() {
return IntegrationFlows.from(() -> new GenericMessage<>(""),
c -> c.id("flow1")
.poller(
Pollers.fixedDelay(period, TimeUnit.MILLISECONDS, initialDelay).taskExecutor(taskExecutor)))
.handle(Http.outboundGateway("http://....)
.charset("UTF-8")
.expectedResponseType(String.class)
.httpMethod(HttpMethod.GET))
.transform(new transformer())
.split()
.channel(CHANNEL_1)
.controlBus()
.get();
}
#Bean
public IntegrationFlow toKafka(KafkaTemplate<?, ?> kafkaTemplate) {
return IntegrationFlows.from(CHANNEL_1)
.handle(Kafka.outboundChannelAdapter(kafkaTemplate)
// KAFKA SPECIFIC
.get();
}
Essentially, when all the messages have been sent (note that I'm using .split), I need to call a Spring bean to update some data.
Thanks

If you talk about "when all the messages" after splitter, then you need to take a look into an .aggregate(): https://docs.spring.io/spring-integration/docs/5.0.3.RELEASE/reference/html/messaging-routing-chapter.html#aggregator.
The splitter populates special sequence details headers into each splitted item and an aggregator is able to gather them to a single entity by default using those headers from received message.
Since you talk about the process after sending to Kafka, you should make your CHANNEL_1 as a PublishSubscribeChannel and have the mentioned .aggregate() as the last subscriber to this channel. The service to update some data should be already after this aggregator.

Related

Use of ConsumerEndpointSpec.transactional() in Spring Integration DSL

I have a question about the behavior of transactional() in the following example:
#Bean
IntegrationFlow myFlow(
EntityManagerFactory entityManagerFactory,
TransactionManager transactionManager
) {
return IntegrationFlows.from(MY_CHANNEL)
.routeToRecipients(route -> route
.recipientFlow(flow -> flow
.handle(Jpa.updatingGateway(entityManagerFactory)
.namedQuery(DELETE_EVERYTHING)))
.recipientFlow(flow -> flow
.handle(Jpa.updatingGateway(entityManagerFactory)))
.transactional(transactionManager))
.get();
}
The idea is that I'm first deleting the contents of a database table, and immediately after I'm filling that same table with new data. Will .transactional() in this example make sure that the first step (deletion) is only committed to the DB if the second step (inserting new data) is successful? What part of the documentation can I refer to for this behavior?
You need to read this documentation: https://docs.spring.io/spring-integration/docs/current/reference/html/transactions.html#transactions.
You assumption is correct as long as both recipients are performed on the same thread and, therefore, sequentially.
The TransactionInterceptor is going to be applied for the RecipientListRouter.handleMessage(). And that's where that "auction" for recipient happens.
Do you have any problem with that configuration since you have came to us with such a question?

Spring integration: How to implement a JDBCOutboundGateway in the middle of a MessageChain?

This appears, to me, to be a simple problem that is probably replicated all over the place. A very basic application of the MessageHandlerChain, probably using nothing more than out of the box functionality.
Conceptually, what I need is this:
(1) Polled JDBC reader (sets parameters for integration pass)
|
V
(2) JDBC Reader (uses input from (1) to fetch data to feed through channel
|
V
(3) JDBC writer (writes data fetched by (2) to target)
|
V
(4) JDBC writer (writes additional data from the original parameters fetched in (1))
What I think I need is
Flow:
From: JdbcPollingChannelAdapter (setup adapter)
Handler: messageHandlerChain
Handlers (
JdbcPollingChannelAdapter (inbound adapter)
JdbcOutboundGateway (outbound adapter)
JdbcOutboundGateway (cleanup gateway)
)
The JdbcPollingChannelAdapter does not implement the MessageHandler API, so I am at a loss how to read the actual data based on the setup step.
Since the JdbcOutboundGateway does not implement the MessageProducer API, I am at a bit of a loss as to what I need to use for the outbound adapter.
Are there OOB classes I should be using? Or do I need to somehow wrap the two adapters in BridgeHandlers to make this work?
Thanks in advance
EDIT (2)
Additional configuration problem
The setup adapter is pulling a single row back with two timestamp columns. They are being processed correctly by the "enrich headers" piece.
However, when the inbound adapter is executing, the framework is passing in java.lang.Object as parameters. Not String, not Timestamp, but an actual java.lang.Object as in new Object ().
It is passing the correct number of objects, but the content and datatypes are lost. Am I correct that the ExpressionEvaluatingSqlParameterSourceFactory needs to be configured?
Message:
GenericMessage [payload=[{startTime=2020-11-18 18:01:34.90944, endTime=2020-11-18 18:01:34.90944}], headers={startTime=2020-11-18 18:01:34.90944, id=835edf42-6f69-226a-18f4-ade030c16618, timestamp=1605897225384}]
SQL in the JdbcOutboundGateway:
Select t.*, w.operation as "ops" from ADDRESS t
Inner join TT_ADDRESS w
on (t.ADDRESSID = w.ADDRESSID)
And (w.LASTUPDATESTAMP >= :payload.from[0].get("startTime") and w.LASTUPDATESTAMP <= :payload.from[0].get("endTime") )
Edit: added solution java DSL configuration
private JdbcPollingChannelAdapter setupAdapter; // select only
private JdbcOutboundGateway inboundAdapter; // select only
private JdbcOutboundGateway insertUpdateAdapter; // update only
private JdbcOutboundGateway deleteAdapter; // update only
private JdbcMessageHandler cleanupAdapter; // update only
setFlow(IntegrationFlows
.from(setupAdapter, c -> c.poller(Pollers.fixedRate(1000L, TimeUnit.MILLISECONDS).maxMessagesPerPoll(1)))
.enrichHeaders(h -> h.headerExpression("ALC_startTime", "payload.from[0].get(\"ALC_startTime\")")
.headerExpression("ALC_endTime", "payload.from[0].get(\"ALC_endTime\")"))
.handle(inboundAdapter)
.enrichHeaders(h -> h.headerExpression("ALC_operation", "payload.from[0].get(\"ALC_operation\")"))
.handle(insertUpdateAdapter)
.handle(deleteAdapter)
.handle(cleanupAdapter)
.get());
flowContext.registration(flow).id(this.getId().toString()).register();
If you would like to carry the original arguments down to the last gateway in your flow, you need to store those arguments in the headers since after every step the payload of reply message is going to be different and you won't have original setup data over there any more. That's first.
Second: if you deal with IntegrationFlow and Java DSL, you don't need to worry about messageHandlerChain since conceptually the IntegrationFlow is a chain by itself but much more advance.
I'm not sure why you need to use a JdbcPollingChannelAdapter to request data on demand according incoming message from the source in the beginning of your flow.
You definitely still need to use a JdbcOutboundGateway for just SELECT mode. The updateQuery is optional, so that gateway is just going to perform SELECT and return a data for you in a payload of the reply message.
If you two next steps are just "write" and you don't care about the result, you probably can just take a look into a PublishSubscribeChannel and two JdbcMessageHandler as subscribers to it. Without a provided Executor for the PublishSubscribeChannel they are going to be executed one-by-one.

Is there a way to flush correlation IDs or channels after Aggregation?

My Spring Integration (with a HTTP inbound gateway) application has the following flow:
The end-client POSTs an XML input message with a UUID for unique identification (let us call this XML, "whole") and other business elements
A Splitter component breaks it down into smaller XMLs with each "part" XML having the same UUID as the "whole" XML
The Aggregator aggregates the parts with some business logic along with a #CorrelationStrategy that uses the UUID of the incoming "part" messages
I am trying to explore a way to clear/flush the Aggregator's inputChannel or reset the Correlation Strategy after the correlation operation to enable different POST requests to reuse the UUIDs in the incoming "whole" XML input messages.
At this point, what is happening is that once a "whole" XML input message is processed successfully, if another HTTP POST sends a "whole" XML input message with the same UUID, the flow is hanging, probably because it is unable to understand what to do with the #CorrelationStrategy or is in a stale state in terms of UUIDs.
Frankly, the very definition of UUID means it has to be uniquely generated for every HTTP request but just wanted to accommodate a case where the end-client ends up sending the same UUID for every POST.
This is how my Aggregator class and its methods look:
#MessageEndpoint
public class ProductAggregator {
...
...
#Aggregator(inputChannel="compositeMessagesForAggregation",
outputChannel="upstreamResponses")
public Message<BusinessComposite>
generateAggregatedResponse(List<Message<BusinessComposite>>
listOfCompositeMessagesForAggregation) {
...
...
}
#CorrelationStrategy
public UUID correlateByUUID(Message<BusinessComposite>
compositeMessageForAggregation) {
...
...
}
Thanks in advance!
Sincerely,
Bharath
Set expireGroupsUponCompletion and expireGroupsUponExpiry to true on the aggregator to reuse the same correlation id; otherwise "late arriving" messages are discarded.

Spring Integration customise Feed Inbound Channel Adapter

I am using Spring Integration and the Feed Inbound Channel Adapter to process news RSS feeds (which I think is fantastic :). I would also like to consume some additional news feeds which are available by an API into the same channel. The API is just a HTTP endpoint which returns a list of news articles in JSON. The fields are very similar to RSS i.e. there is a title, description, published date which could be mapped to the SyndEntry object.
Ideally I want to use the same functionality available in feed inbound channel adapter which deals with duplicate entries etc. Is it possible to customise Feed Inbound Channel Adaptor to process and map the JSON?
Any sample code or pointers would be really helpful.
Well, no. FeedEntryMessageSource is fully based on the Rome Tools which deals only with the XML model.
I'm afraid you have to create your own component which will produce SyndEntry instances for those JSON records. That can really be something like HttpRequestExecutingMessageHandler, based on the RestTemplate with the MappingJackson2HttpMessageConverter which is present by default anyway.
You can try to configure HttpRequestExecutingMessageHandler to the setExpectedResponseType(SyndFeedImpl.class) and expect to have application/json content-type in the response.
To achieve the "deals with duplicate entries" you can consider to use Idempotent Receiver pattern afterwards. Where the MessageSelector should be based on the MetadaStore and really preform logic similar to the one in the FeedEntryMessageSource:
if (lastModifiedDate != null) {
this.lastTime = lastModifiedDate.getTime();
}
else {
this.lastTime += 1; //NOSONAR - single poller thread
}
this.metadataStore.put(this.metadataKey, this.lastTime + "");
...
if ((entryDate != null && entryDate.getTime() > this.lastTime)
where entry is a payload (FeedEntry) of splitted result from mentioned HttpRequestExecutingMessageHandler.

Log spring integration message lifecycle travelling through IntegrationFlow

Is there a way how to log the lifecycle of the spring integration message?
e.g. given DSL IntegrationFlow
jmsFlowsUtils.jmsXmlInputFlow(queue)
.<Object, Class<?>>route(Object::getClass, firstFlow -> firstFlow
.subFlowMapping(SomeClass.class.getName(), amEventFlow -> amEventFlow
.filter(...)
.transform(...)
.channel(...)
.handle(...)
)
.subFlowMapping(OtherClass.class.getName(), secondFlow -> secondFlow
.filter(...)
.transform(...)
.channel(...)
.handle(...)
)
)
.get();
The log message would be a string containing EIP endpoint ids/names the message travelled through including timestamps
This would build the basis for some other tool to extract information from logs and present it to users in nice form (e.g. ELK stack might be used for that purpose: https://www.elastic.co/products)
Does anybody have any experience with this use-case?
You can use #EnableMessageHistory to turn on message history tracking and that information is added to the headers as the message progresses through the flow.
See Message History.
You can use component name patterns to restrict the history tracking to certain components.

Resources