My Spring Integration (with a HTTP inbound gateway) application has the following flow:
The end-client POSTs an XML input message with a UUID for unique identification (let us call this XML, "whole") and other business elements
A Splitter component breaks it down into smaller XMLs with each "part" XML having the same UUID as the "whole" XML
The Aggregator aggregates the parts with some business logic along with a #CorrelationStrategy that uses the UUID of the incoming "part" messages
I am trying to explore a way to clear/flush the Aggregator's inputChannel or reset the Correlation Strategy after the correlation operation to enable different POST requests to reuse the UUIDs in the incoming "whole" XML input messages.
At this point, what is happening is that once a "whole" XML input message is processed successfully, if another HTTP POST sends a "whole" XML input message with the same UUID, the flow is hanging, probably because it is unable to understand what to do with the #CorrelationStrategy or is in a stale state in terms of UUIDs.
Frankly, the very definition of UUID means it has to be uniquely generated for every HTTP request but just wanted to accommodate a case where the end-client ends up sending the same UUID for every POST.
This is how my Aggregator class and its methods look:
#MessageEndpoint
public class ProductAggregator {
...
...
#Aggregator(inputChannel="compositeMessagesForAggregation",
outputChannel="upstreamResponses")
public Message<BusinessComposite>
generateAggregatedResponse(List<Message<BusinessComposite>>
listOfCompositeMessagesForAggregation) {
...
...
}
#CorrelationStrategy
public UUID correlateByUUID(Message<BusinessComposite>
compositeMessageForAggregation) {
...
...
}
Thanks in advance!
Sincerely,
Bharath
Set expireGroupsUponCompletion and expireGroupsUponExpiry to true on the aggregator to reuse the same correlation id; otherwise "late arriving" messages are discarded.
Related
This appears, to me, to be a simple problem that is probably replicated all over the place. A very basic application of the MessageHandlerChain, probably using nothing more than out of the box functionality.
Conceptually, what I need is this:
(1) Polled JDBC reader (sets parameters for integration pass)
|
V
(2) JDBC Reader (uses input from (1) to fetch data to feed through channel
|
V
(3) JDBC writer (writes data fetched by (2) to target)
|
V
(4) JDBC writer (writes additional data from the original parameters fetched in (1))
What I think I need is
Flow:
From: JdbcPollingChannelAdapter (setup adapter)
Handler: messageHandlerChain
Handlers (
JdbcPollingChannelAdapter (inbound adapter)
JdbcOutboundGateway (outbound adapter)
JdbcOutboundGateway (cleanup gateway)
)
The JdbcPollingChannelAdapter does not implement the MessageHandler API, so I am at a loss how to read the actual data based on the setup step.
Since the JdbcOutboundGateway does not implement the MessageProducer API, I am at a bit of a loss as to what I need to use for the outbound adapter.
Are there OOB classes I should be using? Or do I need to somehow wrap the two adapters in BridgeHandlers to make this work?
Thanks in advance
EDIT (2)
Additional configuration problem
The setup adapter is pulling a single row back with two timestamp columns. They are being processed correctly by the "enrich headers" piece.
However, when the inbound adapter is executing, the framework is passing in java.lang.Object as parameters. Not String, not Timestamp, but an actual java.lang.Object as in new Object ().
It is passing the correct number of objects, but the content and datatypes are lost. Am I correct that the ExpressionEvaluatingSqlParameterSourceFactory needs to be configured?
Message:
GenericMessage [payload=[{startTime=2020-11-18 18:01:34.90944, endTime=2020-11-18 18:01:34.90944}], headers={startTime=2020-11-18 18:01:34.90944, id=835edf42-6f69-226a-18f4-ade030c16618, timestamp=1605897225384}]
SQL in the JdbcOutboundGateway:
Select t.*, w.operation as "ops" from ADDRESS t
Inner join TT_ADDRESS w
on (t.ADDRESSID = w.ADDRESSID)
And (w.LASTUPDATESTAMP >= :payload.from[0].get("startTime") and w.LASTUPDATESTAMP <= :payload.from[0].get("endTime") )
Edit: added solution java DSL configuration
private JdbcPollingChannelAdapter setupAdapter; // select only
private JdbcOutboundGateway inboundAdapter; // select only
private JdbcOutboundGateway insertUpdateAdapter; // update only
private JdbcOutboundGateway deleteAdapter; // update only
private JdbcMessageHandler cleanupAdapter; // update only
setFlow(IntegrationFlows
.from(setupAdapter, c -> c.poller(Pollers.fixedRate(1000L, TimeUnit.MILLISECONDS).maxMessagesPerPoll(1)))
.enrichHeaders(h -> h.headerExpression("ALC_startTime", "payload.from[0].get(\"ALC_startTime\")")
.headerExpression("ALC_endTime", "payload.from[0].get(\"ALC_endTime\")"))
.handle(inboundAdapter)
.enrichHeaders(h -> h.headerExpression("ALC_operation", "payload.from[0].get(\"ALC_operation\")"))
.handle(insertUpdateAdapter)
.handle(deleteAdapter)
.handle(cleanupAdapter)
.get());
flowContext.registration(flow).id(this.getId().toString()).register();
If you would like to carry the original arguments down to the last gateway in your flow, you need to store those arguments in the headers since after every step the payload of reply message is going to be different and you won't have original setup data over there any more. That's first.
Second: if you deal with IntegrationFlow and Java DSL, you don't need to worry about messageHandlerChain since conceptually the IntegrationFlow is a chain by itself but much more advance.
I'm not sure why you need to use a JdbcPollingChannelAdapter to request data on demand according incoming message from the source in the beginning of your flow.
You definitely still need to use a JdbcOutboundGateway for just SELECT mode. The updateQuery is optional, so that gateway is just going to perform SELECT and return a data for you in a payload of the reply message.
If you two next steps are just "write" and you don't care about the result, you probably can just take a look into a PublishSubscribeChannel and two JdbcMessageHandler as subscribers to it. Without a provided Executor for the PublishSubscribeChannel they are going to be executed one-by-one.
I use a queue channel backed up with a message store and found that the timestamp header does not keep with the original value, is there a way to keep it?
here's my configuration
<int:channel id="myChannel">
<int:queue message-store="myStore"/>
</int:channel>
thanks in advance!
What you are looking for really has been fixed in the version 5.0: https://github.com/spring-projects/spring-integration/pull/1916.
Pay attention to my phrase there in the end of PR dfescription:
With this fix we persist message in the store as is without any modifications when we perform standard serialization procedure.
Any custom serializers should consider to use MutableMessageBuilder if there is a requirement to retain ID and TIMESTAMP
A default one is there like this:
public Message<?> mapRow(ResultSet rs, int rowNum) throws SQLException {
return (Message<?>) this.deserializer.convert(this.lobHandler.getBlobAsBytes(rs, "MESSAGE_BYTES"));
}
So, we definitelly retain there incoming ID and TIMESTAMP.
I have a Spring Itegration (5.0.2) flow that reads data from an HTTP endpoint and publish Kafka messages using the (split) data from the HTTP response.
I would like to execute a "final" action before the flow completes, some sort of "try, catch, finally" but I'm not sure what is the best way to achieve this.
This is my code:
#Bean
public IntegrationFlow startFlow() {
return IntegrationFlows.from(() -> new GenericMessage<>(""),
c -> c.id("flow1")
.poller(
Pollers.fixedDelay(period, TimeUnit.MILLISECONDS, initialDelay).taskExecutor(taskExecutor)))
.handle(Http.outboundGateway("http://....)
.charset("UTF-8")
.expectedResponseType(String.class)
.httpMethod(HttpMethod.GET))
.transform(new transformer())
.split()
.channel(CHANNEL_1)
.controlBus()
.get();
}
#Bean
public IntegrationFlow toKafka(KafkaTemplate<?, ?> kafkaTemplate) {
return IntegrationFlows.from(CHANNEL_1)
.handle(Kafka.outboundChannelAdapter(kafkaTemplate)
// KAFKA SPECIFIC
.get();
}
Essentially, when all the messages have been sent (note that I'm using .split), I need to call a Spring bean to update some data.
Thanks
If you talk about "when all the messages" after splitter, then you need to take a look into an .aggregate(): https://docs.spring.io/spring-integration/docs/5.0.3.RELEASE/reference/html/messaging-routing-chapter.html#aggregator.
The splitter populates special sequence details headers into each splitted item and an aggregator is able to gather them to a single entity by default using those headers from received message.
Since you talk about the process after sending to Kafka, you should make your CHANNEL_1 as a PublishSubscribeChannel and have the mentioned .aggregate() as the last subscriber to this channel. The service to update some data should be already after this aggregator.
I am using Spring Integration and the Feed Inbound Channel Adapter to process news RSS feeds (which I think is fantastic :). I would also like to consume some additional news feeds which are available by an API into the same channel. The API is just a HTTP endpoint which returns a list of news articles in JSON. The fields are very similar to RSS i.e. there is a title, description, published date which could be mapped to the SyndEntry object.
Ideally I want to use the same functionality available in feed inbound channel adapter which deals with duplicate entries etc. Is it possible to customise Feed Inbound Channel Adaptor to process and map the JSON?
Any sample code or pointers would be really helpful.
Well, no. FeedEntryMessageSource is fully based on the Rome Tools which deals only with the XML model.
I'm afraid you have to create your own component which will produce SyndEntry instances for those JSON records. That can really be something like HttpRequestExecutingMessageHandler, based on the RestTemplate with the MappingJackson2HttpMessageConverter which is present by default anyway.
You can try to configure HttpRequestExecutingMessageHandler to the setExpectedResponseType(SyndFeedImpl.class) and expect to have application/json content-type in the response.
To achieve the "deals with duplicate entries" you can consider to use Idempotent Receiver pattern afterwards. Where the MessageSelector should be based on the MetadaStore and really preform logic similar to the one in the FeedEntryMessageSource:
if (lastModifiedDate != null) {
this.lastTime = lastModifiedDate.getTime();
}
else {
this.lastTime += 1; //NOSONAR - single poller thread
}
this.metadataStore.put(this.metadataKey, this.lastTime + "");
...
if ((entryDate != null && entryDate.getTime() > this.lastTime)
where entry is a payload (FeedEntry) of splitted result from mentioned HttpRequestExecutingMessageHandler.
In spring-batch, data can be passed between various steps via ExecutionContext. You can set the details in one step and retrieve in the next. Do we have anything of this sort in spring-integration ?
My use case is that I have to pick up a file from ftp location, then split it based on certain business logic and then process them. Depending on the file names client id would be derived. This client id would be used in splitter, service activator and aggregator components.
From my newbie level of expertise I have in spring, I could not find anything which help me share state for a particular run.I wanted to know if spring-integration provides this state sharing context in some way.
Please let me know if there is a way to do in spring-context.
In Spring Integration applications there is no single ExecutionContext for state sharing. Instead, as Gary Russel mentioned, each message carries all the information within its payload or its headers.
If you use Spring Integration Java DSL and want to transport the clientId by message header you can use enrichHeader transformer. Being supplied with a HeaderEnricherSpec, it can accept a function which returns dynamically determined value for the specified header. As of your use case this might look like:
return IntegrationFlows
.from(/*ftp source*/)
.enrichHeaders(e -> e.headerFunction("clientId", this::deriveClientId))
./*split, aggregate, etc the file according to clientId*/
, where deriveClientId method might be a sort of:
private String deriveClientId(Message<File> fileMessage) {
String fileName = fileMessage.getHeaders().get(FileHeaders.FILENAME, String.class);
String clientId = /*some other logic for deriving clientId from*/fileName;
return clientId;
}
(FILENAME header is provided by FTP message source)
When you need to access the clientId header somewhere in the downstream flow you can do it the same way as file name mentioned above:
String clientId = message.getHeaders().get("clientId", String.class);
But make sure that the message still contains such header as it could have been lost somewhere among intermediate flow items. This is likely to happen if at some point you construct a message manually and send it further. In order not to loose any headers from the preceding message you can copy them during the building:
Message<PayloadType> newMessage = MessageBuilder
.withPayload(payloadValue)
.copyHeaders(precedingMessage.getHeaders())
.build();
Please note that message headers are immutable in Spring Integration. It means you can't just add or change a header of the existing message. You should create a new message or use HeaderEnricher for that purpose. Examples of both approaches are presented above.
Typically you convey information between components in the message payload itself, or often via message headers - see Message Construction and Header Enricher