I am using Spring Integration and the Feed Inbound Channel Adapter to process news RSS feeds (which I think is fantastic :). I would also like to consume some additional news feeds which are available by an API into the same channel. The API is just a HTTP endpoint which returns a list of news articles in JSON. The fields are very similar to RSS i.e. there is a title, description, published date which could be mapped to the SyndEntry object.
Ideally I want to use the same functionality available in feed inbound channel adapter which deals with duplicate entries etc. Is it possible to customise Feed Inbound Channel Adaptor to process and map the JSON?
Any sample code or pointers would be really helpful.
Well, no. FeedEntryMessageSource is fully based on the Rome Tools which deals only with the XML model.
I'm afraid you have to create your own component which will produce SyndEntry instances for those JSON records. That can really be something like HttpRequestExecutingMessageHandler, based on the RestTemplate with the MappingJackson2HttpMessageConverter which is present by default anyway.
You can try to configure HttpRequestExecutingMessageHandler to the setExpectedResponseType(SyndFeedImpl.class) and expect to have application/json content-type in the response.
To achieve the "deals with duplicate entries" you can consider to use Idempotent Receiver pattern afterwards. Where the MessageSelector should be based on the MetadaStore and really preform logic similar to the one in the FeedEntryMessageSource:
if (lastModifiedDate != null) {
this.lastTime = lastModifiedDate.getTime();
}
else {
this.lastTime += 1; //NOSONAR - single poller thread
}
this.metadataStore.put(this.metadataKey, this.lastTime + "");
...
if ((entryDate != null && entryDate.getTime() > this.lastTime)
where entry is a payload (FeedEntry) of splitted result from mentioned HttpRequestExecutingMessageHandler.
Related
This appears, to me, to be a simple problem that is probably replicated all over the place. A very basic application of the MessageHandlerChain, probably using nothing more than out of the box functionality.
Conceptually, what I need is this:
(1) Polled JDBC reader (sets parameters for integration pass)
|
V
(2) JDBC Reader (uses input from (1) to fetch data to feed through channel
|
V
(3) JDBC writer (writes data fetched by (2) to target)
|
V
(4) JDBC writer (writes additional data from the original parameters fetched in (1))
What I think I need is
Flow:
From: JdbcPollingChannelAdapter (setup adapter)
Handler: messageHandlerChain
Handlers (
JdbcPollingChannelAdapter (inbound adapter)
JdbcOutboundGateway (outbound adapter)
JdbcOutboundGateway (cleanup gateway)
)
The JdbcPollingChannelAdapter does not implement the MessageHandler API, so I am at a loss how to read the actual data based on the setup step.
Since the JdbcOutboundGateway does not implement the MessageProducer API, I am at a bit of a loss as to what I need to use for the outbound adapter.
Are there OOB classes I should be using? Or do I need to somehow wrap the two adapters in BridgeHandlers to make this work?
Thanks in advance
EDIT (2)
Additional configuration problem
The setup adapter is pulling a single row back with two timestamp columns. They are being processed correctly by the "enrich headers" piece.
However, when the inbound adapter is executing, the framework is passing in java.lang.Object as parameters. Not String, not Timestamp, but an actual java.lang.Object as in new Object ().
It is passing the correct number of objects, but the content and datatypes are lost. Am I correct that the ExpressionEvaluatingSqlParameterSourceFactory needs to be configured?
Message:
GenericMessage [payload=[{startTime=2020-11-18 18:01:34.90944, endTime=2020-11-18 18:01:34.90944}], headers={startTime=2020-11-18 18:01:34.90944, id=835edf42-6f69-226a-18f4-ade030c16618, timestamp=1605897225384}]
SQL in the JdbcOutboundGateway:
Select t.*, w.operation as "ops" from ADDRESS t
Inner join TT_ADDRESS w
on (t.ADDRESSID = w.ADDRESSID)
And (w.LASTUPDATESTAMP >= :payload.from[0].get("startTime") and w.LASTUPDATESTAMP <= :payload.from[0].get("endTime") )
Edit: added solution java DSL configuration
private JdbcPollingChannelAdapter setupAdapter; // select only
private JdbcOutboundGateway inboundAdapter; // select only
private JdbcOutboundGateway insertUpdateAdapter; // update only
private JdbcOutboundGateway deleteAdapter; // update only
private JdbcMessageHandler cleanupAdapter; // update only
setFlow(IntegrationFlows
.from(setupAdapter, c -> c.poller(Pollers.fixedRate(1000L, TimeUnit.MILLISECONDS).maxMessagesPerPoll(1)))
.enrichHeaders(h -> h.headerExpression("ALC_startTime", "payload.from[0].get(\"ALC_startTime\")")
.headerExpression("ALC_endTime", "payload.from[0].get(\"ALC_endTime\")"))
.handle(inboundAdapter)
.enrichHeaders(h -> h.headerExpression("ALC_operation", "payload.from[0].get(\"ALC_operation\")"))
.handle(insertUpdateAdapter)
.handle(deleteAdapter)
.handle(cleanupAdapter)
.get());
flowContext.registration(flow).id(this.getId().toString()).register();
If you would like to carry the original arguments down to the last gateway in your flow, you need to store those arguments in the headers since after every step the payload of reply message is going to be different and you won't have original setup data over there any more. That's first.
Second: if you deal with IntegrationFlow and Java DSL, you don't need to worry about messageHandlerChain since conceptually the IntegrationFlow is a chain by itself but much more advance.
I'm not sure why you need to use a JdbcPollingChannelAdapter to request data on demand according incoming message from the source in the beginning of your flow.
You definitely still need to use a JdbcOutboundGateway for just SELECT mode. The updateQuery is optional, so that gateway is just going to perform SELECT and return a data for you in a payload of the reply message.
If you two next steps are just "write" and you don't care about the result, you probably can just take a look into a PublishSubscribeChannel and two JdbcMessageHandler as subscribers to it. Without a provided Executor for the PublishSubscribeChannel they are going to be executed one-by-one.
My Spring Integration (with a HTTP inbound gateway) application has the following flow:
The end-client POSTs an XML input message with a UUID for unique identification (let us call this XML, "whole") and other business elements
A Splitter component breaks it down into smaller XMLs with each "part" XML having the same UUID as the "whole" XML
The Aggregator aggregates the parts with some business logic along with a #CorrelationStrategy that uses the UUID of the incoming "part" messages
I am trying to explore a way to clear/flush the Aggregator's inputChannel or reset the Correlation Strategy after the correlation operation to enable different POST requests to reuse the UUIDs in the incoming "whole" XML input messages.
At this point, what is happening is that once a "whole" XML input message is processed successfully, if another HTTP POST sends a "whole" XML input message with the same UUID, the flow is hanging, probably because it is unable to understand what to do with the #CorrelationStrategy or is in a stale state in terms of UUIDs.
Frankly, the very definition of UUID means it has to be uniquely generated for every HTTP request but just wanted to accommodate a case where the end-client ends up sending the same UUID for every POST.
This is how my Aggregator class and its methods look:
#MessageEndpoint
public class ProductAggregator {
...
...
#Aggregator(inputChannel="compositeMessagesForAggregation",
outputChannel="upstreamResponses")
public Message<BusinessComposite>
generateAggregatedResponse(List<Message<BusinessComposite>>
listOfCompositeMessagesForAggregation) {
...
...
}
#CorrelationStrategy
public UUID correlateByUUID(Message<BusinessComposite>
compositeMessageForAggregation) {
...
...
}
Thanks in advance!
Sincerely,
Bharath
Set expireGroupsUponCompletion and expireGroupsUponExpiry to true on the aggregator to reuse the same correlation id; otherwise "late arriving" messages are discarded.
In spring-batch, data can be passed between various steps via ExecutionContext. You can set the details in one step and retrieve in the next. Do we have anything of this sort in spring-integration ?
My use case is that I have to pick up a file from ftp location, then split it based on certain business logic and then process them. Depending on the file names client id would be derived. This client id would be used in splitter, service activator and aggregator components.
From my newbie level of expertise I have in spring, I could not find anything which help me share state for a particular run.I wanted to know if spring-integration provides this state sharing context in some way.
Please let me know if there is a way to do in spring-context.
In Spring Integration applications there is no single ExecutionContext for state sharing. Instead, as Gary Russel mentioned, each message carries all the information within its payload or its headers.
If you use Spring Integration Java DSL and want to transport the clientId by message header you can use enrichHeader transformer. Being supplied with a HeaderEnricherSpec, it can accept a function which returns dynamically determined value for the specified header. As of your use case this might look like:
return IntegrationFlows
.from(/*ftp source*/)
.enrichHeaders(e -> e.headerFunction("clientId", this::deriveClientId))
./*split, aggregate, etc the file according to clientId*/
, where deriveClientId method might be a sort of:
private String deriveClientId(Message<File> fileMessage) {
String fileName = fileMessage.getHeaders().get(FileHeaders.FILENAME, String.class);
String clientId = /*some other logic for deriving clientId from*/fileName;
return clientId;
}
(FILENAME header is provided by FTP message source)
When you need to access the clientId header somewhere in the downstream flow you can do it the same way as file name mentioned above:
String clientId = message.getHeaders().get("clientId", String.class);
But make sure that the message still contains such header as it could have been lost somewhere among intermediate flow items. This is likely to happen if at some point you construct a message manually and send it further. In order not to loose any headers from the preceding message you can copy them during the building:
Message<PayloadType> newMessage = MessageBuilder
.withPayload(payloadValue)
.copyHeaders(precedingMessage.getHeaders())
.build();
Please note that message headers are immutable in Spring Integration. It means you can't just add or change a header of the existing message. You should create a new message or use HeaderEnricher for that purpose. Examples of both approaches are presented above.
Typically you convey information between components in the message payload itself, or often via message headers - see Message Construction and Header Enricher
Does getstream provide a way to retrieve the number of activities within a feed? I have a notification feed setup. I can retrieve the activities using paginated get. However, I would like to display the number of items within the feed.
Unfortunately not, there is no API endpoint to retrieve the amount of activities within a feed.
At the moment there's no way to count activities directly. You can try
Use a custom feed and use reactions. So now you call count from reactions.
Other way is store in your app (cache/db/etc).
It worked for me:
signature = Stream::Signer.create_jwt_token('activities', '*', '<your_secret>', '*')
uri = URI("https://us-east-api.stream-io-api.com/api/v1.0/enrich/activities/?api_key=<api_key>&ids=<id1,id2,...>&withReactionCounts=true")
req = Net::HTTP::Get.new(uri)
req['Content-Type'] = "application/json"
req['Stream-Auth-Type'] = "jwt"
req['Authorization'] = signature
res = Net::HTTP.start(uri.hostname, uri.port, :use_ssl => true) {|http|
http.request(req)
}
puts JSON.parse(res.body)
References:
Retrieve
reactions_read-feeds
activities
ruby client
Update 2022
You can only get the number of activities in groups within aggregated or notification feeds. Flat feeds are still not supported.
Source
The best solution is to store important numbers in the database which Stream provides.
Source
I want to write a batch which reads website access log files(csv file)from a path every day and do some analysis using spring integration.
this is the simplified version of the input csv file.
srcIp1,t1,path1
srcIp2,t2,path2
srcIp1,t3,path2
srcIp1,t4,path1
The access number per source ip and path is to be calculated after some filtering logic.
I made a input channel whose payload is the parsed log line,and a filter is applied,and finally an aggregator to calculate the final result.
The problem is what should be the right group release stragety,the default release stragety(SequenceSizeReleaseStrategy) does not work.
Also any of other spring integraion out of box release
strategies(ExpressionEvaluatingReleaseStrategy,
MessageCountReleaseStrategy, MethodInvokingReleaseStrategy,
SequenceSizeReleaseStrategy, TimeoutCountSequenceSizeReleaseStrategy)
does not seem to fit my needs.
Or Spring integration assumed that a channel carries a message stream where there is no concept of "ending of message" and is not suitalbe for my problem here ?
You can write a custom ReleaseStrategy if you have some way to tell when the group is complete. It is consulted each time a message is added to the group.
Or, you can use a group-timeout to release a partial group after some time when no messages arrived.