I am polling for files for a service-activator, using a PseudoTransactionManager to move them into processed/failed directories.
If/when the move fails, I would like to log this, including the file name.
As the information being passed around the flows is the Message object, I tried enriching the file name onto its header, but as we make copies of it for each step, this won't work unless I can move the header-enricher between the inbound channel adapter and transaction manager.
In simplified form the main flow I now have is this:
inbound-channel-adapter -> a) header-enricher -> service-activator
Because I want the files moved to a processed or failed directory, there is a second flow:
inbound-channel-adapter -> b) pseudo transaction-manager -> logging-channel-adapter (in case of problems moving the processed file).
I think this follows because the transaction manager definition is nested within the channel adapter definition in the xml.
How can I pass this information through the example setup here to a logging channel adapter?
The transaction stuff only has access to the original message. You can add an error-channel to the poller; the default error channel (errorChannel) is a pub/sub channel and has a logging channel adapter subscribed to it.
When an exception occurs, an ErrorMessage is sent to the error channel (if configured); the payload is a MessagingException with cause and failedMessage properties. The failedMessage is the message at the point the failure occurred.
The default error flow will simply log the message so your "transaction" will "commit".
Instead, you need a custom error flow; log what you want and then re-throw the cause and your "transaction" will "rollback".
Related
I am trying to build a Spring Integration Flow with DSL where a part of the flow calls an existing service that will process data asynchronously and return the response on a channel. The service call returns a task ID that can be used as a correlation ID to obtain the correct response message on the channel.
I am unsure how to build a flow (which components to use) that will call the service (I assume with a service activator), then take the returned ID and then wait for a message on a different channel that has that ID in the correlation ID header (maybe some sort of aggregator?). I have googled and cannot find anything that seems similar.
Also, my flow will receive a request object that I would like to pair up with the response object to pass along the flow after the response is received.
Request -> service call -> returns task ID ---->|
| |---- (Request+Response) --> More processing
| (async) |
---------> Response ----------->|
for task ID
on task complete channel
You are correct. The best way to solve your task is really an aggregator pattern:
https://www.enterpriseintegrationpatterns.com/Aggregator.html
https://docs.spring.io/spring-integration/docs/current/reference/html/message-routing.html#aggregator
So, you probably need to use a header enricher to instead of a service activator to obtain that task ID in the reply and store it in a header for future correlation of this request and some reply later with the same task ID. Or if you have a property on the request object for this task ID, you can use a content enricher instead: https://docs.spring.io/spring-integration/docs/current/reference/html/message-transformation.html#content-enricher
Then you send this request object with the task ID to an aggregator where this task ID must be used as a correlation key. The group size of course is just 2 - request and reply.
Your async service must send a completion to the same aggregator's input channel with. When aggregator encounters a proper correlation key, it will complete the group of two messages and send a single one to its output channel.
UPDATE
The aggregator we are talking about must have its own input channel, so you can send a request with task ID for correlation and then from your async service a reply must be sent to the same channel. With Java DSL it is a matter of exposing that input channel for your convenience:
#Bean
IntegrationFlow aggregatorFlow() {
return f -> f
.aggregate(...)
.channel("correlatedReplyChannel")
}
This flow implicitly starts with a channel like aggregatorFlow.input. So, you use this name in a channel() definition of request and reply flows as the last EIP-method in their definitions.
I am using SpringIntegration's IntegrationFlows to define the message flow, and used Jms.messageDrivenChannelAdapter to get the message from the MQ, now I need to parse it, send it to KAFKA and update couchbase.
IntegrationFlows
.from(Jms.messageDrivenChannelAdapter(this.acarsMqListener)) //MQ Listener with session transacted=true
.wireTap(ACARS_WIRE_TAP_CHNL) // Logging the message
.transform(agmTransformer, "parseXMLMessage") .filter(acarsFilter,"filterMessageOnSmiImi") // Filter the message based on condition
.handle(acarsProcessor, "processEvent") // Create the message
.handle(Kafka.outboundChannelAdapter(kafkaTemplate).messageKey(MESSAGE_KEY).topic(acarsKafkaTopic)) //send it to kafka
.handle(updateCouchbase, "saveToDB") // Update couchbase
.get();
For each message received we want to log it using MDC to help us to collect/aggregate it based on UUID.
Please suggest how to put the UUID in MDC and then clear out the MDC for each message in the above flow
You just can configure a global WireTap and do an appropriate transformation over there in that wire-tapped flow before logging the message.
On the other hand there might be just need to play with MDC since you can inject something like For header into the message and log() them as usual, so you will see messages in logs and would be able to correlate using that header.
I did one simple DSL which retrieves the data from database and doing simple conversion in the service activator.
#Bean
public IntegrationFlow mainFlow() {
return IntegrationFlows.from("userChannel")
.channel("queryChannel")
.handle("sampleConvertor","convertUser")
.get();
queryChannel is a jdbc outbound gateway and sampleConverter is the service Activator.
<int-jdbc:outbound-gateway query="select * from employee where employee_id=:payload"
request-channel="queryChannel" data-source="dataSource"/>
The issue is after retrieving the data from database, the flow is not going to serviceActivator and it simply returns back the database response.
In xml configuration, I used to invoke gateway inside the chain like below.
<int:gateway id="query.gateway" request-channel="queryChannel"/>
Please suggest what I am doing wrong here. Thanks in advance.
That's a bit unusual to combine Java DSL and XML configuration, but they still work together.
Your problem I think that you are missing the fact that your queryChannel has two subscriber at runtime, not a chain of call.
The first one is <int-jdbc:outbound-gateway> and the second is that .handle("sampleConvertor","convertUser"). Right, when you declare a channel in the IntegrationFlow, the next EIP-method produces a subscriber for this channel. At the same time when you use a channel like request-channel or input-channel in the XML configuration that brings a subscriber as well.
So, you have two subscriber on the DirectChannel with the RoundRobinLoadBalancingStrategy and therefore only one of them will handle a message and if it is a request-replly component, like that <int-jdbc:outbound-gateway> it will produce a message into the output-channel or to the replyChannel in the headers. In your case the story is exactly about a replyChannel and therefore you don't go to the .handle("sampleConvertor","convertUser") because it's not the next in the chain, but just a parallel universe by the round-robin algorithm.
If you really would like to reach that .handle("sampleConvertor","convertUser") after calling the <int-jdbc:outbound-gateway>, you should consider to use .gateway("queryChannel") instead of that .channel().
http://www.enterpriseintegrationpatterns.com/patterns/messaging/DataEnricher.html
http://www.enterpriseintegrationpatterns.com/patterns/messaging/MessagingAdapter.html
If there is a reponse channel, service activator seems completely identical to enricher.
See the Spring Integration documentation.
With a Service Activator the input message is replaced by the output message.
With the enricher; the input message is parked; we send a message (which could be different to the input message) to some downstream flow which returns a result.
We then "enrich" the input message (by adding information from the reply) to produce the output message.
For example; let's say you have an Order object with a customer ID and you want to enhance it by adding the customer name; you can use an enricher to send a lookup request for the customer; then set the customer name property on the order.
You can, of course, do the same thing within your service but with the enricher you don't have to write any code - it's your choice.
I have a flow that is similar to
IntegrationFlows.from(
Http.inboundGateway("/events")
.requestMapping(requestMappingSpec -> {
requestMappingSpec.methods(HttpMethod.POST);
requestMappingSpec.consumes(MediaType.APPLICATION_JSON_VALUE);
requestMappingSpec.produces(MediaType.APPLICATION_JSON_VALUE);
})
.requestPayloadType(PushEvent.class)
.errorChannel(ERROR_CHANNEL))
.channel(ReleaseFlow.REQUEST_CHANNEL)
.enrichHeaders(h -> h
.header(HttpHeaders.STATUS_CODE, HttpStatus.ACCEPTED))
.get();
When submitting multiple requests, a request will be processed by the flow attached to the REQUEST_CHANNEL and the following request will be processed by just the enrichedHeaders. My understanding is that the endpoints in this example should be processed serially ...
A request arrives at the /events endpoint
The request is processed by the flow listening to REQUEST_CHANNEL
The response from the previous flow will then have its headers enriched
The flow ends and the response is returned to the remote requestor
I appreciate your help in understanding why request n is processed by the channel (and not enrichHeaders()), request n + 1 is being processed by enrichHeaders() (and not the flow listening to the REQUEST_CHANNEL), request n + 2 processed by the channel (and not enrichHeaders()), ...
UPDATE 1
I am new to Spring Integration, but thought it was appropriate to collect events from a GitHub server and then create a release using an external service. The integration service would be responsible for determining the appropriate workflow based upon the data associated to the commit. The endpoint in question would receive a push event and forward it to the flow attached to the subscribable request channel (REQUEST_CHANNEL). This second flow will make a number of outbound requests to collect the appropriate release template and construct and start the pipeline.
UPDATE 2
I have not developed the second flow completely at this point, but here is a first version that simply performs a transformation based upon data associated with the commit.
return f -> f
.route(branchRouter(), mapping -> mapping
.subFlowMapping(true, t -> t
.transform(pushEventToEventTransformer()))
.subFlowMapping(false, n -> n
.transform(skip -> "")));
When the code has been submitted to a "monitored" branch the actions described in the first update will be performed. I am attempting to build the flows incrementally given my limited knowledge of the framework.
Subscribable channels are point-to-point by default, which means if there are two subscribers, messages will be distributed in round-robin fashion.
If you have another flow " ... attached to the REQUEST_CHANNEL " then you have two subscribers - that flow and the header-enricher.
Perhaps if you can describe what you are trying to do we can help.
With the header enricher after the channel, all that happens is the headers are enriched and the inbound message is returned to the gateway.
Perhaps you want this... ?
IntegrationFlows.from(
Http.inboundGateway("/events")
.requestMapping(requestMappingSpec -> {
requestMappingSpec.methods(HttpMethod.POST);
requestMappingSpec.consumes(MediaType.APPLICATION_JSON_VALUE);
requestMappingSpec.produces(MediaType.APPLICATION_JSON_VALUE);
})
.requestPayloadType(PushEvent.class)
.errorChannel(ERROR_CHANNEL))
.enrichHeaders(h -> h
.header(HttpHeaders.STATUS_CODE, HttpStatus.ACCEPTED))
.channel(ReleaseFlow.REQUEST_CHANNEL)
.get();
Which means all messages with the enriched headers will be sent to the channel.
EDIT
If you want the message to go to both places, you need to use a publish-subscribe channel.