Spring Integration - Barrier - spring-integration

I have a thread that uses gateway (void) to send a message to both (pub/sub):
barrier, to hold the thread during the real execution (requires-reply="true" timeout="XXXX", output-channel="nullChannel"
and to
splitter which next sends splits as messages to the service activator (direct channel) with poller and a thread executor for the actual processing/execution
How to properly configure handling the exceptions that might be thrown by the executor threads and catch them in the below catch block:
try {
gateway.trigger()
} catch (ReplyRequiredException e) {
//fine here
} catch (Throwable t) {
// catch every exception here... or somehow configure these exceptions to discard the thread that waits on the barrier and throw below business exception
throw new SomeExecutionFailedException()
}
EDIT
<!--gateway.trigger()—>
<int:gateway id=“gateway"
service-interface="com.Gateway"
default-request-channel=“channel1"
default-reply-timeout="0"/>
<int:publish-subscribe-channel id=“channel1"/>
<int:splitter input-channel=“channel1" output-channel=“channel2"
order="1">
<bean class=“com.Splitter"/>
</int:splitter>
<int:barrier id=“barrier" input-channel=“channel1"
output-channel="nullChannel"
correlation-strategy-expression=“'XXX’” <!--hack here-->
requires-reply="true"
timeout=“40000"
order="2">
</int:barrier>
<int:channel id=“channel2">
<int:queue capacity="30"/>
</int:channel>
<!— actual processing/execution —>
<int:service-activator id=“executionAct" input-channel=“channel2"
output-channel=“channel3" ref=“executionService">
<int:poller fixed-rate="111" time-unit="MILLISECONDS" max-messages-per-poll="22"
task-executor=“exec"/>
</int:service-activator>
<bean id=“executionService" class=“com.SomeExecService"/>
<bean id=“exec" class="org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor">
<property name="threadFactory" ref=“execFactory"/>
...
<property name="rejectedExecutionHandler">
<bean class="java.util.concurrent.ThreadPoolExecutor.CallerRunsPolicy"/>
</property>
</bean>
<bean id=“execFactory"
class="org.springframework.scheduling.concurrent.CustomizableThreadFactory">
...
</bean>
<int:channel id=“channel3"/>
<int:chain input-channel=“channel3" output-channel=“channel4">
...
<int:aggregator
group-timeout=“30000"
discard-channel=“discardChannel" release-strategy=“com.ReleaseStrategy"
send-partial-result-on-expiry="false">
<bean class="com.Aggregator"/>
</int:aggregator>
</int:chain>
<int:channel id=“discardChannel”/>
<int:channel id=“channel4"/>
<!— processing done - wake up barrier —>
<int:service-activator id=“barrierReleaseAct" input-channel=“channel4" output-channel="nullChannel">
<bean class="com.ServiceThatSendsXXXMessageToChannel5ToReleaseBarrier"/>
</int:service-activator>
<int:channel id=“channel5"/>
<int:outbound-channel-adapter channel=“channel5"
ref=“barrier" method="trigger"/>

You need to provide much more information, configuration etc.
What releases the barrier, and when?
Do you want to propagate the exception(s) to the main thread?
What if multiple splits fail, etc, etc.
The general answer is sending a message with a Throwable payload to the barrier's trigger method will release the thread by throwing a MessagingException with the Throwable as its cause. The gateway unwraps the MessagingException and throws the cause (which is the original payload sent to the barrier's trigger method).
EDIT
I have added a pull request to the barrier sample app to show one technique of collecting exceptions on the async threads and causing the barrier to throw a consolidated exception back to the gateway caller.

Related

Spring Integration Aggregator Continuing to output channel without release-strategy-expression being True

I asked a question a bit ago about a specific use case when multi-threading in spring integration, basically I have to have one thread of execution that stays on the initial and then a second that spawns a thread. I have implemented this like so
<int:channel id="newThread" >
<int:dispatcher task-executor="workerThreadPoolAdapter"/>
</int:channel>
<!--This bean is used to split incoming messages -->
<bean id="splitterBean" class="orchestration.MessageSplitter">
<property name="channels" ref="splitterChannelsList" />
</bean>
<util:list id="splitterChannelsList" value-type="java.lang.String">
<value>newThread</value>
<value>mainThread</value>
</util:list>
<!-- This bean is used to aggregate incoming messages -->
<bean id="aggregator" class="orchestration.MessageAggregator">
<property name="wrapperNode" value="container" />
</bean>
<!-- Channel for aggregator output and that will be input for response transformer -->
<int:publish-subscribe-channel id="gatherChannel" apply-sequence="true"/>
<!-- This splitter splits request and send -->
<!-- will add a header called channelHeader which is the channel the message should be routed to using the recipient-list-router -->
<int:splitter input-channel="splitter"
output-channel="recipientListRouter"
apply-sequence="true"
ref="splitterBean" method="split" />
<!-- Aggregator that aggregates responses received from the calls -->
<int:aggregator input-channel="gatherChannel"
output-channel="transformResponse"
ref="aggregator"
method="aggregateMessages"
send-partial-result-on-expiry="false"
expire-groups-upon-completion="true"
message-store="removeMessageFromStore"
release-strategy-expression="size() == 2"/>
<int:recipient-list-router input-channel="recipientListRouter">
<int:recipient channel="mainThread"
selector-expression="headers.get('channelHeader') == 'mainThread'"/>
<int:recipient channel="newThread"
selector-expression="headers.get('channelHeader') == 'newThread'"/>
</int:recipient-list-router>
<!-- only route to call if boolean is populated -->
<int:header-value-router header-name="shouldMakeExtraOutboundCall"
input-channel="newThread"
default-output-channel="gatherChannel" >
<int:mapping value="true" channel="outboundCall" />
</int:header-value-router>
<int:chain input-channel="outboundCall" output-channel="gatherChannel">
<!-- make an outbound call -->
</int:chain>
<int:chain input-channel="mainThread" output-channel="gatherChannel">
<!-- make a bunch of outbound calls -->
</int:chain>
<int:chain input-channel="transformResponse" output-channel="backToClient">
<!-- do some stuff and respond back to client -->
</int:chain>
I've had the output channel of the aggregator as both a publish-subscribe and a direct channel and had the issue for both.
When I look at the logs I can see that one of the threads has a preSend to the 'gatherChannel' then AggregatingMessageHandler saying it received the message, then another log from AggregatingMessageHandler saying Handling message with correlationKey [f2b16b6a-3605-778f-a628-870ed8ce3f5e] then a postSend (sent=True) on channel 'gatherChannel'.
I thought that it would not send to the transformResponse channel until both messages that got split out from the splitter got to it. I even added the size() == 2 as the release strategy expression as an extra layer but that doesn't seem to be causing the aggregator to wait either.
I'm a little perplexed why this is happening, it's happening when both the main thread or the spawned thread gets to the aggregator, I'm trying to figure out how to get that aggregator to wait to send to the output channel until BOTH messages that were split from the splitter are received.
According to your current configuration it really does not happen.
Do you really have some evidences that first message in the group is really sent to that transformResponse unconditionally?
You probably is a bit confused with those preSend and postSend since an aggregator is really non-blocking until release. It does accept the message and store it into the MessageStore for grouping if release condition is false. Just after this it returns immediately to the caller, which is that gatherChannel and therefore your see postSend on a first message.

How to define an exception router in chain after aggregator

I using Spring integration to aggregate messages into one and then send by FTP out bound adapter, I want to move the aggregated messages in a specific folder when outbound FTP server is not available(org.springframework.messaging.MessageDeliveryException), other exceptions will log by console.
Here is my Configuration
<int:chain id="transformChain" input-channel="inboundChannel">
<int:header-enricher>
<int:header name="file_name" expression="payload.name"/>
<int:header name="correlationId" expression="${header.enricher.correlationId}"/>
<int:header name="sum" expression="${header.enricher.sum}"/>
</int:header-enricher>
<int:transformer ref="fileNameToContentTransformer"/>
<int:aggregator send-partial-result-on-expiry="true"
release-strategy-expression="#this.size() == new Integer([0].headers.sum)"
group-timeout="${aggregator.group-timeout}"
message-store="messageStore"
expire-groups-upon-completion="true"
correlation-strategy-expression="headers.correlationId"/>
<int:transformer ref="xmlToJsonTransformer"/>
<ftp:outbound-channel-adapter remote-directory="${ftp.out.remote.directory}"
session-factory="ftpOutClientSessionFactory" auto-create-directory="true"
remote-filename-generator="fileNameGenerator" charset="UTF-8"
temporary-file-suffix=".writing">
</ftp:outbound-channel-adapter>
<int:exception-type-router >
<int:mapping exception-type="org.springframework.messaging.MessageDeliveryException" channel="undeliveredChannel"/>
<int:mapping exception-type="java.lang.Exception" channel="myErrorChannel"/>
</int:exception-type-router>
</int:chain>
However I met such exception when try to start.
Caused by: java.lang.IllegalArgumentException: All handlers except for the last one in the chain must implement the MessageProducer interface. Object of class [org.springframework.integration.ftp.outbound.FtpMessageHandler] must be an instance of interface org.springframework.integration.core.MessageProducer
at org.springframework.util.Assert.instanceCheckFailed(Assert.java:389)
at org.springframework.util.Assert.isInstanceOf(Assert.java:327)
at org.springframework.integration.handler.MessageHandlerChain.configureChain(MessageHandlerChain.java:119)
at org.springframework.integration.handler.MessageHandlerChain.onInit(MessageHandlerChain.java:99)
at org.springframework.integration.context.IntegrationObjectSupport.afterPropertiesSet(IntegrationObjectSupport.java:176)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1687)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1624)
Anyone can tell me how to achieve this?
thanks in advance.
You can't add a component to the chain after the ftp:outbound-channel-adapter because it produces no result.
See the retry-and-more sample for an example of how to handle exceptions by adding an ExpressionEvaluatingRequestHandlerAdvice to the outbound adapter.

Tomcat complains of a memory leak with spring integration while shutting down

I am getting the following error while shutting down tomcat
SEVERE: The web application [/TestService] appears to have started a thread named [SimpleAsyncTaskExecutor-3] but has failed to stop it. This is very likely to create a memory leak.
<gateway id="testGateway" service-interface="org.example.TestGateway"
default-request-channel="requestChannel" error-channel="errorChannel"/>
public interface TestGateway {
Future execute(Request request);
}
<int:chain input-channel="requestChannel" output-channel="routerChannelA">
<int:service-activator ...
<int:transformer ....
</int:chain>
<int:router input-channel="routerChannelA" expression="payload.name" resolution-required="true">
<int:mapping value="B" channel="channelB" />
....
</int:router>
<int:chain input-channel="channelB" output-channel="channelD">
<int:transformer ......
<int:gateway request-channel="channelC" .....
<int:filter expression="headers['RELEASE'] != null" discard-="nullChannel"/>
</int:chain>
<int:recipient-list-router id="customRouter" input-channel="channelD"
timeout="1234"
ignore-send-failures="true"
apply-sequence="false" >
<int:recipient channel="splitterRequestChannel"/>
<int:recipient channel="completeChannel"/>
</int:recipient-list-router>
<int:splitter expression="payload" input-channel="splitterRequestChannel"
output-channel="splitterResponseChannel" ></int:splitter>
<int:channel id="splitterResponseChannel">
<int:dispatcher task-executor="splitterChannelTaskExecutor"/>
</int:channel>
<bean id="splitterChannelTaskExecutor" class="org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor" >
<property name="corePoolSize" value="10" />
<property name="daemon" value="false"/>
</bean>
<int:chain input-channel="splitterResponseChannel">
......
......
</int:chain>
The client sends requests to TestGateway. I dont have to send any reply back to client, but I want to return back immediately.
The Future return type serves my purpose of returning immediately. However I feel it blocks the main thread.
The request moves from a series of chains and finally reaches splitterRequestChannel. This channel delegates its work to the threads initiated by splitterChannelTaskExecutor, they do their respective jobs now. I feel the main thread should be released now as it has delegated its task, but it doesn't look its getting released.
Edit:
public interface TestGateway {
void execute(Request request);
}
<bean id="requestChannelTaskExecutor" class="org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor" >
<property name="corePoolSize" value="10" />
<property name="daemon" value="false"/>
</bean>
<int:channel id="requestChannel">
<int:dispatcher task-executor="requestChannelTaskExecutor"/>
</int:channel>
Some thoughts...
Just return void if you don't care about the result.
How are you instantiating the application context? If using normal spring web, spring should take care of closing the context; if you are creating it yourself, it's up to you to close it.
Take a thread dump to see what the thread is doing.
EDIT:
If your flow never returns a reply, the TE within the gateway is stuck waiting for the reply.
So; return void and you won't have the problem.

Spring integration: how to handle exceptions in services after an aggregator?

I have an application relying on Spring Integration (4.0.4.RELEASE) and RabbitMQ. My flow is as follow:
Messages are put in queue via a process (they do not expect any answer):
Gateway -> Channel -> RabbitMQ
And then drained by another process:
RabbitMQ --1--> inbound-channel-adapter A --2--> chain B --3--> aggregator C --4--> service-activator D --5--> final service-activator E
Explanations & context
The specific thing is that nowhere in my application I am using a splitter: aggregator C just waits for enough messages to come, or for a timeout to expire, and then forwards the batch to service D. Messages can get stuck in aggregator C for quite a long time, and should NOT be considered as consumed there. They should only be consumed once service D successfully completes. Therefore, I am using MANUAL acknowledgement on inbound-channel-adapter A and service E is in charge of acknowledging the batch.
Custom aggregator
I solved the acknowledgement issue I had when set to AUTO by redefining the aggregator. Indeed, messages are acknowledged immediately if any asynchronous process occurs in the flow (see question here). Therefore, I switched to MANUAL acknowledgement and implemented the aggregator like this:
<bean class="org.springframework.integration.config.ConsumerEndpointFactoryBean">
<property name="inputChannel" ref="channel3"/>
<property name="handler">
<bean class="org.springframework.integration.aggregator.AggregatingMessageHandler">
<constructor-arg name="processor">
<bean class="com.test.AMQPAggregator"/>
</constructor-arg>
<property name="correlationStrategy">
<bean class="com.test.AggregatorDefaultCorrelationStrategy" />
</property>
<property name="releaseStrategy">
<bean class="com.test.AggregatorMongoReleaseStrategy" />
</property>
<property name="messageStore" ref="messageStoreBean"/>
<property name="expireGroupsUponCompletion" value="true"/>
<property name="sendPartialResultOnExpiry" value="true"/>
<property name="outputChannel" ref="channel4"/>
</bean>
</property>
</bean>
<bean id="messageStoreBean" class="org.springframework.integration.store.SimpleMessageStore"/>
<bean id="messageStoreReaperBean" class="org.springframework.integration.store.MessageGroupStoreReaper">
<property name="messageGroupStore" ref="messageStore" />
<property name="timeout" value="${myapp.timeout}" />
</bean>
<task:scheduled-tasks>
<task:scheduled ref="messageStoreReaperBean" method="run" fixed-rate="2000" />
</task:scheduled-tasks>
I wanted indeed to aggregate the headers in a different way, and keep the highest value of all the amqp_deliveryTag for later multi-acknoledgement in service E (see this thread). This works great so far, apart from the fact that it is far more verbose than the typical aggregator namespace (see this old Jira ticket).
Services
I am just using basic configurations:
chain-B
<int:chain input-channel="channel2" output-channel="channel3">
<int:header-enricher>
<int:error-channel ref="errorChannel" /> // Probably useless
</int:header-enricher>
<int:json-to-object-transformer/>
<int:transformer ref="serviceABean"
method="doThis" />
<int:transformer ref="serviceBBean"
method="doThat" />
</int:chain>
service-D
<int:service-activator ref="serviceDBean"
method="doSomething"
input-channel="channel4"
output-channel="channel5" />
Error management
As I rely on MANUAL acknowledgement, I need to manually reject messages as well in case an exception occurs. I have the following definition for inbound-channel-adapter A:
<int-amqp:inbound-channel-adapter channel="channel2"
queue-names="si.queue1"
error-channel="errorChannel"
mapped-request-headers="*"
acknowledge-mode="MANUAL"
prefetch-count="${properties.prefetch_count}"
connection-factory="rabbitConnectionFactory"/>
I use the following definition for errorChannel:
<int:chain input-channel="errorChannel">
<int:transformer ref="errorUnwrapperBean" method="unwrap" />
<int:service-activator ref="amqpAcknowledgerBean" method="rejectMessage" />
</int:chain>
ErrorUnwrapper is based on this code and the whole exception detection and message rejection works well until messages reach aggregator C.
Problem
If an exception is raised while processing the messages in service-activator D, then I see this exception but errorChannel does not seem to receive any message, and my ErrorUnwrapper unwrap() method is not called. The tailored stack traces I see when an Exception("ahahah") is thrown are as follow:
2014-09-23 16:41:18,725 ERROR o.s.i.s.SimpleMessageStore:174: Exception in expiry callback
org.springframework.messaging.MessageHandlingException: java.lang.Exception: ahahaha
at org.springframework.integration.handler.MethodInvokingMessageProcessor.processMessage(MethodInvokingMessageProcessor.java:78)
at org.springframework.integration.handler.ServiceActivatingHandler.handleRequestMessage(ServiceActivatingHandler.java:71)
at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:170)
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:78)
(...)
Caused by: java.lang.Exception: ahahaha
at com.myapp.ServiceD.doSomething(ServiceD.java:153)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
(...)
2014-09-23 16:41:18,733 ERROR o.s.s.s.TaskUtils$LoggingErrorHandler:95: Unexpected error occurred in scheduled task.
org.springframework.messaging.MessageHandlingException: java.lang.Exception: ahahaha
(...)
Question
How can one tell the services that process messages coming from such an aggregator to publish errors to errorChannel? I tried to specify in the header via a header-enricher the error-channel with no luck. I am using the default errorChannel definition, but I tried as well to change its name and redefine it. I am clueless here, and even though I found this and that, I have not managed to get it to work. Thanks in advance for your help!
As you see by StackTrace your process is started from the MessageGroupStoreReaper Thread, which is initiated from the default ThreadPoolTaskScheduler.
So, you must provide a custom bean for that:
<bean id="scheduler" class="org.springframework.scheduling.concurrent.ThreadPoolTaskScheduler">
<property name="errorHandler">
<bean class="org.springframework.integration.channel.MessagePublishingErrorHandler">
<property name="defaultErrorChannel" ref="errorChannel"/>
</bean>
</property>
</bean>
<task:scheduled-tasks scheduler="scheduler">
<task:scheduled ref="messageStoreReaperBean" method="run" fixed-rate="2000" />
</task:scheduled-tasks>
However I see the benefits from having the error-channel on the <aggregator>, where we really have several points from different detached Threads, with wich we can't get deal normally.

Right way to use transactional and request-handler-advice-chain in a JPAOutboundGateway

I've got for a JPA Outbound-channel-adapter both transactional and request-handler-advice-chain. In the advice-chain I try to log the Exception, when it happens.
It iss not logged, but I know that it actually happend since the Message was sent to failover clickDbFailoverChannel . What can be a problem with it? Is it a bug in Spring Integration?
<int:channel id="clickDbWithFailoverChannelSite-1">
<int:dispatcher load-balancer="none" task-executor="clickDbSiteRouterExecutor"/>
</int:channel>
<int:bridge input-channel="clickDbWithFailoverChannelSite-1"
output-channel="jpaOutboundChannelSite-1" order="1" send-timeout="100" />
<int:bridge input-channel="clickDbWithFailoverChannelSite-1"
output-channel="clickDbFailoverChannel" order="2" />
<int-jpa:outbound-channel-adapter id="jpaOutboundChannelSite-1"
persist-mode="PERSIST" flush-size="100" entity-manager-factory="emfSite-1">
<int-jpa:transactional transaction-manager="transactionManagerSite-1" />
<int-jpa:request-handler-advice-chain>
<bean class="org.springframework.integration.handler.advice.ExpressionEvaluatingRequestHandlerAdvice">
<property name="failureChannel" ref="clickDbFailureLogger"/>
<property name="onFailureExpression" value="#exception"/>
</bean>
</int-jpa:request-handler-advice-chain>
</int-jpa:outbound-channel-adapter>
OK. I can guess where is your issue. The real exception to rollback the transaction is caused before an internal logic, where <request-handler-advice-chain> does the stuff. That's why your ExpressionEvaluatingRequestHandlerAdvice doesn't get a failure message.
To workaround your rollback issue, you should replace <int-jpa:transactional> with <tx:advice> within <int-jpa:request-handler-advice-chain>.
You should understand here that <int-jpa:transactional> is for entire MessageHandler.handleMessage, but <int-jpa:request-handler-advice-chain> is just for its part - AbstractReplyProducingMessageHandler.handleRequestMessage.
UPDATE
TX Advice should be like this:
<tx:advice transaction-manager="transactionManagerSite-1"/>
<tx:attributes>
<tx:method name="*"/>
</tx:attributes>
</tx:advice>

Resources