I am trying to configure a poller that queries a bean to get a List into a channel every X seconds. This channel has a downstream flow which splits the list and outputs to a pub/sub channel (further async flow)
How can I make sure that at any given time only once execution of flow is in flight and poller has to wait/block until the flow completes until it is ready for the next poll (fixed rate/delay)?
<int:channel id="configListChannel" />
<task:executor id="pollExecutor" pool-size="1" queue-capacity="1" rejection-policy="ABORT" />
<int:inbound-channel-adapter expression="configMap().values()" auto-startup="true" channel="configListChannel">
<int:poller fixed-delay="30" time-unit="SECONDS" task-executor="pollExecutor"/>
</int:inbound-channel-adapter>
<task:executor id="configExecutor" pool-size="5"/>
<int:channel id="configChannel" >
<int:dispatcher task-executor="configExecutor"/>
</int:channel>
<int:chain input-channel="configListChannel" output-channel="configChannel" id="configChain">
<int:splitter/>
<int:filter expression="payload.enablePolling"/>
</int:chain>
... further async flow on configChannel to send outbound messages
Any examples of a blocking poller with async hand off and using barrier to signal flow complete to the poller thread? Also only one poll at a time.
I would suggest you to implement a ReceiveMessageAdvice (since 5.3 or AbstractMessageSourceAdvice otherwise). It's afterReceive() should just return the message as is, but beforeReceive() should check some state and return false if you can't poll at the moment.
You probably don't need a barrier for that task, but simple AtomicBoolean bean to check the state in that beforeReceive() to false and bring it back to true when you finish your task downstream.
Related
I asked a question a bit ago about a specific use case when multi-threading in spring integration, basically I have to have one thread of execution that stays on the initial and then a second that spawns a thread. I have implemented this like so
<int:channel id="newThread" >
<int:dispatcher task-executor="workerThreadPoolAdapter"/>
</int:channel>
<!--This bean is used to split incoming messages -->
<bean id="splitterBean" class="orchestration.MessageSplitter">
<property name="channels" ref="splitterChannelsList" />
</bean>
<util:list id="splitterChannelsList" value-type="java.lang.String">
<value>newThread</value>
<value>mainThread</value>
</util:list>
<!-- This bean is used to aggregate incoming messages -->
<bean id="aggregator" class="orchestration.MessageAggregator">
<property name="wrapperNode" value="container" />
</bean>
<!-- Channel for aggregator output and that will be input for response transformer -->
<int:publish-subscribe-channel id="gatherChannel" apply-sequence="true"/>
<!-- This splitter splits request and send -->
<!-- will add a header called channelHeader which is the channel the message should be routed to using the recipient-list-router -->
<int:splitter input-channel="splitter"
output-channel="recipientListRouter"
apply-sequence="true"
ref="splitterBean" method="split" />
<!-- Aggregator that aggregates responses received from the calls -->
<int:aggregator input-channel="gatherChannel"
output-channel="transformResponse"
ref="aggregator"
method="aggregateMessages"
send-partial-result-on-expiry="false"
expire-groups-upon-completion="true"
message-store="removeMessageFromStore"
release-strategy-expression="size() == 2"/>
<int:recipient-list-router input-channel="recipientListRouter">
<int:recipient channel="mainThread"
selector-expression="headers.get('channelHeader') == 'mainThread'"/>
<int:recipient channel="newThread"
selector-expression="headers.get('channelHeader') == 'newThread'"/>
</int:recipient-list-router>
<!-- only route to call if boolean is populated -->
<int:header-value-router header-name="shouldMakeExtraOutboundCall"
input-channel="newThread"
default-output-channel="gatherChannel" >
<int:mapping value="true" channel="outboundCall" />
</int:header-value-router>
<int:chain input-channel="outboundCall" output-channel="gatherChannel">
<!-- make an outbound call -->
</int:chain>
<int:chain input-channel="mainThread" output-channel="gatherChannel">
<!-- make a bunch of outbound calls -->
</int:chain>
<int:chain input-channel="transformResponse" output-channel="backToClient">
<!-- do some stuff and respond back to client -->
</int:chain>
I've had the output channel of the aggregator as both a publish-subscribe and a direct channel and had the issue for both.
When I look at the logs I can see that one of the threads has a preSend to the 'gatherChannel' then AggregatingMessageHandler saying it received the message, then another log from AggregatingMessageHandler saying Handling message with correlationKey [f2b16b6a-3605-778f-a628-870ed8ce3f5e] then a postSend (sent=True) on channel 'gatherChannel'.
I thought that it would not send to the transformResponse channel until both messages that got split out from the splitter got to it. I even added the size() == 2 as the release strategy expression as an extra layer but that doesn't seem to be causing the aggregator to wait either.
I'm a little perplexed why this is happening, it's happening when both the main thread or the spawned thread gets to the aggregator, I'm trying to figure out how to get that aggregator to wait to send to the output channel until BOTH messages that were split from the splitter are received.
According to your current configuration it really does not happen.
Do you really have some evidences that first message in the group is really sent to that transformResponse unconditionally?
You probably is a bit confused with those preSend and postSend since an aggregator is really non-blocking until release. It does accept the message and store it into the MessageStore for grouping if release condition is false. Just after this it returns immediately to the caller, which is that gatherChannel and therefore your see postSend on a first message.
I am new to spring integration and we have created an SI flow where we have Splitter and Aggregator also recipient-list-router and Aggregator.
Today, while checking a code I got confused about how Aggregator will clean its store if we have an exception in between flow.
I am worried about the scenario where we got an exception between the flow and that creates stale state object in the system.
I have checked the spring integration doc but no luck (https://docs.spring.io/spring-integration/docs/2.0.0.RC1/reference/html/aggregator.html).
I can see only one topic "Managing State in an Aggregator: MessageGroupStore" but that is for "application shots down".
Also, I did google for the same and I found one thread https://dzone.com/articles/spring-integration-robust but not able to folow much. Sure, I will come back if I am able to find some solution.
I am using OOB Splitter, recipient-list-router and Aggregator. Considering pattern should have mechanism handle this common scenario.
Can you please guide me
i.e:
<int:recipient-list-router input-channel="inputChannel"
default-output-channel="nullChannel">
<int:recipient channel="aInputChannel" />
<int:recipient channel="bInputChannel" />
</int:recipient-list-router>
<int:service-activator ref="aHandler"
input-channel="aInputChannel" output-channel="aggregatorOutputChannel" />
<!-- we have exception in the bHandler -->
<int:service-activator ref="bHandler"
input-channel="bInputChannel" output-channel="aggregatorOutputChannel" />
<int:aggregator input-channel="aggregatorOutputChannel"
output-channel="outputChannel" />
OR
<int-file:splitter id="splitile"
charset="UTF-8" apply-sequence="true" iterator="false"
input-channel="inputChannel"
output-channel="bTransformerChannel" />
<!-- consider we have exception at 4th chunk -->
<int:service-activator ref="transform"
input-channel="bTransformerChannel" output-channel="aggregatorOutputChannel" />
<int:aggregator input-channel="aggregatorOutputChannel"
output-channel="outputChannel" />
Yes; the aggregator is a "passive" component by default - all actions are taken when a message arrives.
To time out stale groups you can use a reaper or, with more recent versions, a group-timeout.
I have a thread that uses gateway (void) to send a message to both (pub/sub):
barrier, to hold the thread during the real execution (requires-reply="true" timeout="XXXX", output-channel="nullChannel"
and to
splitter which next sends splits as messages to the service activator (direct channel) with poller and a thread executor for the actual processing/execution
How to properly configure handling the exceptions that might be thrown by the executor threads and catch them in the below catch block:
try {
gateway.trigger()
} catch (ReplyRequiredException e) {
//fine here
} catch (Throwable t) {
// catch every exception here... or somehow configure these exceptions to discard the thread that waits on the barrier and throw below business exception
throw new SomeExecutionFailedException()
}
EDIT
<!--gateway.trigger()—>
<int:gateway id=“gateway"
service-interface="com.Gateway"
default-request-channel=“channel1"
default-reply-timeout="0"/>
<int:publish-subscribe-channel id=“channel1"/>
<int:splitter input-channel=“channel1" output-channel=“channel2"
order="1">
<bean class=“com.Splitter"/>
</int:splitter>
<int:barrier id=“barrier" input-channel=“channel1"
output-channel="nullChannel"
correlation-strategy-expression=“'XXX’” <!--hack here-->
requires-reply="true"
timeout=“40000"
order="2">
</int:barrier>
<int:channel id=“channel2">
<int:queue capacity="30"/>
</int:channel>
<!— actual processing/execution —>
<int:service-activator id=“executionAct" input-channel=“channel2"
output-channel=“channel3" ref=“executionService">
<int:poller fixed-rate="111" time-unit="MILLISECONDS" max-messages-per-poll="22"
task-executor=“exec"/>
</int:service-activator>
<bean id=“executionService" class=“com.SomeExecService"/>
<bean id=“exec" class="org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor">
<property name="threadFactory" ref=“execFactory"/>
...
<property name="rejectedExecutionHandler">
<bean class="java.util.concurrent.ThreadPoolExecutor.CallerRunsPolicy"/>
</property>
</bean>
<bean id=“execFactory"
class="org.springframework.scheduling.concurrent.CustomizableThreadFactory">
...
</bean>
<int:channel id=“channel3"/>
<int:chain input-channel=“channel3" output-channel=“channel4">
...
<int:aggregator
group-timeout=“30000"
discard-channel=“discardChannel" release-strategy=“com.ReleaseStrategy"
send-partial-result-on-expiry="false">
<bean class="com.Aggregator"/>
</int:aggregator>
</int:chain>
<int:channel id=“discardChannel”/>
<int:channel id=“channel4"/>
<!— processing done - wake up barrier —>
<int:service-activator id=“barrierReleaseAct" input-channel=“channel4" output-channel="nullChannel">
<bean class="com.ServiceThatSendsXXXMessageToChannel5ToReleaseBarrier"/>
</int:service-activator>
<int:channel id=“channel5"/>
<int:outbound-channel-adapter channel=“channel5"
ref=“barrier" method="trigger"/>
You need to provide much more information, configuration etc.
What releases the barrier, and when?
Do you want to propagate the exception(s) to the main thread?
What if multiple splits fail, etc, etc.
The general answer is sending a message with a Throwable payload to the barrier's trigger method will release the thread by throwing a MessagingException with the Throwable as its cause. The gateway unwraps the MessagingException and throws the cause (which is the original payload sent to the barrier's trigger method).
EDIT
I have added a pull request to the barrier sample app to show one technique of collecting exceptions on the async threads and causing the barrier to throw a consolidated exception back to the gateway caller.
I have a rest call to do inside splitter and then aggregate. I planned to implement parallelism for this rest call. So i introduced task executor as per this link. Now parallelism works but sometimes it is working and sometimes not. Guessing the aggregator is not waiting for all the thread to finish. Not sure exactly the problem. Can you help me here.
<int:enricher ipChann="som" opCahnnel="inputChannel" />
<int:splitter ipchan="inputChannel" opChannel="opchannel" ref="customersplitter" />
<int:channel id="opchannel" >
<int:dispatcher task-executor="exec" />
</int:channel>
<task:executor id="exec" pool-size="4" queue-capacity="10"/>
<int:enricher ipChannel="opchannel" opChannel="aggregatorChan"></int:enricher>
<int:aggregator ipChann="aggregatorChan" />
For simplicity i didnt expand enricher, but the flow are same.
I have been following previous answers that talk about inserting gateway / service-activator to start a new transaction mid way in SI processing. I have tried below code but for some reason it does not work, if you could point to error in this configuration please.
All the threads seems to be waiting for something to happen and the sp outbound gateway is not invoked, but i cant figure out what.
The idea here is to process each task produced by splitter in a thread pool under a new transaction.
<int:splitter...output-channel="taskChannel"/>
<int:channel id="taskChannel">
<int:dispatcher task-executor="taskExecutor"/>
</int:channel>
<int:gateway id="txGw" service-interface="com.some.test.StartTransactionalGateway"
default-request-channel="taskChannel" default-reply-channel="individualTask">
</int:gateway>
<int:service-activator ref="txGw"
input-channel="taskChannel"
output-channel="individualTask"
method="startTx"
auto-startup="true">
</int:service-activator>
<int-jdbc:stored-proc-outbound-gateway ...request-channel="individualTask" .....
interface StartTransactionalGateway {
#Transactional
Message<?> startTx(Message<?> m);
}
default-request-channel="taskChannel" the gateway is sending messages to itself.
You can't mix channels in the subflow with the main channels. You need something like...
<int:splitter...output-channel="taskChannel"/>
<int:channel id="taskChannel">
<int:dispatcher task-executor="taskExecutor"/>
</int:channel>
<int:service-activator ref="txGw"
input-channel="taskChannel"
output-channel="whereWeWantTheResultsToGo"
method="startTx"
reply-timeout="0"
auto-startup="true" />
<int:gateway id="txGw" service-interface="com.some.test.StartTransactionalGateway"
default-request-channel="toStoredProc" />
<int-jdbc:stored-proc-outbound-gateway ...request-channel="toStoredProd" .....
You don't need a default-reply-channel; simply omit the reply-channel on the stored proc gateway and the reply will automatically go back.
Well, Don't forget that you can mark something transactional not only using #Transactional annotation. There is <tx:advice> for the XML declaration.
If you need to wrap to the TX just only that <int-jdbc:stored-proc-outbound-gateway>, there is <request-handler-advice> sub-element to wrap the handleRequestMessage of the underlying component to any Advice:
<int-jdbc:request-handler-advice-chain>
<tx:advice>
<tx:attributes>
<tx:method name="*"/>
</tx:attributes>
</tx:advice>
</int-jdbc:request-handler-advice-chain>
From other side your code can not work, because <int-jdbc:stored-proc-outbound-gateway> may not return result. The procedure is one-way. But the <gateway> interface is request-reply.
UPDATE
See my answer here Keep transaction within Spring Integration flow. It is another trick how to make down-stream flow transactional.
With <gateway> you should be sure that your transactional subflow returns to the replyChannel in the end. Or... Make your gateway's method as void to achieve one-way behaviour.