I have a rest call to do inside splitter and then aggregate. I planned to implement parallelism for this rest call. So i introduced task executor as per this link. Now parallelism works but sometimes it is working and sometimes not. Guessing the aggregator is not waiting for all the thread to finish. Not sure exactly the problem. Can you help me here.
<int:enricher ipChann="som" opCahnnel="inputChannel" />
<int:splitter ipchan="inputChannel" opChannel="opchannel" ref="customersplitter" />
<int:channel id="opchannel" >
<int:dispatcher task-executor="exec" />
</int:channel>
<task:executor id="exec" pool-size="4" queue-capacity="10"/>
<int:enricher ipChannel="opchannel" opChannel="aggregatorChan"></int:enricher>
<int:aggregator ipChann="aggregatorChan" />
For simplicity i didnt expand enricher, but the flow are same.
Related
I asked a question a bit ago about a specific use case when multi-threading in spring integration, basically I have to have one thread of execution that stays on the initial and then a second that spawns a thread. I have implemented this like so
<int:channel id="newThread" >
<int:dispatcher task-executor="workerThreadPoolAdapter"/>
</int:channel>
<!--This bean is used to split incoming messages -->
<bean id="splitterBean" class="orchestration.MessageSplitter">
<property name="channels" ref="splitterChannelsList" />
</bean>
<util:list id="splitterChannelsList" value-type="java.lang.String">
<value>newThread</value>
<value>mainThread</value>
</util:list>
<!-- This bean is used to aggregate incoming messages -->
<bean id="aggregator" class="orchestration.MessageAggregator">
<property name="wrapperNode" value="container" />
</bean>
<!-- Channel for aggregator output and that will be input for response transformer -->
<int:publish-subscribe-channel id="gatherChannel" apply-sequence="true"/>
<!-- This splitter splits request and send -->
<!-- will add a header called channelHeader which is the channel the message should be routed to using the recipient-list-router -->
<int:splitter input-channel="splitter"
output-channel="recipientListRouter"
apply-sequence="true"
ref="splitterBean" method="split" />
<!-- Aggregator that aggregates responses received from the calls -->
<int:aggregator input-channel="gatherChannel"
output-channel="transformResponse"
ref="aggregator"
method="aggregateMessages"
send-partial-result-on-expiry="false"
expire-groups-upon-completion="true"
message-store="removeMessageFromStore"
release-strategy-expression="size() == 2"/>
<int:recipient-list-router input-channel="recipientListRouter">
<int:recipient channel="mainThread"
selector-expression="headers.get('channelHeader') == 'mainThread'"/>
<int:recipient channel="newThread"
selector-expression="headers.get('channelHeader') == 'newThread'"/>
</int:recipient-list-router>
<!-- only route to call if boolean is populated -->
<int:header-value-router header-name="shouldMakeExtraOutboundCall"
input-channel="newThread"
default-output-channel="gatherChannel" >
<int:mapping value="true" channel="outboundCall" />
</int:header-value-router>
<int:chain input-channel="outboundCall" output-channel="gatherChannel">
<!-- make an outbound call -->
</int:chain>
<int:chain input-channel="mainThread" output-channel="gatherChannel">
<!-- make a bunch of outbound calls -->
</int:chain>
<int:chain input-channel="transformResponse" output-channel="backToClient">
<!-- do some stuff and respond back to client -->
</int:chain>
I've had the output channel of the aggregator as both a publish-subscribe and a direct channel and had the issue for both.
When I look at the logs I can see that one of the threads has a preSend to the 'gatherChannel' then AggregatingMessageHandler saying it received the message, then another log from AggregatingMessageHandler saying Handling message with correlationKey [f2b16b6a-3605-778f-a628-870ed8ce3f5e] then a postSend (sent=True) on channel 'gatherChannel'.
I thought that it would not send to the transformResponse channel until both messages that got split out from the splitter got to it. I even added the size() == 2 as the release strategy expression as an extra layer but that doesn't seem to be causing the aggregator to wait either.
I'm a little perplexed why this is happening, it's happening when both the main thread or the spawned thread gets to the aggregator, I'm trying to figure out how to get that aggregator to wait to send to the output channel until BOTH messages that were split from the splitter are received.
According to your current configuration it really does not happen.
Do you really have some evidences that first message in the group is really sent to that transformResponse unconditionally?
You probably is a bit confused with those preSend and postSend since an aggregator is really non-blocking until release. It does accept the message and store it into the MessageStore for grouping if release condition is false. Just after this it returns immediately to the caller, which is that gatherChannel and therefore your see postSend on a first message.
I have registered an AsyncHandler and also added a success-channel to an SQS outbound flow. The success-channel has a int:logging-channel-adapter endpoint. However I am not able to see any logs from this adapter. The AsyncHandler is able to receive the call-backs but nothing on the success-channel.
In the SqsMessageHandler I see that we are setting an output channel in the obtainAsyncHandler method, but I did not see the success-channel set anywhere. Am I missing something?
I would prefer using the success and failure channels and not AsyncHandler call-back Impl to avoid having AWS specific code in my classes.
Also my <int-aws:sqs-outbound-channel-adapter> is inside a <int:chain> which has no output channel, since the flow ends when the message is sent.
EDIT - Added Config
This is the only way I can get it to log the callback.
<int:channel id="chainChannel" />
<int:channel id="successChannel" />
<bean class="ServiceTransformer" id="serviceTransformer" />
<int:chain input-channel="serviceChannel" id="sendToServiceSqsChain" output-channel="chainChannel">
<int:transformer ref="serviceTransformer" method="transform" />
<int:header-filter header-names="config" />
<int-aws:sqs-outbound-channel-adapter sqs="amazonSQS" queue="some-queue" async-handler="sqsPublishCallbackHandler" success-channel="successChannel"/>
</int:chain>
<int:logging-channel-adapter log-full-message="true" channel="chainChannel" />
Here I can just use the same channel in both chain (outbound channel) and sqs-outbound (success-channel)
Unable to get it to work like below:
<int:channel id="successChannel" />
<bean class="ServiceTransformer" id="serviceTransformer" />
<int:chain input-channel="serviceChannel" id="sendToServiceSqsChain" >
<int:transformer ref="serviceTransformer" method="transform" />
<int:header-filter header-names="config" />
<int-aws:sqs-outbound-channel-adapter sqs="amazonSQS" queue="some-queue" async-handler="sqsPublishCallbackHandler" success-channel="successChannel"/>
</int:chain>
<int:logging-channel-adapter log-full-message="true" channel="successChannel" />
The <int-aws:sqs-outbound-channel-adapter> component is one-way, therefore there is no outputChannel option expose. However the target class is AbstractMessageProducingHandler. To avoid code duplication we reuse an existing outputChannel internally for that AsyncHandler.
In the XML parser we just remap one to another:
IntegrationNamespaceUtils.setReferenceIfAttributeDefined(builder, element, "success-channel", "outputChannel");
You probably don't see anything in logs because you need to adjust logging config respectively for the appropriate category and level.
UPDATE
According my testing this is definitely not possible to configure such a component with XML DSL within the <chain>. That <int-aws:sqs-outbound-channel-adapter> has to be presented outside of the <chain>.
Consider to more your configuration to Java DSL instead: https://docs.spring.io/spring-integration/docs/5.3.2.RELEASE/reference/html/dsl.html#java-dsl.
I am trying to configure a poller that queries a bean to get a List into a channel every X seconds. This channel has a downstream flow which splits the list and outputs to a pub/sub channel (further async flow)
How can I make sure that at any given time only once execution of flow is in flight and poller has to wait/block until the flow completes until it is ready for the next poll (fixed rate/delay)?
<int:channel id="configListChannel" />
<task:executor id="pollExecutor" pool-size="1" queue-capacity="1" rejection-policy="ABORT" />
<int:inbound-channel-adapter expression="configMap().values()" auto-startup="true" channel="configListChannel">
<int:poller fixed-delay="30" time-unit="SECONDS" task-executor="pollExecutor"/>
</int:inbound-channel-adapter>
<task:executor id="configExecutor" pool-size="5"/>
<int:channel id="configChannel" >
<int:dispatcher task-executor="configExecutor"/>
</int:channel>
<int:chain input-channel="configListChannel" output-channel="configChannel" id="configChain">
<int:splitter/>
<int:filter expression="payload.enablePolling"/>
</int:chain>
... further async flow on configChannel to send outbound messages
Any examples of a blocking poller with async hand off and using barrier to signal flow complete to the poller thread? Also only one poll at a time.
I would suggest you to implement a ReceiveMessageAdvice (since 5.3 or AbstractMessageSourceAdvice otherwise). It's afterReceive() should just return the message as is, but beforeReceive() should check some state and return false if you can't poll at the moment.
You probably don't need a barrier for that task, but simple AtomicBoolean bean to check the state in that beforeReceive() to false and bring it back to true when you finish your task downstream.
I am new to spring integration and we have created an SI flow where we have Splitter and Aggregator also recipient-list-router and Aggregator.
Today, while checking a code I got confused about how Aggregator will clean its store if we have an exception in between flow.
I am worried about the scenario where we got an exception between the flow and that creates stale state object in the system.
I have checked the spring integration doc but no luck (https://docs.spring.io/spring-integration/docs/2.0.0.RC1/reference/html/aggregator.html).
I can see only one topic "Managing State in an Aggregator: MessageGroupStore" but that is for "application shots down".
Also, I did google for the same and I found one thread https://dzone.com/articles/spring-integration-robust but not able to folow much. Sure, I will come back if I am able to find some solution.
I am using OOB Splitter, recipient-list-router and Aggregator. Considering pattern should have mechanism handle this common scenario.
Can you please guide me
i.e:
<int:recipient-list-router input-channel="inputChannel"
default-output-channel="nullChannel">
<int:recipient channel="aInputChannel" />
<int:recipient channel="bInputChannel" />
</int:recipient-list-router>
<int:service-activator ref="aHandler"
input-channel="aInputChannel" output-channel="aggregatorOutputChannel" />
<!-- we have exception in the bHandler -->
<int:service-activator ref="bHandler"
input-channel="bInputChannel" output-channel="aggregatorOutputChannel" />
<int:aggregator input-channel="aggregatorOutputChannel"
output-channel="outputChannel" />
OR
<int-file:splitter id="splitile"
charset="UTF-8" apply-sequence="true" iterator="false"
input-channel="inputChannel"
output-channel="bTransformerChannel" />
<!-- consider we have exception at 4th chunk -->
<int:service-activator ref="transform"
input-channel="bTransformerChannel" output-channel="aggregatorOutputChannel" />
<int:aggregator input-channel="aggregatorOutputChannel"
output-channel="outputChannel" />
Yes; the aggregator is a "passive" component by default - all actions are taken when a message arrives.
To time out stale groups you can use a reaper or, with more recent versions, a group-timeout.
We are using header-value-router. Configuration:
<int:header-value-router input-channel="accountSummeryRequest"
header-name="word"
default-output-channel="accountSummeryRequest"
resolution-required="false">
<int:mapping value="xx" channel="accountSummeryRequest" />
<int:mapping value="yy" channel="newRequestChannel" />
</int:header-value-router>
<int:service-activator id="accountServiceActivator"
input-channel="accountSummeryRequest"
output-channel="accountSummeryResponse"
ref="serviceGatewayAdapter"
method="requestHandler"
send-timeout="60000"/>
<int:service-activator id="caRequestActivator"
input-channel="newRequestChannel"
output-channel="accountSummeryResponse"
ref="caServiceGatewayAdapter"
method="requestHandler"
send-timeout="60000"/>
now if i give word as yy,first time header-value-router enter code hereis getting called and exact service activator,in this case is caRequestActivator ,is called. But i try again with word=yy header-value-router is not getting called and request goes through accountServiceActivator. Alternate requests works correctly.
I don't know what's the problem here.
Your issue is around a round-robin dispatcher for accountSummeryRequest channel and its two subscribers: <int:header-value-router> and accountServiceActivator.
To fix it you should change the input-channel of that <service-activator> to some different channel. And, of course, don't fogtet to change the <header-value-router>
accordingly.