I have been following previous answers that talk about inserting gateway / service-activator to start a new transaction mid way in SI processing. I have tried below code but for some reason it does not work, if you could point to error in this configuration please.
All the threads seems to be waiting for something to happen and the sp outbound gateway is not invoked, but i cant figure out what.
The idea here is to process each task produced by splitter in a thread pool under a new transaction.
<int:splitter...output-channel="taskChannel"/>
<int:channel id="taskChannel">
<int:dispatcher task-executor="taskExecutor"/>
</int:channel>
<int:gateway id="txGw" service-interface="com.some.test.StartTransactionalGateway"
default-request-channel="taskChannel" default-reply-channel="individualTask">
</int:gateway>
<int:service-activator ref="txGw"
input-channel="taskChannel"
output-channel="individualTask"
method="startTx"
auto-startup="true">
</int:service-activator>
<int-jdbc:stored-proc-outbound-gateway ...request-channel="individualTask" .....
interface StartTransactionalGateway {
#Transactional
Message<?> startTx(Message<?> m);
}
default-request-channel="taskChannel" the gateway is sending messages to itself.
You can't mix channels in the subflow with the main channels. You need something like...
<int:splitter...output-channel="taskChannel"/>
<int:channel id="taskChannel">
<int:dispatcher task-executor="taskExecutor"/>
</int:channel>
<int:service-activator ref="txGw"
input-channel="taskChannel"
output-channel="whereWeWantTheResultsToGo"
method="startTx"
reply-timeout="0"
auto-startup="true" />
<int:gateway id="txGw" service-interface="com.some.test.StartTransactionalGateway"
default-request-channel="toStoredProc" />
<int-jdbc:stored-proc-outbound-gateway ...request-channel="toStoredProd" .....
You don't need a default-reply-channel; simply omit the reply-channel on the stored proc gateway and the reply will automatically go back.
Well, Don't forget that you can mark something transactional not only using #Transactional annotation. There is <tx:advice> for the XML declaration.
If you need to wrap to the TX just only that <int-jdbc:stored-proc-outbound-gateway>, there is <request-handler-advice> sub-element to wrap the handleRequestMessage of the underlying component to any Advice:
<int-jdbc:request-handler-advice-chain>
<tx:advice>
<tx:attributes>
<tx:method name="*"/>
</tx:attributes>
</tx:advice>
</int-jdbc:request-handler-advice-chain>
From other side your code can not work, because <int-jdbc:stored-proc-outbound-gateway> may not return result. The procedure is one-way. But the <gateway> interface is request-reply.
UPDATE
See my answer here Keep transaction within Spring Integration flow. It is another trick how to make down-stream flow transactional.
With <gateway> you should be sure that your transactional subflow returns to the replyChannel in the end. Or... Make your gateway's method as void to achieve one-way behaviour.
Related
I asked a question a bit ago about a specific use case when multi-threading in spring integration, basically I have to have one thread of execution that stays on the initial and then a second that spawns a thread. I have implemented this like so
<int:channel id="newThread" >
<int:dispatcher task-executor="workerThreadPoolAdapter"/>
</int:channel>
<!--This bean is used to split incoming messages -->
<bean id="splitterBean" class="orchestration.MessageSplitter">
<property name="channels" ref="splitterChannelsList" />
</bean>
<util:list id="splitterChannelsList" value-type="java.lang.String">
<value>newThread</value>
<value>mainThread</value>
</util:list>
<!-- This bean is used to aggregate incoming messages -->
<bean id="aggregator" class="orchestration.MessageAggregator">
<property name="wrapperNode" value="container" />
</bean>
<!-- Channel for aggregator output and that will be input for response transformer -->
<int:publish-subscribe-channel id="gatherChannel" apply-sequence="true"/>
<!-- This splitter splits request and send -->
<!-- will add a header called channelHeader which is the channel the message should be routed to using the recipient-list-router -->
<int:splitter input-channel="splitter"
output-channel="recipientListRouter"
apply-sequence="true"
ref="splitterBean" method="split" />
<!-- Aggregator that aggregates responses received from the calls -->
<int:aggregator input-channel="gatherChannel"
output-channel="transformResponse"
ref="aggregator"
method="aggregateMessages"
send-partial-result-on-expiry="false"
expire-groups-upon-completion="true"
message-store="removeMessageFromStore"
release-strategy-expression="size() == 2"/>
<int:recipient-list-router input-channel="recipientListRouter">
<int:recipient channel="mainThread"
selector-expression="headers.get('channelHeader') == 'mainThread'"/>
<int:recipient channel="newThread"
selector-expression="headers.get('channelHeader') == 'newThread'"/>
</int:recipient-list-router>
<!-- only route to call if boolean is populated -->
<int:header-value-router header-name="shouldMakeExtraOutboundCall"
input-channel="newThread"
default-output-channel="gatherChannel" >
<int:mapping value="true" channel="outboundCall" />
</int:header-value-router>
<int:chain input-channel="outboundCall" output-channel="gatherChannel">
<!-- make an outbound call -->
</int:chain>
<int:chain input-channel="mainThread" output-channel="gatherChannel">
<!-- make a bunch of outbound calls -->
</int:chain>
<int:chain input-channel="transformResponse" output-channel="backToClient">
<!-- do some stuff and respond back to client -->
</int:chain>
I've had the output channel of the aggregator as both a publish-subscribe and a direct channel and had the issue for both.
When I look at the logs I can see that one of the threads has a preSend to the 'gatherChannel' then AggregatingMessageHandler saying it received the message, then another log from AggregatingMessageHandler saying Handling message with correlationKey [f2b16b6a-3605-778f-a628-870ed8ce3f5e] then a postSend (sent=True) on channel 'gatherChannel'.
I thought that it would not send to the transformResponse channel until both messages that got split out from the splitter got to it. I even added the size() == 2 as the release strategy expression as an extra layer but that doesn't seem to be causing the aggregator to wait either.
I'm a little perplexed why this is happening, it's happening when both the main thread or the spawned thread gets to the aggregator, I'm trying to figure out how to get that aggregator to wait to send to the output channel until BOTH messages that were split from the splitter are received.
According to your current configuration it really does not happen.
Do you really have some evidences that first message in the group is really sent to that transformResponse unconditionally?
You probably is a bit confused with those preSend and postSend since an aggregator is really non-blocking until release. It does accept the message and store it into the MessageStore for grouping if release condition is false. Just after this it returns immediately to the caller, which is that gatherChannel and therefore your see postSend on a first message.
I have registered an AsyncHandler and also added a success-channel to an SQS outbound flow. The success-channel has a int:logging-channel-adapter endpoint. However I am not able to see any logs from this adapter. The AsyncHandler is able to receive the call-backs but nothing on the success-channel.
In the SqsMessageHandler I see that we are setting an output channel in the obtainAsyncHandler method, but I did not see the success-channel set anywhere. Am I missing something?
I would prefer using the success and failure channels and not AsyncHandler call-back Impl to avoid having AWS specific code in my classes.
Also my <int-aws:sqs-outbound-channel-adapter> is inside a <int:chain> which has no output channel, since the flow ends when the message is sent.
EDIT - Added Config
This is the only way I can get it to log the callback.
<int:channel id="chainChannel" />
<int:channel id="successChannel" />
<bean class="ServiceTransformer" id="serviceTransformer" />
<int:chain input-channel="serviceChannel" id="sendToServiceSqsChain" output-channel="chainChannel">
<int:transformer ref="serviceTransformer" method="transform" />
<int:header-filter header-names="config" />
<int-aws:sqs-outbound-channel-adapter sqs="amazonSQS" queue="some-queue" async-handler="sqsPublishCallbackHandler" success-channel="successChannel"/>
</int:chain>
<int:logging-channel-adapter log-full-message="true" channel="chainChannel" />
Here I can just use the same channel in both chain (outbound channel) and sqs-outbound (success-channel)
Unable to get it to work like below:
<int:channel id="successChannel" />
<bean class="ServiceTransformer" id="serviceTransformer" />
<int:chain input-channel="serviceChannel" id="sendToServiceSqsChain" >
<int:transformer ref="serviceTransformer" method="transform" />
<int:header-filter header-names="config" />
<int-aws:sqs-outbound-channel-adapter sqs="amazonSQS" queue="some-queue" async-handler="sqsPublishCallbackHandler" success-channel="successChannel"/>
</int:chain>
<int:logging-channel-adapter log-full-message="true" channel="successChannel" />
The <int-aws:sqs-outbound-channel-adapter> component is one-way, therefore there is no outputChannel option expose. However the target class is AbstractMessageProducingHandler. To avoid code duplication we reuse an existing outputChannel internally for that AsyncHandler.
In the XML parser we just remap one to another:
IntegrationNamespaceUtils.setReferenceIfAttributeDefined(builder, element, "success-channel", "outputChannel");
You probably don't see anything in logs because you need to adjust logging config respectively for the appropriate category and level.
UPDATE
According my testing this is definitely not possible to configure such a component with XML DSL within the <chain>. That <int-aws:sqs-outbound-channel-adapter> has to be presented outside of the <chain>.
Consider to more your configuration to Java DSL instead: https://docs.spring.io/spring-integration/docs/5.3.2.RELEASE/reference/html/dsl.html#java-dsl.
I am trying to configure a poller that queries a bean to get a List into a channel every X seconds. This channel has a downstream flow which splits the list and outputs to a pub/sub channel (further async flow)
How can I make sure that at any given time only once execution of flow is in flight and poller has to wait/block until the flow completes until it is ready for the next poll (fixed rate/delay)?
<int:channel id="configListChannel" />
<task:executor id="pollExecutor" pool-size="1" queue-capacity="1" rejection-policy="ABORT" />
<int:inbound-channel-adapter expression="configMap().values()" auto-startup="true" channel="configListChannel">
<int:poller fixed-delay="30" time-unit="SECONDS" task-executor="pollExecutor"/>
</int:inbound-channel-adapter>
<task:executor id="configExecutor" pool-size="5"/>
<int:channel id="configChannel" >
<int:dispatcher task-executor="configExecutor"/>
</int:channel>
<int:chain input-channel="configListChannel" output-channel="configChannel" id="configChain">
<int:splitter/>
<int:filter expression="payload.enablePolling"/>
</int:chain>
... further async flow on configChannel to send outbound messages
Any examples of a blocking poller with async hand off and using barrier to signal flow complete to the poller thread? Also only one poll at a time.
I would suggest you to implement a ReceiveMessageAdvice (since 5.3 or AbstractMessageSourceAdvice otherwise). It's afterReceive() should just return the message as is, but beforeReceive() should check some state and return false if you can't poll at the moment.
You probably don't need a barrier for that task, but simple AtomicBoolean bean to check the state in that beforeReceive() to false and bring it back to true when you finish your task downstream.
I am new to spring integration and we have created an SI flow where we have Splitter and Aggregator also recipient-list-router and Aggregator.
Today, while checking a code I got confused about how Aggregator will clean its store if we have an exception in between flow.
I am worried about the scenario where we got an exception between the flow and that creates stale state object in the system.
I have checked the spring integration doc but no luck (https://docs.spring.io/spring-integration/docs/2.0.0.RC1/reference/html/aggregator.html).
I can see only one topic "Managing State in an Aggregator: MessageGroupStore" but that is for "application shots down".
Also, I did google for the same and I found one thread https://dzone.com/articles/spring-integration-robust but not able to folow much. Sure, I will come back if I am able to find some solution.
I am using OOB Splitter, recipient-list-router and Aggregator. Considering pattern should have mechanism handle this common scenario.
Can you please guide me
i.e:
<int:recipient-list-router input-channel="inputChannel"
default-output-channel="nullChannel">
<int:recipient channel="aInputChannel" />
<int:recipient channel="bInputChannel" />
</int:recipient-list-router>
<int:service-activator ref="aHandler"
input-channel="aInputChannel" output-channel="aggregatorOutputChannel" />
<!-- we have exception in the bHandler -->
<int:service-activator ref="bHandler"
input-channel="bInputChannel" output-channel="aggregatorOutputChannel" />
<int:aggregator input-channel="aggregatorOutputChannel"
output-channel="outputChannel" />
OR
<int-file:splitter id="splitile"
charset="UTF-8" apply-sequence="true" iterator="false"
input-channel="inputChannel"
output-channel="bTransformerChannel" />
<!-- consider we have exception at 4th chunk -->
<int:service-activator ref="transform"
input-channel="bTransformerChannel" output-channel="aggregatorOutputChannel" />
<int:aggregator input-channel="aggregatorOutputChannel"
output-channel="outputChannel" />
Yes; the aggregator is a "passive" component by default - all actions are taken when a message arrives.
To time out stale groups you can use a reaper or, with more recent versions, a group-timeout.
I using Spring integration to aggregate messages into one and then send by FTP out bound adapter, I want to move the aggregated messages in a specific folder when outbound FTP server is not available(org.springframework.messaging.MessageDeliveryException), other exceptions will log by console.
Here is my Configuration
<int:chain id="transformChain" input-channel="inboundChannel">
<int:header-enricher>
<int:header name="file_name" expression="payload.name"/>
<int:header name="correlationId" expression="${header.enricher.correlationId}"/>
<int:header name="sum" expression="${header.enricher.sum}"/>
</int:header-enricher>
<int:transformer ref="fileNameToContentTransformer"/>
<int:aggregator send-partial-result-on-expiry="true"
release-strategy-expression="#this.size() == new Integer([0].headers.sum)"
group-timeout="${aggregator.group-timeout}"
message-store="messageStore"
expire-groups-upon-completion="true"
correlation-strategy-expression="headers.correlationId"/>
<int:transformer ref="xmlToJsonTransformer"/>
<ftp:outbound-channel-adapter remote-directory="${ftp.out.remote.directory}"
session-factory="ftpOutClientSessionFactory" auto-create-directory="true"
remote-filename-generator="fileNameGenerator" charset="UTF-8"
temporary-file-suffix=".writing">
</ftp:outbound-channel-adapter>
<int:exception-type-router >
<int:mapping exception-type="org.springframework.messaging.MessageDeliveryException" channel="undeliveredChannel"/>
<int:mapping exception-type="java.lang.Exception" channel="myErrorChannel"/>
</int:exception-type-router>
</int:chain>
However I met such exception when try to start.
Caused by: java.lang.IllegalArgumentException: All handlers except for the last one in the chain must implement the MessageProducer interface. Object of class [org.springframework.integration.ftp.outbound.FtpMessageHandler] must be an instance of interface org.springframework.integration.core.MessageProducer
at org.springframework.util.Assert.instanceCheckFailed(Assert.java:389)
at org.springframework.util.Assert.isInstanceOf(Assert.java:327)
at org.springframework.integration.handler.MessageHandlerChain.configureChain(MessageHandlerChain.java:119)
at org.springframework.integration.handler.MessageHandlerChain.onInit(MessageHandlerChain.java:99)
at org.springframework.integration.context.IntegrationObjectSupport.afterPropertiesSet(IntegrationObjectSupport.java:176)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1687)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1624)
Anyone can tell me how to achieve this?
thanks in advance.
You can't add a component to the chain after the ftp:outbound-channel-adapter because it produces no result.
See the retry-and-more sample for an example of how to handle exceptions by adding an ExpressionEvaluatingRequestHandlerAdvice to the outbound adapter.