I have a message driven channel adapter configured to pick messages from queue with acknowledge set to transacted. Is it possible to send different messages to different queues down stream holding the same transaction also have database inserts between in the flow?. If message delivery fails to any queue , the transaction must roll back (including database entries) as well the message send to other queues.
Example : queue(receive)--->insert Db-->send to(queue1,queue2.. etc sending different message to each queue)
if any send call fails for queue1, queue2 etc the transaction should roll back..
I am able to do the configuration with single queue (ie only with queue1). But how to do if multiple queues are involved and holding the transaction boundaries.
Thanks
Vaidya
Below is the configuration
<int-jms:message-driven-channel-adapter
id="MessageDrivenAdapter" channel="injmsChannel" destination="testQ"
concurrent-consumers="5" max-concurrent-consumers="10" acknowledge="transacted"
error-channel="Error" />
<int:channel id="injmsChannel" />
<int:chain input-channel="injmsChannel" id="Chain1">
<int-jpa:retrieving-outbound-gateway
entity-class="entity.TestTable8"
entity-manager-factory="entityManagerFactory" id-expression="payload.getSno()">
<int-jpa:transactional transaction-manager="transactionManager" />
</int-jpa:retrieving-outbound-gateway>
<int:recipient-list-router id="ROUTE_1_2">
<int:recipient channel="SuccessChannel"
selector-expression="payload.getSno()==1" />
/> -->
</int:recipient-list-router>
</int:chain>
<int:channel id="SuccessChannel" />
<int:chain id="MessageProcessingChain" input-channel="SuccessChannel"
output-channel="putMsgChannel">
<int:service-activator id="a1" ref="taskexe"
method="processTable8_1" requires-reply="true" />
<int-jpa:retrieving-outbound-gateway
id="table7" entity-class="entity.TestTable7"
entity-manager-factory="entityManagerFactory" id-expression="payload.getSno()">
<int-jpa:transactional transaction-manager="transactionManager" />
</int-jpa:retrieving-outbound-gateway>
<int:service-activator id="a2" ref="taskexe"
method="processTable8_2" requires-reply="true" />
<int-jpa:updating-outbound-gateway
id="table6" entity-class="entity.TestTable6"
entity-manager-factory="entityManagerFactory" flush="true">
<int-jpa:transactional transaction-manager="transactionManager" />
</int-jpa:updating-outbound-gateway>
<int:service-activator id="a3" ref="taskexe"
method="processTable6_1" requires-reply="true" />
<int-jpa:updating-outbound-gateway
id="uptable6" entity-class="entity.TestTable6"
entity-manager-factory="entityManagerFactory" flush="true">
<int-jpa:transactional transaction-manager="transactionManager" />
</int-jpa:updating-outbound-gateway>
<int:service-activator id="a4" ref="taskexe"
method="processTable6_2" requires-reply="true" />
<int-jpa:updating-outbound-gateway
id="uptable4" entity-class="entity.TestTable4"
entity-manager-factory="entityManagerFactory" flush="true">
<int-jpa:transactional transaction-manager="transactionManager" />
</int-jpa:updating-outbound-gateway>
<int:service-activator ref="taskexe" method="processTable4_1"
requires-reply="true" />
</int:chain>
<int:channel id="putMsgChannel" />
<int-jms:outbound-channel-adapter id="sendsomemsg"
channel="putMsgChannel" connection-factory="connectionFactory"
session-transacted="true" destination-expression="headers['somequeue']" />
How to add another int-jms:outbound-channel-adapte for other queue having same transaction boundaries of message driven adapter?. Also flush=true been set so that message is not passed downstream if there is any jpa adapter exception.
As long as the queue sends are performed using a JmsTemplate (including the use of a JMS outbound channel adapter), on the same thread, they will be performed in the same transactional session as the message-driven adapter's message delivery.
If you add the JDBC transaction manager to the message-driven adapter, its transaction will be synchronized with the JMS transaction.
This provides "Best Effort 1PC" as discussed in Dave Syer's JavaWorld article: Distributed transactions in Spring, with and without XA.
There is a small possibility that the DB commit will succeed and the JMS commit fails, so you need to deal with duplicates. To avoid that you need a full XA solution.
Related
I asked a question a bit ago about a specific use case when multi-threading in spring integration, basically I have to have one thread of execution that stays on the initial and then a second that spawns a thread. I have implemented this like so
<int:channel id="newThread" >
<int:dispatcher task-executor="workerThreadPoolAdapter"/>
</int:channel>
<!--This bean is used to split incoming messages -->
<bean id="splitterBean" class="orchestration.MessageSplitter">
<property name="channels" ref="splitterChannelsList" />
</bean>
<util:list id="splitterChannelsList" value-type="java.lang.String">
<value>newThread</value>
<value>mainThread</value>
</util:list>
<!-- This bean is used to aggregate incoming messages -->
<bean id="aggregator" class="orchestration.MessageAggregator">
<property name="wrapperNode" value="container" />
</bean>
<!-- Channel for aggregator output and that will be input for response transformer -->
<int:publish-subscribe-channel id="gatherChannel" apply-sequence="true"/>
<!-- This splitter splits request and send -->
<!-- will add a header called channelHeader which is the channel the message should be routed to using the recipient-list-router -->
<int:splitter input-channel="splitter"
output-channel="recipientListRouter"
apply-sequence="true"
ref="splitterBean" method="split" />
<!-- Aggregator that aggregates responses received from the calls -->
<int:aggregator input-channel="gatherChannel"
output-channel="transformResponse"
ref="aggregator"
method="aggregateMessages"
send-partial-result-on-expiry="false"
expire-groups-upon-completion="true"
message-store="removeMessageFromStore"
release-strategy-expression="size() == 2"/>
<int:recipient-list-router input-channel="recipientListRouter">
<int:recipient channel="mainThread"
selector-expression="headers.get('channelHeader') == 'mainThread'"/>
<int:recipient channel="newThread"
selector-expression="headers.get('channelHeader') == 'newThread'"/>
</int:recipient-list-router>
<!-- only route to call if boolean is populated -->
<int:header-value-router header-name="shouldMakeExtraOutboundCall"
input-channel="newThread"
default-output-channel="gatherChannel" >
<int:mapping value="true" channel="outboundCall" />
</int:header-value-router>
<int:chain input-channel="outboundCall" output-channel="gatherChannel">
<!-- make an outbound call -->
</int:chain>
<int:chain input-channel="mainThread" output-channel="gatherChannel">
<!-- make a bunch of outbound calls -->
</int:chain>
<int:chain input-channel="transformResponse" output-channel="backToClient">
<!-- do some stuff and respond back to client -->
</int:chain>
I've had the output channel of the aggregator as both a publish-subscribe and a direct channel and had the issue for both.
When I look at the logs I can see that one of the threads has a preSend to the 'gatherChannel' then AggregatingMessageHandler saying it received the message, then another log from AggregatingMessageHandler saying Handling message with correlationKey [f2b16b6a-3605-778f-a628-870ed8ce3f5e] then a postSend (sent=True) on channel 'gatherChannel'.
I thought that it would not send to the transformResponse channel until both messages that got split out from the splitter got to it. I even added the size() == 2 as the release strategy expression as an extra layer but that doesn't seem to be causing the aggregator to wait either.
I'm a little perplexed why this is happening, it's happening when both the main thread or the spawned thread gets to the aggregator, I'm trying to figure out how to get that aggregator to wait to send to the output channel until BOTH messages that were split from the splitter are received.
According to your current configuration it really does not happen.
Do you really have some evidences that first message in the group is really sent to that transformResponse unconditionally?
You probably is a bit confused with those preSend and postSend since an aggregator is really non-blocking until release. It does accept the message and store it into the MessageStore for grouping if release condition is false. Just after this it returns immediately to the caller, which is that gatherChannel and therefore your see postSend on a first message.
I have registered an AsyncHandler and also added a success-channel to an SQS outbound flow. The success-channel has a int:logging-channel-adapter endpoint. However I am not able to see any logs from this adapter. The AsyncHandler is able to receive the call-backs but nothing on the success-channel.
In the SqsMessageHandler I see that we are setting an output channel in the obtainAsyncHandler method, but I did not see the success-channel set anywhere. Am I missing something?
I would prefer using the success and failure channels and not AsyncHandler call-back Impl to avoid having AWS specific code in my classes.
Also my <int-aws:sqs-outbound-channel-adapter> is inside a <int:chain> which has no output channel, since the flow ends when the message is sent.
EDIT - Added Config
This is the only way I can get it to log the callback.
<int:channel id="chainChannel" />
<int:channel id="successChannel" />
<bean class="ServiceTransformer" id="serviceTransformer" />
<int:chain input-channel="serviceChannel" id="sendToServiceSqsChain" output-channel="chainChannel">
<int:transformer ref="serviceTransformer" method="transform" />
<int:header-filter header-names="config" />
<int-aws:sqs-outbound-channel-adapter sqs="amazonSQS" queue="some-queue" async-handler="sqsPublishCallbackHandler" success-channel="successChannel"/>
</int:chain>
<int:logging-channel-adapter log-full-message="true" channel="chainChannel" />
Here I can just use the same channel in both chain (outbound channel) and sqs-outbound (success-channel)
Unable to get it to work like below:
<int:channel id="successChannel" />
<bean class="ServiceTransformer" id="serviceTransformer" />
<int:chain input-channel="serviceChannel" id="sendToServiceSqsChain" >
<int:transformer ref="serviceTransformer" method="transform" />
<int:header-filter header-names="config" />
<int-aws:sqs-outbound-channel-adapter sqs="amazonSQS" queue="some-queue" async-handler="sqsPublishCallbackHandler" success-channel="successChannel"/>
</int:chain>
<int:logging-channel-adapter log-full-message="true" channel="successChannel" />
The <int-aws:sqs-outbound-channel-adapter> component is one-way, therefore there is no outputChannel option expose. However the target class is AbstractMessageProducingHandler. To avoid code duplication we reuse an existing outputChannel internally for that AsyncHandler.
In the XML parser we just remap one to another:
IntegrationNamespaceUtils.setReferenceIfAttributeDefined(builder, element, "success-channel", "outputChannel");
You probably don't see anything in logs because you need to adjust logging config respectively for the appropriate category and level.
UPDATE
According my testing this is definitely not possible to configure such a component with XML DSL within the <chain>. That <int-aws:sqs-outbound-channel-adapter> has to be presented outside of the <chain>.
Consider to more your configuration to Java DSL instead: https://docs.spring.io/spring-integration/docs/5.3.2.RELEASE/reference/html/dsl.html#java-dsl.
<context:annotation-config/>
<context:component-scan base-package="ru.*"/>
<int:poller id="defaultPoller" default="true"
fixed-delay="1000" task-executor="taskExecutor"/>
<task:executor id="taskExecutor" pool-size="1-5" queue-capacity="200" rejection-policy="CALLER_RUNS"/>
<int:channel id="listFiles">
<int:queue capacity="1000"/>
</int:channel>
<int:channel id="pages">
<int:queue capacity="1000"/>
</int:channel>
<int:channel id="resultWithHeader">
<int:queue capacity="1000"/>
</int:channel>
<int:channel id="toBD">
<int:queue capacity="1000"/>
</int:channel>
<int:channel id="unparsedToUniq">
<int:queue capacity="1000"/>
</int:channel>
<task:scheduled-tasks>
<task:scheduled ref="getFilesList"
method="getList"
cron="0 0 12 * * WED"/>
</task:scheduled-tasks>
<int:splitter id="splitterPdf" ref="split" method="receive" input-channel="listFiles" output-channel="pages"/>
<int:transformer id="transfMain" input-channel="pages" ref="firstTransform" method="transform"
output-channel="resultWithHeader"/>
<int:header-value-router id="routedParsingDocument" input-channel="resultWithHeader" header-name="parsed">
<int:mapping value="yes" channel="toBD"/>
<int:mapping value="no" channel="unparsedToUniq"/>
</int:header-value-router>
<int:transformer id="transfUniq" input-channel="unparsedToUniq" ref="secondTransform" method="transform"
output-channel="toBD"/>
<int:service-activator id="bd_activator" input-channel="toBD" method="receive" ref="bd" > <int:poller task-executor="taskBD" fixed-delay="1500"/></int:service-activator>
<bean id="getFilesList" class="ru.*.GetAndReadFiles"/>
<bean id="split" class="ru.*.SplitDocument"/>
<bean id="firstTransform" class="ru.*.MainParser"/>
<bean id="secondTransform" class="ru.*.UniqParser"/>
<bean id="bd" class="ru.*.BDWriter"/>
<task:executor id="taskBD" pool-size="1-10" queue-capacity="100" rejection-policy="CALLER_RUNS"/>
Hello.
Program read all files on directory, split pages, transform information(main or uniq) and write info to base.
I have problem with this config.
06-03-2019 13:01:24.773 [task-scheduler-7] ERROR o.s.i.handler.LoggingHandler.handleMessageInternal - org.springframework.core.task.TaskRejectedException: Executor [java.util.concurrent.ThreadPoolExecutor#19c520db[Running, pool size = 5, active threads = 5, queued tasks = 200, completed tasks = 1353]] did not accept task: org.springframework.integration.util.ErrorHandlingTaskExecutor$$Lambda$241/1065676784#6991f347
What am I doing wrong? If i don't use taskExecutor, program work. But I need to increase the speed of execution. In other configuration work< but i have this problem:ERROR o.s.i.handler.LoggingHandler.handleMessageInternal - java.lang.OutOfMemoryError: Java heap space
Thanks for you help.
You should consider do not use queue channels in between. There is just enough to distribute polling results into that taskExecutor from the poller.
You can leave a queue channel after splitter though, since you are going to have one-to-many after splitting, so would be great to make them parallel. Although you may also consider to use there an ExecutorChannel instead.
Not clear, though, why you have a rejected task since your policy is CALLER_RUNS...
I am getting the following error while shutting down tomcat
SEVERE: The web application [/TestService] appears to have started a thread named [SimpleAsyncTaskExecutor-3] but has failed to stop it. This is very likely to create a memory leak.
<gateway id="testGateway" service-interface="org.example.TestGateway"
default-request-channel="requestChannel" error-channel="errorChannel"/>
public interface TestGateway {
Future execute(Request request);
}
<int:chain input-channel="requestChannel" output-channel="routerChannelA">
<int:service-activator ...
<int:transformer ....
</int:chain>
<int:router input-channel="routerChannelA" expression="payload.name" resolution-required="true">
<int:mapping value="B" channel="channelB" />
....
</int:router>
<int:chain input-channel="channelB" output-channel="channelD">
<int:transformer ......
<int:gateway request-channel="channelC" .....
<int:filter expression="headers['RELEASE'] != null" discard-="nullChannel"/>
</int:chain>
<int:recipient-list-router id="customRouter" input-channel="channelD"
timeout="1234"
ignore-send-failures="true"
apply-sequence="false" >
<int:recipient channel="splitterRequestChannel"/>
<int:recipient channel="completeChannel"/>
</int:recipient-list-router>
<int:splitter expression="payload" input-channel="splitterRequestChannel"
output-channel="splitterResponseChannel" ></int:splitter>
<int:channel id="splitterResponseChannel">
<int:dispatcher task-executor="splitterChannelTaskExecutor"/>
</int:channel>
<bean id="splitterChannelTaskExecutor" class="org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor" >
<property name="corePoolSize" value="10" />
<property name="daemon" value="false"/>
</bean>
<int:chain input-channel="splitterResponseChannel">
......
......
</int:chain>
The client sends requests to TestGateway. I dont have to send any reply back to client, but I want to return back immediately.
The Future return type serves my purpose of returning immediately. However I feel it blocks the main thread.
The request moves from a series of chains and finally reaches splitterRequestChannel. This channel delegates its work to the threads initiated by splitterChannelTaskExecutor, they do their respective jobs now. I feel the main thread should be released now as it has delegated its task, but it doesn't look its getting released.
Edit:
public interface TestGateway {
void execute(Request request);
}
<bean id="requestChannelTaskExecutor" class="org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor" >
<property name="corePoolSize" value="10" />
<property name="daemon" value="false"/>
</bean>
<int:channel id="requestChannel">
<int:dispatcher task-executor="requestChannelTaskExecutor"/>
</int:channel>
Some thoughts...
Just return void if you don't care about the result.
How are you instantiating the application context? If using normal spring web, spring should take care of closing the context; if you are creating it yourself, it's up to you to close it.
Take a thread dump to see what the thread is doing.
EDIT:
If your flow never returns a reply, the TE within the gateway is stuck waiting for the reply.
So; return void and you won't have the problem.
I've got for a JPA Outbound-channel-adapter both transactional and request-handler-advice-chain. In the advice-chain I try to log the Exception, when it happens.
It iss not logged, but I know that it actually happend since the Message was sent to failover clickDbFailoverChannel . What can be a problem with it? Is it a bug in Spring Integration?
<int:channel id="clickDbWithFailoverChannelSite-1">
<int:dispatcher load-balancer="none" task-executor="clickDbSiteRouterExecutor"/>
</int:channel>
<int:bridge input-channel="clickDbWithFailoverChannelSite-1"
output-channel="jpaOutboundChannelSite-1" order="1" send-timeout="100" />
<int:bridge input-channel="clickDbWithFailoverChannelSite-1"
output-channel="clickDbFailoverChannel" order="2" />
<int-jpa:outbound-channel-adapter id="jpaOutboundChannelSite-1"
persist-mode="PERSIST" flush-size="100" entity-manager-factory="emfSite-1">
<int-jpa:transactional transaction-manager="transactionManagerSite-1" />
<int-jpa:request-handler-advice-chain>
<bean class="org.springframework.integration.handler.advice.ExpressionEvaluatingRequestHandlerAdvice">
<property name="failureChannel" ref="clickDbFailureLogger"/>
<property name="onFailureExpression" value="#exception"/>
</bean>
</int-jpa:request-handler-advice-chain>
</int-jpa:outbound-channel-adapter>
OK. I can guess where is your issue. The real exception to rollback the transaction is caused before an internal logic, where <request-handler-advice-chain> does the stuff. That's why your ExpressionEvaluatingRequestHandlerAdvice doesn't get a failure message.
To workaround your rollback issue, you should replace <int-jpa:transactional> with <tx:advice> within <int-jpa:request-handler-advice-chain>.
You should understand here that <int-jpa:transactional> is for entire MessageHandler.handleMessage, but <int-jpa:request-handler-advice-chain> is just for its part - AbstractReplyProducingMessageHandler.handleRequestMessage.
UPDATE
TX Advice should be like this:
<tx:advice transaction-manager="transactionManagerSite-1"/>
<tx:attributes>
<tx:method name="*"/>
</tx:attributes>
</tx:advice>