Spring Integration : How to handle multiple Subscribers in one transaction? - spring-integration

My Spring Integration Xml.
<int:channel id="request-write-to-PMSQueueChannel" >
<int:queue message-store="channelStore" />
</int:channel>
<int:bridge input-channel="request-write-to-PMSQueueChannel" output-channel="writetoPMSChannel">
<int:poller fixed-rate="5000" max-messages-per-poll="-1">
<int:transactional propagation="REQUIRED" transaction-manager="transactionManager"/>
</int:poller>
</int:bridge>
<int:channel id="redBlue-error-channel"/>
<int:service-activator id="errorServiceActivator" input-channel ="redBlue-error-channel">
<bean id="errorSVC"
class="com.sds.redBlue.core.module.analyzer.sample.ErrorServiceActivator"/>
</int:service-activator>
<bean id="channelStore" class="org.springframework.integration.jdbc.store.JdbcChannelMessageStore">
<property name="dataSource" ref="dataSource" />
<property name="channelMessageStoreQueryProvider" ref="queryProvider" />
</bean>
<bean id="queryProvider" class="org.springframework.integration.jdbc.store.channel.DerbyChannelMessageStoreQueryProvider"/>
<int:publish-subscribe-channel id="writetoPMSChannel" ignore-failures = "false"/>
<int:chain input-channel="writetoPMSChannel"
output-channel="writetoPMS001Channel">
<int:service-activator method="exectue">
<bean id=""
class="com.sds.redBlue.core.module.analyzer.convert.ModelingConvertSVC">
</bean>
</int:service-activator>
<int:splitter ref="fromListToRowSplitter" />
</int:chain>
<int:chain input-channel="writetoPMSChannel"
output-channel="writetoPMS002Channel">
<int:service-activator method="exectue002">
<bean id=""
class="com.sds.redBlue.core.module.analyzer.convert.ModelingConvertSVC">
</bean>
</int:service-activator>
<int:splitter ref="fromListToRowSplitter" />
</int:chain>
<int:chain input-channel="writetoPMSChannel"
output-channel="writetoPMS003Channel">
<int:service-activator method="exectue003">
<bean id=""
class="com.sds.redBlue.core.module.analyzer.convert.ModelingConvertSVC">
</bean>
</int:service-activator>
<int:splitter ref="fromListToRowSplitter" />
</int:chain>
<int:chain input-channel="writetoPMSChannel"
output-channel="writetoPMS004Channel">
<int:service-activator method="exectue004">
<bean id=""
class="com.sds.redBlue.core.module.analyzer.convert.ModelingConvertSVC">
</bean>
</int:service-activator>
<int:splitter ref="fromListToRowSplitter" />
</int:chain>
<int:chain input-channel="writetoPMSChannel"
output-channel="writetoPMS005Channel">
<int:service-activator method="exectue005">
<bean id=""
class="com.sds.redBlue.core.module.analyzer.convert.ModelingConvertSVC">
</bean>
</int:service-activator>
<int:splitter ref="fromListToRowSplitter" />
</int:chain>
I want to 5 subscribers have to be executed at once or have to be rollback if one subscribe throws any exception. But I can't find the way to sovle.
related article:
Pub-Sub error handling strategy
I already asked with the the eip pattern pic and Gary helped. but, I stucked.
(Spring Integration : How to guarantee the transaction two more jdbc-outbound-gateway?)

Since transaction is bounded to the Thread, you should be sure that all your subscribers are withing the same direct flow. In this case they will be invoked one by one.
However I see that you use ignore-failures = "false". Having you ignore all downstream exception and allow for your subscribers to do their work. Although you lose TX rollback, of course.
So, revise your use-case if you really want to have automatic rallback.
There is an async trick to control TX, but it is a bit complex and require some logic with aggregator.

Related

Configure Spring integration. ErrorHandlingTaskExecutor

<context:annotation-config/>
<context:component-scan base-package="ru.*"/>
<int:poller id="defaultPoller" default="true"
fixed-delay="1000" task-executor="taskExecutor"/>
<task:executor id="taskExecutor" pool-size="1-5" queue-capacity="200" rejection-policy="CALLER_RUNS"/>
<int:channel id="listFiles">
<int:queue capacity="1000"/>
</int:channel>
<int:channel id="pages">
<int:queue capacity="1000"/>
</int:channel>
<int:channel id="resultWithHeader">
<int:queue capacity="1000"/>
</int:channel>
<int:channel id="toBD">
<int:queue capacity="1000"/>
</int:channel>
<int:channel id="unparsedToUniq">
<int:queue capacity="1000"/>
</int:channel>
<task:scheduled-tasks>
<task:scheduled ref="getFilesList"
method="getList"
cron="0 0 12 * * WED"/>
</task:scheduled-tasks>
<int:splitter id="splitterPdf" ref="split" method="receive" input-channel="listFiles" output-channel="pages"/>
<int:transformer id="transfMain" input-channel="pages" ref="firstTransform" method="transform"
output-channel="resultWithHeader"/>
<int:header-value-router id="routedParsingDocument" input-channel="resultWithHeader" header-name="parsed">
<int:mapping value="yes" channel="toBD"/>
<int:mapping value="no" channel="unparsedToUniq"/>
</int:header-value-router>
<int:transformer id="transfUniq" input-channel="unparsedToUniq" ref="secondTransform" method="transform"
output-channel="toBD"/>
<int:service-activator id="bd_activator" input-channel="toBD" method="receive" ref="bd" > <int:poller task-executor="taskBD" fixed-delay="1500"/></int:service-activator>
<bean id="getFilesList" class="ru.*.GetAndReadFiles"/>
<bean id="split" class="ru.*.SplitDocument"/>
<bean id="firstTransform" class="ru.*.MainParser"/>
<bean id="secondTransform" class="ru.*.UniqParser"/>
<bean id="bd" class="ru.*.BDWriter"/>
<task:executor id="taskBD" pool-size="1-10" queue-capacity="100" rejection-policy="CALLER_RUNS"/>
Hello.
Program read all files on directory, split pages, transform information(main or uniq) and write info to base.
I have problem with this config.
06-03-2019 13:01:24.773 [task-scheduler-7] ERROR o.s.i.handler.LoggingHandler.handleMessageInternal - org.springframework.core.task.TaskRejectedException: Executor [java.util.concurrent.ThreadPoolExecutor#19c520db[Running, pool size = 5, active threads = 5, queued tasks = 200, completed tasks = 1353]] did not accept task: org.springframework.integration.util.ErrorHandlingTaskExecutor$$Lambda$241/1065676784#6991f347
What am I doing wrong? If i don't use taskExecutor, program work. But I need to increase the speed of execution. In other configuration work< but i have this problem:ERROR o.s.i.handler.LoggingHandler.handleMessageInternal - java.lang.OutOfMemoryError: Java heap space
Thanks for you help.
You should consider do not use queue channels in between. There is just enough to distribute polling results into that taskExecutor from the poller.
You can leave a queue channel after splitter though, since you are going to have one-to-many after splitting, so would be great to make them parallel. Although you may also consider to use there an ExecutorChannel instead.
Not clear, though, why you have a rejected task since your policy is CALLER_RUNS...

confirm-ack-channel is not getting invoked

Below is my config. In case of successful posting to queue the flow is not passing to confirm-ack-channel. Can anyone suggest where am going wrong?
<int-amqp:outbound-channel-adapter channel="fulfillmentOutboundChannel"
routing-key="xyz"
amqp-template="transactionalRabbitTemplate"
confirm-correlation-expression="payload"
confirm-ack-channel="successRespTransformChannel"
confirm-nack-channel="failureRespTransformChannel"
return-channel="failureRespTransformChannel"
mapped-request-headers="*"
/>
<int:channel id="successRespTransformChannel">
<int:interceptors><int:wire-tap channel="loggerChannel"/></int:interceptors>
</int:channel>
<chain input-channel="successRespTransformChannel">
<int:header-enricher>
<error-channel ref="failed-publishing" />
</int:header-enricher>
<service-activator id="successResp" expression="#abc.addRequestTracking(payload.id,'success')"/>
</chain>
In order to get acks/nacks, in addition to setting up the adapter with confirm-correlation, You also need to enable publisherConfirms on the CachingConnectionFactory.
From the documentation:
For Publisher Confirms (aka Publisher Acknowledgements), the template requires a CachingConnectionFactory that has its publisherConfirms property set to true.
Thanks for your response Gary. I have already made publish-confirms=true as below..
<rabbit:connection-factory id="fufillConnectionFactory"
connection-factory="rabbitClientConnectionFactory"
publisher-confirms="true"
publisher-returns="true"/>
<bean id="rabbitClientConnectionFactory" class="com.rabbitmq.client.ConnectionFactory" >
<property name="uri" value="${mq.uri}" />
<property name="requestedHeartbeat" value="30" />
</bean>

messages not getting consumed depending whether the message is of type text or string

Below i have the program to send a message and consume a message from queue , rite now i have commented out the sending part and only want to consume the messages from queue , the message can be type of string or object message for which i have configure router.
Now there are text message in the queue which my below program is not consuming please advise how to overcome from this below is my configuration , as you can see i have commented out the sender part so the only reading part from the queue is the active one
and also when rite now i have observe that messages are getting consumed but files are not being generated so this means that there is some error after payload router i have configured
http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/integration http://www.springframework.org/schema/integration/spring-integration.xsd
http://www.springframework.org/schema/integration/jms
http://www.springframework.org/schema/integration/jms/spring-integration-jms.xsd
http://www.springframework.org/schema/integration/spring-integration.xsd
http://www.springframework.org/schema/context
http://www.springframework.org/schema/integration/file
http://www.springframework.org/schema/integration/file/spring-integration-file.xsd
http://www.springframework.org/schema/context/spring-context.xsd">
<int:poller id="poller" default="true">
<int:interval-trigger interval="200" />
</int:poller>
<int:channel id="input">
<int:queue capacity="10" />
</int:channel>
<bean id="tibcoEMSJndiTemplate" class="org.springframework.jndi.JndiTemplate">
<property name="environment">
<props>
<prop key="java.naming.factory.initial">com.tibco.tibjms.naming.TibjmsInitialContextFactory
</prop>
<prop key="java.naming.provider.url">tcp://lsdrtems2.fm.crdgrp.net:7333</prop>
<prop key="java.naming.security.principal">acfgtir</prop>
<prop key="java.naming.security.credentials">acfgtir</prop>
</props>
</property>
</bean>
<bean id="tibcoEMSConnFactory" class="org.springframework.jndi.JndiObjectFactoryBean">
<property name="jndiTemplate">
<ref bean="tibcoEMSJndiTemplate" />
</property>
<property name="jndiName">
<value>GenericConnectionFactory</value>
</property>
</bean>
<bean id="tibcosendJMSTemplate" class="org.springframework.jms.core.JmsTemplate">
<property name="connectionFactory">
<ref bean="tibcoEMSConnFactory" />
</property>
<property name="defaultDestinationName">
<value>acfgtirrtyation.ioa.swretift_publish_poc1</value>
</property>
<property name="pubSubDomain">
<value>false</value>
</property>
<property name="receiveTimeout">
<value>120000</value>
</property>
</bean>
<!-- <jms:outbound-channel-adapter channel="input"
destination-name="acfgtirrtyation.ioa.swretift_publish_poc1" connection-factory="tibcoEMSConnFactory" /> -->
<int:channel id="objetChannel"></int:channel>
<int:channel id="StringChannel"></int:channel>
<int:channel id="jmsInChannel" />
<jms:message-driven-channel-adapter id="jmsIn" concurrent-consumers="10"
destination-name="acfgtirrtyation.ioa.swretift_publish_poc1" connection-factory="tibcoEMSConnFactory" extract-payload="false"
channel="jmsInChannel" />
<int:payload-type-router input-channel="jmsInChannel">
<int:mapping type="javax.jms.ObjectMessage" channel="objetChannel" />
<int:mapping type="javax.jms.TextMessage" channel="StringChannel" />
</int:payload-type-router>
<file:outbound-channel-adapter id="filesoutOject" channel="objetChannel" directory="C:\\abcsaral"
filename-generator="generatorr" />
<file:outbound-channel-adapter id="filesoutString" channel="StringChannel" directory="C:\\abcsaral"
filename-generator="generatorr" />
<bean id="generatorr" class="com.abs.tibco.TimestampTextGenerator">
</bean>
</beans>
Folks please advise for this any early help would be much appreciated
below is the timestamp generator class as shown below
public class TimestampTextGenerator implements FileNameGenerator {
#Override
public String generateFileName(Message<?> arg0) {
return new java.text.SimpleDateFormat("yyyyMMdd-HHmmss.SSS")
.format(new java.util.Date()) + ".txt";
}
}
Folks please advise as I am completely stuck up on this..!
Remove this attribute extract-payload="false" in your message-driven-channel-adapter. It changes expected payload type and your payload-type-router doesn't work.
Also look this 20.1. Inbound Channel Adapter manual chapter.
The best way to understand what's wrong with your application is to take a look into logs. I should admit that sometimes Java is more smarter than me :-).
See Gary Russel's answer for you on the matter: Enabling logging in spring integration utiliy
Please, try to follow with all our answers to you around this topic.
I suggested you to use extract-payload="false" to get the whole javax.jms.ObjectMessage or javax.jms.TextMessage as a payload independently of the conversion result, when the javax.jms.ObjectMessage maybe converted to the java.lang.String as well. Although, right, in most case the routing based just only on the java.lang.String and all others should work, too.
Now about the <file:outbound-channel-adapter> problem. If you take a into logs you should see something like:
else {
throw new IllegalArgumentException(
"unsupported Message payload type [" + payload.getClass().getName() + "]");
}
M-m-m, yeah, I mean the result of this Exception in logs. Just because your payload is javax.jms.Message, but to write into file you should supply File, InputStream, byte[] or String.
So, as I said in comments to my answer (consuming all the object type messages from a queue) to your similar question, you should convert the message after routing into some appropriate type for <file:outbound-channel-adapter>.

Tomcat complains of a memory leak with spring integration while shutting down

I am getting the following error while shutting down tomcat
SEVERE: The web application [/TestService] appears to have started a thread named [SimpleAsyncTaskExecutor-3] but has failed to stop it. This is very likely to create a memory leak.
<gateway id="testGateway" service-interface="org.example.TestGateway"
default-request-channel="requestChannel" error-channel="errorChannel"/>
public interface TestGateway {
Future execute(Request request);
}
<int:chain input-channel="requestChannel" output-channel="routerChannelA">
<int:service-activator ...
<int:transformer ....
</int:chain>
<int:router input-channel="routerChannelA" expression="payload.name" resolution-required="true">
<int:mapping value="B" channel="channelB" />
....
</int:router>
<int:chain input-channel="channelB" output-channel="channelD">
<int:transformer ......
<int:gateway request-channel="channelC" .....
<int:filter expression="headers['RELEASE'] != null" discard-="nullChannel"/>
</int:chain>
<int:recipient-list-router id="customRouter" input-channel="channelD"
timeout="1234"
ignore-send-failures="true"
apply-sequence="false" >
<int:recipient channel="splitterRequestChannel"/>
<int:recipient channel="completeChannel"/>
</int:recipient-list-router>
<int:splitter expression="payload" input-channel="splitterRequestChannel"
output-channel="splitterResponseChannel" ></int:splitter>
<int:channel id="splitterResponseChannel">
<int:dispatcher task-executor="splitterChannelTaskExecutor"/>
</int:channel>
<bean id="splitterChannelTaskExecutor" class="org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor" >
<property name="corePoolSize" value="10" />
<property name="daemon" value="false"/>
</bean>
<int:chain input-channel="splitterResponseChannel">
......
......
</int:chain>
The client sends requests to TestGateway. I dont have to send any reply back to client, but I want to return back immediately.
The Future return type serves my purpose of returning immediately. However I feel it blocks the main thread.
The request moves from a series of chains and finally reaches splitterRequestChannel. This channel delegates its work to the threads initiated by splitterChannelTaskExecutor, they do their respective jobs now. I feel the main thread should be released now as it has delegated its task, but it doesn't look its getting released.
Edit:
public interface TestGateway {
void execute(Request request);
}
<bean id="requestChannelTaskExecutor" class="org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor" >
<property name="corePoolSize" value="10" />
<property name="daemon" value="false"/>
</bean>
<int:channel id="requestChannel">
<int:dispatcher task-executor="requestChannelTaskExecutor"/>
</int:channel>
Some thoughts...
Just return void if you don't care about the result.
How are you instantiating the application context? If using normal spring web, spring should take care of closing the context; if you are creating it yourself, it's up to you to close it.
Take a thread dump to see what the thread is doing.
EDIT:
If your flow never returns a reply, the TE within the gateway is stuck waiting for the reply.
So; return void and you won't have the problem.

Right way to use transactional and request-handler-advice-chain in a JPAOutboundGateway

I've got for a JPA Outbound-channel-adapter both transactional and request-handler-advice-chain. In the advice-chain I try to log the Exception, when it happens.
It iss not logged, but I know that it actually happend since the Message was sent to failover clickDbFailoverChannel . What can be a problem with it? Is it a bug in Spring Integration?
<int:channel id="clickDbWithFailoverChannelSite-1">
<int:dispatcher load-balancer="none" task-executor="clickDbSiteRouterExecutor"/>
</int:channel>
<int:bridge input-channel="clickDbWithFailoverChannelSite-1"
output-channel="jpaOutboundChannelSite-1" order="1" send-timeout="100" />
<int:bridge input-channel="clickDbWithFailoverChannelSite-1"
output-channel="clickDbFailoverChannel" order="2" />
<int-jpa:outbound-channel-adapter id="jpaOutboundChannelSite-1"
persist-mode="PERSIST" flush-size="100" entity-manager-factory="emfSite-1">
<int-jpa:transactional transaction-manager="transactionManagerSite-1" />
<int-jpa:request-handler-advice-chain>
<bean class="org.springframework.integration.handler.advice.ExpressionEvaluatingRequestHandlerAdvice">
<property name="failureChannel" ref="clickDbFailureLogger"/>
<property name="onFailureExpression" value="#exception"/>
</bean>
</int-jpa:request-handler-advice-chain>
</int-jpa:outbound-channel-adapter>
OK. I can guess where is your issue. The real exception to rollback the transaction is caused before an internal logic, where <request-handler-advice-chain> does the stuff. That's why your ExpressionEvaluatingRequestHandlerAdvice doesn't get a failure message.
To workaround your rollback issue, you should replace <int-jpa:transactional> with <tx:advice> within <int-jpa:request-handler-advice-chain>.
You should understand here that <int-jpa:transactional> is for entire MessageHandler.handleMessage, but <int-jpa:request-handler-advice-chain> is just for its part - AbstractReplyProducingMessageHandler.handleRequestMessage.
UPDATE
TX Advice should be like this:
<tx:advice transaction-manager="transactionManagerSite-1"/>
<tx:attributes>
<tx:method name="*"/>
</tx:attributes>
</tx:advice>

Resources