Spring Integration AMQP - retries continuously despite retry advice - spring-integration

I've used Spring Integration with JMS very successfully before, but we're now using with RabbitMQ / AMQP and having some issues with error handling.
I have an int-amqp:inbound-channel-adapter with with a errorChannel set up to receive any exception, here an ErrorTransformer class inspects the failed Message's cause exception. Then depending on the type of exception either :-
suppresses the exception and transforms into a JSON object that can go to AMQP outbound-channel-adapter as a business reply explaining the failure. Here I want the original message consumed/ACKed.
Or Re-throws the causing exception to let RabbitMQ re-deliver the message, a certain number of times.
I found that re-throwing caused the message to be re-delivered continuously, I then read about StatefulRetryOperationsInterceptorFactoryBean, so added an advice chain to retry 3 times, then I got exception about no message-id, so also added a 'MissingMessageIdAdvice' at the start of the advice chain.
Despite the advice, I still get continuous re-tries for a RuntimeException that is re-thrown from errorChannel's ErrorTransformer. I'm publishing the message via RabbitMQ admin just using the defaults. Not sure if the lack of a message-id is making this not work, if so how do I get a message to have an id ?
I'm confused about the differences between the :-
A) ConditionalRejectingErrorHandler (I've set as the inbound adapter's error-handler) which allows me to provide a customFatalExceptionStrategy to implement isFatal(). Where I believe fatal=true (means DISCARD) and message is consumed and discarded, but how can I still send an outbound failure message ?
B) And the errorChannel I have on the inbound adapter which I'm using to inspect an Exception and transform into a outbound fail response message. Here I guess I could throw AmqpRejectAndDontRequeueException, but then why have the ConditionalRejectingErrorHandler as well ? and will thorwing AmqpRejectAndDontRequeueException work
<int-amqp:inbound-channel-adapter id="amqpInRequestPatternValuation" channel="requestAmqpIn" channel-transacted="true" transaction-manager="transactionManager"
queue-names="requestQueue" error-channel="patternValuationErrorChannel" connection-factory="connectionFactory"
receive-timeout="59000" concurrent-consumers="1"
advice-chain="retryChain" error-handler="customErrorHandler" />
<bean id="customErrorHandler" class="org.springframework.amqp.rabbit.listener.ConditionalRejectingErrorHandler">
<constructor-arg ref="customFatalExceptionStrategy"/>
</bean>
<bean id="customFatalExceptionStrategy" class="abc.common.CustomFatalExceptionStrategy"/>
<!-- ADVICE CHAIN FOR CONTROLLING NUMBER OF RE-TRIES before sending to DLQ (or discarding if no DLQ) without this any re-queued fatal message will retry forever -->
<util:list id="retryChain">
<bean class="org.springframework.amqp.rabbit.retry.MissingMessageIdAdvice">
<constructor-arg>
<bean class="org.springframework.retry.policy.MapRetryContextCache" />
</constructor-arg>
</bean>
<ref bean="retryInterceptor" />
</util:list>
<bean id="retryInterceptor"
class="org.springframework.amqp.rabbit.config.StatefulRetryOperationsInterceptorFactoryBean">
<property name="retryOperations" ref="retryTemplate" />
<property name="messageRecoverer" ref="messageRecoverer"/>
</bean>
<bean id="retryTemplate" class="org.springframework.retry.support.RetryTemplate">
<property name="retryPolicy" ref="simpleRetryPolicy" />
<property name="backOffPolicy">
<bean class="org.springframework.retry.backoff.ExponentialBackOffPolicy">
<property name="initialInterval" value="10000" />
</bean>
</property>
</bean>
<bean id="simpleRetryPolicy" class="org.springframework.retry.policy.SimpleRetryPolicy">
<property name="maxAttempts" value="3" />
</bean>

You have to use RejectAndDontRequeueRecoverer to stop re-delivery in the end of retries:
* MessageRecover that causes the listener container to reject
* the message without requeuing. This enables failed messages
* to be sent to a Dead Letter Exchange/Queue, if the broker is
* so configured.
Yes, the messageId is important for that retry use-case.
You can inject custom MessageKeyGenerator strategy to determine a unique key from message if you can't supply it manually during sending.

I never got around to posting the solution, so here it is.
Once I had configured the retry advice chain to the AMQP inbound channel adapter which must include a messageRecoverer of RejectAndDontRequeueRecoverer (which I believe is also the dafault). The important point I was missing was to ensure that when sender's send a message they include a message_id. So if publishing via the RabbitMQ Admin Console I needed to include the predefined message_id property and supply a value.
Using the 'MissingMessageIdAdvice' doesn't help (so I removed) as it will generate a different message_id on each re-deliver on the message leading to the re-try count not incrementing as each delivery was considered different from the last

Related

Spring-Integration Default Retry Configuration

I have a spring-integration project which does the following
1.) Read messages from a queue
2.) Transform messages
3.) Send transformed messages to an Api
Relevant Config for Step 1
<bean id="cachingConnectionFactory" class="org.springframework.jms.connection.CachingConnectionFactory">
<property name="targetConnectionFactory" ref="MQConnectionFactory" />
<property name="sessionCacheSize" value="10"/>
</bean>
<bean id="requestQueue" class="com.ibm.mq.jms.MQQueue">
<constructor-arg index="0" value="${queuemanager}"/>
<constructor-arg index="1" value="${incoming.queue}"/>
</bean>
<integration:poller id="poller" default="true" fixed-delay="1000"/>
<jms:message-driven-channel-adapter id="jmsIn"
destination="requestQueue"
channel="inputJsonConversionChannel"
connection-factory="cachingConnectionFactory" />
Step 3 is a Service Activator, and in case of a failure (not 201 HTTP status) I am throwing a custom exception.
Relevant config for step 3
<int:service-activator input-channel="ApiChannel" ref="EventApiClient" method="post"/>
<int:service-activator input-channel="errorChannel" ref="PListenerExceptionHandler" method="handleFailure"/>
The behaviour that occurs is that, it keeps trying to connect and gets the same errors over and over again.
I wanted to know if someone could explain to me
how is this default retry being configured/triggered?
how can I redirect the errors to an error channel, because right now errors in Step 1 use the global error channel and the default error handler I created. But errors from the Service Activator are not.
Cheers
Kris
It's retrying because the message-driven channel adapter is transactional by default, which means the exception causes the message to be rolled back onto the queue.
You can add an error-channel to the adapter and any exceptions will be sent in the form of an ErrorMessage which has a payload MessagingException which has properties failedMessage and cause.
If the integration flow downstream of the error channel "consumes" the error, the transaction will be committed and the message removed. If the error flow throws an exception, the transaction will be rolled back, as before.
There is a default errorChannel which just logs the exception by default.
error-channel="errorChannel"
or you can use a custom channel and put your own logic in a subscriber to that channel.

Synchronizing JMS Message Queue and JDBC transaction- Is Transaction proxy needed

We need to persist message from a JMS Queue to a database in a transaction ensuring that JMS message is not acknowledged in case any error is thrown during DB persist.
Based on the solution provided here- Transaction handling while using message driven channel adapter & service activator
Below is the approach I arrived at
<int-jms:message-driven-channel-adapter id="jmsIn"
transaction-manager="transactionManager"
connection-factory="sConnectionFactory"
destination-name="emsQueue"
acknowledge="transacted" channel="jmsInChannel"/>
<int:channel id=" jmsInChannel " />
<bean id="dataSource"
class="org.springframework.jdbc.datasource.DriverManagerDataSource">
<property name="driverClassName" value="org.hsqldb.jdbcDriver" />
<property name="url" value="jdbc:hsqldb:mem:testdb" />
</bean>
<!-- Transaction manager for a datasource -->
<bean id="transactionManager" class="org.springframework.jdbc.datasource.DataSourceTransactionManager">
<property name="dataSource" ref="dataSource" />
</bean>
<bean id="sconnectionFactory" class="org.springframework.jms.connection.TransactionAwareConnectionFactoryProxy">
<property name="targetConnectionFactory">
<bean class="project.TestConnectionFactory">
</bean>
</property>
<property name="synchedLocalTransactionAllowed" value="true" />
</bean>
Please confirm if the understanding is correct. Also, have the below queries
Is the TransactionAwareConnectionFactoryProxy necessary in this scenario
JMS Queue and JDBC are two separate transactional resources. Is injecting jdbc transaction manager into JMS adapter as shown in this example equivalent to using a ChainedTransactionManager(chaining JMS Transaction Mananger and JDBC transaction manager)
For TransactionAwareConnectionFactoryProxy, please, consult its JavaDocs:
* Proxy for a target CCI {#link javax.resource.cci.ConnectionFactory}, adding
* awareness of Spring-managed transactions. Similar to a transactional JNDI
* ConnectionFactory as provided by a Java EE server.
*
* <p>Data access code that should remain unaware of Spring's data access support
* can work with this proxy to seamlessly participate in Spring-managed transactions.
* Note that the transaction manager, for example the {#link CciLocalTransactionManager},
* still needs to work with underlying ConnectionFactory, <i>not</i> with this proxy.
I'm not sure how is that related to your request about JMS+DB transaction, but I think you should worry about transaction manager which supports both those transactional resources.
For this purpose Spring suggests JtaTransactionManager for XA transactions.
Or you can consider to use ChainedTransactionManager based on the Dave Syer article: http://www.javaworld.com/article/2077963/open-source-tools/distributed-transactions-in-spring--with-and-without-xa.html
UPDATE
If local transaction are fine for you, you really can go ahead with the TransactionAwareConnectionFactoryProxy and its synchedLocalTransactionAllowed to true. And, of course, use that DataSourceTransactionManager directly from the <int-jms:message-driven-channel-adapter>.
Otherwise you have to go with the ChainedTransactionManager. This solution really adds some overhead over the first TransactionAwareConnectionFactoryProxy-based one.

Logging SOAP message on http gateway spring integration

I am using spring integration (4.0.6) to make SOAP calls from rest service using int-http:outbound-gateway and int-ws:outbound-gateway. Is there a way to log SOAP request message only in case of exception.
<int:gateway id="myGateway" service-interface="myservice">
<int:method name="getData" request-channel="reqChannel" reply-channel="repChannel">
</int-gateway>
my outbound-gateway configured as below
<int-http:outbound-gateway id="og1" request-channel="reqChannel" url="xxx" http-method="POST" reply-channel="repChannel" reply-timeout="10000" message-converters="converter" request-factory="reqFactory">
<bean id="reqFactory"
class="org.springframework.http.client.SimpleClientHttpRequestFactory">
<property name="cTimeout" value="20000"/>
<property name="rTimeout" value="20000"/>
</bean>
Ok, Try this method.
Create your own error Handler which implements the spring Error Handler Interface. In your context file
<bean id="soapErrorHandlingSyncTaskExecutor"
class="org.springframework.integration.util.ErrorHandlingTaskExecutor">
<constructor-arg>
<bean class="org.springframework.core.task.SyncTaskExecutor" />
</constructor-arg>
<constructor-arg>
<bean class="ErrorHandle"
p:errorSender-ref="errorSender"/>
</constructor-arg>
</bean>
<bean id="errorSender" class="org.springframework.integration.core.MessagingTemplate"
p:defaultChannel-ref="errors" />
When ever your WS adapter throws any exception then org.springframework.integration.util.ErrorHandlingTaskExecutor catches this exception now its upto you how you handle the exception in your error handler.
The SOAP faultcode and faultstring is found in the MessagingException's cause (if a SoapFaultClientException).
Here i am sending the error messages to a channel for me to monitor.
Hope this helps.
Would be better to share some config and the point where would you like to get a log/exception.
Both, <int-http:outbound-gateway> and <int-ws:outbound-gateway> have <request-handler-advice-chain> ability, where you could use ExpressionEvaluatingRequestHandlerAdvice and send an ErrorMessage with the request (guilty?) message for any diagnostics.
You should configure the readTimeout on the appropriate ClientHttpRequestFactory, e.g. HttpComponentsClientHttpRequestFactory#setReadTimeout(int timeout).
Regarding converters and the wish to log their results in case of error...
How about to implement ClientHttpRequestInterceptor with the desired logic:
try {
return execution.execute(request, body);
catch (Exception e) {
this.errorChannel.send(...);
}
I'm prety sure that the byte[] body is your SOAP request.
See ClientHttpRequestInterceptor JavaDocs for more info.
The ClientHttpRequestInterceptor is part of RestTemplate, which can be injected to the http:outbound-gateway.
For the ws:outbound-gateway you should use ClientInterceptor with the handleFault implementation and extract MessageContext.getRequest() for your purpose.

how to add a retry advice with jms:message-driven-channel-adapter

I am new to spring integration. My requirement is that if there is a connection problem to the jms q then it should try to connect 3 times then log it and exit the process. I am not able to do it. It throws an error saying it needs the ref attribute for service:activator. But I don't have/know reference of which class to provide here. Is there any other way of doing it?
<int-jms:message-driven-channel-adapter id="msgIn" channel="toRoute" container="messageListenerContainer" />
<int:service-activator id="service" input-channel="toRoute" >
<int:request-handler-advice-chain>
<bean class="org.springframework.integration.handler.advice.RequestHandlerRetryAdvice">
<property name="recoveryCallback">
<bean class="org.springframework.integration.handler.advice.ErrorMessageSendingRecoverer">
<constructor-arg ref=“errorChannel" />
</bean>
</property>
</bean>
</request-handler-advice-chain>
</int:service-activator>
You seem to have completely misunderstood what the framework does.
The service-activator gets a message when one is received from JMS (which implies the connection is good), and needs "something" (a reference to a bean or an expression) to invoke as a result of receiving that message.
The retry advice is to retry calling that service if it fails to process the message for some reason. It is unrelated to whatever is the source of the message (JMS in this case).
It's not clear why you are trying to use Spring Integration for something as simple as testing whether a JMS broker is available.
Perhaps if you can provide some larger context someone might be able to help.

Spring Integration JMS message driven channel adapter stops picking message after certain interval of time

I am using spring integration to read message from TIBCO EMS queue using jms-int:message-driven-channel-adapter. I am facing issue : After certain interval of time say 5-10 hours(occurs at random interval), jms channel adapter stops picking the message even if there is message in the jms queue
Below is my spring integration context :
<bean id="jmsConnectionFactory" class="org.springframework.jms.connection.CachingConnectionFactory">
<property name="targetConnectionFactory" ref="connectionFactory"/>
<property name="sessionCacheSize" value="${sessionCacheSize}"/>
<property name="cacheProducers" value="${cacheProducers}"/>
<property name="cacheConsumers" value="${cacheConsumers}"/>
</bean>
<bean id="jmsQueue" class="${queueClassName}">
<constructor-arg value="${jmsQueue}" />
</bean>
<int-jms:message-driven-channel-adapter
id="jmsMessageDrivenAdapter" connection-factory="jmsConnectionFactory" channel="jmsListenerChannel" destination="jmsQueue"
error-channel="integrationErrorChannel" max-concurrent-consumers="${maxConcurrentConsumers}" auto-startup="${jms.autostart}"/>
I have feature to start/stop jmsMessageDrivenAdapter, and the adapter can be started/stopped anytime while it is picking messages but at this point, start/stop feature does not works too. Please suggest !
This seems to be a duplicate of Spring Integration jms message driven channel adapter fails (but a different contributor, so I'll answer again here)...
This is generally caused by some component in the downstream flow hanging the container thread uninterruptibly (e.g. reading from a socket with no timeout, where no data arrives).
To diagnose, take a thread dump (e.g. jstack ) when the hang occurs, to find out what the listener container threads are doing.

Resources