I have an integration flow like below and I would like to add a retry for whole the integration flow. Like for any exception in the flow, we do retry and when retries exhausted send it to dead-letter queue.
IntegrationFlows
.from(kinesisInboundChannelAdapter(amazonKinesis(), streamNames))
.transform(new IssuanceTransformer())
.route(router())
.get();
Would you please help me out how to configure both retries and dead-letter queue?
Thank you
Your understanding of integration flow and retry is slightly wrong. The retry it applied for some service call. When we have a complex flow like yours we have also channels between service calls. So, what you can do is something like a retry for each component you think is wrong. For that purpose you need to take a look into the RequestHandlerRetryAdvice and an advice() option of the ConsumerEndpointSpec.
If you really have all the logic in a single thread and the same call stack, we really can come up with the solution like "retry the whole sub-flow", but from the Spring Integration perspective it still looks like a service call. For this purpose you need to take a look into the .gateway() EIP-method of the IntegrationFlowDefinition:
IntegrationFlows
.from(kinesisInboundChannelAdapter(amazonKinesis(), streamNames))
.gateway(sf -> sf
.transform(new IssuanceTransformer())
.route(router())
, e -> e.advice(requestHandlerRetryAdvice()))
.get();
Related
I have to listen a queue using spring integration flow and intgeration sqs. Once message is received from queue it should trigger a integration flow. Below is the things which I am trying but everythings fine in but afater receiving test it is not triggering any Integration flow. Please let me know where I am doing wrong:
UPDATED as per comment from Artem
Adapter for SQS.
#Bean
public MessageProducerSupport sqsMessageDrivenChannelAdapter() {
SqsMessageDrivenChannelAdapter adapter = new SqsMessageDrivenChannelAdapter(amazonSQSAsync, "Main");
adapter.setOutputChannel(inputChannel());
adapter.setAutoStartup(true);
adapter.setMessageDeletionPolicy(SqsMessageDeletionPolicy.NEVER);
adapter.setMaxNumberOfMessages(10);
adapter.setVisibilityTimeout(5);
return adapter;
}
Queue configured:
#Bean
public MessageChannel inputChannel() {
return new DirectChannel();
}
Now the main integration flow trigger point:
#Bean
public IntegrationFlow inbound() {
return IntegrationFlows.from("inputChannel").transform(i -> "TEST_FLOW").get();
}
}
Appreciate any type of help.
The sqsMessageDrivenChannelAdapter() must be declared as a #Bean
The inbound() must be declared as a #Bean
This one fully does not make sense IntegrationFlows.from(MessageChannels.queue()). What is the point to start the flow from anonymous channel? Who and how is going to produce messages to that channel?
Make yourself familiar with different channels: https://docs.spring.io/spring-integration/docs/current/reference/html/core.html#channel-implementations
Pay attention that QueueChannel must be consumed via polling endpoint.
Right, there is a default poller auto-configured by Spring Boot, but it is based on a single thread in the TaskScheduler and has a polling period as 10 millis.
I wouldn't recommend to hand off SQS messages to the QueueChannel: when consumer fails, you lose the data. It is better to process those messages in the consumer thread.
Otherwise your intention is not clear in the provided code.
Can you, please, share with us what error you get or anything else?
You also can turn on DEBUG logging level for org.springframework.integration to see how your messages are processed.
My beans/listeners are built using IntegrationFlows
ex.
#Bean
IntegrationFlow registerDevices() {
return IntegrationFlows
.from(adapter)
.channel(channel)
.get();
}
there is such a way to stop listeners after something happens, endpoint call/event
RabbitListenerEndpointRegistry does not work, because I'm not using #RabbitListener
There is the possibility to set autostart up - false, but then how to manipulate it nicely during runtime?
First of all, please, be careful when you choose tags for your questions over here. You concern so far really belongs to Spring Integration and has nothing to do with Spring AMQP. Although you are right: the lifecycle control really goes in the end to the ListenerContainer from Spring AMQP.
Anyway: the end-user API for your use-case is a part of Spring Integration logic.
See the second argument of that from():
.from(adapter, e -> e.id("myAmqpAdapter"))
So, having that id yo are able to reach your AmqpInboundChannelAdapter (implement a Lifecycle) at runtime and stop & start it whenever you need.
See more in docs:
https://docs.spring.io/spring-integration/docs/current/reference/html/dsl.html#java-dsl-endpoints
And this pattern is also good to have on board:
https://docs.spring.io/spring-integration/docs/current/reference/html/system-management.html#control-bus
I need to throttle the movement of messages between some JMS (activeMQ) queues to ensure I dont overrun an external service used during message processing.
I have done this in the past with Camel but given that this project is otherwise entirely Spring based, I figured I'd give spring-integration a whirl.
I am happy to see that in 5.0.7 the Java DSL is in core and would really like to use it instead of xml.
But...I cant seem to find good/current docs for using the DSL to do even simple things like create the input and output messageChannels for JMS.
Could anyone point me to any current example of using the java DSL to create channels that I can use to consume and produce messages with...and then later use in a bridge with some throttling applied?
Well, looks like our JMS chapter in the Reference Manual leaks of the Java DSL samples, similar to what we have so far with AMQP, for example:
https://docs.spring.io/spring-integration/docs/5.0.7.RELEASE/reference/html/amqp.html#_configuring_with_the_java_dsl_2
https://docs.spring.io/spring-integration/docs/5.0.7.RELEASE/reference/html/amqp.html#_configuring_with_the_java_dsl_4
I believe we can add similar paragraphs into the JMS chapter as well. Please, raise a JIRA on the matter and we will address it soon. Meanwhile I suggest you to open a org.springframework.integration.jms.dsl.Jms factory for appropriate builder to use.
On the other hand I can suggest you to take a look into the existing test-case of some possible configurations: https://github.com/spring-projects/spring-integration/blob/master/spring-integration-jms/src/test/java/org/springframework/integration/jms/dsl/JmsTests.java
For example to read from the queue you need a configuration like this:
#Bean
public IntegrationFlow jmsMessageDrivenFlow() {
return IntegrationFlows
.from(Jms.messageDrivenChannelAdapter(jmsConnectionFactory(), DefaultMessageListenerContainer.class)
.outputChannel(jmsMessageDrivenInputChannel())
.destination("jmsMessageDriven")
.configureListenerContainer(c -> c.clientId("foo")))
.<String, String>transform(String::toLowerCase)
.channel(jmsOutboundInboundReplyChannel())
.get();
}
To send to the JMS you need something like this:
#Bean
public IntegrationFlow jmsOutboundFlow() {
return f -> f
.handle(Jms.outboundAdapter(jmsConnectionFactory())
.destinationExpression("headers." + SimpMessageHeaderAccessor.DESTINATION_HEADER)
.configureJmsTemplate(t -> t.id("jmsOutboundFlowTemplate")));
}
I did one simple DSL which retrieves the data from database and doing simple conversion in the service activator.
#Bean
public IntegrationFlow mainFlow() {
return IntegrationFlows.from("userChannel")
.channel("queryChannel")
.handle("sampleConvertor","convertUser")
.get();
queryChannel is a jdbc outbound gateway and sampleConverter is the service Activator.
<int-jdbc:outbound-gateway query="select * from employee where employee_id=:payload"
request-channel="queryChannel" data-source="dataSource"/>
The issue is after retrieving the data from database, the flow is not going to serviceActivator and it simply returns back the database response.
In xml configuration, I used to invoke gateway inside the chain like below.
<int:gateway id="query.gateway" request-channel="queryChannel"/>
Please suggest what I am doing wrong here. Thanks in advance.
That's a bit unusual to combine Java DSL and XML configuration, but they still work together.
Your problem I think that you are missing the fact that your queryChannel has two subscriber at runtime, not a chain of call.
The first one is <int-jdbc:outbound-gateway> and the second is that .handle("sampleConvertor","convertUser"). Right, when you declare a channel in the IntegrationFlow, the next EIP-method produces a subscriber for this channel. At the same time when you use a channel like request-channel or input-channel in the XML configuration that brings a subscriber as well.
So, you have two subscriber on the DirectChannel with the RoundRobinLoadBalancingStrategy and therefore only one of them will handle a message and if it is a request-replly component, like that <int-jdbc:outbound-gateway> it will produce a message into the output-channel or to the replyChannel in the headers. In your case the story is exactly about a replyChannel and therefore you don't go to the .handle("sampleConvertor","convertUser") because it's not the next in the chain, but just a parallel universe by the round-robin algorithm.
If you really would like to reach that .handle("sampleConvertor","convertUser") after calling the <int-jdbc:outbound-gateway>, you should consider to use .gateway("queryChannel") instead of that .channel().
I am using
spring 4.2.4.RELEASE and
spring-integration-java-dsl:1.1.2.RELEASE
I have Jms.inboundGateway and some transformer. When transformer fails to transform a message, inbound gateway retries again and again. But I want to stop the entire flow in case of exception. Is it possible to do?
This is my flow configuration:
#Bean
public IntegrationFlow nonStop {
return IntegrationFlows
.from(Jms.inboundGateway(emsConnectionFactory)
.destination(myDestination)
.configureListenerContainer(spec -> spec
.sessionTransacted(true)
.subscriptionDurable(true)
.durableSubscriptionName(durableSubscriptionName)
.errorHandler((ErrorHandler) t -> {
t.printStackTrace();
throw new RuntimeException(t);
}))
.errorChannel(errorChannel)
.autoStartup(true)
.id(myNonStoppableFlow))
.filter(...)
.transform(...)
.handle(Jms.outboundAdapter(emsConnectionFactory)
.destination(myOnotherDestination))
.get();
}
One interesting notice. When errorHandler swallows an exception inbound gateway retires without any delay. When it throws Runtime exception there is delay about 5 seconds (which is not configured anywhere).
That retry in JMS is called like redelivery: http://www.javaworld.com/article/2074123/java-web-development/transaction-and-redelivery-in-jms.html.
So, any downstream exception makes for your message rollback and redelivery, which can be configured on Broker for the destination.
Actually if your .errorChannel(errorChannel) flow doesn't re-throw the exception, it will be treated as a successful handling and as commit.
From other side you should reconsider if you really need there a Jms.inboundGateway(). Because this one requires a reply, which doesn't look possible with your Jms.outboundAdapter() in the end.