My beans/listeners are built using IntegrationFlows
ex.
#Bean
IntegrationFlow registerDevices() {
return IntegrationFlows
.from(adapter)
.channel(channel)
.get();
}
there is such a way to stop listeners after something happens, endpoint call/event
RabbitListenerEndpointRegistry does not work, because I'm not using #RabbitListener
There is the possibility to set autostart up - false, but then how to manipulate it nicely during runtime?
First of all, please, be careful when you choose tags for your questions over here. You concern so far really belongs to Spring Integration and has nothing to do with Spring AMQP. Although you are right: the lifecycle control really goes in the end to the ListenerContainer from Spring AMQP.
Anyway: the end-user API for your use-case is a part of Spring Integration logic.
See the second argument of that from():
.from(adapter, e -> e.id("myAmqpAdapter"))
So, having that id yo are able to reach your AmqpInboundChannelAdapter (implement a Lifecycle) at runtime and stop & start it whenever you need.
See more in docs:
https://docs.spring.io/spring-integration/docs/current/reference/html/dsl.html#java-dsl-endpoints
And this pattern is also good to have on board:
https://docs.spring.io/spring-integration/docs/current/reference/html/system-management.html#control-bus
Related
I have set up a simple Spring Integration flow which is composed of such steps:
poll a rest api periodically then
do some processing on the payload
and land it on a Kafka topic.
Please observe the code below:
#Component
public class MyIntegrationFlow extends IntegrationFlowAdapter {
#Override
protected IntegrationFlowDefinition<?> buildFlow() {
return from(() -> List.of("pathVariable1", "pathVariable2"), c -> c.poller(Pollers.fixedDelay(5, TimeUnit.SECONDS)))
.split()
.handle(httpRequest(), c -> c.advice(new RequestHandlerRetryAdvice()))
.transform(Tranformers.fromJson(Foo.class))
.filter(payload -> payload.isValid())
.log()
.transform(Tranformers.toJson())
.channel(Source.OUTPUT); // output channel for kafka topic
}
private HttpMessageHandlerSpec httpRequest() {
return Http.outboundGateway("http://somehost:8080/{pathVariable}")
.httpMethod(GET)
.uriVariable("pathVariable", Message::getPayload)
.expectedResponseType(String.class);
}
}
This works brilliantly, however, I am struggling to come up with some good tests.
How am I supposed to mock the external REST API?
How am I supposed to test that the retry policy does kick in and the desired number of http requests are made?
How do I change the MessageSource of the flow (list of path vars) that is polled periodically?
How do I check if the payload has successfully made it to the Kafka topic?
Too much questions and some of them requires too broad explanation. Anyway I think you can start from Spring Integration Testing Framework and its documentation.
How am I supposed to mock the external REST API?
I think you can just consider to use a Mock MVC from Spring Framework and its MockMvcClientHttpRequestFactory to inject into the HttpRequestExecutingMessageHandler based on the HttpMessageHandlerSpec.
retry policy does kick
Well, I guess the same mocked MVC endpoint can verify how many times it has been called and fail for first several times to initiate that retry.
How do I change the MessageSource
This is exactly a part of Spring Integration Testing Framework with its MockIntegration.mockMessageSource() and MockIntegrationContext: https://docs.spring.io/spring-integration/docs/5.1.6.RELEASE/reference/html/#mockintegration
made it to the Kafka topic?
Or you the mentioned MockIntegration.mockMessageHandler() to verify that endpoint for Kafka is called. Or use an Embedded Kafka from Spring Kafka project: https://docs.spring.io/spring-kafka/docs/2.2.7.RELEASE/reference/html/#embedded-kafka-annotation
I am using following to define my integration flow:
#Bean
public IntegrationFlow pollingFlow(MessageSource<Object> jdbcMessageSource) {
return IntegrationFlows.from(jdbcMessageSource,
c -> c.poller(Pollers.fixedRate(250, TimeUnit.MILLISECONDS)
.maxMessagesPerPoll(1)
.transactional()))
.split()
.channel(taskSourceChannel())
.get();
}
I would like to make call to service activator that reads from taskSourceChannel as transactional. Also, I want to use following with my transaction.
#Bean
public TransactionSynchronizationFactory transactionSynchronizationFactory() {
ExpressionEvaluatingTransactionSynchronizationProcessor syncProcessor
= new ExpressionEvaluatingTransactionSynchronizationProcessor();
syncProcessor.setAfterCommitChannel(successChannel());
syncProcessor.setAfterRollbackChannel(failureChannel());
return new DefaultTransactionSynchronizationFactory(syncProcessor);
}
The taskSourceChannel is an executor channel.
#Bean
public MessageChannel taskSourceChannel() {
return new ExecutorChannel(executor());
}
How can I add transaction support after split while using TransactionSynchronizationFactory. I don't want to make polling transacational. The only solution I can think of is putting transactional on activator but that won't solve my problem. I would like to make it applicable to any service activator uses this channel.
You question is not so clear, but you definitely need to consider to add transaction into the service activator. Although you don't show what is the subscriber for that taskSourceChannel, but you need to think do not have several subscribers on it.
Nevertheless I think your point is to apply TX into the service activator on this taskSourceChannel and everything after that one.
For this purpose Spring Integration provides a TransactionHandleMessageAdvice. See more info the Reference Manual: https://docs.spring.io/spring-integration/reference/html/messaging-endpoints-chapter.html#tx-handle-message-advice.
The TransactionSynchronizationFactory is only used from the AbstractPollingEndpoint implementations. However you can still utilize it in your transactional context relying on the TransactionSynchronizationManager.registerSynchronization().
I need to throttle the movement of messages between some JMS (activeMQ) queues to ensure I dont overrun an external service used during message processing.
I have done this in the past with Camel but given that this project is otherwise entirely Spring based, I figured I'd give spring-integration a whirl.
I am happy to see that in 5.0.7 the Java DSL is in core and would really like to use it instead of xml.
But...I cant seem to find good/current docs for using the DSL to do even simple things like create the input and output messageChannels for JMS.
Could anyone point me to any current example of using the java DSL to create channels that I can use to consume and produce messages with...and then later use in a bridge with some throttling applied?
Well, looks like our JMS chapter in the Reference Manual leaks of the Java DSL samples, similar to what we have so far with AMQP, for example:
https://docs.spring.io/spring-integration/docs/5.0.7.RELEASE/reference/html/amqp.html#_configuring_with_the_java_dsl_2
https://docs.spring.io/spring-integration/docs/5.0.7.RELEASE/reference/html/amqp.html#_configuring_with_the_java_dsl_4
I believe we can add similar paragraphs into the JMS chapter as well. Please, raise a JIRA on the matter and we will address it soon. Meanwhile I suggest you to open a org.springframework.integration.jms.dsl.Jms factory for appropriate builder to use.
On the other hand I can suggest you to take a look into the existing test-case of some possible configurations: https://github.com/spring-projects/spring-integration/blob/master/spring-integration-jms/src/test/java/org/springframework/integration/jms/dsl/JmsTests.java
For example to read from the queue you need a configuration like this:
#Bean
public IntegrationFlow jmsMessageDrivenFlow() {
return IntegrationFlows
.from(Jms.messageDrivenChannelAdapter(jmsConnectionFactory(), DefaultMessageListenerContainer.class)
.outputChannel(jmsMessageDrivenInputChannel())
.destination("jmsMessageDriven")
.configureListenerContainer(c -> c.clientId("foo")))
.<String, String>transform(String::toLowerCase)
.channel(jmsOutboundInboundReplyChannel())
.get();
}
To send to the JMS you need something like this:
#Bean
public IntegrationFlow jmsOutboundFlow() {
return f -> f
.handle(Jms.outboundAdapter(jmsConnectionFactory())
.destinationExpression("headers." + SimpMessageHeaderAccessor.DESTINATION_HEADER)
.configureJmsTemplate(t -> t.id("jmsOutboundFlowTemplate")));
}
I have an integration flow like below and I would like to add a retry for whole the integration flow. Like for any exception in the flow, we do retry and when retries exhausted send it to dead-letter queue.
IntegrationFlows
.from(kinesisInboundChannelAdapter(amazonKinesis(), streamNames))
.transform(new IssuanceTransformer())
.route(router())
.get();
Would you please help me out how to configure both retries and dead-letter queue?
Thank you
Your understanding of integration flow and retry is slightly wrong. The retry it applied for some service call. When we have a complex flow like yours we have also channels between service calls. So, what you can do is something like a retry for each component you think is wrong. For that purpose you need to take a look into the RequestHandlerRetryAdvice and an advice() option of the ConsumerEndpointSpec.
If you really have all the logic in a single thread and the same call stack, we really can come up with the solution like "retry the whole sub-flow", but from the Spring Integration perspective it still looks like a service call. For this purpose you need to take a look into the .gateway() EIP-method of the IntegrationFlowDefinition:
IntegrationFlows
.from(kinesisInboundChannelAdapter(amazonKinesis(), streamNames))
.gateway(sf -> sf
.transform(new IssuanceTransformer())
.route(router())
, e -> e.advice(requestHandlerRetryAdvice()))
.get();
I did one simple DSL which retrieves the data from database and doing simple conversion in the service activator.
#Bean
public IntegrationFlow mainFlow() {
return IntegrationFlows.from("userChannel")
.channel("queryChannel")
.handle("sampleConvertor","convertUser")
.get();
queryChannel is a jdbc outbound gateway and sampleConverter is the service Activator.
<int-jdbc:outbound-gateway query="select * from employee where employee_id=:payload"
request-channel="queryChannel" data-source="dataSource"/>
The issue is after retrieving the data from database, the flow is not going to serviceActivator and it simply returns back the database response.
In xml configuration, I used to invoke gateway inside the chain like below.
<int:gateway id="query.gateway" request-channel="queryChannel"/>
Please suggest what I am doing wrong here. Thanks in advance.
That's a bit unusual to combine Java DSL and XML configuration, but they still work together.
Your problem I think that you are missing the fact that your queryChannel has two subscriber at runtime, not a chain of call.
The first one is <int-jdbc:outbound-gateway> and the second is that .handle("sampleConvertor","convertUser"). Right, when you declare a channel in the IntegrationFlow, the next EIP-method produces a subscriber for this channel. At the same time when you use a channel like request-channel or input-channel in the XML configuration that brings a subscriber as well.
So, you have two subscriber on the DirectChannel with the RoundRobinLoadBalancingStrategy and therefore only one of them will handle a message and if it is a request-replly component, like that <int-jdbc:outbound-gateway> it will produce a message into the output-channel or to the replyChannel in the headers. In your case the story is exactly about a replyChannel and therefore you don't go to the .handle("sampleConvertor","convertUser") because it's not the next in the chain, but just a parallel universe by the round-robin algorithm.
If you really would like to reach that .handle("sampleConvertor","convertUser") after calling the <int-jdbc:outbound-gateway>, you should consider to use .gateway("queryChannel") instead of that .channel().