I am refactoring some of my Spring Integration RSS feed code which uses feed inbound channel adaptor to a microservice. I want the feeds to be stored internally in a mongodb database (in case of failures, audit etc) and also to write the feed (in JSON) to a kafka topic for onward processing.
How do I do this with Spring Integration? Do I need a pub/subscribe queue with two handlers?
Any example code using Java DSL would be most helpful.
Not sure if you need pub/sub for such a use-case, but definitely you can use QueueChannel in between Feed Inbound Channel Adapter and Kafka Outbound Channel Adapter. That QueueChannel can really be supplied with the MongoDbChannelMessageStore for persistence in case of failures.
The Java DSL sample is like this:
#Bean
public IntegrationFlow feedFlow(MongoDbChannelMessageStore messageStore) {
return IntegrationFlows
.from(Feed.inboundAdapter(...),
e -> e.poller(p -> p.fixedDelay(1000)))
.channel(c -> c.queue(messageStore, "entries"))
.handle(Kafka.outboundChannelAdapter(...),
e -> e.poller(p -> p.fixedDelay(1000)))
.get();
}
https://docs.spring.io/spring-integration/docs/5.0.3.RELEASE/reference/html/feed.html#feed-java-configuration
https://docs.spring.io/spring-integration/docs/5.0.3.RELEASE/reference/html/mongodb.html#mongodb-priority-channel-message-store
https://docs.spring.io/spring-kafka/docs/2.1.4.RELEASE/reference/html/_spring_integration.html#si-outbound
Related
we used to configure the poller in int:chain like below, the inboundChannel is configured with queue
<int:chain id="messageProcessChain" input-channel="inboundChannel" >
<int:poller fixed-delay="1" max-messages-per-poll="1"/>
This time, we'd like to programmatically initialize int flows, so using DSL. my test code as below:
Jms.messageDrivenChannelAdapter(inConnFactory).destination(flowProperties.getInputQueue()))
.channel(MessageChannels.queue(mongoDbChannelMessageStore,"groupId"))
// .enrichHeaders(testHeaders)
.handle(m -> {
System.out.println("test handler");
logger.info("test dsl flow:"+m.getPayload().toString());
},c -> c.poller(Pollers.fixedRate(flowProperties.getInboudFixedRate()).maxMessagesPerPoll(10)))
.get()
it cannot work with the ".enrichHeaders(testHeaders)", due to'No poller has been defined for endpoint', but neither can I configure a poller for the header richer because it has an implicit SubscribableChannel?
In such case, can only use a bridge to connect the two channel? Is there some other approaches?
Use .enrichHeaders(headers, e -> e.poller(...)).
However, you should NOT use a queue channel with a message-driven channel adapter, you will likely lose messages in the event of a failure.
To achieve concurrency, increase concurrency on the adapter.
I have set up a simple Spring Integration flow which is composed of such steps:
poll a rest api periodically then
do some processing on the payload
and land it on a Kafka topic.
Please observe the code below:
#Component
public class MyIntegrationFlow extends IntegrationFlowAdapter {
#Override
protected IntegrationFlowDefinition<?> buildFlow() {
return from(() -> List.of("pathVariable1", "pathVariable2"), c -> c.poller(Pollers.fixedDelay(5, TimeUnit.SECONDS)))
.split()
.handle(httpRequest(), c -> c.advice(new RequestHandlerRetryAdvice()))
.transform(Tranformers.fromJson(Foo.class))
.filter(payload -> payload.isValid())
.log()
.transform(Tranformers.toJson())
.channel(Source.OUTPUT); // output channel for kafka topic
}
private HttpMessageHandlerSpec httpRequest() {
return Http.outboundGateway("http://somehost:8080/{pathVariable}")
.httpMethod(GET)
.uriVariable("pathVariable", Message::getPayload)
.expectedResponseType(String.class);
}
}
This works brilliantly, however, I am struggling to come up with some good tests.
How am I supposed to mock the external REST API?
How am I supposed to test that the retry policy does kick in and the desired number of http requests are made?
How do I change the MessageSource of the flow (list of path vars) that is polled periodically?
How do I check if the payload has successfully made it to the Kafka topic?
Too much questions and some of them requires too broad explanation. Anyway I think you can start from Spring Integration Testing Framework and its documentation.
How am I supposed to mock the external REST API?
I think you can just consider to use a Mock MVC from Spring Framework and its MockMvcClientHttpRequestFactory to inject into the HttpRequestExecutingMessageHandler based on the HttpMessageHandlerSpec.
retry policy does kick
Well, I guess the same mocked MVC endpoint can verify how many times it has been called and fail for first several times to initiate that retry.
How do I change the MessageSource
This is exactly a part of Spring Integration Testing Framework with its MockIntegration.mockMessageSource() and MockIntegrationContext: https://docs.spring.io/spring-integration/docs/5.1.6.RELEASE/reference/html/#mockintegration
made it to the Kafka topic?
Or you the mentioned MockIntegration.mockMessageHandler() to verify that endpoint for Kafka is called. Or use an Embedded Kafka from Spring Kafka project: https://docs.spring.io/spring-kafka/docs/2.2.7.RELEASE/reference/html/#embedded-kafka-annotation
I see many examples related to spring-integration-file. But I am looking for an example application where it uses spring-integration-jpa to pull data from database using Inbound Channel Adapter and create a Java object out of it.
Any help is much appreciated.
There is a basic JPA sample in the official Spring Integration Samples repository: https://github.com/spring-projects/spring-integration-samples/tree/master/basic/jpa.
The simple Java DSL sample for Inbound Channel Adapter might look like this:
#Bean
public IntegrationFlow pollingAdapterFlow(EntityManagerFactory entityManagerFactory) {
return IntegrationFlows
.from(Jpa.inboundAdapter(entityManagerFactory)
.entityClass(StudentDomain.class)
.maxResults(1)
.expectSingleResult(true),
e -> e.poller(p -> p.trigger(new OnlyOnceTrigger())))
.channel(c -> c.queue("pollingResults"))
.get();
}
I have a model object which is populated after several transformation and parsing. Now I need to send the message attribute within the model to kafka using spring integration.
I am able to construct the key using the method messageKey but how can i get the actual message from the model like m.getPayload().getMessage() and send it to kafka.
.publishSubscribeChannel(pubSub -> pubSub
.subscribe(flow -> flow
.bridge(e -> e.order(Ordered.HIGHEST_PRECEDENCE))
.handle(Kafka.outboundChannelAdapter(kafkaTemplate).
messageKey(m -> ((AcarsFlightInformation) m.getPayload()).getFlightNbr()).topic(acarsKafkaTopic)))
It's not entirely clear what you are asking. The payload of the message sent to the adapter becomes the value of the producer record.
I think what you are asking is that you only want to send part of the payload.
Use a header enricher and transformer before the .handle...
.enrichHeaders(h -> h.headerExpression(KafkaHeaders.MESSAGE_KEY, "payload.flightNumber")
.transform("payload.message")
.handle(Kafka.outboundChannelAdapter(kafkaTemplate)
.topic(acarsKafkaTopic))
.get();
The adapter will look for that header for the key.
I need to throttle the movement of messages between some JMS (activeMQ) queues to ensure I dont overrun an external service used during message processing.
I have done this in the past with Camel but given that this project is otherwise entirely Spring based, I figured I'd give spring-integration a whirl.
I am happy to see that in 5.0.7 the Java DSL is in core and would really like to use it instead of xml.
But...I cant seem to find good/current docs for using the DSL to do even simple things like create the input and output messageChannels for JMS.
Could anyone point me to any current example of using the java DSL to create channels that I can use to consume and produce messages with...and then later use in a bridge with some throttling applied?
Well, looks like our JMS chapter in the Reference Manual leaks of the Java DSL samples, similar to what we have so far with AMQP, for example:
https://docs.spring.io/spring-integration/docs/5.0.7.RELEASE/reference/html/amqp.html#_configuring_with_the_java_dsl_2
https://docs.spring.io/spring-integration/docs/5.0.7.RELEASE/reference/html/amqp.html#_configuring_with_the_java_dsl_4
I believe we can add similar paragraphs into the JMS chapter as well. Please, raise a JIRA on the matter and we will address it soon. Meanwhile I suggest you to open a org.springframework.integration.jms.dsl.Jms factory for appropriate builder to use.
On the other hand I can suggest you to take a look into the existing test-case of some possible configurations: https://github.com/spring-projects/spring-integration/blob/master/spring-integration-jms/src/test/java/org/springframework/integration/jms/dsl/JmsTests.java
For example to read from the queue you need a configuration like this:
#Bean
public IntegrationFlow jmsMessageDrivenFlow() {
return IntegrationFlows
.from(Jms.messageDrivenChannelAdapter(jmsConnectionFactory(), DefaultMessageListenerContainer.class)
.outputChannel(jmsMessageDrivenInputChannel())
.destination("jmsMessageDriven")
.configureListenerContainer(c -> c.clientId("foo")))
.<String, String>transform(String::toLowerCase)
.channel(jmsOutboundInboundReplyChannel())
.get();
}
To send to the JMS you need something like this:
#Bean
public IntegrationFlow jmsOutboundFlow() {
return f -> f
.handle(Jms.outboundAdapter(jmsConnectionFactory())
.destinationExpression("headers." + SimpMessageHeaderAccessor.DESTINATION_HEADER)
.configureJmsTemplate(t -> t.id("jmsOutboundFlowTemplate")));
}