Spring Kafka outboundChannelAdapter's control does not return back in the integration flow - spring-integration

After a messsage is sent, it gets published to Kafka topic but the Message from KafkaSuccessTransformer does not return back to the REST controller. I am trying to return the message as-is if sent successfully but nothing after Kafka handler seems to be invoked.
#MessagingGateway
public interface MyGateway<String, Message<?>> {
#Gateway(requestChannel = "enrollChannel")
Message<?> sendMsg(#Payload String payload);
}
------------------------
#RestController
public class Controller {
MyGateway<String, Message<?>> myGateway;
#PostMapping
public Message<?> send(#RequestBody String request) throws Exception {
Message<?> resp = myGateway.sendMsg(request);
log.info("I am back"); // control doesn't come to this point
return resp;
}
}
--------------------------
#Component
public class MyIntegrationFlow {
KafkaSuccessTransformer stransformer;
#Bean
public MessageChannel enrollChannel() {
return new DirectChannel();
}
#Bean
public MessageChannel kafkaSuccessChannel() {
return new DirectChannel();
}
#Bean
public IntegrationFlow enrollIntegrationFlow() {
return IntegrationFlows.from("enrollChannel")
//another transformer which turns the string to Message<?>
.handle(Kafka.outboundChannelAdapter(kafkaTemplate) //kafkaTemplate has the necesssary config
.topic("topic1")
.messageKey(messageKeyFunction -> messageKeyFunction.getHeaders()
.get("key1")
.sendSuccessChannel("kafkaSuccessChannel"));
}
#Bean
public IntegrationFlow successfulKafkaSends() {
return f -> IntegrationFlows.from("kafkaSuccessChannel").transform(stransformer);
}
}
--------------
#Component
public class KafkaSuccessTransformer {
#Transformer
public Message<?> transform(Message<?> message) {
log.info("Message is sent to Kafka");
return message; //control comes here but does not return to REST controller
}
}

Channel adapters are for one-way traffic; there is no result.
Add a publishSubscribe channel with two subflows; the second one can be just a bridge to nowhere - .bridge() ends the flow. It will then return the outbound message to the gateway.
See https://docs.spring.io/spring-integration/docs/current/reference/html/dsl.html#java-dsl-subflows
Per Artem:
Something is off in the configuration or code. The logic is like this: processSendResult(message, producerRecord, sendFuture, getSendSuccessChannel());. Then: getMessageBuilderFactory().fromMessage(message). So, the replyChannel header is present in this "success" message. Therefore that transform(stransformer) should really produce its return to the replyChannel for a gateway in the beginning. Only the problem could be in the KafkaSuccessTransformer code where it does not copy request message headers for reply message. Please, share its whole code.

Related

Spring integration Java DSL http outbound

How to resend same or modified message from outbound http call in case of specific client error responses like 400, 413 etc
#Bean
private IntegrationFlow myChannel() {
IntegrationFlowBuilder builder =
IntegrationFlows.from(queue)
.handle(//http post method config)
...
.expectedResponseType(String.class))
.channel(MessageChannels.publishSubscribe(channel2));
return builder.get();
}
#Bean
private IntegrationFlow defaultErrorChannel() {
}
EDIT: Added end point to handle method
#Bean
private IntegrationFlow myChannel() {
IntegrationFlowBuilder builder =
IntegrationFlows.from(queue)
.handle(//http post method config)
...
.expectedResponseType(String.class),
e -> e.advice(myRetryAdvice()))
.channel(MessageChannels.publishSubscribe(channel2));
return builder.get();
}
#Bean
public Advice myRetryAdvice(){
... // set custom retry policy
}
Custom Retry policy:
class InternalServerExceptionClassifierRetryPolicy extends
ExceptionClassifierRetryPolicy {
public InternalServerExceptionClassifierRetryPolicy() {
final SimpleRetryPolicy simpleRetryPolicy =
new SimpleRetryPolicy();
simpleRetryPolicy.setMaxAttempts(2);
this.setExceptionClassifier(new Classifier<Throwable, RetryPolicy>() {
#Override
public RetryPolicy classify(Throwable classifiable) {
if (classifiable instanceof HttpServerErrorException) {
// For specifically 500 and 504
if (((HttpServerErrorException) classifiable).getStatusCode() == HttpStatus.INTERNAL_SERVER_ERROR
|| ((HttpServerErrorException) classifiable)
.getStatusCode() == HttpStatus.GATEWAY_TIMEOUT) {
return simpleRetryPolicy;
}
return new NeverRetryPolicy();
}
return new NeverRetryPolicy();
}
});
}}
EDIT 2: Override open() to modify the original message
RequestHandlerRetryAdvice retryAdvice = new
RequestHandlerRetryAdvice(){
#Override
public<T, E extends Throwable> boolean open(RetryContext
retryContext, RetryCallback<T,E> callback){
Message<String> originalMsg =
(Message) retryContext.getAttribute(ErrorMessageUtils.FAILED_MESSAGE_CONTEXT);
Message<String> updatedMsg = //some updated message
retryContext.setAttribute(ErrorMessageUtils.FAILED_MESSAGE_CONTEXT,up datedMsg);
return super.open(retryContext, callback);
}
See a RequestHandlerRetryAdvice: https://docs.spring.io/spring-integration/reference/html/messaging-endpoints.html#message-handler-advice-chain. So, you configure some RetryPolicy to check those HttpClientErrorException for retry and the framework will re-send for you.
Java DSL allows us to configure it via second handle() argument - endpoint configurer: .handle(..., e -> e.advice(myRetryAdvice)): https://docs.spring.io/spring-integration/reference/html/dsl.html#java-dsl-endpoints

Vaadin and receiving email asynchronouosly

My Vaadin 14 application should receive emails in the background. If emails with a certain subject have been received, the user should be informed about this via PUSH message on the UI.
For the entire email handling I implemented the email / message handling from Spring integration and that works too. Two beans (IntegrationFlow and a ServiceActivator) are generated via #Configuration and #Bean annotation in the Spring Application Context like so:
#Configuration
public class EmailReceiver {
#Bean
public HeaderMapper<MimeMessage> mailHeaderMapper() {
return new DefaultMailHeaderMapper();
}
#Bean
public IntegrationFlow imapMailFlow() {
IntegrationFlow flow = IntegrationFlows
.from(Mail.imapInboundAdapter("imaps://user:pass#imap.ionos.de/INBOX")
.userFlag("testSIUserFlag")
.javaMailProperties(new Properties()),
e -> e.autoStartup(true)
.poller(p -> p.fixedDelay(5000)))
.transform(Mail.toStringTransformer())
.channel(MessageChannels.queue("imapChannel"))
.get();
return flow;
}
#Bean(name = PollerMetadata.DEFAULT_POLLER)
public PollerMetadata defaultPoller() {
PollerMetadata pollerMetadata = new PollerMetadata();
pollerMetadata.setTrigger(new PeriodicTrigger(1000));
return pollerMetadata;
}
#Bean
#ServiceActivator(inputChannel = "imapChannel")
public MessageHandler processNewEmail() {
MessageHandler messageHandler = new MessageHandler() {
#Override
public void handleMessage(org.springframework.messaging.Message<?> message) throws MessagingException {
System.out.println("new email received");
}
};
return messageHandler;
}
}
See also here: https://docs.spring.io/spring-integration/docs/current/reference/html/mail.html#mail-java-dsl-configuration
With such a #Configuration annotated class, the emails are received in the background of the Vaadin app. Check.
But how can I integrate a callback into a Vaadin view in the method EmailReceiver.processNewEmail?
#Bean
#ServiceActivator(inputChannel = "imapChannel")
public MessageHandler processNewEmail(UI ui) {
This always throws an error at application start: Scope vaadin-ui is not active for the current thread; consider defining a scoped proxy for this bean.
There is the example for asynchronous updates with Vaadin https://vaadin.com/docs/v14/flow/advanced/tutorial-push-access.
In contrast to this, I have to create a #Bean for #ServiceActivator handling. As soon as that is the case, there is always the error There is no UI available. The UI scope is not active.
If I move the method processNewEmail() into a separate class I still cannot reference a Vaadin UI:
#MessageEndpoint
class EmailMessageHandler {
private UI ui;
public EmailMessageHandler(UI ui) {
this.ui = ui;
}
#Bean
#ServiceActivator(inputChannel = "imapChannel")
public MessageHandler processNewEmail() {
MessageHandler messageHandler = new MessageHandler() {
#Override
public void handleMessage(org.springframework.messaging.Message<?> message) throws MessagingException {
System.out.println("new email received" + message);
}
};
return messageHandler;
}
}
How can I combine Vaadin asynchronous handling and Spring-Integration Email/ServiceActivator processing?
The point is that your mail receiving functionality is singleton per your application. On the other hand you are going to have as many UIs as many users do HTTP requests to your application. So, you need to think about some intermediary to dump email and get them from there when UI request happens.
You already have that imapChannel as a QueueChannel. So, you can take it from your UI scoped code and call its receive() API to pull the next message. Only the problem that it is a queue: as long as one call receive(), the other won't see the same message. Probably this is OK for your so far, but better to think about something what could be treated as topic in messaging terms. A good candidate easy to use is a Reactor's Sinks.Many: https://projectreactor.io/docs/core/release/reference/#sinks

Spring Integration - Messaging Gateway with a returning value

I'm new with Spring Integration. I need to implement a Messaging Gateway with a returning value. In order to continue some processing asynchronously after executing some synchronous steps. So I made 2 activators
#Slf4j
#MessageEndpoint
public class Activator1 {
#ServiceActivator(inputChannel = "asyncChannel")
public void async(){
log.info("Just async message");
try {
Thread.sleep(500);
} catch (InterruptedException e) {
log.error("I don't want to sleep now");
}
}
}
and
#Slf4j
#MessageEndpoint
public class Activator2 {
#ServiceActivator(inputChannel = "syncChannel")
public ResponseEntity sync(){
try {
Thread.sleep(500);
return ResponseEntity.ok("Return Http Message");
} catch (InterruptedException e) {
log.error("I don't want to sleep");
}
return ResponseEntity.badRequest().build();
}
}
The pipeline
#Configuration
public class Pipeline {
#Bean
MessageChannel asyncChannel() {
return new DirectChannel();
}
#Bean
public MessageChannel syncChannel() {
return MessageChannels.direct().get();
}
}
the gateway
#MessagingGateway
public interface ReturningGateway {
#Gateway(requestChannel = "asyncChannel", replyChannel = "syncChannel")
public ResponseEntity getSyncHttpResponse();
}
And Controller
#Slf4j
#RestController
#RequestMapping("/sync")
public class ResponseController {
#Autowired
ReturningGateway returningGateway;
#PostMapping("/http-response")
public ResponseEntity post() {
return returningGateway.getSyncHttpResponse();
}
}
So I'm not sure if thats the correct way to do what I want to do
Can you give me hand?
Let me try to explain some things first of all!
#Gateway(requestChannel = "asyncChannel", replyChannel = "syncChannel")
The requestChannel is where a gateway sends a message. But since you don't have any arguments in the gateway method and there is no a payloadExpression, the behavior is to "receive" from that channel. See docs for more info: https://docs.spring.io/spring-integration/docs/current/reference/html/messaging-endpoints.html#gateway-calling-no-argument-methods.
The replyChannel is where to wait for a reply, not send. In most cases a gateway relies on the replyChannel header for correlation. The request-reply pattern in messaging. We need an explicit replyChannel if it is a PublishSubscribeChannel to track a reply somehow or when we deal with the flow which we can't modify to rely on the replyChannel header. See the same gateway chapter in the docs.
Your use-case is not clear for me: you say async continuation, but at the same time the return from your gateway contract looks like a result of that sync() method. From here, please make yourself familiar with the gateway contract and then come back to us with refreshed vision for your solution.

spring-integration-kafka: KafkaTemplate#setMessageConverter(RecordMessageConverter) has no effect

I'm trying to set a custom message converter for my Spring Integration Kafka message handler (yes, I know I can supply serializer configsā€”I'm trying to do something a little different).
I have the following:
#Bean
public KafkaTemplate<String, String> kafkaTemplate() {
final KafkaTemplate<String, String> kafkaTemplate = new KafkaTemplate<>(producerFactory());
kafkaTemplate.setMessageConverter(new MessagingMessageConverter() {
#Override
public ProducerRecord<?, ?> fromMessage(final Message<?> message, final String s) {
LOGGER.info("fromMessage({}, {})", message, s);
return super.fromMessage(message, s);
}
});
return kafkaTemplate;
}
#Bean
#ServiceActivator(inputChannel = "kafkaMessageChannel")
public MessageHandler kafkaMessageHandler() {
final KafkaProducerMessageHandler<String, String> handler = new KafkaProducerMessageHandler<>(kafkaTemplate());
handler.setTopicExpression(new LiteralExpression(getTopic()));
handler.setSendSuccessChannel(kafkaSuccessChannel());
return handler;
}
When a message is sent to kafkaMessageChannel, the handler sends it and the result shows up in kafkaSuccessChannel, but the RecordMessageConverter I set in the template was never called
The template message converter is only used when using template.send(Message<?>) which is not used by the outbound channel adapter.
The outbound adapter maps the headers itself using its header mapper; there is no conversion performed on the message payload.
What documentation leads you to believe the converter is used in this context?

Calls to gateway result never return to caller when successful

I am using Spring Integration DSL and have a simple Gateway:
#MessagingGateway(name = "eventGateway", defaultRequestChannel = "inputChannel")
public interface EventProcessorGateway {
#Gateway(requestChannel="inputChannel")
public void processEvent(Message message)
}
My spring integration flow is defined as:
#Bean MessageChannel inputChannel() { return new DirectChannel(); }
#Bean MessageChannel errorChannel() { return new DirectChannel(); }
#Bean MessageChannel retryGatewayChannel() { return new DirectChannel(); }
#Bean MessageChannel jsonChannel() { return new DirectChannel(); }
#Bean
public IntegrationFlow postEvents() {
return IntegrationFlows.from(inputChannel())
.route("headers.contentType", m -> m.channelMapping(MediaType.APPLICATION_JSON_VALUE, "json")
)
.get();
}
#Bean
public IntegrationFlow retryGateway() {
return IntegrationFlows.from("json")
.gateway(retryGatewayChannel(), e -> e.advice(retryAdvice()))
.get();
}
#Bean
public IntegrationFlow transformJsonEvents() {
return IntegrationFlows
.from(retryGatewayChannel())
.transform(new JsonTransformer())
.handle(new JsonHandler())
.get();
}
The JsonTransformer is a simple AbstractTransformer that transforms the JSON data and passes it to the JsonHandler.
class JsonHandler extends AbstractMessageHandler {
public void handleMessageInternal(Message message) throws Exception {
// do stuff, return nothing if success else throw Exception
}
}
I call my gateway from code as such:
try {
Message<List<EventRecord>> message = MessageBuilder.createMessage(eventList, new MessageHeaders(['contentType': contentType]))
eventProcessorGateway.processEvent(message)
logSuccess(eventList)
} catch (Exception e) {
logError(eventList)
}
I want the entire call and processing to be synchronous, and any errors that occur to be caught so I can handle them appropriately. The call to the gateway works, the message gets sent to through the Transformer and to the Handler, processed and if an Exception occurs it bubbles back and is caught and logError() is called. However if the call is successful, the call to logSuccess() never occurs. It is like execution stops/hangs after the Handler processes the message and never returns. I do not need to actually get any response, I am more concerned if something fails to process. Do I need to send something back to the initial EventProcessorGateway?
Your issue is here:
return IntegrationFlows.from("json")
.gateway(retryGatewayChannel(), e -> e.advice(retryAdvice()))
.get();
where that .gateway() is request/reply because it is a part of the main flow.
It is something similar to the <gateway> within <chain>.
So, even if your main flow is one-way, using .gateway() inside that requires from your sub-flow some reply, but this one:
.handle(new JsonHandler())
.get();
doesn't do that.
Because it is one-way MessageHandler.
From other side, even if you'd make the last one as request-reply (AbstractReplyProducingMessageHandler), it won't help you because you don't know what to do with that reply after the mid-flow gateway. Just because your main flow is the one-way.
You must re-think your desing a bit more and try to get rid of that mid-flow gateway. I see that you try to make some logic with retryAdvice().
But how about to move it to the .handle(new JsonHandler()) instead of that wrong .gateway()?

Resources