I am processing messages from batch.
Defined advice for MessageHandler
ExpressionEvaluatingRequestHandlerAdvice advice = new ExpressionEvaluatingRequestHandlerAdvice();
advice.setFailureChannelName("errorChannel");
Once there is an error processing one or more messages from they payload, serviceActivator is triggered
#ServiceActivator(inputChannel = "errorChannel")
public void handleFailure (Message<?> message){
ExpressionEvaluatingRequestHandlerAdvice.MessageHandlingExpressionEvaluatingAdviceException adviceException = (ExpressionEvaluatingRequestHandlerAdvice.MessageHandlingExpressionEvaluatingAdviceException) message.getPayload();
//throw CustomException
So the above gives me the error but not which item from payload caused it (can be more than one).
What is the correct way to retry the payload(one by one)?
Should I somehow 'Split' the payload ? if yes what is the way to do this.
I tried replacing
#ServiceActivator(inputChannel = "errorChannel")
public void handleFailure (Message<?> message){
with
#Splitter(inputChannel = "errorChannel", outputChannel = "outboundChannel")
public List<Message<?>> handleFailure2(Message<?> message)
But couldn't split the messages from there as message's type is ExpressionEvaluatingRequestHandlerAdvice.MessageHandlingExpressionEvaluatingAdviceException
Spring Integration deals with a Message as a unit of work. It doesn't matter for the framework if payload is a batch of data or not: the service throws an exception, the whole failed message is sent to the error channel with a single exception.
You may consider to use an AggregateMessageDeliveryException in your service to gather all the errors in a single exception and then throw it for processing in that error handler. There you can inspect such an aggregate to determine what items in batch have been failed.
If you'd like to go a splitter way, then it is going to be a general splitter for your batch and then you process in your service items one by one. Not sure if that is what you'd like to do. However splitter can be configured for an ExecutorChannel as an output to process items in parallel.
Related
just want to ask what is the redisQueueInboundGateway.setReplyChannelName for
I got a log B and then a log.
1.My question is in what situation will the log C be printed when I set it to the RedisQueueInboundGateway.
the doc in "https://docs.spring.io/spring-integration/reference/html/redis.html#redis-queue-inbound-gateway" seems incorrect for class name and class explanation such like:
2.1 the 'RedisOutboundChannelAdapter' is named in 'RedisPublishingMessageHandler'.
2.2 the 'RedisQueueOutboundChannelAdapter' is named in 'RedisQueueMessageDrivenEndpoint'.
2.3 the explanation of Redis Queue Outbound Gateway is exactly the copy of Redis Queue Inbound Gateway.
#GetMapping("test")
public void test() {
this.teller.test("testing 1");
#Gateway(requestChannel = "inputA")
void test(String transaction);
#Bean("A")
PublishSubscribeChannel getA() {
return new PublishSubscribeChannel();
}
#Bean("B")
PublishSubscribeChannel getB() {
return new PublishSubscribeChannel();
}
#Bean("C")
PublishSubscribeChannel getC() {
return new PublishSubscribeChannel();
}
#ServiceActivator(inputChannel = "A")
void aTesting(Message message) {
System.out.println("A");
System.out.println(message);
}
#ServiceActivator(inputChannel = "B")
String bTesting(Message message) {
System.out.println("B");
System.out.println(message);
return message.getPayload() + "Basdfasdfasdfadsfasdf";
}
#ServiceActivator(inputChannel = "C")
void cTesting(Message message) {
System.out.println("C");
System.out.println(message);
}
#ServiceActivator(inputChannel = "inputA")
#Bean
RedisQueueOutboundGateway getRedisQueueOutboundGateway(RedisConnectionFactory connectionFactory) {
val redisQueueOutboundGateway = new RedisQueueOutboundGateway(Teller.CHANNEL_CREATE_INVOICE, connectionFactory);
redisQueueOutboundGateway.setReceiveTimeout(5);
redisQueueOutboundGateway.setOutputChannelName("A");
redisQueueOutboundGateway.setSerializer(new GenericJackson2JsonRedisSerializer(new ObjectMapper()));
return redisQueueOutboundGateway;
}
#Bean
RedisQueueInboundGateway getRedisQueueInboundGateway(RedisConnectionFactory connectionFactory) {
val redisQueueInboundGateway = new RedisQueueInboundGateway(Teller.CHANNEL_CREATE_INVOICE, connectionFactory);
redisQueueInboundGateway.setReceiveTimeout(5);
redisQueueInboundGateway.setRequestChannelName("B");
redisQueueInboundGateway.setReplyChannelName("C");
redisQueueInboundGateway.setSerializer(new GenericJackson2JsonRedisSerializer(new ObjectMapper()));
return redisQueueInboundGateway;
}
Your concern is not clear.
2.1
There is a component (pattern name) and there is a class on background covering the logic.
Sometime they are not the same.
So, Redis Outbound Channel Adapter is covered by the RedisPublishingMessageHandler, just because there is a ConsumerEndpointFactoryBean to consume messages from the input channel and RedisPublishingMessageHandler to handle them. In other words the framework creates two beans to make such a Redis interaction working. In fact all the outbound channel adapters (gateways) are handled the same way: endpoint plus handler. Together they are called adapter or gateway depending on the type of the interaction.
2.2
I don't see such a misleading in the docs.
2.3
That's not true.
See difference:
Spring Integration introduced the Redis queue outbound gateway to perform request and reply scenarios. It pushes a conversation UUID to the provided queue,
Spring Integration 4.1 introduced the Redis queue inbound gateway to perform request and reply scenarios. It pops a conversation UUID from the provided queue
All the inbound gateways are supplied with an optional replyChannel to track the replies. It is not where this type of gateways is going to send something. It is fully opposite: the place where this inbound gateway is going to take a reply channel to send a reply message into Redis back. The Inbound gateway is initiated externally. In our case as a request message in the configured Redis list. When you Integration flow does its work, it sends a reply message to this gateway. In most cases it is done automatically using a replyChannel header from the message. But if you would like to track that reply, you add a PublishSubscribeChannel as that replyChannel option on the inbound gateway and both your service activator and the gateway get the same message.
The behavior behind that replyChannel option is explained in the Messaging Gateway chapter: https://docs.spring.io/spring-integration/reference/html/messaging-endpoints.html#gateway-default-reply-channel
You probably right about this section in the docs https://docs.spring.io/spring-integration/reference/html/redis.html#redis-queue-inbound-gateway and those requestChannel and replyChannel are really a copy of the text from the Outbound Gateway section. That has to be fixed. Feel free to raise a GH issue so we won't forget to address it.
The C logs are going to be printed when you send a message into that C channel, but again: if you want to make a reply correlation working for the Redis Inbound Gateway it has to be as a PublishSubscribeChannel. Otherwise just omit it and your String bTesting(Message message) { will send its result to the replyChannel header.
I am developing an API in spring-integration using DSL, this how it works
JDBC Polling Adapter initiates the flow and gets some data from tables and send it to DefaultRequestChannel, from here the message is handled/flowing thru various channels.
Now I am trying to
1. send a email, if any errors (e.g connectivity issue, bad record found while polling the data) occurred/detected in my error channel.
After sending email to my support group, I want to suspend my flow for 15 mins and then resume automatically.
I tried creating a sendEmailChannel (recipient of my errorChannel), it doesn't work for me. So just created a transformer method like below
this code is running fine, but is it a good practice?
#
#Transformer(inputChannel="errorChannel", outputChannel="suspendChannel")
public Message<?> errorChannelHandler(ErrorMessage errorMessage) throws RuntimeException, MessagingException, InterruptedException {
Exception exception = (Exception) errorMessage.getPayload();
String errorMsg = errorMessage.toString();
String subject = "API issue";
if (exception instanceof RuntimeException) {
errorMsg = "Run time exception";
subject = "Critical Alert";
}
if (exception instanceof JsonParseException) {
errorMsg = ....;
subject = .....;
}
MimeMessage message = sender.createMimeMessage();
MimeMessageHelper helper = new MimeMessageHelper(message);
helper.setFrom(senderEmail);
helper.setTo(receiverEmail);
helper.setText(errorMsg);
helper.setSubject(subject);
sender.send(message);
kafkaProducerSwitch.isKafkaDown());
return MessageBuilder.withPayload(exception.getMessage())
.build();
}
I am looking for some better way of handling the above logic.
And also any suggestions to suspend my flow for few mins.
You definitely can use a mail sending channel adapter from Spring Integration box to send those messages from the error channel: https://docs.spring.io/spring-integration/docs/5.1.5.RELEASE/reference/html/#mail-outbound. The Java DSL variant is like this:
.handle(Mail.outboundAdapter("gmail")
.port(smtpServer.getPort())
.credentials("user", "pw")
.protocol("smtp")))
The suspend can be done via CompoundTriggerAdvice extension, when you check the some AtimocBoolean bean for the state to activate one or another trigger in the beforeReceive() implementation. Such a AtimocBoolean can change its state in one more subscriber to that errorChannel because this one is a PublishSubscribeChannel by default. Don't forget to bring the state back to normal after that you return a false from the beforeReceive(). Just because that is enough to mark your system as normal at this moment since it is is going to work only after 15 mins.
I have a spring integration flow which produces messages that should be kept around waiting for an appropriate consumer to come along and consume them.
#Bean
public IntegrationFlow messagesPerCustomerFlow() {
return IntegrationFlows.
from(WebFlux.inboundChannelAdapter("/messages/{customer}")
.requestMapping(r -> r
.methods(HttpMethod.POST)
)
.requestPayloadType(JsonNode.class)
.headerExpression("customer", "#pathVariables.customer")
)
.channel(messagesPerCustomerQueue())
.get();
}
#Bean(name = PollerMetadata.DEFAULT_POLLER)
public PollerSpec poller() {
return Pollers.fixedRate(100);
}
#Bean
public QueueChannel messagesPerCustomerQueue() {
return MessageChannels.queue()
.get();
}
The messages in the queue should be delivered as server-sent events via http as shown below.
The PublisherSubscription is just a holder for the Publisher and the IntegrationFlowRegistration, the latter is used to destroy the dynamically created flow when it is no longer needed (note that the incoming message for the GET has no content, which is not handled properly ATM by the Webflux integration, hence a small workaround is necessary to get access to the path variable shoved to the customer header):
#Bean
public IntegrationFlow eventMessagesPerCustomer() {
return IntegrationFlows
.from(WebFlux.inboundGateway("/events/{customer}")
.requestMapping(m -> m.produces(TEXT_EVENT_STREAM_VALUE))
.headerExpression("customer", "#pathVariables.customer")
.payloadExpression("''") // neeeded to make handle((p,h) work
)
.log()
.handle((p, h) -> {
String customer = h.get("customer").toString();
PublisherSubscription<JsonNode> publisherSubscription =
subscribeToMessagesPerCustomer(customer);
return Flux.from(publisherSubscription.getPublisher())
.map(Message::getPayload)
.doFinally(signalType ->
publisherSubscription.unsubscribe());
})
.get();
}
The above request for server-sent events dynamically registers a flow which subscribes to the queue channel on demand with a selective consumer realized by a filter with throwExceptionOnRejection(true). Following the spec for Message Handler chain that should ensure that the message is offered to all consumers until one accepts it.
public PublisherSubscription<JsonNode> subscribeToMessagesPerCustomer(String customer) {
IntegrationFlowBuilder flow = IntegrationFlows.from(messagesPerCustomerQueue())
.filter("headers.customer=='" + customer + "'",
filterEndpointSpec -> filterEndpointSpec.throwExceptionOnRejection(true));
Publisher<Message<JsonNode>> messagePublisher = flow.toReactivePublisher();
IntegrationFlowRegistration registration = integrationFlowContext.registration(flow.get())
.register();
return new PublisherSubscription<>(messagePublisher, registration);
}
This construct works in principle, but with the following issues:
Messages sent to the queue while there are no subscribers at all lead to a MessageDeliveryException: Dispatcher has no subscribers for channel 'application.messagesPerCustomerQueue'
Messages sent to the queue while no matching subscriber is present yet lead to an AggregateMessageDeliveryException: All attempts to deliver Message to MessageHandlers failed.
What I want is that the message remains in the queue and is repeatedly offered to all subscribers until it is either consumed or expires (a proper selective consumer). How can I do that?
note that the incoming message for the GET has no content, which is not handled properly ATM by the Webflux integration
I don't understand this concern.
The WebFluxInboundEndpoint works with this algorithm:
if (isReadable(request)) {
...
else {
return (Mono<T>) Mono.just(exchange.getRequest().getQueryParams());
}
Where GET method really goes to the else branch. And the payload of the message to send is a MultiValueMap. And also we recently fixed with you the problem for the POST, which is released already as well in version 5.0.5: https://jira.spring.io/browse/INT-4462
Dispatcher has no subscribers
Can't happen on the QueueChannel in principle. There is no any dispatcher on there at all. It is just queue and sender offers message to be stored. You are missing something else to share with us. But let's call things with its own names: the messagesPerCustomerQueue is not a QueueChannel in your application.
UPDATE
Regarding:
What I want is that the message remains in the queue and is repeatedly offered to all subscribers until it is either consumed or expires (a proper selective consumer)
Only what we see is a PollableJmsChannel based on the embedded ActiveMQ to honor TTL for messages. As a consumer of this queue you should have a PublishSubscribeChannel with the setMinSubscribers(1) to make MessagingTemplate to throw a MessageDeliveryException when there is no subscribers yet. This way a JMS transaction will be rolled back and message will return to the queue for the next polling cycle.
The problem with in-memory QueueChannel that there is no transactional redelivery and message once polled from that queue is going to be lost.
Another option is similar to JMS (transactional) is a JdbcChannelMessageStore for the QueueChannel. Although this way we don't have a TTL functionality...
I would like to time an API request I have setup in Spring Integration using an . The response time should be added as a message header so it can be audited downstream and recorded along with other data.
Both HTTP success and error responses should be timed, e.g. I want to know how long it took for a server to issue a 500 response. I'm also already using the Retry and CircuitBreaker advice (as I want that functionality).
I've written a custom advice (from reading the docs) and added to my :
public class RequestResponseTimeInterceptor extends AbstractRequestHandlerAdvice {
private static final Logger LOG = Logger.getLogger(RequestResponseTimeInterceptor.class);
protected Object doInvoke(ExecutionCallback callback, Object target, Message<?> message) throws Exception {
long before = System.currentTimeMillis();
try {
Object result = callback.execute();
long after = System.currentTimeMillis();
message = MessageBuilder.fromMessage(message).setHeader("requestResponseTime", after - before).build();
return result;
}
catch (MessageHandlingException e) {
long after = System.currentTimeMillis();
Message modifiedFailedMessage = MessageBuilder.fromMessage(e.getFailedMessage()).setHeader("requestResponseTime", after - before).build();
throw new MessageHandlingException(modifiedFailedMessage, e.getMessage(), e.getCause());
}
}
<bean id="calculationRetry" class="org.springframework.integration.handler.advice.RequestHandlerRetryAdvice"/>
<bean id="calculationCircuitBreaker" class="org.springframework.integration.handler.advice.RequestHandlerCircuitBreakerAdvice"/>
<bean id="timer" class="com.integration.audit.RequestResponseTimeInterceptor"/>
<int-http:request-handler-advice-chain>
<ref bean="timer"/>
<ref bean="calculationRetry"/>
<ref bean="calculationCircuitBreaker"/>
</int-http:request-handler-advice-chain>
It works by timing between the callback execution. I had a problem that failed HTTP responses (e.g. HTTP 500) are wrapped in a MessageHandlingException (in Spring Integration, by design), which stops the rest of the advice from finishing and timing the response.
I have solved this in the code sample by catching the exception, adding the response time as a message header, re-creating the exception and throwing it (so it gets picked up by my error channel). This seems to work OK - but feel like it's not the best solution. It also times the full duration of any retries (from my Retry advice) rather than the last retry.
Is there a better way to time HTTP request/response that will work nicely with existing Retry advice, and with failed HTTP response codes?
Appreciate your feedback!
Edit: Based on Artem's information.
The order of the advice in request-handler-advice-chain is important (it is an ordered chain after all). So putting the 'timer' at the end of the chain means it only times request/responses, rather than multiple retries or circuit breakers.
I updated the advice based on Artem's comments so it is now actually returning a Message. I can then read the new "requestResponseTime" header downstream in the pipeline as needed.
Here's the new code:
public class RequestResponseTimeInterceptor extends AbstractRequestHandlerAdvice {
private static final Logger LOG = Logger.getLogger(RequestResponseTimeInterceptor.class);
protected Object doInvoke(ExecutionCallback callback, Object target, Message<?> requestMessage) throws Exception {
long before = System.currentTimeMillis();
try {
Object result = callback.execute();
long after = System.currentTimeMillis();
Message<?> responseMessage = (Message<?>) result;
return MessageBuilder.withPayload(responseMessage.getPayload())
.setHeader(MessageHeaderConstants.REQUEST_RESPONSE_TIME, after - before).build();
}
catch (MessageHandlingException e) {
//Catch HTTP errors (e.g. 500s) so we can also time the response
long after = System.currentTimeMillis();
Message modifiedFailedMessage = MessageBuilder.fromMessage(e.getFailedMessage()).setHeader(MessageHeaderConstants.REQUEST_RESPONSE_TIME, after - before).build();
//rethrow new exception for error handling
throw new MessageHandlingException(modifiedFailedMessage, e.getMessage(), e.getCause());
}
}
<int-http:request-handler-advice-chain>
<ref bean="calculationCircuitBreaker"/>
<ref bean="calculationRetry"/>
<ref bean="timer"/>
</int-http:request-handler-advice-chain>
First of all think if your timer really should include all the retries. And that doesn’t seem for me valuable to count timeout for opened circuit breaker. I mean that it would be much realist to reorder your advices different way: circuit first, retry and only then timer. This way you will count only real http requests and won’t go to that part with opened circuit.
Regarding your exception concern. That’s definitely is OK to rebuild message with required headers. For this purpose we have introduced a ErrorMessageStrategy: https://docs.spring.io/spring-integration/docs/latest-ga/reference/html/whats-new.html#_errormessagepublisher_and_errormessagestrategy
On the other hand, please, pay attention to the Spring Cloud Sleuth project: https://cloud.spring.io/spring-cloud-sleuth/
I'd like to be able to requeue a message from within my Service Endpoint that has been wired up through the RegisterHandler method of RabbitMQ Server. e.g.
mqServer.RegisterHandler<OutboundILeadPhone>(m =>
{
var db = container.Resolve<IFrontEndRepository>();
db.SaveMessage(m as Message);
return ServiceController.ExecuteMessage(m);
}, noOfThreads: 1);
or here.
public object Post(OutboundILeadPhone request)
{
throw new OutBoundAgentNotFoundException(); // added after mythz posted his first response
}
I don't see any examples how this is accomplished, so I'm starting to believe that it may not be possible with the ServiceStack abstraction. On the other hand, this looks promising.
Thank you, Stephen
Update
Throwing an exception in the Service does nak it, but then the message is sent to the OutboundILeadPhone.dlq which is normal ServiceStack behavior. Guess what I'm looking for is a way for the message to stay in the OutboundILeadPhone.inq queue.
Throwing an exception in your Service will automatically Nak the message. This default exception handling behavior can also be overridden with RabbitMqServer's RegisterHandler API that takes an Exception callback, i.e:
void RegisterHandler<T>(
Func<IMessage<T>, object> processMessageFn,
Action<IMessage<T>, Exception> processExceptionEx);
void RegisterHandler<T>(
Func<IMessage<T>, object> processMessageFn,
Action<IMessage<T>, Exception> processExceptionEx,
int noOfThreads)