I've the following flow:
#Resource(name = S3_CLIENT_BEAN)
private MessageSource<InputStream> messageSource;
public IntegrationFlow fileStreamingFlow() {
return IntegrationFlows.from(s3Properties.getFileStreamingInputChannel())
.enrichHeaders(spec -> spec.header(ERROR_CHANNEL, S3_ERROR_CHANNEL, true))
.handle(String.class, (fileName, h) -> {
if (messageSource instanceof S3StreamingMessageSource) {
S3StreamingMessageSource s3StreamingMessageSource = (S3StreamingMessageSource) messageSource;
ChainFileListFilter<S3ObjectSummary> chainFileListFilter = new ChainFileListFilter<>();
chainFileListFilter.addFilters(...);
s3StreamingMessageSource.setFilter(chainFileListFilter);
return s3StreamingMessageSource.receive();
}
return messageSource.receive();
}, spec -> spec
.requiresReply(false) // in case all messages got filtered out
)
.channel(s3Properties.getFileStreamingOutputChannel())
.get();
}
I found that if s3StreamingMessageSource.receive throws an exception, the error ends up in the error channel configured for the previous flow in the pipeline, not the S3_ERROR_CHANNEL that's configured for this flow. Not sure if it's related to this question.
The s3StreamingMessageSource.receive() is called from the SourcePollingChannelAdapter:
protected Message<?> receiveMessage() {
return this.source.receive();
}
This one called from the AbstractPollingEndpoint:
private boolean doPoll() {
message = this.receiveMessage();
...
this.handleMessage(message);
...
}
That handleMessage() does this:
this.messagingTemplate.send(getOutputChannel(), message);
So, that is definitely still far away from the mentioned .enrichHeaders(spec -> spec.header(ERROR_CHANNEL, S3_ERROR_CHANNEL, true)) downstream.
However you still can catch an exception in that S3_ERROR_CHANNEL. Pay attention to the second argument of the IntegrationFlows.from():
IntegrationFlows.from(s3Properties.getFileStreamingInputChannel(),
e -> e.poller(Pollers.fixedDelay(...)
.errorChannel(S3_ERROR_CHANNEL)))
Or, according your current you have somewhere a global poller, so configure an errorChannel there.
Related
We are facing a strange behavior in our integration flows where the errorChannel does not receive a message in case an exception is thrown in a step after an aggregation.
This is the (reduced) flow:
#Bean
public StandardIntegrationFlow startKafkaInbound() {
return IntegrationFlows.from(Kafka
.messageDrivenChannelAdapter(
kafkaConsumerFactory,
ListenerMode.record,
serviceProperties.getInputTopic().getName())
.errorChannel(errorHandler.getInputChannel())
)
.channel(nextChannel().getInputChannel())
.get();
}
#Bean
public IntegrationFlow nextChannel() {
return IntegrationFlows.from("next")
.transform(Transformers.fromJson(MyObject.class)) // An exception here is sent to errorChannel
.aggregate(aggregatorSpec ->
aggregatorSpec
.releaseStrategy(new MessageCountReleaseStrategy(100))
.sendPartialResultOnExpiry(true)
.groupTimeout(2000L)
.expireGroupsUponCompletion(true)
.correlationStrategy(message -> KafkaHeaderUtils.getOrDefault(message.getHeaders(), MY_CORRELATION_HEADER, ""))
)
.transform(myObjectTransformer) // Exception here is not sent to errorChannel
.channel(acknowledgeMyObjectFlow().getInputChannel())
.get();
}
If we add an explicit channel which is not of type DirectChannel the errorHandling is working as expected. Working code looks like:
// ...
.aggregate(aggregatorSpec -> ...)
.channel(MessageChannels.queue())
.transform(myObjectTransformer) // Now the exception is sent to errorChannel
.channel(acknowledgeMyObjectFlow().getInputChannel())
// ...
Also we'd like to mention, that we have a very similar flow with an aggregation where errorHandling works as expected (Exception sent to errorChannel)
So we were actually able to get the code running, but since errorHandling is a very critical part of the application we'd really like to understand how we can ensure each error will be sent to the configured channel and why explicitly setting a QueueChannel leads to the wanted behavior.
Thanks in advance
You can add this
.enrichHeaders(headers -> headers.header(MessageHeaders.ERROR_CHANNEL, (errorHandler.getInputChannel()))
before an aggregator.
The .channel(MessageChannels.queue()) is misleading over here because the error is sent to the global errorChannel, which is apparently is the same as yours errorHandler.getInputChannel().
The problem that .groupTimeout(2000L) is done on a separate TaskScheduler thread and when an error happens downstream there is no knowledge about try..catch in that Kafka.messageDrivenChannelAdapter.
Feel free to raise a GH issue, so we will think about populating that errorChannel into message headers from the MessageProducerSupport, like that Kafka.messageDrivenChannelAdapter. So, the error handling would be the same independently of the async nature of the downstream flow.
UPDATE
Please, try this as a solution:
.transform(Transformers.fromJson(MyDataObject.class)) // An exception here is sent to errorChannel
.enrichHeaders(headers -> headers.header(MessageHeaders.ERROR_CHANNEL, (errorHandler.getInputChannel())))
.aggregate(aggregatorSpec ->
The enrichHeaders() should do the trick to determine a proper error channel to send error.
Plus your MyDataObjectTransformer has to be modified to this:
throw new MessageTransformationException(source, "test");
The point is that there is a logic like this when exception is caught by the endpoint:
if (handler != null) {
try {
handler.handleMessage(message);
return true;
}
catch (Exception e) {
throw IntegrationUtils.wrapInDeliveryExceptionIfNecessary(message,
() -> "Dispatcher failed to deliver Message", e);
}
}
where:
if (!(ex instanceof MessagingException) ||
((MessagingException) ex).getFailedMessage() == null) {
runtimeException = new MessageDeliveryException(message, text.get(), ex);
}
And then in the AbstractCorrelatingMessageHandler:
catch (MessageDeliveryException ex) {
logger.warn(ex, () ->
"The MessageGroup [" + groupId +
"] is rescheduled by the reason of: ");
scheduleGroupToForceComplete(groupId);
}
That's how your exception does not reach the error channel.
You may consider to not use that MessageTransformationException. The logic in the wrapping handler is like this:
protected Object handleRequestMessage(Message<?> message) {
try {
return this.transformer.transform(message);
}
catch (Exception e) {
if (e instanceof MessageTransformationException) { // NOSONAR
throw (MessageTransformationException) e;
}
throw new MessageTransformationException(message, "Failed to transform Message in " + this, e);
}
}
UPDATE 2
OK. I see that you use Spring Boot and that one does not register a respective ErrorHandler to the TaskScheduler used in the aggregator for group timeout feature.
Please, consider to add this bean into your configuration:
#Bean
TaskSchedulerCustomizer taskSchedulerCustomizer(ErrorHandler integrationMessagePublishingErrorHandler) {
return taskScheduler -> taskScheduler.setErrorHandler(integrationMessagePublishingErrorHandler);
}
And then feel free to raise a GH issue for Spring Boot to make this customization as a default one in the auto-configuration.
If an exception occurs in a spring-integration webflux flow, the exception itself (with stacktrace) is sent back to the caller as payload through MessagePublishingErrorHandler, which uses an error channel from the "errorChannel" header, not the default error channel.
How can I set up an error handler similar to WebExceptionHandler? I want to produce an Http status code and possibly a DefaultErrorAttributes object as response.
Simply defining a flow that starts from the errorChannel doesn't work, the error message won't end up there. I tried to define my own fluxErrorChannel, but it appears that it is also not used as error channel, the errors do not end up in my errorFlow:
#Bean
public IntegrationFlow fooRestFlow() {
return IntegrationFlows.from(
WebFlux.inboundGateway("/foo")
.requestMapping(r -> r.methods(HttpMethod.POST))
.requestPayloadType(Map.class)
.errorChannel(fluxErrorChannel()))
.channel(bazFlow().getInputChannel())
.get();
}
#Bean
public MessageChannel fluxErrorChannel() {
return MessageChannels.flux().get();
}
#Bean
public IntegrationFlow errorFlow() {
return IntegrationFlows.from(fluxErrorChannel())
.transform(source -> source)
.enrichHeaders(h -> h.header(HttpHeaders.STATUS_CODE, HttpStatus.BAD_GATEWAY))
.get();
}
#Bean
public IntegrationFlow bazFlow() {
return f -> f.split(Map.class, map -> map.get("items"))
.channel(MessageChannels.flux())
.<String>handle((p, h) -> throw new RuntimeException())
.aggregate();
}
UPDATE
In MessagingGatewaySupport.doSendAndReceiveMessageReactive my error channel defined on the WebFlux.inboundGateway is never used to set the error channel, rather the error channel is always the replyChannel which is being created here:
FutureReplyChannel replyChannel = new FutureReplyChannel();
Message<?> requestMessage = MutableMessageBuilder.fromMessage(message)
.setReplyChannel(replyChannel)
.setHeader(this.messagingTemplate.getSendTimeoutHeader(), null)
.setHeader(this.messagingTemplate.getReceiveTimeoutHeader(), null)
.setErrorChannel(replyChannel)
.build();
The error channel is ultimately being reset to the originalErrorChannelHandler in Mono.fromFuture, but that error channel is ˋnullˋ in my case. Also, the onErrorResume lambda is never invoked:
return Mono.fromFuture(replyChannel.messageFuture)
.doOnSubscribe(s -> {
if (!error && this.countsEnabled) {
this.messageCount.incrementAndGet();
}
})
.<Message<?>>map(replyMessage ->
MessageBuilder.fromMessage(replyMessage)
.setHeader(MessageHeaders.REPLY_CHANNEL, originalReplyChannelHeader)
.setHeader(MessageHeaders.ERROR_CHANNEL, originalErrorChannelHeader)
.build())
.onErrorResume(t -> error ? Mono.error(t) : handleSendError(requestMessage, t));
How is this intended to work?
It's a bug; the ErrorMessage created for the exception by the error handler is sent to the errorChannel header (which has to be the replyChannel so the gateway gets the result). The gateway should then invoke the error flow (if present) and return the result of that.
https://jira.spring.io/browse/INT-4541
I'm having an issue with the SFTP outbound gateway using DSL.
I want to use a outbound gateway to send a file, then continue my flow.
The problem is that I have an exception telling me:
IllegalArgumentException: 'remoteDirectoryExpression' is required
I saw that I can use a RemoteFileTemplate where I can set the sftp session factory plus the remote directory information, but the directory I wan't is defined in my flow by the code put in the header just before the launch of the batch.
#Bean
public IntegrationFlow orderServiceFlow() {
return f -> f
.handleWithAdapter(h -> h.httpGateway("myUrl")
.httpMethod(HttpMethod.GET)
.expectedResponseType(List.class)
)
.split()
.channel(batchLaunchChannel());
}
#Bean
public DirectChannel batchLaunchChannel() {
return MessageChannels.direct("batchLaunchChannel").get();
}
#Bean
public IntegrationFlow batchLaunchFlow() {
return IntegrationFlows.from(batchLaunchChannel())
.enrichHeaders(h -> h
.headerExpression("oiCode", "payload")
)
.transform((GenericTransformer<String, JobLaunchRequest>) message -> {
JobParameters jobParameters = new JobParametersBuilder()
.addDate("exec_date", new Date())
.addString("oiCode", message)
.toJobParameters();
return new JobLaunchRequest(orderServiceJob, jobParameters);
})
.handle(new JobLaunchingMessageHandler(jobLauncher))
.enrichHeaders(h -> h
.headerExpression("jobExecution", "payload")
)
.handle((p, h) -> {
//Logic to retreive file...
return new File("");
})
.handle(Sftp.outboundGateway(sftpSessionFactory,
AbstractRemoteFileOutboundGateway.Command.PUT,
"payload")
)
.get();
}
I don't see how I can tell my outbound gateway which will be the directory depending what is in my header.
The Sftp.outboundGateway() has an overloaded version with the RemoteFileTemplate. So, you need to instantiate SftpRemoteFileTemplate bean and configure its:
/**
* Set the remote directory expression used to determine the remote directory to which
* files will be sent.
* #param remoteDirectoryExpression the remote directory expression.
*/
public void setRemoteDirectoryExpression(Expression remoteDirectoryExpression) {
This one can be like FunctionExpression:
setRemoteDirectoryExpression(m -> m.getHeaders().get("remoteDireHeader"))
Before I got an answer, I came with this solution but I'm not sure it's a good one.
// flow //
.handle((p, h) -> {
//Logic to retreive file...
return new File("");
})
.handle(
Sftp.outboundGateway(
remoteFileTemplate(new SpelExpressionParser().parseExpression("headers['oiCode']")),
AbstractRemoteFileOutboundGateway.Command.PUT,
"payload")
)
.handle(// next steps //)
.get();
public RemoteFileTemplate remoteFileTemplate(Expression directory) throws Exception {
RemoteFileTemplate template = new SftpRemoteFileTemplate(sftpSessionFactory);
template.setRemoteDirectoryExpression(directory);
template.setAutoCreateDirectory(true);
template.afterPropertiesSet();
return template;
}
But this provoke a warn because of exception thrown by the ExpresionUtils
java.lang.RuntimeException: No beanFactory
I'm working on a service which sends emails using the spring integration java dsl.
I have a batch message which is split into a collection of individual messages which will be turned into emails.
The issue I am experiencing is that if one of these individual messages throws an error, the other messages in the batch are not processed.
Is there a way to configure the flow so that when a message throws an exception, the exception is handled gracefully and the next message in the batch is processed?
The following code achieves the functionality I would like but I'm wondering if there is an easier / better way to achieve this, ideally in a single IntegrationFlow? :
#Bean
public MessageChannel individualFlowInputChannel() {
return MessageChannels.direct().get();
}
#Bean
public IntegrationFlow batchFlow() {
return f -> f
.split()
.handle(message -> {
try {
individualFlowInputChannel().send(message);
} catch (Exception e) {
e.printStackTrace();
}
});
}
#Bean
public IntegrationFlow individualFlow() {
return IntegrationFlows.from(individualFlowInputChannel())
.handle((payload, headers) -> {
throw new RuntimeException("BOOM!");
}).get();
}
You can add ExpressionEvaluatingRequestHandlerAdvice to the last handle() definition with its trapException option:
/**
* If true, any exception will be caught and null returned.
* Default false.
* #param trapException true to trap Exceptions.
*/
public void setTrapException(boolean trapException) {
On the other hand, if you are talking about "sends emails", wouldn't it be better to consider to do that in the separate thread for each splitted item? In this case the ExecutorChannel after .split() comes to the rescue!
I'm trying to test some stuff with spring-integration using the DSL. This is only a test so far, the flow is simple:
create some messages
process (log) them in parallel
aggregate them
log the aggregate
Apart from the aggregator, it is working fine:
#Bean
public IntegrationFlow integrationFlow() {
return IntegrationFlows
.from(integerMessageSource(), c -> c.poller(Pollers.fixedRate(1, TimeUnit.SECONDS)))
.channel(MessageChannels.executor(Executors.newCachedThreadPool()))
.handle((GenericHandler<Integer>) (payload, headers) -> {
System.out.println("\t delaying message:" + payload + " on thread "
+ Thread.currentThread().getName());
try {
Thread.sleep(2000);
} catch (InterruptedException e) {
System.err.println(e.getMessage());
}
return payload;
})
.handle(this::logMessage)
.aggregate(a ->
a.releaseStrategy(g -> g.size()>10)
.outputProcessor(g ->
g.getMessages()
.stream()
.map(e -> e.getPayload().toString())
.collect(Collectors.joining(",")))
)
.handle(this::logMessage)
.get();
}
If I leave out the .aggregate(..), part, the sample is working.
Withe the aggregator, I get the following exception:
Caused by: org.springframework.beans.factory.BeanCreationException: The 'currentComponent' (org.faboo.test.ParallelIntegrationApplication$$Lambda$9/1341404543#6fe1b4fb) is a one-way 'MessageHandler' and it isn't appropriate to configure 'outputChannel'. This is the end of the integration flow.
As far as I understand, it complains that there is no output from the aggregator?
The full source can be found here: hithub
The problem is the handle() before the aggregator - it produces no result so there is nothing to aggregate...
.handle(this::logMessage)
.aggregate(a ->
Presumably logMessage(Message<?>) has a void return type.
If you want to log before the aggregator, use a wireTap, or change logMessage to return the Message<?> after logging.
.wireTap(sf -> sf.handle(this::logMessage))