I'm looking to implement an integration flow where at some point a split/aggregate writes a bunch of files, returning true or false if an exception occurred in writing one of the files.
It works fine when all files are written without error, I get a list full of trues and can validate my business logic.
I've tried to use a ExpressionEvaluatingRequestHandlerAdvice to get a false when writing one the files failed (regardless of the details of the failure), to no avail:
#Bean
public IntegrationFlow mainFlow() {
return IntegrationFlows.from(...)
...
.split()
.<MyClass, Boolean>route(pp -> pp.getPattern == null,
mm -> mm.subFlowMapping(true, aSubflow()))
.subFlowMapping(false, anotherSubflow()))
// the following line works when I don't use an advice, but does not handle exceptions from above
//.handle((p, h) -> true)
.aggregate()
.<List<Boolean>>handle((p, h) -> p.stream().allMatch(b -> b))
...
.get();
}
public IntegrationFlow aSubflow() {
return f -> f.handle(Files.outboundGateway(...),
c -> c.advice(fileSystemAdvice()));
}
#Bean
public Advice advice() {
var advice = new ExpressionEvaluatingRequestHandlerAdvice();
advice.setSuccessChannelName("filesystemSuccess.input");
advice.setOnSuccessExpressionString("payload + ' was successful'");
advice.setFailureChannelName("filesystemFailure.input");
advice.setOnFailureExpressionString("payload + ' failed'");
advice.setTrapException(true);
return advice;
}
#Bean
public IntegrationFlow success() {
// logging works fine, but that's not what I want
//return f -> f.handle(System.out::println);
return f -> f.handle((p, h) -> true);
}
#Bean
public IntegrationFlow failure() {
// logging works fine
//return f -> f.handle(System.out::println);
return f -> f.handle((p, h) -> false);
}
With this configuration, I get the a DestinationResolutionException no output-channel or replyChannel header available.
What's the correct way to branch the outbound of the success/failure flows back to the split/aggregate flow (just before the aggregate)?
UPDATE:
I also tried to use a gateway as mentioned in the documentation (the splitRouteAggregate() example), but still get the same exception.
UPDATE #2
Here is a minimal failing example: no route, no subflow, just moving a file:
#SpringBootApplication
public class Application {
public static void main(String[] args) {
SpringApplication.run(Application.class, args);
}
#Bean
public IntegrationFlow myFlow() {
return IntegrationFlows
.from(Files.inboundAdapter(new File("./")).patternFilter("dummy.txt")
.autoCreateDirectory(true).recursive(false)
.useWatchService(true).watchEvents(FileReadingMessageSource.WatchEventType.CREATE))
.handle(Files.outboundGateway(new File("./out"))
.deleteSourceFiles(true)
.fileExistsMode(FileExistsMode.REPLACE)
.autoCreateDirectory(false),
c -> c.advice(advice())
)
.<Object, Boolean>transform(p -> !Boolean.FALSE.equals(p))
.log(LoggingHandler.Level.INFO, m -> m.getHeaders().getId() + ": " + m.getPayload())
.get();
}
#Bean
public Advice advice() {
var advice = new ExpressionEvaluatingRequestHandlerAdvice();
advice.setOnFailureExpressionString("false");
return advice;
}
}
Create the out directory at the root of the project, run the project, and then create a dummy.txt at the root of the project, it works fine (though I get two lines of log and not one as I expected).
2022-09-16 23:55:23.678 INFO 21564 --- [ scheduling-1] o.s.integration.handler.LoggingHandler : 1d253af9-625a-3745-edbb-0438224a9163: true
2022-09-16 23:55:23.700 INFO 21564 --- [ scheduling-1] o.s.integration.handler.LoggingHandler : 9d2486fc-f776-a5c7-fb21-f16b9ced3094: true
Now, create the out directory at the root of the project, run the project, rmdir the out directory, and then create a dummy.txt at the root of the project, to force an IllegalArgumentException when trying to write the file. I don't get a false payload, the workflow is interrupted by the exception.
2022-09-17 21:19:00.895 ERROR 14548 --- [ scheduling-1] o.s.integration.handler.LoggingHandler : org.springframework.messaging.MessageHandlingException: error occurred in message handler [bean 'myFlow.file:outbound-gateway#0' for component 'myFlow.org.springframework.integration.config.ConsumerEndpointFactoryBean#0'; defined in: 'integrationtest.Application'; from source: 'bean method myFlow']; nested exception is java.lang.IllegalArgumentException: Destination directory [.\out] does not exist., failedMessage=GenericMessage [payload=C:\Users\*****\Dev\ldd\so73745736\.\dummy.txt, headers={file_originalFile=C:\Users\*****\Dev\ldd\so73745736\.\dummy.txt, id=909a6ffe-6598-da35-33ff-90ebb9c68d67, file_name=dummy.txt, file_relativePath=dummy.txt, timestamp=1663442340858}]
at org.springframework.integration.support.utils.IntegrationUtils.wrapInHandlingExceptionIfNecessary(IntegrationUtils.java:191)
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:65)
at org.springframework.integration.dispatcher.AbstractDispatcher.tryOptimizedDispatch(AbstractDispatcher.java:115)
at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:133)
at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:106)
at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:72)
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:317)
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:272)
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:187)
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:166)
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:47)
at org.springframework.messaging.core.AbstractMessageSendingTemplate.send(AbstractMessageSendingTemplate.java:109)
at org.springframework.integration.endpoint.SourcePollingChannelAdapter.handleMessage(SourcePollingChannelAdapter.java:196)
at org.springframework.integration.endpoint.AbstractPollingEndpoint.messageReceived(AbstractPollingEndpoint.java:475)
at org.springframework.integration.endpoint.AbstractPollingEndpoint.doPoll(AbstractPollingEndpoint.java:461)
at org.springframework.integration.endpoint.AbstractPollingEndpoint.pollForMessage(AbstractPollingEndpoint.java:413)
at org.springframework.integration.endpoint.AbstractPollingEndpoint.lambda$createPoller$4(AbstractPollingEndpoint.java:348)
at org.springframework.integration.util.ErrorHandlingTaskExecutor.lambda$execute$0(ErrorHandlingTaskExecutor.java:57)
at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50)
at org.springframework.integration.util.ErrorHandlingTaskExecutor.execute(ErrorHandlingTaskExecutor.java:55)
at org.springframework.integration.endpoint.AbstractPollingEndpoint.lambda$createPoller$5(AbstractPollingEndpoint.java:341)
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:95)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.IllegalArgumentException: Destination directory [.\out] does not exist.
at org.springframework.util.Assert.isTrue(Assert.java:139)
at org.springframework.integration.file.FileWritingMessageHandler.validateDestinationDirectory(FileWritingMessageHandler.java:496)
at org.springframework.integration.file.FileWritingMessageHandler.evaluateDestinationDirectoryExpression(FileWritingMessageHandler.java:884)
at org.springframework.integration.file.FileWritingMessageHandler.handleRequestMessage(FileWritingMessageHandler.java:510)
at org.springframework.integration.handler.AbstractReplyProducingMessageHandler$AdvisedRequestHandler.handleRequestMessage(AbstractReplyProducingMessageHandler.java:208)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.integration.handler.advice.AbstractRequestHandlerAdvice$CallbackImpl.execute(AbstractRequestHandlerAdvice.java:151)
at org.springframework.integration.handler.advice.ExpressionEvaluatingRequestHandlerAdvice.doInvoke(ExpressionEvaluatingRequestHandlerAdvice.java:215)
at org.springframework.integration.handler.advice.AbstractRequestHandlerAdvice.invoke(AbstractRequestHandlerAdvice.java:67)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at jdk.proxy2/jdk.proxy2.$Proxy46.handleRequestMessage(Unknown Source)
at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.doInvokeAdvisedRequestHandler(AbstractReplyProducingMessageHandler.java:155)
at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:139)
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:56)
... 27 more
The logic of the ExpressionEvaluatingRequestHandlerAdvice is like this:
#Override
protected Object doInvoke(ExecutionCallback callback, Object target, Message<?> message) {
try {
Object result = callback.execute();
if (this.onSuccessExpression != null) {
evaluateSuccessExpression(message);
}
return result;
}
catch (RuntimeException e) {
Exception actualException = unwrapExceptionIfNecessary(e);
if (this.onFailureExpression != null) {
Object evalResult = evaluateFailureExpression(message, actualException);
if (this.returnFailureExpressionResult) {
return evalResult;
}
}
if (!this.trapException) {
if (e instanceof ThrowableHolderException) { // NOSONAR
throw (ThrowableHolderException) e;
}
else {
throw new ThrowableHolderException(actualException); // NOSONAR lost stack trace
}
}
return null;
}
}
So, if callback.execute() is OK, we try to call an onSuccessExpression (if any) and still return the result of the callback.execute(). Looks like in your case it is whatever an HTTP call replies. I don't think we need this onSuccessExpression to return just true. We can do that a bit later with a transform(). Stay tuned.
If callback.execute() fails, we call onFailureExpression, which in your case could be as simple as: setOnFailureExpressionString("false").
Therefore your advice should be configured like this:
#Bean
public Advice advice() {
var advice = new ExpressionEvaluatingRequestHandlerAdvice();
advice.setOnFailureExpressionString("false");
advice.setReturnFailureExpressionResult(true);
return advice;
}
Now, after that Http.outboundGateway() and before an aggregator, you have to add just this:
.<Object, Boolean>transform(p -> !Boolean.FALSE.equals(p))
Related
I want to upload all files from the local folder ~/sftp-outbound/Export
to a SFTP server. The folder contains two files:
foo1.TEST.txt
foo2.TEST.txt
I am doing it with the MPUT command of a SftpOutboundGateway within a Flow/Subflow DSL style (actually there are more gateway methods which I have removed for better focus and readability).
My configuration is this:
#Bean
public TransferChannel myChannel() {
LOG.debug("myChannel");
TransferChannel channel = new TransferChannel();
channel.setHost(myEnv.getSftpHost());
channel.setPort(myEnv.getSftpPort());
channel.setUser(myEnv.getSftpUser());
channel.setPrivateKey(myEnv.getSftpPrivateKey());
channel.setPassword(myEnv.getSftpPassword());
return channel;
}
#Bean
public TransferContext myContext(TransferChannel myChannel) {
LOG.debug("myContext");
TransferContext context = new TransferContext();
context.setEnabled(env.isEnabled());
context.setChannel(myChannel);
context.setPreserveTimestamp(true);
context.setLocalDir(env.getLocalDir());
context.setLocalFilename(env.getLocalFilename());
context.setRemoteDir(env.getRemoteDir());
return context;
}
#Bean
public SessionFactory<LsEntry> myFactory(TransferChannel myChannel) {
LOG.debug("myFactory");
DefaultSftpSessionFactory sf = new DefaultSftpSessionFactory();
sf.setHost(myChannel.getHost());
sf.setPort(myChannel.getPort());
sf.setUser(myChannel.getUser());
if (myChannel.getPrivateKey() != null) {
sf.setPrivateKey(myChannel.getPrivateKey());
} else {
sf.setPassword(myChannel.getPassword());
}
sf.setAllowUnknownKeys(true);
return new CachingSessionFactory<LsEntry>(sf);
}
#Bean
public IntegrationFlow myFlow(SessionFactory<LsEntry> myFactory, TransferContext myContext) {
LOG.debug("myFlow");
return IntegrationFlows.from(myGateway.class, g -> g
.header("method", args -> args.getMethod().getName()))
.log()
.route(Message.class, m -> m.getHeaders().get("method", String.class),
r -> r
.subFlowMapping("mput", f2 -> f2
.handle(Sftp.outboundGateway(
remoteFileTemplate(myFactory,
new SpelExpressionParser().parseExpression(
"headers['" + FileHeaders.REMOTE_DIRECTORY + "']")),
Command.MPUT, "payload"))
))
.get();
}
#Bean
public RemoteFileTemplate<LsEntry> remoteFileTemplate(SessionFactory<LsEntry> sessionFactory,
Expression directory) {
RemoteFileTemplate<LsEntry> template = new SftpRemoteFileTemplate(sessionFactory);
template.setRemoteDirectoryExpression(directory);
template.setAutoCreateDirectory(false);
template.afterPropertiesSet();
return template;
}
public interface MyGateway {
List<String> mput(String localDir,
#Header(FileHeaders.REMOTE_DIRECTORY) String remoteDirectory);
}
The call of the MyGateway method is:
#Autowired
private MyGateway gate;
...
String localFilename = "~/sftp-outbound/Export";
LOG.debug("runAsTask mput={}", localFilename);
jobLOG.info("put files to SFTP: {}", localFilename);
List<String> result = gate.mput(localFilename, env.getRemoteDir());
LOG.debug("runAsTask, files transferred, result={}", result);
It comes with the following LOG output:
2022-10-21 00:02:00.000 INFO [] --- [pool-125-thread-1] job-ExportData : put files to SFTP: ~/sftp-outbound/Export
2022-10-21 00:02:00.001 INFO [] --- [pool-125-thread-1] o.s.integration.handler.LoggingHandler : GenericMessage [payload=~/cdb/sftp-outbound/Export, headers={replyChannel=org.springframework.messaging.core.GenericMessagingTemplate$TemporaryReplyChannel#ce6c62d4, errorChannel=org.springframework.messaging.core.GenericMessagingTemplate$TemporaryReplyChannel#ce6c62d4, id=02f7972e-ef06-2297-b1dd-fed5cd75a2d2, method=mput, file_remoteDirectory=data/export, timestamp=1666303320001}]
2022-10-21 00:02:00.008 DEBUG [] --- [pool-125-thread-1] .l.c.c.j.ExportDataTask : runAsTask, files transferred, result=[data/export/02f7972e-ef06-2297-b1dd-fed5cd75a2d2.msg]
One sees two things:
there is only one file transferred instead of two
the remote filename is
02f7972e-ef06-2297-b1dd-fed5cd75a2d2.msg instead of
foo1.TEST.txt or foo2.TEST.txt
What am I doing wrong?
EDIT 1
Trying to find the problem I slightly changed the Integration Flow configuration, setting the remote directory inline to make the RemoteFileTemplate bean obsolete:
.subFlowMapping("mput", f2 -> f2
.handle(Sftp.outboundGateway(myFactory, Command.MPUT, "payload")
.autoCreateDirectory(false)
.remoteDirectoryExpression(myContext.getRemoteDir()))
Following your answer, Artem, I changed the SftpOutboundGateway method's parameter to type java.io.File as follows:
public interface MyGateway {
List<String> mput(File localDir);
}
The mput gateway method call is now:
List<String> result = gate.mput(new File(localFilename));
But this did not resolve my problem. Now I get the following error:
.subFlowMapping("mput", f2 -> f2
.handle(Sftp.outboundGateway(myFactory, Command.MPUT, "payload")
.autoCreateDirectory(false)
.remoteDirectoryExpression(myContext.getRemoteDir()))
2022-10-22 00:11:30.321 ERROR [] --- [pool-18-thread-1] .l.c.c.j.MyExportTask : MyExports failed! Exception: {}
org.springframework.expression.spel.SpelEvaluationException: EL1008E: Property or field 'data' cannot be found on object of type 'org.springframework.integration.support.MutableMessage' - maybe not public or not valid?
at org.springframework.expression.spel.ast.PropertyOrFieldReference.readProperty(PropertyOrFieldReference.java:217)
at org.springframework.expression.spel.ast.PropertyOrFieldReference.getValueInternal(PropertyOrFieldReference.java:104)
at org.springframework.expression.spel.ast.PropertyOrFieldReference.getValueInternal(PropertyOrFieldReference.java:91)
at org.springframework.expression.spel.ast.OpDivide.getValueInternal(OpDivide.java:49)
at org.springframework.expression.spel.ast.OpMinus.getValueInternal(OpMinus.java:98)
at org.springframework.expression.spel.ast.SpelNodeImpl.getTypedValue(SpelNodeImpl.java:117)
at org.springframework.expression.spel.standard.SpelExpression.getValue(SpelExpression.java:375)
at org.springframework.integration.util.AbstractExpressionEvaluator.evaluateExpression(AbstractExpressionEvaluator.java:171)
at org.springframework.integration.util.AbstractExpressionEvaluator.evaluateExpression(AbstractExpressionEvaluator.java:129)
at org.springframework.integration.handler.ExpressionEvaluatingMessageProcessor.processMessage(ExpressionEvaluatingMessageProcessor.java:107)
at org.springframework.integration.file.remote.RemoteFileTemplate.doSend(RemoteFileTemplate.java:325)
at org.springframework.integration.file.remote.RemoteFileTemplate.lambda$send$0(RemoteFileTemplate.java:301)
at org.springframework.integration.file.remote.RemoteFileTemplate.execute(RemoteFileTemplate.java:439)
at org.springframework.integration.file.remote.RemoteFileTemplate.send(RemoteFileTemplate.java:301)
at org.springframework.integration.file.remote.RemoteFileTemplate.send(RemoteFileTemplate.java:289)
at org.springframework.integration.file.remote.gateway.AbstractRemoteFileOutboundGateway.put(AbstractRemoteFileOutboundGateway.java:807)
at org.springframework.integration.file.remote.gateway.AbstractRemoteFileOutboundGateway.lambda$doPut$11(AbstractRemoteFileOutboundGateway.java:793)
at org.springframework.integration.file.remote.RemoteFileTemplate.invoke(RemoteFileTemplate.java:471)
at org.springframework.integration.file.remote.gateway.AbstractRemoteFileOutboundGateway.doPut(AbstractRemoteFileOutboundGateway.java:792)
at org.springframework.integration.file.remote.RemoteFileTemplate.send(RemoteFileTemplate.java:301)
at org.springframework.integration.file.remote.RemoteFileTemplate.send(RemoteFileTemplate.java:289)
at org.springframework.integration.file.remote.gateway.AbstractRemoteFileOutboundGateway.put(AbstractRemoteFileOutboundGateway.java:807)
at org.springframework.integration.file.remote.gateway.AbstractRemoteFileOutboundGateway.lambda$doPut$11(AbstractRemoteFileOutboundGateway.java:793)
at org.springframework.integration.file.remote.RemoteFileTemplate.invoke(RemoteFileTemplate.java:471)
at org.springframework.integration.file.remote.gateway.AbstractRemoteFileOutboundGateway.doMput(AbstractRemoteFileOutboundGateway.java:854)
at org.springframework.integration.file.remote.gateway.AbstractRemoteFileOutboundGateway.handleRequestMessage(AbstractRemoteFileOutboundGateway.java:594)
at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:134)
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:62)
at org.springframework.integration.dispatcher.AbstractDispatcher.tryOptimizedDispatch(AbstractDispatcher.java:115)
at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:133)
at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:106)
at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:72)
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:570)
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:520)
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:187)
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:166)
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:47)
at org.springframework.messaging.core.AbstractMessageSendingTemplate.send(AbstractMessageSendingTemplate.java:109)
at org.springframework.integration.router.AbstractMessageRouter.doSend(AbstractMessageRouter.java:213)
at org.springframework.integration.router.AbstractMessageRouter.handleMessageInternal(AbstractMessageRouter.java:195)
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:62)
at org.springframework.integration.dispatcher.AbstractDispatcher.tryOptimizedDispatch(AbstractDispatcher.java:115)
at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:133)
at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:106)
at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:72)
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:570)
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:520)
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:187)
at org.springframework.messaging.core.GenericMessagingTemplate.doSendAndReceive(GenericMessagingTemplate.java:233)
at org.springframework.messaging.core.GenericMessagingTemplate.doSendAndReceive(GenericMessagingTemplate.java:47)
at org.springframework.messaging.core.AbstractMessagingTemplate.sendAndReceive(AbstractMessagingTemplate.java:46)
at org.springframework.integration.core.MessagingTemplate.sendAndReceive(MessagingTemplate.java:97)
at org.springframework.integration.core.MessagingTemplate.sendAndReceive(MessagingTemplate.java:38)
at org.springframework.messaging.core.AbstractMessagingTemplate.convertSendAndReceive(AbstractMessagingTemplate.java:96)
at org.springframework.messaging.core.AbstractMessagingTemplate.convertSendAndReceive(AbstractMessagingTemplate.java:86)
at org.springframework.integration.gateway.MessagingGatewaySupport.doSendAndReceive(MessagingGatewaySupport.java:514)
at org.springframework.integration.gateway.MessagingGatewaySupport.sendAndReceive(MessagingGatewaySupport.java:488)
at org.springframework.integration.gateway.GatewayProxyFactoryBean.sendOrSendAndReceive(GatewayProxyFactoryBean.java:648)
at org.springframework.integration.gateway.GatewayProxyFactoryBean.invokeGatewayMethod(GatewayProxyFactoryBean.java:573)
at org.springframework.integration.gateway.GatewayProxyFactoryBean.doInvoke(GatewayProxyFactoryBean.java:540)
at org.springframework.integration.gateway.GatewayProxyFactoryBean.doInvoke(GatewayProxyFactoryBean.java:540)
at org.springframework.integration.gateway.GatewayProxyFactoryBean.invoke(GatewayProxyFactoryBean.java:529)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at com.sun.proxy.$Proxy1232.mput(Unknown Source)
at com.lhsystems.cdb.cdbjob.job.MyExportTask.runAsTask(MyExportTask.java:73)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:84)
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:93)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:840)
I have no clue what causes that exception. The changed SubflowMapping even does not use a SPEL expression anymore. Any help is highly welcome!
So, you send a String localFilename to that gateway.
The logic there fails to this then:
else if (payload instanceof String) {
file = new File((String) payload);
}
And according to your observation and execution result we end up here:
else if (!file.isDirectory()) {
return doPut(requestMessage);
}
Some way it does not recognize the file against your ~/sftp-outbound/Export as a dir and just performs a plain PUT with a single file.
Try to resolve your ~/sftp-outbound/Export as a File object and send already that one to this gateway.
I've the following flow:
#Resource(name = S3_CLIENT_BEAN)
private MessageSource<InputStream> messageSource;
public IntegrationFlow fileStreamingFlow() {
return IntegrationFlows.from(s3Properties.getFileStreamingInputChannel())
.enrichHeaders(spec -> spec.header(ERROR_CHANNEL, S3_ERROR_CHANNEL, true))
.handle(String.class, (fileName, h) -> {
if (messageSource instanceof S3StreamingMessageSource) {
S3StreamingMessageSource s3StreamingMessageSource = (S3StreamingMessageSource) messageSource;
ChainFileListFilter<S3ObjectSummary> chainFileListFilter = new ChainFileListFilter<>();
chainFileListFilter.addFilters(...);
s3StreamingMessageSource.setFilter(chainFileListFilter);
return s3StreamingMessageSource.receive();
}
return messageSource.receive();
}, spec -> spec
.requiresReply(false) // in case all messages got filtered out
)
.channel(s3Properties.getFileStreamingOutputChannel())
.get();
}
I found that if s3StreamingMessageSource.receive throws an exception, the error ends up in the error channel configured for the previous flow in the pipeline, not the S3_ERROR_CHANNEL that's configured for this flow. Not sure if it's related to this question.
The s3StreamingMessageSource.receive() is called from the SourcePollingChannelAdapter:
protected Message<?> receiveMessage() {
return this.source.receive();
}
This one called from the AbstractPollingEndpoint:
private boolean doPoll() {
message = this.receiveMessage();
...
this.handleMessage(message);
...
}
That handleMessage() does this:
this.messagingTemplate.send(getOutputChannel(), message);
So, that is definitely still far away from the mentioned .enrichHeaders(spec -> spec.header(ERROR_CHANNEL, S3_ERROR_CHANNEL, true)) downstream.
However you still can catch an exception in that S3_ERROR_CHANNEL. Pay attention to the second argument of the IntegrationFlows.from():
IntegrationFlows.from(s3Properties.getFileStreamingInputChannel(),
e -> e.poller(Pollers.fixedDelay(...)
.errorChannel(S3_ERROR_CHANNEL)))
Or, according your current you have somewhere a global poller, so configure an errorChannel there.
I'm having an issue with the SFTP outbound gateway using DSL.
I want to use a outbound gateway to send a file, then continue my flow.
The problem is that I have an exception telling me:
IllegalArgumentException: 'remoteDirectoryExpression' is required
I saw that I can use a RemoteFileTemplate where I can set the sftp session factory plus the remote directory information, but the directory I wan't is defined in my flow by the code put in the header just before the launch of the batch.
#Bean
public IntegrationFlow orderServiceFlow() {
return f -> f
.handleWithAdapter(h -> h.httpGateway("myUrl")
.httpMethod(HttpMethod.GET)
.expectedResponseType(List.class)
)
.split()
.channel(batchLaunchChannel());
}
#Bean
public DirectChannel batchLaunchChannel() {
return MessageChannels.direct("batchLaunchChannel").get();
}
#Bean
public IntegrationFlow batchLaunchFlow() {
return IntegrationFlows.from(batchLaunchChannel())
.enrichHeaders(h -> h
.headerExpression("oiCode", "payload")
)
.transform((GenericTransformer<String, JobLaunchRequest>) message -> {
JobParameters jobParameters = new JobParametersBuilder()
.addDate("exec_date", new Date())
.addString("oiCode", message)
.toJobParameters();
return new JobLaunchRequest(orderServiceJob, jobParameters);
})
.handle(new JobLaunchingMessageHandler(jobLauncher))
.enrichHeaders(h -> h
.headerExpression("jobExecution", "payload")
)
.handle((p, h) -> {
//Logic to retreive file...
return new File("");
})
.handle(Sftp.outboundGateway(sftpSessionFactory,
AbstractRemoteFileOutboundGateway.Command.PUT,
"payload")
)
.get();
}
I don't see how I can tell my outbound gateway which will be the directory depending what is in my header.
The Sftp.outboundGateway() has an overloaded version with the RemoteFileTemplate. So, you need to instantiate SftpRemoteFileTemplate bean and configure its:
/**
* Set the remote directory expression used to determine the remote directory to which
* files will be sent.
* #param remoteDirectoryExpression the remote directory expression.
*/
public void setRemoteDirectoryExpression(Expression remoteDirectoryExpression) {
This one can be like FunctionExpression:
setRemoteDirectoryExpression(m -> m.getHeaders().get("remoteDireHeader"))
Before I got an answer, I came with this solution but I'm not sure it's a good one.
// flow //
.handle((p, h) -> {
//Logic to retreive file...
return new File("");
})
.handle(
Sftp.outboundGateway(
remoteFileTemplate(new SpelExpressionParser().parseExpression("headers['oiCode']")),
AbstractRemoteFileOutboundGateway.Command.PUT,
"payload")
)
.handle(// next steps //)
.get();
public RemoteFileTemplate remoteFileTemplate(Expression directory) throws Exception {
RemoteFileTemplate template = new SftpRemoteFileTemplate(sftpSessionFactory);
template.setRemoteDirectoryExpression(directory);
template.setAutoCreateDirectory(true);
template.afterPropertiesSet();
return template;
}
But this provoke a warn because of exception thrown by the ExpresionUtils
java.lang.RuntimeException: No beanFactory
What do I need to do to prevent the following Exception which is presumably thrown by RabbitMQ.
org.springframework.amqp.rabbit.listener.exception.ListenerExecutionFailedException: Listener threw exception
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.wrapToListenerExecutionFailedExceptionIfNeeded(AbstractMessageListenerContainer.java:877)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:787)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:707)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$001(SimpleMessageListenerContainer.java:98)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$1.invokeListener(SimpleMessageListenerContainer.java:189)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.invokeListener(SimpleMessageListenerContainer.java:1236)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListener(AbstractMessageListenerContainer.java:688)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.doReceiveAndExecute(SimpleMessageListenerContainer.java:1190)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.receiveAndExecute(SimpleMessageListenerContainer.java:1174)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$1200(SimpleMessageListenerContainer.java:98)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1363)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.springframework.messaging.MessageDeliveryException: failed to send Message to channel 'amqpLaunchSpringBatchJobFlow.channel#0'; nested exception is jp.ixam_drive.batch.service.JobExecutionRuntimeException: Failed to start job with name ads-insights-import and parameters {accessToken=<ACCESS_TOKEN>, id=act_1234567890, classifier=stats, report_run_id=1482330625184792, job_request_id=32}
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:449)
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:373)
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:115)
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:45)
at org.springframework.messaging.core.AbstractMessageSendingTemplate.send(AbstractMessageSendingTemplate.java:105)
at org.springframework.integration.endpoint.MessageProducerSupport.sendMessage(MessageProducerSupport.java:171)
at org.springframework.integration.amqp.inbound.AmqpInboundChannelAdapter.access$400(AmqpInboundChannelAdapter.java:45)
at org.springframework.integration.amqp.inbound.AmqpInboundChannelAdapter$1.onMessage(AmqpInboundChannelAdapter.java:95)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:784)
... 10 common frames omitted
Caused by: jp.ixam_drive.batch.service.JobExecutionRuntimeException: Failed to start job with name ads-insights-import and parameters {accessToken=<ACCESS_TOKEN>, id=act_1234567890, classifier=stats, report_run_id=1482330625184792, job_request_id=32}
at jp.ixam_drive.facebook.SpringBatchLauncher.launchJob(SpringBatchLauncher.java:42)
at jp.ixam_drive.facebook.AmqpBatchLaunchIntegrationFlows.lambda$amqpLaunchSpringBatchJobFlow$1(AmqpBatchLaunchIntegrationFlows.java:71)
at org.springframework.integration.dispatcher.AbstractDispatcher.tryOptimizedDispatch(AbstractDispatcher.java:116)
at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:148)
at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:121)
at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:89)
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:423)
... 18 common frames omitted
Caused by: org.springframework.batch.core.repository.JobInstanceAlreadyCompleteException: A job instance already exists and is complete for parameters={accessToken=<ACCESS_TOKEN>, id=act_1234567890, classifier=stats, report_run_id=1482330625184792, job_request_id=32}. If you want to run this job again, change the parameters.
at org.springframework.batch.core.repository.support.SimpleJobRepository.createJobExecution(SimpleJobRepository.java:126)
at sun.reflect.GeneratedMethodAccessor193.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.batch.core.repository.support.AbstractJobRepositoryFactoryBean$1.invoke(AbstractJobRepositoryFactoryBean.java:172)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
at com.sun.proxy.$Proxy125.createJobExecution(Unknown Source)
at org.springframework.batch.core.launch.support.SimpleJobLauncher.run(SimpleJobLauncher.java:125)
at jp.ixam_drive.batch.service.JobOperationsService.launch(JobOperationsService.java:64)
at jp.ixam_drive.facebook.SpringBatchLauncher.launchJob(SpringBatchLauncher.java:37)
... 24 common frames omitted
when I have 2 instances of Spring Boot application both of which run the following code in parallel to execute Spring Batch Jobs?
#Configuration
#Conditional(AmqpBatchLaunchCondition.class)
#Slf4j
public class AmqpAsyncAdsInsightsConfiguration {
#Autowired
ObjectMapper objectMapper;
#Value("${batch.launch.amqp.routing-keys.async-insights}")
String routingKey;
#Bean
public IntegrationFlow amqpOutboundAsyncAdsInsights(AmqpTemplate amqpTemplate) {
return IntegrationFlows.from("async_ads_insights")
.<JobParameters, byte[]>transform(SerializationUtils::serialize)
.handle(Amqp.outboundAdapter(amqpTemplate).routingKey(routingKey)).get();
}
#Bean
public IntegrationFlow amqpAdsInsightsAsyncJobRequestFlow(FacebookMarketingServiceProvider serviceProvider,
JobParametersToApiParametersTransformer transformer, ConnectionFactory connectionFactory) {
return IntegrationFlows.from(Amqp.inboundAdapter(connectionFactory, routingKey))
.<byte[], JobParameters>transform(SerializationUtils::deserialize)
.<JobParameters, ApiParameters>transform(transformer)
.<ApiParameters>handle((payload, header) -> {
String accessToken = (String) header.get("accessToken");
String id = (String) header.get("object_id");
FacebookMarketingApi api = serviceProvider.getApi(accessToken);
String reportRunId = api.asyncRequestOperations().getReportRunId(id, payload.toMap());
ObjectNode objectNode = objectMapper.createObjectNode();
objectNode.put("accessToken", accessToken);
objectNode.put("id", id);
objectNode.put("report_run_id", reportRunId);
objectNode.put("classifier", (String) header.get("classifier"));
objectNode.put("job_request_id", (Long) header.get("job_request_id"));
return serialize(objectNode);
}).channel("ad_report_run_polling_channel").get();
}
#SneakyThrows
private String serialize(JsonNode jsonNode) {
return objectMapper.writeValueAsString(jsonNode);
}
}
#Configuration
#Conditional(AmqpBatchLaunchCondition.class)
#Slf4j
public class AmqpBatchLaunchIntegrationFlows {
#Autowired
SpringBatchLauncher batchLauncher;
#Value("${batch.launch.amqp.routing-keys.job-launch}")
String routingKey;
#Bean(name = "batch_launch_channel")
public MessageChannel batchLaunchChannel() {
return MessageChannels.executor(Executors.newSingleThreadExecutor()).get();
}
#Bean
public IntegrationFlow amqpOutbound(AmqpTemplate amqpTemplate,
#Qualifier("batch_launch_channel") MessageChannel batchLaunchChannel) {
return IntegrationFlows.from(batchLaunchChannel)
.<JobParameters, byte[]>transform(SerializationUtils::serialize)
.handle(Amqp.outboundAdapter(amqpTemplate).routingKey(routingKey)).get();
}
#Bean
public IntegrationFlow amqpLaunchSpringBatchJobFlow(ConnectionFactory connectionFactory) {
return IntegrationFlows.from(Amqp.inboundAdapter(connectionFactory, routingKey))
.handle(message -> {
String jobName = (String) message.getHeaders().get("job_name");
byte[] bytes = (byte[]) message.getPayload();
JobParameters jobParameters = SerializationUtils.deserialize(bytes);
batchLauncher.launchJob(jobName, jobParameters);
}).get();
}
}
#Configuration
#Slf4j
public class AsyncAdsInsightsConfiguration {
#Value("${batch.core.pool.size}")
public Integer batchCorePoolSize;
#Value("${ixam_drive.facebook.api.ads-insights.async-poll-interval}")
public String asyncPollInterval;
#Autowired
ObjectMapper objectMapper;
#Autowired
private DataSource dataSource;
#Bean(name = "async_ads_insights")
public MessageChannel adsInsightsAsyncJobRequestChannel() {
return MessageChannels.direct().get();
}
#Bean(name = "ad_report_run_polling_channel")
public MessageChannel adReportRunPollingChannel() {
return MessageChannels.executor(Executors.newFixedThreadPool(batchCorePoolSize)).get();
}
#Bean
public IntegrationFlow adReportRunPollingLoopFlow(FacebookMarketingServiceProvider serviceProvider) {
return IntegrationFlows.from(adReportRunPollingChannel())
.<String>handle((payload, header) -> {
ObjectNode jsonNode = deserialize(payload);
String accessToken = jsonNode.get("accessToken").asText();
String reportRunId = jsonNode.get("report_run_id").asText();
try {
AdReportRun adReportRun = serviceProvider.getApi(accessToken)
.fetchObject(reportRunId, AdReportRun.class);
log.debug("ad_report_run: {}", adReportRun);
return jsonNode.set("ad_report_run", objectMapper.valueToTree(adReportRun));
} catch (Exception e) {
log.error("failed while polling for ad_report_run.id: {}", reportRunId);
throw new RuntimeException(e);
}
}).<JsonNode, Boolean>route(payload -> {
JsonNode adReportRun = payload.get("ad_report_run");
return adReportRun.get("async_percent_completion").asInt() == 100 &&
"Job Completed".equals(adReportRun.get("async_status").asText());
}, rs -> rs.subFlowMapping(true,
f -> f.transform(JsonNode.class,
source -> {
JobParametersBuilder jobParametersBuilder = new JobParametersBuilder();
jobParametersBuilder
.addString("accessToken", source.get("accessToken").asText());
jobParametersBuilder.addString("id", source.get("id").asText());
jobParametersBuilder
.addString("classifier", source.get("classifier").asText());
jobParametersBuilder
.addLong("report_run_id", source.get("report_run_id").asLong());
jobParametersBuilder
.addLong("job_request_id", source.get("job_request_id").asLong());
return jobParametersBuilder.toJobParameters();
}).channel("batch_launch_channel"))
.subFlowMapping(false,
f -> f.transform(JsonNode.class, this::serialize)
.<String>delay("delay", asyncPollInterval, c -> c.transactional()
.messageStore(jdbcMessageStore()))
.channel(adReportRunPollingChannel()))).get();
}
#SneakyThrows
private String serialize(JsonNode jsonNode) {
return objectMapper.writeValueAsString(jsonNode);
}
#SneakyThrows
private ObjectNode deserialize(String payload) {
return objectMapper.readerFor(ObjectNode.class).readValue(payload);
}
#Bean
public JdbcMessageStore jdbcMessageStore() {
JdbcMessageStore jdbcMessageStore = new JdbcMessageStore(dataSource);
return jdbcMessageStore;
}
#Bean
public JobParametersToApiParametersTransformer jobParametersToApiParametersTransformer() {
return new JobParametersToApiParametersTransformer() {
#Override
protected ApiParameters transform(JobParameters jobParameters) {
ApiParameters.ApiParametersBuilder builder = ApiParameters.builder();
MultiValueMap<String, String> multiValueMap = new LinkedMultiValueMap<>();
String level = jobParameters.getString("level");
if (!StringUtils.isEmpty(level)) {
multiValueMap.set("level", level);
}
String fields = jobParameters.getString("fields");
if (!StringUtils.isEmpty(fields)) {
multiValueMap.set("fields", fields);
}
String filter = jobParameters.getString("filter");
if (filter != null) {
try {
JsonNode jsonNode = objectMapper.readTree(filter);
if (jsonNode != null && jsonNode.isArray()) {
List<ApiFilteringParameters> filteringParametersList = new ArrayList<>();
List<ApiSingleValueFilteringParameters> singleValueFilteringParameters = new ArrayList<>();
ArrayNode arrayNode = (ArrayNode) jsonNode;
arrayNode.forEach(node -> {
String field = node.get("field").asText();
String operator = node.get("operator").asText();
if (!StringUtils.isEmpty(field) && !StringUtils.isEmpty(operator)) {
String values = node.get("values").asText();
String[] valuesArray = !StringUtils.isEmpty(values) ? values.split(",") : null;
if (valuesArray != null) {
if (valuesArray.length > 1) {
filteringParametersList.add(ApiFilteringParameters
.of(field, Operator.valueOf(operator), valuesArray));
} else {
singleValueFilteringParameters.add(ApiSingleValueFilteringParameters
.of(field, Operator.valueOf(operator), valuesArray[0]));
}
}
}
});
if (!filteringParametersList.isEmpty()) {
builder.filterings(filteringParametersList);
}
if (!singleValueFilteringParameters.isEmpty()) {
builder.filterings(singleValueFilteringParameters);
}
}
} catch (IOException e) {
throw new UncheckedIOException(e);
}
}
String start = jobParameters.getString("time_ranges.start");
String end = jobParameters.getString("time_ranges.end");
String since = jobParameters.getString("time_range.since");
String until = jobParameters.getString("time_range.until");
if (!StringUtils.isEmpty(start) && !StringUtils.isEmpty(end)) {
builder.timeRanges(ApiParameters.timeRanges(start, end));
} else if (!StringUtils.isEmpty(since) && !StringUtils.isEmpty(until)) {
builder.timeRange(new TimeRange(since, until));
}
String actionBreakdowns = jobParameters.getString("action_breakdowns");
if (!StringUtils.isEmpty(actionBreakdowns)) {
multiValueMap.set("action_breakdowns", actionBreakdowns);
}
String attributionWindows = jobParameters.getString("action_attribution_windows");
if (attributionWindows != null) {
try {
multiValueMap
.set("action_attribution_windows",
objectMapper.writeValueAsString(attributionWindows.split(",")));
} catch (JsonProcessingException e) {
e.printStackTrace();
}
}
builder.multiValueMap(multiValueMap);
String pageSize = jobParameters.getString("pageSize");
if (!StringUtils.isEmpty(pageSize)) {
builder.limit(pageSize);
}
return builder.build();
}
};
}
}
Here is how message flows:
1. channel[async_ads_insights] ->IntegrationFlow[amqpOutboundAsyncAdsInsights]->[AMQP]->IntegrationFlow[amqpAdsInsightsAsyncJobRequestFlow]->channel[ad_report_run_polling_channel]->IntegrationFlow[adReportRunPollingLoopFlow]-IF END LOOP->channel[batch_launch_channel] ELSE -> channel[ad_report_run_polling_channel]
2. channel[batch_launch_channel] -> IntegrationFlow[amqpOutbound]-> IntegrationFlow[amqpLaunchSpringBatchJobFlow]
3. Spring Batch Job is launched.
The exception isn't thrown immediately after both instances are started, but after a while. Launching Spring Batch Jobs do succeeds but then start to fail with "A job instance already exists and is complete for..."
The job is for retrieving facebook ads results.
I would appreciate your insights into what is causing the error above.
I also have this configuration which does not use AMQP and works without any problem, but it is only for one instance.
#Configuration
#Conditional(SimpleBatchLaunchCondition.class)
#Slf4j
public class SimpleBatchLaunchIntegrationFlows {
#Autowired
SpringBatchLauncher batchLauncher;
#Autowired
DataSource dataSource;
#Bean(name = "batch_launch_channel")
public MessageChannel batchLaunchChannel() {
return MessageChannels.queue(jdbcChannelMessageStore(), "batch_launch_channel").get();
}
#Bean
public ChannelMessageStoreQueryProvider channelMessageStoreQueryProvider() {
return new MySqlChannelMessageStoreQueryProvider();
}
#Bean
public JdbcChannelMessageStore jdbcChannelMessageStore() {
JdbcChannelMessageStore channelMessageStore = new JdbcChannelMessageStore(dataSource);
channelMessageStore.setChannelMessageStoreQueryProvider(channelMessageStoreQueryProvider());
channelMessageStore.setUsingIdCache(true);
channelMessageStore.setPriorityEnabled(true);
return channelMessageStore;
}
#Bean
public IntegrationFlow launchSpringBatchJobFlow(#Qualifier("batch_launch_channel")
MessageChannel batchLaunchChannel) {
return IntegrationFlows.from(batchLaunchChannel)
.handle(message -> {
String jobName = (String) message.getHeaders().get("job_name");
JobParameters jobParameters = (JobParameters) message.getPayload();
batchLauncher.launchJob(jobName, jobParameters);
}, e->e.poller(Pollers.fixedRate(500).receiveTimeout(500))).get();
}
}
Refer to the Spring Batch documentation. When launching a new instance of a job, the job parameters must be unique.
A common solution is to add a dummy parameter with a UUID or similar but batch provides a strategy, e.g to increment a numeric parameter each time.
EDIT
There is a certain class of exceptions where the members of which are considered irrecoverable (fatal) and it makes no sense to attempt redelivery.
Examples include MessageConversionException - if we can't convert it the first time, we probably can't convert on a redelivery. The ConditionalRejectingErrorHandler is the mechanism by which we detect such exceptions, and cause them to be permanently rejected (and not redelivered).
Other exceptions cause the message to be redelivered by default - there is another property defaultRequeuRejected which can be set to false to permanently reject all failures (not recommended).
You can customize the error handler by subclassing its DefaultExceptionStrategy - override isUserCauseFatal(Throwable cause) to scan the cause tree to look for a JobInstanceAlreadyCompleteException and return true (cause.getCause().getCause() instanceof ...)
I think it was triggered by the error thrown by the "SpringBatch job running already" exception.
That still indicates you have somehow received a second message with the same parameters; it's a different error because the original job is still running; that message is rejected (and requeued) but on subsequent deliveries you get the already completed exception.
So, I still say the root cause of your problem is duplicate requests, but you can avoid the behavior with a customized error handler in the channel adapter's listener container.
I suggest you log the duplicate message so you can figure out why you are getting them.
I'm trying to test some stuff with spring-integration using the DSL. This is only a test so far, the flow is simple:
create some messages
process (log) them in parallel
aggregate them
log the aggregate
Apart from the aggregator, it is working fine:
#Bean
public IntegrationFlow integrationFlow() {
return IntegrationFlows
.from(integerMessageSource(), c -> c.poller(Pollers.fixedRate(1, TimeUnit.SECONDS)))
.channel(MessageChannels.executor(Executors.newCachedThreadPool()))
.handle((GenericHandler<Integer>) (payload, headers) -> {
System.out.println("\t delaying message:" + payload + " on thread "
+ Thread.currentThread().getName());
try {
Thread.sleep(2000);
} catch (InterruptedException e) {
System.err.println(e.getMessage());
}
return payload;
})
.handle(this::logMessage)
.aggregate(a ->
a.releaseStrategy(g -> g.size()>10)
.outputProcessor(g ->
g.getMessages()
.stream()
.map(e -> e.getPayload().toString())
.collect(Collectors.joining(",")))
)
.handle(this::logMessage)
.get();
}
If I leave out the .aggregate(..), part, the sample is working.
Withe the aggregator, I get the following exception:
Caused by: org.springframework.beans.factory.BeanCreationException: The 'currentComponent' (org.faboo.test.ParallelIntegrationApplication$$Lambda$9/1341404543#6fe1b4fb) is a one-way 'MessageHandler' and it isn't appropriate to configure 'outputChannel'. This is the end of the integration flow.
As far as I understand, it complains that there is no output from the aggregator?
The full source can be found here: hithub
The problem is the handle() before the aggregator - it produces no result so there is nothing to aggregate...
.handle(this::logMessage)
.aggregate(a ->
Presumably logMessage(Message<?>) has a void return type.
If you want to log before the aggregator, use a wireTap, or change logMessage to return the Message<?> after logging.
.wireTap(sf -> sf.handle(this::logMessage))