I am doing a local file copy using Spring integration. My class is presented later in this post.
I initiate the file copy by issuing an adapter.start() from another class. That works fine, and the adapter (localFileTransferAdapter) runs once (based on the FireOnceTrigger) copying all the files I expect. But I then want to stop the adapter after all the files have been copied.
What can I do to detect that the adapter has copied the files so I can then stop the adapter? The FireOnceTrigger will never fire again - but the adapter still shows running when I query it from the class I use to start the adapter. I could wait some number of seconds and stop the adapter - but it could stop the adapter in the middle of copying files if there are many large files to copy - which I don't want to happen.
I've reviewed How to stop polling after a message is received? Spring Integration but it does not appear to match my use case.
Thanks in advance for for any assistance.
public class LocalFileTransfer {
#Value("${source.directory:c:/source}")
private String sourceDirectory;
#Value("${target.directory:c:/target}")
private String targetDirectory;
#Bean
public MessageSource<File> sourceDirectory() {
FileReadingMessageSource messageSource = new FileReadingMessageSource();
messageSource.setDirectory(new File(sourceDirectory));
return messageSource;
}
#Bean
public IntegrationFlow fileMover() {
return IntegrationFlows.from(sourceDirectory(),
c -> c.autoStartup(false)
.id("localFileTransferAdapter")
.poller(Pollers.trigger(new FireOnceTrigger())
.maxMessagesPerPoll(-1)))
.filter(source -> ((File) source).getName().endsWith(".txt"))
.log(LoggingHandler.Level.ERROR, "localfile.category", m -> m.getPayload())
.log(LoggingHandler.Level.ERROR, "localfile.category", m -> m.getHeaders())
.handle(targetDirectory())
.get();
}
#Bean
public MessageHandler targetDirectory() {
FileWritingMessageHandler handler = new FileWritingMessageHandler(new File(targetDirectory));
handler.setFileExistsMode(FileExistsMode.REPLACE);
handler.setExpectReply(false);
return handler;
}
}
You are on the right track with the FireOnceTrigger and maxMessagesPerPoll(-1). Also that answer with an AbstractMessageSourceAdvice sample fully fits to your scenario.
In the MessageSourceMutator.afterReceive() you just check for the null of result argument and call a stop() of that localFileTransferAdapter.
Since everything happens on the same thread, you are safe to call stop just when you meet a null: no need to introduce some delay and the channel adapter doesn't exist until it finishes producing messages in that one poll cycle.
Related
My Message handler for publishing messages to the kinesis stream is as follows
public MessageHandler kinesisMessageHandler(final AmazonKinesisAsync amazonKinesis,
#Qualifier("successChannel") MessageChannel successChannel,
#Qualifier("errorChannel") MessageChannel errorChannel) {
KinesisMessageHandler kinesisMessageHandler = new KinesisMessageHandler(amazonKinesis);
kinesisMessageHandler.setSync(false);
kinesisMessageHandler.setOutputChannel(successChannel);
kinesisMessageHandler.setFailureChannel(errorChannel);
return kinesisMessageHandler;
}
#Bean(name = "errorChannel")
public MessageChannel errorChannel() {
return MessageChannels.direct().get();
}
#Bean(name = "successChannel")
public MessageChannel successChannel() {
return MessageChannels.direct().get();
}
The setSync flag is set as false so that the messages are getting processed asynchronously.Also, I have created separate IntegrationFlow to receive and process Kinesis response from the success & error channel.
public IntegrationFlow successMessageIntegrationFlow(MessageChannel successChannel,
MessageChannel inboundKinesisMessageChannel,
MessageReceiverServiceActivator kinesisMessageReceiverServiceActivator) {
return IntegrationFlows.from(successChannel).channel(inboundKinesisMessageChannel)
.handle(kinesisMessageReceiverServiceActivator, "receiveMessage").get();
}
#Bean
public IntegrationFlow errorMessageIntegrationFlow(MessageChannel errorChannel,
MessageChannel inboundKinesisErrorChannel,
MessageReceiverServiceActivator kinesisErrorReceiverServiceActivator
) {
return IntegrationFlows.from(errorChannel).channel(inboundKinesisErrorChannel)
.handle(kinesisErrorReceiverServiceActivator, "receiveMessage").get();
}
I wanted to know if you see any issues in using Direct Channel to receive success & error responses from Kinesis and processing it using an IntegrationFlow. As far as I know, with Direct Channel a producer is a blocker during send until the consumer finishes its work and returns management to the producer caller back. Is it a correct assumption that here the producer is executed in a different set of thread pools by the AmazonKinesisAsyncClient and the producer will not wait for the IntegrationFlow to process the messages? Let me know If I need to implement it differently
Your assumption about blocking is correct: the control does not come back to the producing thread. So, if have a limited number of threads in that Kinesis client, you need to be sure that you free them as soon as possible. You might consider to have those callbacks in the queue channel instead. They are asynchronous anyway, but won’t hold Kinesis client if that.
You still have a flaw in your flows: .channel(inboundKinesisMessageChannel) . That means the same channel in the middle if two different flows . And if it is a direct one , then you end up with round robin distribution. I would just remove it altogether .
I have the following Spring Integration flow:
It gathers records from one database, converts to json and sends to another database.
The idea is to have 10 pollers (channel0 to 9). Each one is a pollingFlowChanN Bean. But I suspect they are sharing the same thread.
How to I make the polling multi-thread in this scenario?
private IntegrationFlow getChannelPoller(final int channel, final int pollSize, final long delay) {
return IntegrationFlows.from(jdbcMessageSource(channel, pollSize), c -> c.poller(Pollers.fixedDelay(delay)
.transactional(transactionManager)))
.split()
.handle(intControleToJson())
.handle(pgsqlSink)
.get();
}
#Bean
public IntegrationFlow pollingFlowChan0() {
return getChannelPoller(0, properties.getChan0PollSize(), properties.getChan0Delay());
}
#Bean
public IntegrationFlow pollingFlowChan1() {
return getChannelPoller(1, properties.getChan1PollSize(), properties.getChan1Delay());
}
....
I assume you use the latest Spring Boot, which have a TaskScheduler auto-configured with one thread: https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#features.spring-integration. That's the best guess why your tasks use the same thread.
See also answer here: Why does the SFTP Outbound Gateway not start working as soon as I start its Integration Flow?
The last element in the code for the following DSL flow is Service Activator (.handle method).
Is there a default output direct channel to which I can subscribe here? If I understand things correctly, the output channel must be present
I know I can add .channel("name") at the end but the question is what if it's not written explicitly.
Here is the code:
#SpringBootApplication
#IntegrationComponentScan
public class QueueChannelResearch {
#Bean
public IntegrationFlow lambdaFlow() {
return f -> f.channel(c -> c.queue(50))
.handle(System.out::println);
}
public static void main(String[] args) {
ConfigurableApplicationContext ctx = SpringApplication.run(QueueChannelResearch.class, args);
MessageChannel inputChannel = ctx.getBean("lambdaFlow.input", MessageChannel.class);
for (int i = 0; i < 1000; i++) {
inputChannel.send(MessageBuilder.withPayload("w" + i)
.build());
}
ctx.close();
}
Another question is about QueueChannel. The program hangs if comment handle() and completes if uncomment it. Does that mean that handle() add a default Poller before it?
return f -> f.channel(c -> c.queue(50));
// .handle(System.out::println);
No, that doesn't work that way.
Just recall that integration flow is a filter-pipes architecture and result of the current step is going to be sent to next one. Since you use .handle(System.out::println) there is no output from that println() method call therefore nothing is returned to build a Message to sent to the next channel if any. So, the flow stops here. The void return type or null returned value is a signal for service activator to stop the flow. Consider your .handle(System.out::println) as an <outbound-channel-adapter> in the XML configuration.
And yes: there is no any default channels, unless you define one via replyChannel header in advance. But again: your service method must return something valuable.
The output from service activator is optional, that's why we didn't introduce extra operator for the Outbound Channel Adapter.
The question about QueueChannel would be better to handle in the separate SO thread. There is no default poller unless you declare one as a PollerMetadata.DEFAULT_POLLER. You might use some library which delcares that one for you.
I have an outbound gateway for soap calls (MarshallingWebServiceOutboundGateway) with elaborate setup. I need to use that gateway definition from multiple flows.
The question spring-integration: MessageProducer may only be referenced once is somewhat similar, but this question is about the proper use of the spring bean scope prototype for spring integration collaborators.
I have a separate config file which sets up the gateway and its dependencies:
#Bean
public MarshallingWebServiceOutboundGateway myServiceGateway() {
Jaxb2Marshaller marshaller = new Jaxb2Marshaller();
marshaller.setPackagesToScan("blah.*");
MarshallingWebServiceOutboundGateway gateway = new MarshallingWebServiceOutboundGateway(
serviceEndpoint, marshaller, messageFactory);
gateway.setMessageSender(messageSender);
gateway.setRequestCallback(messageCallback);
return gateway;
}
This is how I initially tried to wire up the outbound gateway from two different flows in two different config files.
In one config file:
#Bean
public IntegrationFlow flow1() {
MarshallingWebServiceOutboundGateway myServiceGateway = context.getBean("myServiceGateway", MarshallingWebServiceOutboundGateway.class);
return IntegrationFlows
.from(Http.inboundGateway("/res1")
.requestMapping(r -> r.methods(HttpMethod.GET))
.transform(soapRequestTransformer)
.handle(myServiceGateway) // wrong: cannot be same bean
.transform(widgetTransformer)
.get();
}
In a separate config file:
#Bean
public IntegrationFlow flow2() {
MarshallingWebServiceOutboundGateway myServiceGateway = context.getBean("myServiceGateway", MarshallingWebServiceOutboundGateway.class);
return IntegrationFlows
.from(Http.inboundGateway("/res2")
.requestMapping(r -> r.methods(HttpMethod.GET))
.transform(soapRequestTransformer)
.handle(myServiceGateway) // wrong: cannot be same bean
.transform(widgetTransformer)
.handle(servicePojo)
.get();
}
This is a problem because - as I understand it - myServiceGateway cannot be the same instance, since that instance has only one outbound channel and cannot belong to two different flows.
In the related question spring-integration: MessageProducer may only be referenced once, #artem-bilan advised not to create the outbound gateway in an #Bean method, rather to use a plain method which creates new instances for every call.
That works, but it is inconvenient in my case. I need to reuse the outbound gateway from several flows in different config files and I would have to copy the code to create the gateway into each config file. Also, the gateway dependencies inflate my Configuration file constructors, making Sonar bail.
Since the error message coming out of IntegrationFlowDefinition.checkReuse() says A reply MessageProducer may only be referenced once (myServiceGateway) - use #Scope(ConfigurableBeanFactory.SCOPE_PROTOTYPE) on #Bean definition. I wanted to give the scope prototype another try.
So I try to make spring integration look up a prototype gateway from the context by name, hoping to get a different gateway instance in flow1 and flow2:
.handle(context.getBean("myServiceGateway",
MarshallingWebServiceOutboundGateway.class))
And I annotated the outbound gateway #Bean definition with
#Scope(value = ConfigurableBeanFactory.SCOPE_PROTOTYPE)
But I can see that the myServiceGateway() method is only invoked once, despite the prototype scope, and application startup still fails with the error message which advises to use the prototype scope - quite confusing, actually ;-)
Based on Mystery around Spring Integration and prototype scope I also tried:
#Scope(value = ConfigurableBeanFactory.SCOPE_PROTOTYPE, proxyMode = ScopedProxyMode.TARGET_CLASS)
The application starts, but the responses never reach the step after the gateway, the widgetTransformer. (Even more strange, exactly the widgetTransformer is skipped: in flow1 the outcome is the untransformed gateway response and in flow2 the untransformed messages hit the step after the widgetTransformer, i.e. the servicePojo). Making a proxy out of a message producer seems not to be a good idea.
I really want to get to the bottom of this. Is the exception message wrong which asks to use the prototype scope or am I just getting it wrong? How can I avoid to repeat the bean definition for message producers if I need several such producers which are all set up the same way?
Using spring-integration 5.0.9.
I am not entirely sure why the #Scope is not working, but here is a work-around...
#SpringBootApplication
public class So52453934Application {
public static void main(String[] args) {
SpringApplication.run(So52453934Application.class, args);
}
#Autowired
private HandlerConfig config;
#Bean
public IntegrationFlow flow1() {
return f -> f.handle(this.config.myHandler())
.handle(System.out::println);
}
#Bean
public IntegrationFlow flow2() {
return f -> f.handle(this.config.myHandler())
.handle(System.out::println);
}
#Bean
public ApplicationRunner runner() {
return args -> {
context.getBean("flow1.input", MessageChannel.class).send(new GenericMessage<>("foo"));
context.getBean("flow2.input", MessageChannel.class).send(new GenericMessage<>("bar"));
};
}
}
#Configuration
class HandlerConfig {
public AbstractReplyProducingMessageHandler myHandler() {
return new AbstractReplyProducingMessageHandler() {
#Override
protected Object handleRequestMessage(Message<?> requestMessage) {
return ((String) requestMessage.getPayload()).toUpperCase();
}
};
}
}
i.e. do as #artem suggested, but inject the bean with the factory method.
I'm extremely new to Spring, and even more-so to Spring Integration, so apologies if this is a very basic question.
I'm wanting to build a very basic log file processor to learn the ropes. Very similar to this: example
I'm also wanting to use a java config approach, and most of the examples I've been following are all XML driven and I'm having a hard time doing the translation.
Ultimately I'd like to recursively poll a source directory for log files and use a persistence store to keep track of what's been found.
Then, copy those files to be processed to a processing folder, and then kick off a spring batch job to process the contents of the file.
When everything completes the processed file can be deleted from the processing location.
I can't seem to figure out the proper way to wire up (using general java config of SpEL) the flow. Also, I'm still very unsure of what the proper pieces should be.
Again something along these basic, high-level lines for the file moving:
file:inbound-channel-adapter -> channel -> file:outbound-adapter
basic sample
Here is what I have so far
EDIT
I've updated with Artem's solution. My source files are now properly copied to the destination location. Thanks Artem!
Ultimately I am still facing the same problem. The files to be scanned are found immediately (and the metadata-store.properties files is populated immediately) but the files are slowly copied to the destination folder. If a crash happens, any source files that have not been copied to the destination folder will essentially be "lost". Perhaps I need to look at other forms of persistence stores, like a custom jdbcfilter.
#Value("${logProcessor.filenamePattern}")
private String filenamePattern;
#Value("${logProcessor.sourceDirectory}")
private String sourceDirectory;
#Value("${logProcessor.processingDirectory}")
private String processingDirectory;
#Bean
#InboundChannelAdapter(channel = "sourceFileChannel", poller = #Poller(fixedRate = "5000"))
public MessageSource<File> sourceFiles() {
CompositeFileListFilter<File> filters = new CompositeFileListFilter<>();
filters.addFilter(new SimplePatternFileListFilter(filenamePattern));
filters.addFilter(persistentFilter());
FileReadingMessageSource source = new FileReadingMessageSource();
source.setAutoCreateDirectory(true);
source.setDirectory(new File(sourceDirectory));
source.setFilter(filters);
source.setUseWatchService(true);
return source;
}
#Bean
#InboundChannelAdapter(channel = "processingFileChannel", poller = #Poller(fixedRate = "5000"))
public MessageSource<File> processingFiles() {
CompositeFileListFilter<File> filters = new CompositeFileListFilter<>();
filters.addFilter(new SimplePatternFileListFilter(filenamePattern));
FileReadingMessageSource source = new FileReadingMessageSource();
source.setAutoCreateDirectory(true);
source.setDirectory(new File(processingDirectory));
source.setFilter(filters);
return source;
}
#Bean
#ServiceActivator(inputChannel = "sourceFileChannel")
public MessageHandler fileOutboundChannelAdapter() {
FileWritingMessageHandler adapter = new FileWritingMessageHandler(new File(processingDirectory));
adapter.setDeleteSourceFiles(false);
adapter.setAutoCreateDirectory(true);
adapter.setExpectReply(false);
return adapter;
}
#Bean
public MessageChannel sourceFileChannel() {
return new DirectChannel();
}
#Bean
public MessageChannel processingFileChannel() {
return new DirectChannel();
}
#Bean
public DefaultDirectoryScanner defaultDirectoryScanner() {
return new DefaultDirectoryScanner();
}
#Bean
public FileSystemPersistentAcceptOnceFileListFilter persistentFilter() {
FileSystemPersistentAcceptOnceFileListFilter fileSystemPersistentAcceptOnceFileListFilter = new FileSystemPersistentAcceptOnceFileListFilter(metadataStore(), "");
fileSystemPersistentAcceptOnceFileListFilter.setFlushOnUpdate(true);
return fileSystemPersistentAcceptOnceFileListFilter;
}
#Bean
public PropertiesPersistingMetadataStore metadataStore(){
PropertiesPersistingMetadataStore metadataStore = new PropertiesPersistingMetadataStore();
metadataStore.setBaseDirectory("C:\\root\\code\\logProcessor");
return metadataStore;
}
You config is good so far.
Having such a complex task, I'm not sure how to help you.
You should ask more specific question. We can't write the solution for you.
Not sure why you need to copy files from one dir to another, if you can simply poll them from the source dir, store in the metadataStore and start a file processing.
So, far I see a small problem in your config. The FileWritingMessageHandler sends results to the processingFileChannel and the same is done by the second FileReadingMessageSource. I'm not sure that it is your intention. Just in case to pay your attention.
You might also need to know about FileSplitter, which lets you to process file line by line.
Also you say processingDirectory, but then you use tmpDir for the FileWritingMessageHandler, which, I guess, assumes your copy logic.
Let's do the task step by step! And then you figure out what, where and how to use!
EDIT
If you need just copy file to the processingDirectory without any reply, you should do one-way adapter:
#Bean
#ServiceActivator(inputChannel = "sourceFileChannel")
public MessageHandler fileOutboundChannelAdapter() {
FileWritingMessageHandler adapter = new FileWritingMessageHandler(new File(processingDirectory));
adapter.setDeleteSourceFiles(true);
adapter.setAutoCreateDirectory(true);
adapter.setExpectReply(false);
return adapter;
}
And then that your #InboundChannelAdapter(channel = "processingFileChannel" is good to pick up files for processing.
Not sure that you need to DeleteSourceFiles though...