Spring Integration: defining inputs for each MessageSource poll - spring-integration

Question

I've spent a fair amount of time over the past few weeks learning about Spring Integration, and something I'm not seeing in the docs is how the various MessageSources prevent themselves (or can be configured to prevent) reading duplicate inputs into the IntegrationFlows.
For instance, let's take an FTP server:
return IntegrationFlows.from("ftp-endpoint")
.handle(fileProcessor())
.get();
Let's say at one point in time the directory on the FTP server (indicated by "ftp-endpoint" above) has the following files on it:
readingDir/
file1.txt
file2.txt
file3.txt
The next time the scheduled FTP message source (ftp-endpoint) runs, it ingests all 3 files. Then some more files get added:
readingDir/
file1.txt
file2.txt
file3.txt
file4.txt
file5.txt
How do we prevent file1.txt, file2.txt and file3.txt from being read the next time the flow polls the FTP server and runs? What if we actually want it to re-run file3.txt for some reason -- how would we tell it to read file3.txt (re-run), file4.txt (new) and file5.txt (new) but skip file1.txt and file2.txt?
And this question is not just for FTP, it would be for any polling endpoint: email, DropBox, S3, etc. Hopefully its the same API/strategy for all of them!
The only thing I can see on the API that jumps out is to provide a SourcePollingChannelAdapterSpec on the endpoint, so:
IntegrationFlows.from("ftp-endpoint", sourcePollingChannelAdapterSpec)...
And then configure the SourcePollingChannelAdapterSpec to have a piece of Advice (setAdvice(Arrays.asList(specialAdvice))) that has logic in it, but that feels kind of clumsy (I'm not a huge fan of tag interfaces like Advice). I'll use it if that's the only solution, but there's gotta be a better way to tell each MessageSource what to consider "valid inputs" on each run!

Answer 1

It is not clear what resources did you investigate to learn about this stuff.
Please, take a look into official FTP Inbound Channel Adapter docs:
https://docs.spring.io/spring-integration/docs/current/reference/html/ftp.html#ftp-inbound
https://docs.spring.io/spring-integration/docs/current/reference/html/file.html#remote-persistent-flf
So, you are fully missing the fact that there is a FileListFilter strategy and there are a lot of out-of-the-box implementations for our consideration. What you are asking is covered by the mentioned in the first doc AcceptOnceFileListFilter. It does hold a reference to the file to skip it on the next poll. If you'd like to re-fetch some old file, you just can call AcceptOnceFileListFilter.remove(). Or look into an FtpPersistentAcceptOnceFileListFilter which is able to track not just file name, but also its timestamp. So, if the content of remote file is updated, then its timestamp is changed - and this filter would treat the file as new one and will fetch it again.