Download multiple files using SourcePollingChannelAdapter- start/stop issue? - spring-integration

Question

My requirement is to donwload multiple files from a remote directory in a file server in my spring-batch application. One of the example here suggests the usage of SourcePollingChannelAdapter. In this example the adapter is started adapter.start() but its not stopped. So how does the lifecycle of this work ? My requirement is to first download & Only when all files are downloaded, then proceed to read the files. But this seems this is kind of async process to download the files. So how would I get notified when all the files are downloaded & ready to proceed further ? I dont see any other methods to inject any handlers to the adapter/pollablechannel.
Sample config used :
<int-sftp:inbound-channel-adapter id="sftpInbondAdapterBean"
auto-startup="false" channel="receiveChannelBean" session-factory="sftpSessionFactory"
local-directory="${sftp.download.localDirResource}"
remote-directory="${sftp.download.remoteFilePath}"
auto-create-local-directory="true" delete-remote-files="false"
filename-regex=".*\.csv$">
<int:poller fixed-rate="5000" max-messages-per-poll="3" />
</int-sftp:inbound-channel-adapter>
<int:channel id="receiveChannelBean">
<int:queue/>
</int:channel>
It throws InterruptedException if adapter.stop() is called explicitly. If stop is not called , the files are downloaded in async way without blocking & hence the next step doesn't know if the download is completed or is in-progress.
EDIT : Code Snippet of DownloadingTasklet
#Override
public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext) throws Exception {
try {
sftpInbondAdapter.start();
Thread.sleep(15000); // ---Is this mandatory ?
} catch (Exception e) {
throw new BatchClientException("Batch File Download was un-successful !", e);
} finally {
// sftpInbondAdapter.stop();
Logger.logDebug("BATCH_INPUT_FILE_DOWNLOAD", "COMPLETED");
}
return RepeatStatus.FINISHED;
}

Answer 1

The SftpInboundFileSynchronizingMessageSource works in two phases:
First of all it does synchronization from the remote directory to the local one:
Message<File> message = this.fileSource.receive();
if (message == null) {
this.synchronizer.synchronizeToLocalDirectory(this.localDirectory);
message = this.fileSource.receive();
}
And as you see only after that it starts to emit files as a messages. But already only local files. So, download from the FTP has been done already.
The next download attempt happens only when all local files are cleared during sending process.
So, what you want is a built-in functionality.
EDIT
Some DEBUG logs from our tests:
19:29:32,005 DEBUG task-scheduler-1 util.SimplePool:190 - Obtained new org.springframework.integration.sftp.session.SftpSession#26f16625.
19:29:32,036 DEBUG task-scheduler-1 inbound.SftpInboundFileSynchronizer:287 - cannot copy, not a file: /sftpSource/subSftpSource
19:29:32,036 DEBUG task-scheduler-1 session.CachingSessionFactory:187 - Releasing Session org.springframework.integration.sftp.session.SftpSession#26f16625 back to the pool.
19:29:32,036 DEBUG task-scheduler-1 util.SimplePool:221 - Releasing org.springframework.integration.sftp.session.SftpSession#26f16625 back to the pool
19:29:32,037 DEBUG task-scheduler-1 inbound.SftpInboundFileSynchronizer:270 - 3 files transferred
19:29:32,037 DEBUG task-scheduler-1 file.FileReadingMessageSource:380 - Added to queue: [local-test-dir\rollback\ sftpSource1.txt, local-test-dir\rollback\sftpSource2.txt]
19:29:32,043 INFO task-scheduler-1 file.FileReadingMessageSource:368 - Created message: [GenericMessage [payload=local-test-dir\rollback\ sftpSource1.txt, headers={id=40f75bd1-2150-72a4-76f0-f5c619a246da, timestamp=1465514972043}]]
It is with config like:
<int-sftp:inbound-channel-adapter
session-factory="sftpSessionFactory"
channel="requestChannel"
remote-directory="/sftpSource"
local-directory="local-test-dir/rollback"
auto-create-local-directory="true"
local-filter="acceptOnceFilter">
<int:poller fixed-delay="1" max-messages-per-poll="0"/>
</int-sftp:inbound-channel-adapter>
Pay attention how it shows 3 files transferred, where one of them is directory, so skipped.
How it says Added to queue:
And only after that it starts to emit them as messages.
And everything is just because I have fixed-delay="1". Even if it is only 1 millisecond.
The max-messages-per-poll="0" means processes everything with one polling task.