ftp outbound channel adapter not transferring same files - spring-integration

Question

I have the following configuration in my application. I am writing a file to the ftp folder and reading from the ftp location.
I want to take the file from the ftp location and save it in a directory decided dynamically. This is done by using two chained channel adapters. The ftp nbound channel adapter picks up the file, puts it in the local directory and then the file inbound channel adapter picks up the file and puts it in its final destination. I need to filter old files for this process. ftp inbound channel adapter custom filter gave me FTPFile object in the filtering method. This object gives the last modified date rather than the date the file was put in the filter. Due to this limitation I had to use file inbound channel adapter as well.
Since I do not know how to generate the source directory dynamically, I am using plain java code to copy the required file to the local-directory from where the ftp outbound channel picks it up and puts it on ftp location.
This is the configuration:
<bean id="fileNameGenerator" class="com.polling.util.FileNameGenerator"/>
<int-file:inbound-channel-adapter id="filesIn" directory="file:${paths.root}" channel="abc" filter="compositeFilter" >
<int:poller id="poller" fixed-rate="500" />
</int-file:inbound-channel-adapter>
<int:channel id="abc"/>
<bean id="compositeFilter" class="org.springframework.integration.file.filters.CompositeFileListFilter">
<constructor-arg>
<list>
<!-- Ensures that the file is whole before processing it -->
<bean class="com.polling.util.CustomFileFilter"/>
<!-- Ensures files are picked up only once from the directory -->
<bean class="org.springframework.integration.file.filters.AcceptOnceFileListFilter" />
</list>
</constructor-arg>
</bean>
<int-file:outbound-channel-adapter channel="abc" id="filesOut"
directory-expression="#outPathBean.getPath()"
delete-source-files="true" filename-generator="fileNameGenerator" >
<int-file:request-handler-advice-chain>
<bean class="org.springframework.integration.handler.advice.ExpressionEvaluatingRequestHandlerAdvice">
<property name="onSuccessExpression" value="payload.delete()" />
</bean>
</int-file:request-handler-advice-chain>
</int-file:outbound-channel-adapter>
<bean id="ftpClientFactory"
class="org.springframework.integration.ftp.session.DefaultFtpSessionFactory">
<property name="host" value="${ftp.ip}"/>
<property name="port" value="${ftp.port}"/>
<property name="username" value="${ftp.username}"/>
<property name="password" value="${ftp.password}"/>
<property name="clientMode" value="0"/>
<property name="fileType" value="2"/>
<property name="bufferSize" value="100000"/>
</bean>
<int:channel id="ftpChannel"/>
<int-ftp:inbound-channel-adapter id="ftpInbound"
channel="ftpChannel"
session-factory="ftpClientFactory"
charset="UTF-8"
local-directory="file:${paths.root}"
delete-remote-files="true"
temporary-file-suffix=".writing"
remote-directory="${file.ftpfolder}"
filter="compositeFilterRemote"
preserve-timestamp="true"
auto-startup="true">
<int:poller fixed-rate="1000"/>
</int-ftp:inbound-channel-adapter>
<int-ftp:outbound-channel-adapter id="ftpOutbound"
channel="ftpChannel"
session-factory="ftpClientFactory"
charset="UTF-8"
remote-file-separator="/"
auto-create-directory="true"
remote-directory="${file.ftpfolder}"
use-temporary-file-name="true"
temporary-file-suffix=".writing">
<int-ftp:request-handler-advice-chain>
<bean class="org.springframework.integration.handler.advice.ExpressionEvaluatingRequestHandlerAdvice">
<property name="onSuccessExpression" value="payload.delete()" />
</bean>
</int-ftp:request-handler-advice-chain>
</int-ftp:outbound-channel-adapter>
Using the suggestion in #Gary's answer, I have added a CustomFilter to the ftp inbound channel filter attribute. This checks the regular expression of the file name as follows
#Override
public List<FTPFile> filterFiles(FTPFile[] files)
{
List<FTPFile> ret = new ArrayList<FTPFile>();
Pattern pattern = Pattern.compile("~.*?~");
Matcher matcher;
for (FTPFile file : files)
{
matcher = pattern.matcher(file.getName());
if(matcher.matches())
{
ret.add(file);
}
}
return ret;
}
Due to this the ~something~ files are picked up and sent to final destination.
My Questions are:
The files being sent from the final destination to the ftp location have a different pattern which I don't know where to put(something#something#something).
Sometimes the ftp folder may be tampered with and the file deleted. If so I would like spring to rewrite the file. The same file gets written again in the local-directory mentioned in the <ftp:inbound-channel-adapter>. But ftp outbound channel adapter does not pick the file up and put it in the ftp location. According to #Gary's answer I need to configure the filter for the inbound channel adapter to avoid the accept once filter.
Does this mean that I should create two separate channels and two different flows one for back and one for forth? Is this the right way forward for my requirement?
Thanks for your help
EDIT::
I have tried to implement two different processes. One to pickup files from remotedir with pattern ~something~ and another from the local directory with pattern something#something#something. Though the basic functionality is working, if the file in the remote dir is deleted i cannot execute the second flow again. As Gary said the accept all filter needs to be in place but I dont know how to use both the custom filter and the accept all filter.
Any suggestions are appreciated

Answer 1

The AcceptOnceFileListFilter is configured by default. Use the local-filter on the inbound adapter to use a different filter (such as an AcceptAllFileListFilter). See the documentation.
That said, your application looks odd - it appears you are fetching files and simply sending them back.

Answer 2

This is what worked
I continued with the two flows and changed my bean configuration to include the accept all filter.
This is the configuration now.
<bean id="fileNameGenerator" class="com.polling.util.FileNameGenerator"/>
<int-file:inbound-channel-adapter id="filesIn" directory="file:${paths.root}" channel="abc" filter="compositeFilter" >
<int:poller id="poller" fixed-rate="500" />
</int-file:inbound-channel-adapter>
<int:channel id="abc"/>
<bean id="compositeFilter" class="org.springframework.integration.file.filters.CompositeFileListFilter">
<constructor-arg>
<list>
<!-- Ensures that the file is whole before processing it -->
<bean class="com.polling.util.CustomFileFilter"/>
<!-- Ensures files are picked up only once from the directory -->
<bean class="org.springframework.integration.file.filters.AcceptOnceFileListFilter" />
</list>
</constructor-arg>
</bean>
<int-file:outbound-channel-adapter channel="abc" id="filesOut"
directory-expression="#outPathBean.getPath()"
delete-source-files="true" filename-generator="fileNameGenerator" >
<int-file:request-handler-advice-chain>
<bean class="org.springframework.integration.handler.advice.ExpressionEvaluatingRequestHandlerAdvice">
<property name="onSuccessExpression" value="payload.delete()" />
</bean>
</int-file:request-handler-advice-chain>
</int-file:outbound-channel-adapter>
<bean id="ftpClientFactory"
class="org.springframework.integration.ftp.session.DefaultFtpSessionFactory">
<property name="host" value="${ftp.ip}"/>
<property name="port" value="${ftp.port}"/>
<property name="username" value="${ftp.username}"/>
<property name="password" value="${ftp.password}"/>
<property name="clientMode" value="0"/>
<property name="fileType" value="2"/>
<property name="bufferSize" value="100000"/>
</bean>
<int:channel id="ftpChannel1"/>
<int-ftp:inbound-channel-adapter id="ftpInbound1"
channel="ftpChannel1"
session-factory="ftpClientFactory"
charset="UTF-8"
local-directory="file:${paths.root}"
delete-remote-files="true"
temporary-file-suffix=".writing"
remote-directory="."
preserve-timestamp="true"
auto-startup="true"
local-filter="compositeFilterLocal">
<int:poller fixed-rate="1000"/>
</int-ftp:inbound-channel-adapter>
<int-ftp:outbound-channel-adapter id="ftpOutbound1"
channel="ftpChannel1"
session-factory="ftpClientFactory"
charset="UTF-8"
remote-file-separator="/"
auto-create-directory="true"
remote-directory="${file.ftpfolder}"
use-temporary-file-name="true"
temporary-file-suffix=".writing">
<int-ftp:request-handler-advice-chain>
<bean class="org.springframework.integration.handler.advice.ExpressionEvaluatingRequestHandlerAdvice">
<property name="onSuccessExpression" value="payload.delete()" />
</bean>
</int-ftp:request-handler-advice-chain>
</int-ftp:outbound-channel-adapter>
<int:channel id="ftpChannel"/>
<int-ftp:inbound-channel-adapter id="ftpInbound"
channel="ftpChannel"
session-factory="ftpClientFactory"
charset="UTF-8"
local-directory="file:${paths.root}"
delete-remote-files="true"
temporary-file-suffix=".writing"
remote-directory="${file.ftpfolder}"
filter="compositeFilterRemote"
preserve-timestamp="true"
auto-startup="true">
<int:poller fixed-rate="1000"/>
</int-ftp:inbound-channel-adapter>
<int-ftp:outbound-channel-adapter id="ftpOutbound"
channel="ftpChannel"
session-factory="ftpClientFactory"
charset="UTF-8"
remote-file-separator="/"
auto-create-directory="true"
remote-directory="${file.ftpfolder}"
use-temporary-file-name="true"
temporary-file-suffix=".writing">
<int-ftp:request-handler-advice-chain>
<bean class="org.springframework.integration.handler.advice.ExpressionEvaluatingRequestHandlerAdvice">
<property name="onSuccessExpression" value="payload.delete()" />
</bean>
</int-ftp:request-handler-advice-chain>
</int-ftp:outbound-channel-adapter>
<bean id="acceptAllFilter" class="org.springframework.integration.file.filters.AcceptAllFileListFilter" />
<bean id="compositeFilterLocal" class="org.springframework.integration.file.filters.CompositeFileListFilter">
<constructor-arg>
<list>
<!-- Ensures that the file is whole before processing it -->
<bean class="com.polling.util.CustomFileFilterLocal"/>
<!-- Ensures files are picked up only once from the directory -->
<bean class="org.springframework.integration.file.filters.AcceptAllFileListFilter" />
</list>
</constructor-arg>
</bean>
<bean id="compositeFilterRemote" class="org.springframework.integration.file.filters.CompositeFileListFilter">
<constructor-arg>
<list>
<!-- Ensures that the file is whole before processing it -->
<bean class="com.polling.util.CustomFileFilterRemote"/>
<!-- Ensures files are picked up only once from the directory -->
<bean class="org.springframework.integration.file.filters.AcceptOnceFileListFilter" />
</list>
</constructor-arg>
</bean>
If anyone has any suggestion to make this more efficient , please let me know. As of now I can repeatedly download files. So am continuing with this..
Thanks