Related
I have a integration flow.
IntegrationFlows.from(
Sftp.inboundAdapter(inboundSftp)
.localDirectory(this.getlocalDirectory(config.getId()))
.deleteRemoteFiles(true)
.autoCreateLocalDirectory(true)
.filter(new CompositeFileListFilter().addFilter(new LastModifiedLsEntryFileListFilter(config.getRegexFilter())))
.remoteDirectory(config.getInboundDirectory())
, e -> e.poller(
Pollers.cron(config.getCron())
.errorChannel(MessageHeaders.ERROR_CHANNEL).errorHandler((ex) -> {
try {
this.destroy(String.valueOf(config.getId()));
configurationService.removeConfigurationChannelById(config.getId());
//loggin here
}
} catch (Exception ex1) {
Logger.getLogger(ExceptionAspect.class.getName()).log(Level.SEVERE, null, ex1);
}
})
.advice(startup.scanRemoteDirectory())
))
.transform(
file -> util.transform((File) file, config.getSourceEncoding(), config.getTargetEncoding(), doEncoding, doZip))
.publishSubscribeChannel(s -> s
.subscribe(f -> {
f.handle(Sftp.outboundAdapter(outboundSftp)
.useTemporaryFileName(false)
.autoCreateDirectory(true)
.remoteDirectory(config.getOutboundDirectory()), c -> c.advice(startup.deleteFileAdvice()));
})
.subscribe(f -> {
if(doArchive) {
f.handle(Sftp.outboundAdapter(inboundSftp)
.useTemporaryFileName(false)
.autoCreateDirectory(true)
.remoteDirectory(config.getInboundArchiveDirectory()));
} else {
f.handle(m -> {});
}
})
.subscribe(f -> f
.handle(m -> {
// file transfer logging here
if(doArchive) {
// file archived logging here
}
}
})
)
)
.get();
in 1st subscriber, if file is failed to upload it still printing logs that file is transferred but actually its not.
what i understand message will be pass to each on subscriber and when 1st subscriber complete its work it will sent to next one.
In my case first subscriber is actually fail to upload file. Advice is not removing file.
I did try few things.
IntegrationFlows.from(
Sftp.inboundAdapter(inboundSftp)
..........
))
.transform(....
)
.publishSubscribeChannel(s -> s
.subscribe(f -> {
f.handle(Sftp.outboundAdapter(outboundSftp)
.useTemporaryFileName(false)
.autoCreateDirectory(true)
.remoteDirectory(config.getOutboundDirectory()), c -> c.advice(startup.deleteFileAdvice()));
}).publishSubcribeChannel(s -> s
.subscribe(f -> {
if(doArchive) {
f.handle(Sftp.outboundAdapter(inboundSftp)
.useTemporaryFileName(false)
.autoCreateDirectory(true)
.remoteDirectory(config.getInboundArchiveDirectory()));
} else {
f.handle(m -> {});
}
})
.subscribe(f -> f
.handle(m -> {
// file transfer logging here
if(doArchive) {
// file archived logging here
}
}
})
)
)
)
.get();
I try to publish again after outboundAdapter subscriber, but in error it still published.
Also try to add .errorChannel in outboundAdapter subscriber, didn't work for me.
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v2.2.2.RELEASE)
2020-08-20 11:10:53,265 INFO c.t.i.s.I2SftpApplication - Starting I2SftpApplication on GSEUC5CG8393GR5 with PID 21636 (C:\Users\MuhammadUmair\IdeaProjects\i2sftpinboudservice\target\classes started by MuhammadUmair in C:\Users\MuhammadUmair\IdeaProjects\i2sftpinboudservice)
2020-08-20 11:10:53,269 INFO c.t.i.s.I2SftpApplication - No active profile set, falling back to default profiles: default
2020-08-20 11:10:54,464 INFO o.s.i.c.DefaultConfiguringBeanFactoryPostProcessor - No bean named 'errorChannel' has been explicitly defined. Therefore, a default PublishSubscribeChannel will be created.
2020-08-20 11:10:54,478 INFO o.s.i.c.DefaultConfiguringBeanFactoryPostProcessor - No bean named 'integrationHeaderChannelRegistry' has been explicitly defined. Therefore, a default DefaultHeaderChannelRegistry will be created.
2020-08-20 11:10:54,577 INFO o.s.c.s.PostProcessorRegistrationDelegate$BeanPostProcessorChecker - Bean 'integrationChannelResolver' of type [org.springframework.integration.support.channel.BeanFactoryChannelResolver] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2020-08-20 11:10:54,584 INFO o.s.c.s.PostProcessorRegistrationDelegate$BeanPostProcessorChecker - Bean 'integrationDisposableAutoCreatedBeans' of type [org.springframework.integration.config.annotation.Disposables] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2020-08-20 11:10:54,597 INFO o.s.c.s.PostProcessorRegistrationDelegate$BeanPostProcessorChecker - Bean 'org.springframework.integration.config.IntegrationManagementConfiguration' of type [org.springframework.integration.config.IntegrationManagementConfiguration] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2020-08-20 11:10:54,906 INFO o.s.b.w.e.t.TomcatWebServer - Tomcat initialized with port(s): 8080 (http)
2020-08-20 11:10:54,917 INFO o.a.c.h.Http11NioProtocol - Initializing ProtocolHandler ["http-nio-8080"]
2020-08-20 11:10:54,917 INFO o.a.c.c.StandardService - Starting service [Tomcat]
2020-08-20 11:10:54,917 INFO o.a.c.c.StandardEngine - Starting Servlet engine: [Apache Tomcat/9.0.29]
2020-08-20 11:10:55,095 INFO o.a.c.c.C.[.[.[/] - Initializing Spring embedded WebApplicationContext
2020-08-20 11:10:55,095 INFO o.s.w.c.ContextLoader - Root WebApplicationContext: initialization completed in 1705 ms
2020-08-20 11:10:56,920 INFO o.s.s.c.ThreadPoolTaskExecutor - Initializing ExecutorService 'applicationTaskExecutor'
2020-08-20 11:10:57,243 INFO o.s.s.c.ThreadPoolTaskScheduler - Initializing ExecutorService 'taskScheduler'
2020-08-20 11:10:57,415 INFO o.s.i.e.EventDrivenConsumer - Adding {logging-channel-adapter:_org.springframework.integration.errorLogger} as a subscriber to the 'errorChannel' channel
2020-08-20 11:10:57,416 INFO o.s.i.c.PublishSubscribeChannel - Channel 'application.errorChannel' has 1 subscriber(s).
2020-08-20 11:10:57,416 INFO o.s.i.e.EventDrivenConsumer - started bean '_org.springframework.integration.errorLogger'
2020-08-20 11:10:57,416 INFO o.s.i.e.EventDrivenConsumer - Adding {message-handler:startup.resultFileHandler.serviceActivator} as a subscriber to the 'fromSftpChannel' channel
2020-08-20 11:10:57,416 INFO o.s.i.c.DirectChannel - Channel 'application.fromSftpChannel' has 1 subscriber(s).
2020-08-20 11:10:57,417 INFO o.s.i.e.EventDrivenConsumer - started bean 'startup.resultFileHandler.serviceActivator'
2020-08-20 11:10:57,422 INFO o.s.i.e.SourcePollingChannelAdapter - started bean 'startup.sftpMessageSource.inboundChannelAdapter'
2020-08-20 11:10:57,441 INFO o.a.c.h.Http11NioProtocol - Starting ProtocolHandler ["http-nio-8080"]
2020-08-20 11:10:57,498 INFO o.s.b.w.e.t.TomcatWebServer - Tomcat started on port(s): 8080 (http) with context path ''
2020-08-20 11:10:57,508 INFO c.t.i.s.I2SftpApplication - Started I2SftpApplication in 9.228 seconds (JVM running for 13.133)
Registering an Integration Flow with id : 1
2020-08-20 11:10:57,689 INFO o.s.i.c.DirectChannel - Channel 'application.1.subFlow#2.channel#0' has 1 subscriber(s).
2020-08-20 11:10:57,690 INFO o.s.i.e.EventDrivenConsumer - started bean '1.subFlow#2.org.springframework.integration.config.ConsumerEndpointFactoryBean#1'
2020-08-20 11:10:57,691 INFO o.s.i.e.EventDrivenConsumer - Adding {bridge} as a subscriber to the '1.subFlow#0.channel#0' channel
2020-08-20 11:10:57,691 INFO o.s.i.c.PublishSubscribeChannel - Channel 'application.1.subFlow#0.channel#0' has 1 subscriber(s).
2020-08-20 11:10:57,691 INFO o.s.i.e.EventDrivenConsumer - started bean '1.subFlow#2.org.springframework.integration.config.ConsumerEndpointFactoryBean#0'
2020-08-20 11:10:57,691 INFO o.s.i.c.DirectChannel - Channel 'application.1.subFlow#1.channel#0' has 1 subscriber(s).
2020-08-20 11:10:57,691 INFO o.s.i.e.EventDrivenConsumer - started bean '1.subFlow#1.org.springframework.integration.config.ConsumerEndpointFactoryBean#1'
2020-08-20 11:10:57,691 INFO o.s.i.e.EventDrivenConsumer - Adding {bridge} as a subscriber to the '1.subFlow#0.channel#0' channel
2020-08-20 11:10:57,692 INFO o.s.i.c.PublishSubscribeChannel - Channel 'application.1.subFlow#0.channel#0' has 2 subscriber(s).
2020-08-20 11:10:57,692 INFO o.s.i.e.EventDrivenConsumer - started bean '1.subFlow#1.org.springframework.integration.config.ConsumerEndpointFactoryBean#0'
2020-08-20 11:10:57,693 INFO o.s.i.e.EventDrivenConsumer - Adding {message-handler} as a subscriber to the '1.subFlow#0.channel#1' channel
2020-08-20 11:10:57,694 INFO o.s.i.c.DirectChannel - Channel 'application.1.subFlow#0.channel#1' has 1 subscriber(s).
2020-08-20 11:10:57,694 INFO o.s.i.e.EventDrivenConsumer - started bean '1.subFlow#0.org.springframework.integration.config.ConsumerEndpointFactoryBean#1'
2020-08-20 11:10:57,694 INFO o.s.i.e.EventDrivenConsumer - Adding {bridge} as a subscriber to the '1.subFlow#0.channel#0' channel
2020-08-20 11:10:57,694 INFO o.s.i.e.EventDrivenConsumer - started bean '1.subFlow#0.org.springframework.integration.config.ConsumerEndpointFactoryBean#0'
2020-08-20 11:10:57,694 INFO o.s.i.e.EventDrivenConsumer - Adding {transformer} as a subscriber to the '1.channel#0' channel
2020-08-20 11:10:57,694 INFO o.s.i.c.DirectChannel - Channel 'application.1.channel#0' has 1 subscriber(s).
2020-08-20 11:10:57,694 INFO o.s.i.e.EventDrivenConsumer - started bean '1.org.springframework.integration.config.ConsumerEndpointFactoryBean#0'
2020-08-20 11:10:57,700 INFO o.s.i.e.SourcePollingChannelAdapter - started bean '1.org.springframework.integration.config.SourcePollingChannelAdapterFactoryBean#0'
2020-08-20 11:10:58,017 INFO c.j.jsch - Connecting to *.*.*.114 port 22
2020-08-20 11:10:58,341 INFO c.j.jsch - Connection established
2020-08-20 11:10:58,672 INFO c.j.jsch - Remote version string: SSH-2.0-OpenSSH_7.4
2020-08-20 11:10:58,673 INFO c.j.jsch - Local version string: SSH-2.0-JSCH-0.1.54
2020-08-20 11:10:58,674 INFO c.j.jsch - CheckCiphers: aes256-ctr,aes192-ctr,aes128-ctr,aes256-cbc,aes192-cbc,aes128-cbc,3des-ctr,arcfour,arcfour128,arcfour256
2020-08-20 11:10:58,728 INFO c.j.jsch - CheckKexes: diffie-hellman-group14-sha1,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521
2020-08-20 11:10:58,815 INFO c.j.jsch - CheckSignatures: ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
2020-08-20 11:10:58,823 INFO c.j.jsch - SSH_MSG_KEXINIT sent
2020-08-20 11:10:59,036 INFO c.j.jsch - SSH_MSG_KEXINIT received
2020-08-20 11:10:59,038 INFO c.j.jsch - kex: server: curve25519-sha256,curve25519-sha256#libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
2020-08-20 11:10:59,038 INFO c.j.jsch - kex: server: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519
2020-08-20 11:10:59,039 INFO c.j.jsch - kex: server: aes256-ctr,aes192-ctr,aes128-ctr
2020-08-20 11:10:59,040 INFO c.j.jsch - kex: server: aes256-ctr,aes192-ctr,aes128-ctr
2020-08-20 11:10:59,040 INFO c.j.jsch - kex: server: hmac-sha2-512,hmac-sha2-256
2020-08-20 11:10:59,041 INFO c.j.jsch - kex: server: hmac-sha2-512,hmac-sha2-256
2020-08-20 11:10:59,042 INFO c.j.jsch - kex: server: none,zlib#openssh.com
2020-08-20 11:10:59,042 INFO c.j.jsch - kex: server: none,zlib#openssh.com
2020-08-20 11:10:59,042 INFO c.j.jsch - kex: server:
2020-08-20 11:10:59,043 INFO c.j.jsch - kex: server:
2020-08-20 11:10:59,043 INFO c.j.jsch - kex: client: ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group14-sha1,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1
2020-08-20 11:10:59,044 INFO c.j.jsch - kex: client: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
2020-08-20 11:10:59,044 INFO c.j.jsch - kex: client: aes128-ctr,aes128-cbc,3des-ctr,3des-cbc,blowfish-cbc,aes192-ctr,aes192-cbc,aes256-ctr,aes256-cbc
2020-08-20 11:10:59,045 INFO c.j.jsch - kex: client: aes128-ctr,aes128-cbc,3des-ctr,3des-cbc,blowfish-cbc,aes192-ctr,aes192-cbc,aes256-ctr,aes256-cbc
2020-08-20 11:10:59,045 INFO c.j.jsch - kex: client: hmac-md5,hmac-sha1,hmac-sha2-256,hmac-sha1-96,hmac-md5-96
2020-08-20 11:10:59,045 INFO c.j.jsch - kex: client: hmac-md5,hmac-sha1,hmac-sha2-256,hmac-sha1-96,hmac-md5-96
2020-08-20 11:10:59,045 INFO c.j.jsch - kex: client: none
2020-08-20 11:10:59,045 INFO c.j.jsch - kex: client: none
2020-08-20 11:10:59,045 INFO c.j.jsch - kex: client:
2020-08-20 11:10:59,046 INFO c.j.jsch - kex: client:
2020-08-20 11:10:59,046 INFO c.j.jsch - kex: server->client aes128-ctr hmac-sha2-256 none
2020-08-20 11:10:59,047 INFO c.j.jsch - kex: client->server aes128-ctr hmac-sha2-256 none
2020-08-20 11:10:59,053 INFO c.j.jsch - SSH_MSG_KEX_ECDH_INIT sent
2020-08-20 11:10:59,053 INFO c.j.jsch - expecting SSH_MSG_KEX_ECDH_REPLY
2020-08-20 11:10:59,385 INFO c.j.jsch - ssh_rsa_verify: signature true
2020-08-20 11:10:59,389 INFO o.s.i.s.s.DefaultSftpSessionFactory - The authenticity of host '*.*.*.114' can't be established.
RSA key fingerprint is 1f:3e:c9:5f:37:00:a1:00:ef:50:59:af:42:98:99:e9.
Are you sure you want to continue connecting?
2020-08-20 11:10:59,390 WARN c.j.jsch - Permanently added '*.*.*.114' (RSA) to the list of known hosts.
2020-08-20 11:10:59,390 INFO c.j.jsch - SSH_MSG_NEWKEYS sent
2020-08-20 11:10:59,390 INFO c.j.jsch - SSH_MSG_NEWKEYS received
2020-08-20 11:10:59,395 INFO c.j.jsch - SSH_MSG_SERVICE_REQUEST sent
2020-08-20 11:10:59,718 INFO c.j.jsch - SSH_MSG_SERVICE_ACCEPT received
2020-08-20 11:11:00,047 INFO c.j.jsch - Authentications that can continue: gssapi-with-mic,publickey,keyboard-interactive,password
2020-08-20 11:11:00,047 INFO c.j.jsch - Next authentication method: gssapi-with-mic
2020-08-20 11:11:00,399 INFO c.j.jsch - Authentications that can continue: password
2020-08-20 11:11:00,399 INFO c.j.jsch - Next authentication method: password
2020-08-20 11:11:00,746 INFO c.j.jsch - Authentication succeeded (password).
Total Files: 1
2020-08-20 11:11:27,623 INFO c.t.i.s.s.LastModifiedLsEntryFileListFilter - [OB.xml] old size [null] increased to [0]...
Total Files: 1
2020-08-20 11:11:29,636 INFO c.t.i.s.s.LastModifiedLsEntryFileListFilter - [OB.xml] old size [0] increased to [38709]...
Total Files: 1
2020-08-20 11:11:34,256 INFO c.j.jsch - Connecting to *.*.*.115 port 22
2020-08-20 11:11:55,282 INFO c.t.i.c.f.a.LoggingAspect - log message Integration Name=CXML invoice DK Service Name=FileTransferService Source=External Source Interface=/home/umair/accruals/14minute/ Target=NA Target Interface=/home/umair/accruals/15minute/ Content ID=OB.xml Message ID=N/A Category=SUCCESS Timestamp=2020-08-20T11:11:55.281 Message=file OB.xml transferred
as per Artem Bilan suggestion, i have created a demo app and uploaded to git repo here
server configs are here
integration flow is defined here
README file
I found your problem:
#Bean
public Advice deleteFileAdvice() {
ExpressionEvaluatingRequestHandlerAdvice advice = new ExpressionEvaluatingRequestHandlerAdvice();
advice.setOnSuccessExpressionString(AppConstants.SUCCESS_EXPRESSION);
advice.setTrapException(true);
return advice;
}
See its JavaDocs:
/**
* If true, any exception will be caught and null returned.
* Default false.
* #param trapException true to trap Exceptions.
*/
public void setTrapException(boolean trapException) {
So, any exceptions from the message handler, including an SFTP conneciton failure is going to be swallowed. Therefore a request message is really moved to the next subscriber in your publishSubcribeChannel configuration.
I have flow that I want to implement JdbcMetadataStore in the filers, I used SimpleMetadataStore() but this is causing issues as it an in memory store, I need to use a shared metadata store, so I have Postgres DB installed and I can see that Jdbc supports it, I declared a bean as per documents to return a JdbcMetadataStore but I'm not sure how I can use this in the filters, tried to search a lot for any example but could not find a one, note that I'm using FileSystemPersistentAcceptOnceFileListFilter as well as my datasource for Postgres is all setup in my application properties. I have pasted my code here anyone can please guide me on how to move forward?
private DataSource dataSource;
public IntegrationFlow localToFtpFlow(Branch myBranch){
return IntegrationFlows.from(Files.inboundAdapter(new File(myBranch.getBranchCode()))
.filter(new ChainFileListFilter<File>()
.addFilter(new RegexPatternFileListFilter("final" + myBranch.getBranchCode() +".csv"))
.addFilter(new FileSystemPersistentAcceptOnceFileListFilter(new SimpleMetadataStore(), "foo"))),
e -> e.poller(Pollers.fixedDelay(10_000)))
.transform( p ->{
LOG1.info("Sending file " + p + " to FTP branch " + myBranch.getBranchCode());
return p;
})
.log()
.handle(Ftp.outboundAdapter(createNewFtpSessionFactory(myBranch),FileExistsMode.REPLACE)
.useTemporaryFileName(true)
.autoCreateDirectory(false)
.remoteDirectory(myBranch.getFolderPath()))
.get();
}
public DefaultFtpSessionFactory createNewFtpSessionFactory(Branch branch){
final DefaultFtpSessionFactory factory = new DefaultFtpSessionFactory();
factory.setHost(branch.getHost());
factory.setUsername(branch.getUsern());
factory.setPort(branch.getFtpPort());
factory.setPassword(branch.getPassword());
return factory;
}
#Bean
public MetadataStore metadataStore(final DataSource dataSource) {
return new JdbcMetadataStore(dataSource);
}
error arising before I created the table manually, but shouldn't it be created automatically in Postgres as the DB is supported:
org.springframework.messaging.MessagingException: nested exception is org.springframework.jdbc.BadSqlGrammarException: PreparedStatementCallback; bad SQL grammar [INSERT INTO INT_METADATA_STORE(METADATA_KEY, METADATA_VALUE, REGION) SELECT ?, ?, ? FROM INT_METADATA_STORE WHERE METADATA_KEY=? AND REGION=? HAVING COUNT(*)=0]; nested exception is org.postgresql.util.PSQLException: ERROR: relation "int_metadata_store" does not exist
Logging info regarding the issue when adding another flow for the second server, it will trigger the first flow handling method and sending the data to ftp server for both:
Saved Branch : BEY
Hibernate: select branch0_._id as _id1_0_0_, branch0_.branch_code as branch_c2_0_0_, branch0_.folder_path as folder_p3_0_0_, branch0_.ftp_port as ftp_port4_0_0_, branch0_.host as host5_0_0_, branch0_.password as password6_0_0_, branch0_.usern as usern7_0_0_ from branch branch0_ where branch0_._id=?
BEY
2019-01-07 15:11:25.816 INFO 12940 --- [nio-8081-exec-5] o.s.integration.channel.DirectChannel : Channel 'application.intermediateChannel' has 2 subscriber(s).
2019-01-07 15:11:25.817 INFO 12940 --- [nio-8081-exec-5] o.s.i.endpoint.EventDrivenConsumer : started 1.org.springframework.integration.config.ConsumerEndpointFactoryBean#1
2019-01-07 15:11:25.817 INFO 12940 --- [nio-8081-exec-5] o.s.i.endpoint.EventDrivenConsumer : Adding {transformer} as a subscriber to the '1.channel#0' channel
2019-01-07 15:11:25.817 INFO 12940 --- [nio-8081-exec-5] o.s.integration.channel.DirectChannel : Channel 'application.1.channel#0' has 1 subscriber(s).
2019-01-07 15:11:25.817 INFO 12940 --- [nio-8081-exec-5] o.s.i.endpoint.EventDrivenConsumer : started 1.org.springframework.integration.config.ConsumerEndpointFactoryBean#0
2019-01-07 15:11:25.829 INFO 12940 --- [nio-8081-exec-5] o.s.i.e.SourcePollingChannelAdapter : started stockInboundPoller
BEY
2019-01-07 15:11:25.984 INFO 12940 --- [nio-8081-exec-5] o.s.i.endpoint.EventDrivenConsumer : Adding {message-handler} as a subscriber to the '1o.channel#2' channel
2019-01-07 15:11:25.984 INFO 12940 --- [nio-8081-exec-5] o.s.integration.channel.DirectChannel : Channel 'application.1o.channel#2' has 1 subscriber(s).
2019-01-07 15:11:25.984 INFO 12940 --- [nio-8081-exec-5] o.s.i.endpoint.EventDrivenConsumer : started 1o.org.springframework.integration.config.ConsumerEndpointFactoryBean#1
2019-01-07 15:11:25.984 INFO 12940 --- [nio-8081-exec-5] o.s.i.endpoint.EventDrivenConsumer : Adding {transformer} as a subscriber to the '1o.channel#0' channel
2019-01-07 15:11:25.984 INFO 12940 --- [nio-8081-exec-5] o.s.integration.channel.DirectChannel : Channel 'application.1o.channel#0' has 1 subscriber(s).
2019-01-07 15:11:25.984 INFO 12940 --- [nio-8081-exec-5] o.s.i.endpoint.EventDrivenConsumer : started 1o.org.springframework.integration.config.ConsumerEndpointFactoryBean#0
2019-01-07 15:11:25.984 INFO 12940 --- [nio-8081-exec-5] o.s.i.e.SourcePollingChannelAdapter : started 1o.org.springframework.integration.config.SourcePollingChannelAdapterFactoryBean#0
2019-01-07 15:11:42.655 INFO 12940 --- [ask-scheduler-4] o.s.integration.ftp.session.FtpSession : File has been successfully transferred from: /ftp/erbranch/EDMS/FEFO/FEFOexportBEY.csv
Hibernate: select branch0_._id as _id1_0_, branch0_.branch_code as branch_c2_0_, branch0_.folder_path as folder_p3_0_, branch0_.ftp_port as ftp_port4_0_, branch0_.host as host5_0_, branch0_.password as password6_0_, branch0_.usern as usern7_0_ from branch branch0_
Hibernate: insert into branch (branch_code, folder_path, ftp_port, host, password, usern) values (?, ?, ?, ?, ?, ?)
Hibernate: select currval('branch__id_seq')
Saved Branch : JNB
Hibernate: select branch0_._id as _id1_0_0_, branch0_.branch_code as branch_c2_0_0_, branch0_.folder_path as folder_p3_0_0_, branch0_.ftp_port as ftp_port4_0_0_, branch0_.host as host5_0_0_, branch0_.password as password6_0_0_, branch0_.usern as usern7_0_0_ from branch branch0_ where branch0_._id=?
JNB
2019-01-07 15:13:36.099 INFO 12940 --- [nio-8081-exec-7] o.s.integration.channel.DirectChannel : Channel 'application.intermediateChannel' has 3 subscriber(s).
2019-01-07 15:13:36.099 INFO 12940 --- [nio-8081-exec-7] o.s.i.endpoint.EventDrivenConsumer : started 2.org.springframework.integration.config.ConsumerEndpointFactoryBean#1
2019-01-07 15:13:36.099 INFO 12940 --- [nio-8081-exec-7] o.s.i.endpoint.EventDrivenConsumer : Adding {transformer} as a subscriber to the '2.channel#0' channel
2019-01-07 15:13:36.099 INFO 12940 --- [nio-8081-exec-7] o.s.integration.channel.DirectChannel : Channel 'application.2.channel#0' has 1 subscriber(s).
2019-01-07 15:13:36.099 INFO 12940 --- [nio-8081-exec-7] o.s.i.endpoint.EventDrivenConsumer : started 2.org.springframework.integration.config.ConsumerEndpointFactoryBean#0
2019-01-07 15:13:36.099 INFO 12940 --- [nio-8081-exec-7] o.s.i.e.SourcePollingChannelAdapter : started stockInboundPoller
JNB
2019-01-07 15:13:36.130 INFO 12940 --- [nio-8081-exec-7] o.s.i.endpoint.EventDrivenConsumer : Adding {message-handler} as a subscriber to the '2o.channel#2' channel
2019-01-07 15:13:36.135 INFO 12940 --- [nio-8081-exec-7] o.s.integration.channel.DirectChannel : Channel 'application.2o.channel#2' has 1 subscriber(s).
2019-01-07 15:13:36.135 INFO 12940 --- [nio-8081-exec-7] o.s.i.endpoint.EventDrivenConsumer : started 2o.org.springframework.integration.config.ConsumerEndpointFactoryBean#1
2019-01-07 15:13:36.135 INFO 12940 --- [nio-8081-exec-7] o.s.i.endpoint.EventDrivenConsumer : Adding {transformer} as a subscriber to the '2o.channel#0' channel
2019-01-07 15:13:36.135 INFO 12940 --- [nio-8081-exec-7] o.s.integration.channel.DirectChannel : Channel 'application.2o.channel#0' has 1 subscriber(s).
2019-01-07 15:13:36.135 INFO 12940 --- [nio-8081-exec-7] o.s.i.endpoint.EventDrivenConsumer : started 2o.org.springframework.integration.config.ConsumerEndpointFactoryBean#0
2019-01-07 15:13:36.135 INFO 12940 --- [nio-8081-exec-7] o.s.i.e.SourcePollingChannelAdapter : started 2o.org.springframework.integration.config.SourcePollingChannelAdapterFactoryBean#0
2019-01-07 15:13:40.981 INFO 12940 --- [ask-scheduler-1] o.s.integration.ftp.session.FtpSession : File has been successfully transferred from: /ftp/erbranch/EDMS/FEFO/FEFOexportJNB.csv
2019-01-07 15:13:46.085 INFO 12940 --- [ask-scheduler-7] o.s.i.file.FileReadingMessageSource : Created message: [GenericMessage [payload=BEY\finalBEY.csv, headers={file_originalFile=BEY\finalBEY.csv, id=42a97889-7bfb-8f77-75d8-4e7988a368f9, file_name=finalBEY.csv, file_relativePath=finalBEY.csv, timestamp=1546866826085}]]
2019-01-07 15:13:46.086 INFO 12940 --- [ask-scheduler-7] o.s.integration.handler.LoggingHandler : GenericMessage [payload=BEY\finalBEY.csv, headers={file_originalFile=BEY\finalBEY.csv, id=108a92b0-db42-620e-1c46-90652a071220, file_name=finalBEY.csv, file_relativePath=finalBEY.csv, timestamp=1546866826086}]
2019-01-07 15:13:46.160 INFO 12940 --- [ask-scheduler-8] o.s.i.file.FileReadingMessageSource : Created message: [GenericMessage [payload=JNB\finalJNB.csv, headers={file_originalFile=JNB\finalJNB.csv, id=d3b2c6a0-2e9c-42a8-c224-0ed9cbbfaabb, file_name=finalJNB.csv, file_relativePath=finalJNB.csv, timestamp=1546866826160}]]
2019-01-07 15:13:46.161 INFO 12940 --- [ask-scheduler-8] o.s.integration.handler.LoggingHandler : GenericMessage [payload=JNB\finalJNB.csv, headers={file_originalFile=JNB\finalJNB.csv, id=e34070c2-e6ff-e5e1-8c64-4af697ab1032, file_name=finalJNB.csv, file_relativePath=finalJNB.csv, timestamp=1546866826161}]
2019-01-07 15:13:47.129 INFO 12940 --- [ask-scheduler-7] o.s.integration.ftp.session.FtpSession : File has been successfully transferred to: /ftp/erbranch/EDMS/FEFO/finalBEY.csv.writing
2019-01-07 15:13:47.534 INFO 12940 --- [ask-scheduler-7] o.s.integration.ftp.session.FtpSession : File has been successfully renamed from: /ftp/erbranch/EDMS/FEFO/finalBEY.csv.writing to /ftp/erbranch/EDMS/FEFO/finalBEY.csv
2019-01-07 15:13:49.772 INFO 12940 --- [ask-scheduler-8] o.s.integration.ftp.session.FtpSession : File has been successfully transferred to: /ftp/erbranch/EDMS/FEFO/finalJNB.csv.writing
2019-01-07 15:13:50.757 INFO 12940 --- [ask-scheduler-8] o.s.integration.ftp.session.FtpSession : File has been successfully renamed from: /ftp/erbranch/EDMS/FEFO/finalJNB.csv.writing to /ftp/erbranch/EDMS/FEFO/finalJNB.csv
You can find my app here
https://github.com/EliasKhattar/Spring-Integration-Project/tree/master/spring4ftpappftp
Just use metadataSource(dataSource) instead of new SimpleMetadataStore() in the filter constructor.
EDIT
I just copied your flow into a new app (made a few changes, but not to the filter) and everything works fine for me...
#SpringBootApplication
public class So54039852Application {
public static void main(String[] args) {
SpringApplication.run(So54039852Application.class, args);
}
#Bean
public IntegrationFlow localToFtpFlow(DataSource dataSource) {
return IntegrationFlows.from(Files.inboundAdapter(new File("/tmp/foo"))
.filter(new ChainFileListFilter<File>()
.addFilter(new RegexPatternFileListFilter(".*\\.csv"))
.addFilter(new FileSystemPersistentAcceptOnceFileListFilter(metadataStore(dataSource), "foo"))),
e -> e.poller(Pollers.fixedDelay(10_000)))
.log()
.get();
}
#Bean
public ConcurrentMetadataStore metadataStore(final DataSource dataSource) {
return new JdbcMetadataStore(dataSource);
}
}
and
$ touch /tmp/foo/foo.csv
...
$ touch /tmp/foo/bar.csv
and
2019-01-09 12:46:26.332 INFO 43329 --- [ask-scheduler-2] o.s.integration.handler.LoggingHandler : GenericMessage [payload=/tmp/foo/foo.csv, headers={file_originalFile=/tmp/foo/foo.csv, id=e0613529-a657-fbd3-5e67-8bb53a58b5ca, file_name=foo.csv, file_relativePath=foo.csv, timestamp=1547055986330}]
2019-01-09 12:47:26.487 INFO 43329 --- [ask-scheduler-5] o.s.integration.handler.LoggingHandler : GenericMessage [payload=/tmp/foo/bar.csv, headers={file_originalFile=/tmp/foo/bar.csv, id=4feb74b6-d711-f028-70c7-83cdfcd0aeec, file_name=bar.csv, file_relativePath=bar.csv, timestamp=1547056046487}]
and
mysql> select * from INT_METADATA_STORE;
+---------------------+----------------+---------+
| METADATA_KEY | METADATA_VALUE | REGION |
+---------------------+----------------+---------+
| foo/tmp/foo/bar.csv | 1547056039000 | DEFAULT |
| foo/tmp/foo/foo.csv | 1547055980000 | DEFAULT |
+---------------------+----------------+---------+
2 rows in set (0.00 sec)
I don't see the file again even after a restart, but if I change the date on one of the files...
$ touch /tmp/foo/bar.csv
and
2019-01-09 12:51:58.534 INFO 44430 --- [ask-scheduler-2] o.s.integration.handler.LoggingHandler : GenericMessage [payload=/tmp/foo/bar.csv, headers={file_originalFile=/tmp/foo/bar.csv, id=f92d6b36-c948-37cc-ca56-9ef28de336f2, file_name=bar.csv, file_relativePath=bar.csv, timestamp=1547056318532}]
I have created an application that uses Spring MVC to add ftp servers through a web page. When I add a server with details, it will start a flow polling a directory on the sever for specific csv file, once that file is there it will pull it to my local folder then a method is created to read that csv file and generate a new csv that will be sent back to the server, now this is accomplished, my issue is that when I add a new ftp server to poll and it finds a csv file on the second server, the application is pulling that file to my local as it should do but the handling method is being triggered again for both csv files while it should only do so for the second server since it is already triggered for the first server, so when I add the second server then again another file for server 1 generated and sent back to ftp and new one for the second server is send to the second ftp, at the end I have two files sent to first serve and one file to the second server, below is the coding and the output consol of the app, please assist on pinpointing my issue.
#Configuration
#EnableIntegration
#ComponentScan
public class FTIntegration {
public static final String TIMEZONE_UTC = "UTC";
public static final String TIMESTAMP_FORMAT_OF_FILES = "yyyyMMddHHmmssSSS";
public static final String TEMPORARY_FILE_SUFFIX = ".part";
public static final int POLLER_FIXED_PERIOD_DELAY = 5000;
public static final int MAX_MESSAGES_PER_POLL = 100;
//private static final Logger LOG = LoggerFactory.getLogger(FTIntegration.class);
private static final Logger LOG1 = Logger.getLogger(FTIntegration.class);
private static final String CHANNEL_INTERMEDIATE_STAGE = "intermediateChannel";
private static final String OUTBOUND_CHANNEL = "outboundChannel";
/* pulling the server config from postgres DB*/
private final BranchRepository branchRepository;
#Autowired
private CSVToCSVNoQ csvToCSVNoQ;
#Value("${app.temp-dir}")
private String localTempPath;
public FTIntegration(BranchRepository branchRepository) {
this.branchRepository = branchRepository;
}
#Bean
public Branch myBranch(){
return new Branch();
}
/**
* The default poller with 5s, 100 messages, RotatingServerAdvice and transaction.
*
* #return default poller.
*/
#Bean(name = PollerMetadata.DEFAULT_POLLER)
public PollerMetadata poller(){
return Pollers
.fixedDelay(POLLER_FIXED_PERIOD_DELAY)
.maxMessagesPerPoll(MAX_MESSAGES_PER_POLL)
.transactional()
.get();
}
/**
* The direct channel for the flow.
*
* #return MessageChannel
*/
#Bean
public MessageChannel stockIntermediateChannel() {
return new DirectChannel();
}
/**
* Get the files from a remote directory. Add a timestamp to the filename
* and write them to a local temporary folder.
*
* #return IntegrationFlow
*/
#Bean
public PropertiesPersistingMetadataStore store() {
PropertiesPersistingMetadataStore store = new PropertiesPersistingMetadataStore();
return store;
}
public IntegrationFlow fileInboundFlowFromFTPServer(Branch myBranch) throws IOException {
final FtpInboundChannelAdapterSpec sourceSpecFtp = Ftp.inboundAdapter(createNewFtpSessionFactory(myBranch))
.preserveTimestamp(true)
//.patternFilter("*.csv")
.maxFetchSize(MAX_MESSAGES_PER_POLL)
.remoteDirectory(myBranch.getFolderPath())
.regexFilter("FEFOexport"+myBranch.getBranchCode()+".csv")
.deleteRemoteFiles(true)
.localDirectory(new File(myBranch.getBranchCode()))
.temporaryFileSuffix(TEMPORARY_FILE_SUFFIX)
/*.localFilenameExpression(new FunctionExpression<String>(s -> {
final int fileTypeSepPos = s.lastIndexOf('.');
return DateTimeFormatter
.ofPattern(TIMESTAMP_FORMAT_OF_FILES)
.withZone(ZoneId.of(TIMEZONE_UTC))
.format(Instant.now())
+ "_"
+ s.substring(0,fileTypeSepPos)
+ s.substring(fileTypeSepPos);
}))*/;
// Poller definition
final Consumer<SourcePollingChannelAdapterSpec> stockInboundPoller = endpointConfigurer -> endpointConfigurer
.id("stockInboundPoller")
.autoStartup(true)
.poller(poller());
IntegrationFlow flow = IntegrationFlows
.from(sourceSpecFtp, stockInboundPoller)
.transform(File.class, p ->{
// log step
LOG1.info("flow=stockInboundFlowFromAFT, message=incoming file: " + p);
return p;
})
.channel(CHANNEL_INTERMEDIATE_STAGE)
.handle(m -> {
try {
this.csvToCSVNoQ.writeCSVfinal("test", myBranch.getBranchCode() + "/final" + myBranch.getBranchCode() + ".csv", myBranch.getBranchCode() + "/FEFOexport" + myBranch.getBranchCode() + ".csv");
LOG1.info("Writing final file .csv " + m);
} catch (IOException e) {
e.printStackTrace();
}
})
//.handle(m -> this.csvToCSVNoQ.writeCSVfinal(m.getPayload(),m.getHeaders().get("csv", "FEFOexport"+myBranch.getBranchCode()+".csv")))
.get();
return flow;
}
#Bean
public IntegrationFlow stockIntermediateStageChannel() {
IntegrationFlow flow = IntegrationFlows
.from(CHANNEL_INTERMEDIATE_STAGE)
.transform(p -> {
//log step
LOG1.info("flow=stockIntermediateStageChannel, message=rename file: " + p);
return p;
})
//TODO
.channel(new NullChannel())
.get();
return flow;
}
/*
* Creating the outbound adaptor to send files from local to FTP server
*
* */
public IntegrationFlow localToFtpFlow(Branch myBranch){
return IntegrationFlows.from(Files.inboundAdapter(new File(myBranch.getBranchCode()))
.filter(new ChainFileListFilter<File>()
.addFilter(new RegexPatternFileListFilter("final" + myBranch.getBranchCode() +".csv"))
.addFilter(new FileSystemPersistentAcceptOnceFileListFilter(new SimpleMetadataStore(), "foo"))),
e -> e.poller(Pollers.fixedDelay(10_000)))
.transform( p ->{
LOG1.info("Sending file " + p + " to FTP branch " + myBranch.getBranchCode());
return p;
})
.log()
.handle(Ftp.outboundAdapter(createNewFtpSessionFactory(myBranch),FileExistsMode.REPLACE)
.useTemporaryFileName(true)
.autoCreateDirectory(false)
.remoteDirectory(myBranch.getFolderPath()))
.get();
}
public DefaultFtpSessionFactory createNewFtpSessionFactory(Branch branch){
final DefaultFtpSessionFactory factory = new DefaultFtpSessionFactory();
factory.setHost(branch.getHost());
factory.setUsername(branch.getUsern());
factory.setPort(branch.getFtpPort());
factory.setPassword(branch.getPassword());
return factory;
}
Console Output:
Saved Branch : BEY
Hibernate: select branch0_._id as _id1_0_0_, branch0_.branch_code as branch_c2_0_0_, branch0_.folder_path as folder_p3_0_0_, branch0_.ftp_port as ftp_port4_0_0_, branch0_.host as host5_0_0_, branch0_.password as password6_0_0_, branch0_.usern as usern7_0_0_ from branch branch0_ where branch0_._id=?
2018-12-20 07:58:39.218 INFO 6668 --- [nio-8081-exec-5] o.s.integration.channel.DirectChannel : Channel 'application.intermediateChannel' has 2 subscriber(s).
2018-12-20 07:58:39.218 INFO 6668 --- [nio-8081-exec-5] o.s.i.endpoint.EventDrivenConsumer : started 1.org.springframework.integration.config.ConsumerEndpointFactoryBean#1
2018-12-20 07:58:39.218 INFO 6668 --- [nio-8081-exec-5] o.s.i.endpoint.EventDrivenConsumer : Adding {transformer} as a subscriber to the '1.channel#0' channel
2018-12-20 07:58:39.218 INFO 6668 --- [nio-8081-exec-5] o.s.integration.channel.DirectChannel : Channel 'application.1.channel#0' has 1 subscriber(s).
2018-12-20 07:58:39.218 INFO 6668 --- [nio-8081-exec-5] o.s.i.endpoint.EventDrivenConsumer : started 1.org.springframework.integration.config.ConsumerEndpointFactoryBean#0
2018-12-20 07:58:39.229 INFO 6668 --- [nio-8081-exec-5] o.s.i.e.SourcePollingChannelAdapter : started stockInboundPoller
2018-12-20 07:58:39.417 INFO 6668 --- [nio-8081-exec-5] o.s.i.endpoint.EventDrivenConsumer : Adding {message-handler} as a subscriber to the '1o.channel#2' channel
2018-12-20 07:58:39.417 INFO 6668 --- [nio-8081-exec-5] o.s.integration.channel.DirectChannel : Channel 'application.1o.channel#2' has 1 subscriber(s).
2018-12-20 07:58:39.418 INFO 6668 --- [nio-8081-exec-5] o.s.i.endpoint.EventDrivenConsumer : started 1o.org.springframework.integration.config.ConsumerEndpointFactoryBean#1
2018-12-20 07:58:39.418 INFO 6668 --- [nio-8081-exec-5] o.s.i.endpoint.EventDrivenConsumer : Adding {transformer} as a subscriber to the '1o.channel#0' channel
2018-12-20 07:58:39.418 INFO 6668 --- [nio-8081-exec-5] o.s.integration.channel.DirectChannel : Channel 'application.1o.channel#0' has 1 subscriber(s).
2018-12-20 07:58:39.418 INFO 6668 --- [nio-8081-exec-5] o.s.i.endpoint.EventDrivenConsumer : started 1o.org.springframework.integration.config.ConsumerEndpointFactoryBean#0
2018-12-20 07:58:39.419 INFO 6668 --- [nio-8081-exec-5] o.s.i.e.SourcePollingChannelAdapter : started 1o.org.springframework.integration.config.SourcePollingChannelAdapterFactoryBean#0
2018-12-20 07:59:41.165 INFO 6668 --- [sk-scheduler-10] o.s.integration.ftp.session.FtpSession : File has been successfully transferred from: /ftp/erbranch/EDMS/FEFO/FEFOexportBEY.csv
2018-12-20 07:59:49.446 INFO 6668 --- [ask-scheduler-1] o.s.i.file.FileReadingMessageSource : Created message: [GenericMessage [payload=BEY\finalBEY.csv, headers={file_originalFile=BEY\finalBEY.csv, id=7f0cccb7-a070-bd4c-d468-977a265ceb2e, file_name=finalBEY.csv, file_relativePath=finalBEY.csv, timestamp=1545285589446}]]
2018-12-20 07:59:49.448 INFO 6668 --- [ask-scheduler-1] o.s.integration.handler.LoggingHandler : GenericMessage [payload=BEY\finalBEY.csv, headers={file_originalFile=BEY\finalBEY.csv, id=d857130a-d4a0-eaeb-19ea-f819924d94e2, file_name=finalBEY.csv, file_relativePath=finalBEY.csv, timestamp=1545285589447}]
2018-12-20 07:59:50.488 INFO 6668 --- [ask-scheduler-1] o.s.integration.ftp.session.FtpSession : File has been successfully transferred to: /ftp/erbranch/EDMS/FEFO/finalBEY.csv.writing
2018-12-20 07:59:50.899 INFO 6668 --- [ask-scheduler-1] o.s.integration.ftp.session.FtpSession : File has been successfully renamed from: /ftp/erbranch/EDMS/FEFO/finalBEY.csv.writing to /ftp/erbranch/EDMS/FEFO/finalBEY.csv
Hibernate: select branch0_._id as _id1_0_, branch0_.branch_code as branch_c2_0_, branch0_.folder_path as folder_p3_0_, branch0_.ftp_port as ftp_port4_0_, branch0_.host as host5_0_, branch0_.password as password6_0_, branch0_.usern as usern7_0_ from branch branch0_
Hibernate: insert into branch (branch_code, folder_path, ftp_port, host, password, usern) values (?, ?, ?, ?, ?, ?)
Hibernate: select currval('branch__id_seq')
Saved Branch : JNB
Hibernate: select branch0_._id as _id1_0_0_, branch0_.branch_code as branch_c2_0_0_, branch0_.folder_path as folder_p3_0_0_, branch0_.ftp_port as ftp_port4_0_0_, branch0_.host as host5_0_0_, branch0_.password as password6_0_0_, branch0_.usern as usern7_0_0_ from branch branch0_ where branch0_._id=?
2018-12-20 08:02:24.966 INFO 6668 --- [nio-8081-exec-8] o.s.integration.channel.DirectChannel : Channel 'application.intermediateChannel' has 3 subscriber(s).
2018-12-20 08:02:24.966 INFO 6668 --- [nio-8081-exec-8] o.s.i.endpoint.EventDrivenConsumer : started 2.org.springframework.integration.config.ConsumerEndpointFactoryBean#1
2018-12-20 08:02:24.966 INFO 6668 --- [nio-8081-exec-8] o.s.i.endpoint.EventDrivenConsumer : Adding {transformer} as a subscriber to the '2.channel#0' channel
2018-12-20 08:02:24.966 INFO 6668 --- [nio-8081-exec-8] o.s.integration.channel.DirectChannel : Channel 'application.2.channel#0' has 1 subscriber(s).
2018-12-20 08:02:24.966 INFO 6668 --- [nio-8081-exec-8] o.s.i.endpoint.EventDrivenConsumer : started 2.org.springframework.integration.config.ConsumerEndpointFactoryBean#0
2018-12-20 08:02:24.966 INFO 6668 --- [nio-8081-exec-8] o.s.i.e.SourcePollingChannelAdapter : started stockInboundPoller
2018-12-20 08:02:24.992 INFO 6668 --- [nio-8081-exec-8] o.s.i.endpoint.EventDrivenConsumer : Adding {message-handler} as a subscriber to the '2o.channel#2' channel
2018-12-20 08:02:24.992 INFO 6668 --- [nio-8081-exec-8] o.s.integration.channel.DirectChannel : Channel 'application.2o.channel#2' has 1 subscriber(s).
2018-12-20 08:02:24.992 INFO 6668 --- [nio-8081-exec-8] o.s.i.endpoint.EventDrivenConsumer : started 2o.org.springframework.integration.config.ConsumerEndpointFactoryBean#1
2018-12-20 08:02:24.992 INFO 6668 --- [nio-8081-exec-8] o.s.i.endpoint.EventDrivenConsumer : Adding {transformer} as a subscriber to the '2o.channel#0' channel
2018-12-20 08:02:24.992 INFO 6668 --- [nio-8081-exec-8] o.s.integration.channel.DirectChannel : Channel 'application.2o.channel#0' has 1 subscriber(s).
2018-12-20 08:02:24.992 INFO 6668 --- [nio-8081-exec-8] o.s.i.endpoint.EventDrivenConsumer : started 2o.org.springframework.integration.config.ConsumerEndpointFactoryBean#0
2018-12-20 08:02:24.992 INFO 6668 --- [nio-8081-exec-8] o.s.i.e.SourcePollingChannelAdapter : started 2o.org.springframework.integration.config.SourcePollingChannelAdapterFactoryBean#0
Hibernate: select branch0_._id as _id1_0_, branch0_.branch_code as branch_c2_0_, branch0_.folder_path as folder_p3_0_, branch0_.ftp_port as ftp_port4_0_, branch0_.host as host5_0_, branch0_.password as password6_0_, branch0_.usern as usern7_0_ from branch branch0_
2018-12-20 08:03:00.225 INFO 6668 --- [ask-scheduler-8] o.s.integration.ftp.session.FtpSession : File has been successfully transferred from: /ftp/erbranch/EDMS/FEFO/FEFOexportJNB.csv
2018-12-20 08:03:00.929 INFO 6668 --- [ask-scheduler-5] o.s.i.file.FileReadingMessageSource : Created message: [GenericMessage [payload=BEY\finalBEY.csv, headers={file_originalFile=BEY\finalBEY.csv, id=6ed554eb-f553-0293-f042-d633155357c0, file_name=finalBEY.csv, file_relativePath=finalBEY.csv, timestamp=1545285780929}]]
2018-12-20 08:03:00.930 INFO 6668 --- [ask-scheduler-5] o.s.integration.handler.LoggingHandler : GenericMessage [payload=BEY\finalBEY.csv, headers={file_originalFile=BEY\finalBEY.csv, id=b2d76ac5-fb85-1313-a37e-8849714a545e, file_name=finalBEY.csv, file_relativePath=finalBEY.csv, timestamp=1545285780930}]
2018-12-20 08:03:01.958 INFO 6668 --- [ask-scheduler-5] o.s.integration.ftp.session.FtpSession : File has been successfully transferred to: /ftp/erbranch/EDMS/FEFO/finalBEY.csv.writing
2018-12-20 08:03:02.373 INFO 6668 --- [ask-scheduler-5] o.s.integration.ftp.session.FtpSession : File has been successfully renamed from: /ftp/erbranch/EDMS/FEFO/finalBEY.csv.writing to /ftp/erbranch/EDMS/FEFO/finalBEY.csv
2018-12-20 08:03:05.033 INFO 6668 --- [ask-scheduler-7] o.s.i.file.FileReadingMessageSource : Created message: [GenericMessage [payload=JNB\finalJNB.csv, headers={file_originalFile=JNB\finalJNB.csv, id=4514e132-5684-9e82-28e7-f75c5c3dcf91, file_name=finalJNB.csv, file_relativePath=finalJNB.csv, timestamp=1545285785033}]]
2018-12-20 08:03:05.034 INFO 6668 --- [ask-scheduler-7] o.s.integration.handler.LoggingHandler : GenericMessage [payload=JNB\finalJNB.csv, headers={file_originalFile=JNB\finalJNB.csv, id=59e62375-f1da-461d-ee61-d105ac3159a0, file_name=finalJNB.csv, file_relativePath=finalJNB.csv, timestamp=1545285785034}]
2018-12-20 08:03:07.530 INFO 6668 --- [ask-scheduler-7] o.s.integration.ftp.session.FtpSession : File has been successfully transferred to: /ftp/erbranch/EDMS/FEFO/finalJNB.csv.writing
2018-12-20 08:03:08.539 INFO 6668 --- [ask-scheduler-7] o.s.integration.ftp.session.FtpSession : File has been successfully renamed from: /ftp/erbranch/EDMS/FEFO/finalJNB.csv.writing to /ftp/erbranch/EDMS/FEFO/finalJNB.csv
.addFilter(new FileSystemPersistentAcceptOnceFileListFilter(new SimpleMetadataStore(), "foo"))),
The SimpleMetadataStore() is in-memory - using a new store for each flow means it starts empty each time - you need to use a shared metadata store across all the flows.
Also, if you need to prevent duplicates across server restarts, you need to use a persistent shared store, e.g. Redis, JDBC etc.
Can I have a job with just the slaves and no master and listen to a rabbitmq queue? I want to listen to a queue and process the messages in chunk oriented manner using spring batch and spring integration in a spring boot app.
I want to use the chunkProcessorChunkHandler configuration explained in the RemoteChunking example for Spring batch by Michael Minella (https://www.youtube.com/watch?v=30Tdp1mfR0g), but without a master configuration.
Below is my configuration for the job.
#Configuration
#EnableIntegration
public class QueueIntegrationConfiguration {
#Autowired
private CassandraItemWriter cassandraItemWriter;
#Autowired
private VendorProcessor vendorProcessor;
#Autowired
ConnectionFactory connectionFactory;
#Bean
public AmqpInboundChannelAdapter inboundChannelAdapter(
SimpleMessageListenerContainer listenerContainer) {
AmqpInboundChannelAdapter adapter = new AmqpInboundChannelAdapter(listenerContainer);
adapter.setOutputChannel(inboundQueueChannel());
adapter.setAutoStartup(true);
return adapter;
}
#Bean
public SimpleMessageListenerContainer listenerContainer(ConnectionFactory connectionFactory, MessageConverter jsonMessageConverter) {
SimpleMessageListenerContainer container = new SimpleMessageListenerContainer(
connectionFactory);
container.setQueueNames("ProductStore_Partial");
container.setAutoStartup(true);
container.setMessageConverter(jsonMessageConverter);
return container;
}
#Bean
#ServiceActivator(inputChannel = "ProductStore_Partial")
public ChunkProcessorChunkHandler chunkProcessorChunkHandler()
throws Exception {
SimpleChunkProcessor chunkProcessor = new SimpleChunkProcessor(vendorProcessor,
cassandraItemWriter);
chunkProcessor.afterPropertiesSet();
ChunkProcessorChunkHandler<Vendor> chunkHandler = new ChunkProcessorChunkHandler<>();
chunkHandler.setChunkProcessor(chunkProcessor);
chunkHandler.afterPropertiesSet();
return chunkHandler;
}
#Bean
public QueueChannel inboundQueueChannel() {
return new QueueChannel().;
}
}
Below is my Application.java class for spring boot.
#SpringBootApplication
#EnableBatchProcessing
public class BulkImportProductApplication {
public static void main(String[] args) {
SpringApplication app = new SpringApplication(BulkImportProductApplication.class);
app.setWebEnvironment(false);
app.run(args).close();
}
}
From what I understand from spring integration, I have an AmqpInboundChannelAdapter for listening to messages from the queue. A ServiceActivator, an inboundQueueChannel, autowired ItemProcessor and ItemWriter. I am not sure what am I missing here.
The batch job starts, consumes one message from the queue and get a cancelOk and my job terminates without processing the message.
I am also sharing my debug logging if that would help.
2017-12-04 09:58:49.679 INFO 7450 --- [ main] c.a.s.p.b.BulkImportProductApplication : Started BulkImportProductApplication in 9.412 seconds (JVM running for 10.39)
2017-12-04 09:58:49.679 INFO 7450 --- [ main] s.c.a.AnnotationConfigApplicationContext : Closing org.springframework.context.annotation.AnnotationConfigApplicationContext#31c88ec8: startup date [Mon Dec 04 09:58:40 PST 2017]; root of context hierarchy
2017-12-04 09:58:49.679 DEBUG 7450 --- [ main] o.s.b.f.s.DefaultListableBeanFactory : Returning cached instance of singleton bean 'org.springframework.integration.config.IdGeneratorConfigurer#0'
2017-12-04 09:58:49.680 DEBUG 7450 --- [ main] o.s.b.f.s.DefaultListableBeanFactory : Returning cached instance of singleton bean 'inboundChannelAdapter'
2017-12-04 09:58:49.680 DEBUG 7450 --- [ main] o.s.b.f.s.DefaultListableBeanFactory : Returning cached instance of singleton bean 'listenerContainer'
2017-12-04 09:58:49.680 DEBUG 7450 --- [ main] o.s.b.f.s.DefaultListableBeanFactory : Returning cached instance of singleton bean 'integrationHeaderChannelRegistry'
2017-12-04 09:58:49.680 DEBUG 7450 --- [ main] o.s.b.f.s.DefaultListableBeanFactory : Returning cached instance of singleton bean 'org.springframework.amqp.rabbit.config.internalRabbitListenerEndpointRegistry'
2017-12-04 09:58:49.680 DEBUG 7450 --- [ main] o.s.b.f.s.DefaultListableBeanFactory : Returning cached instance of singleton bean '_org.springframework.integration.errorLogger'
2017-12-04 09:58:49.680 DEBUG 7450 --- [ main] o.s.b.f.s.DefaultListableBeanFactory : Returning cached instance of singleton bean 'queueIntegrationConfiguration.chunkProcessorChunkHandler.serviceActivator.handler'
2017-12-04 09:58:49.680 DEBUG 7450 --- [ main] o.s.b.f.s.DefaultListableBeanFactory : Returning cached instance of singleton bean 'queueIntegrationConfiguration.chunkProcessorChunkHandler.serviceActivator'
2017-12-04 09:58:49.680 DEBUG 7450 --- [ main] o.s.b.f.s.DefaultListableBeanFactory : Returning cached instance of singleton bean 'lifecycleProcessor'
2017-12-04 09:58:49.680 INFO 7450 --- [ main] o.s.c.support.DefaultLifecycleProcessor : Stopping beans in phase 2147483647
2017-12-04 09:58:49.680 DEBUG 7450 --- [ main] o.s.c.support.DefaultLifecycleProcessor : Asking bean 'inboundChannelAdapter' of type [class org.springframework.integration.amqp.inbound.AmqpInboundChannelAdapter] to stop
2017-12-04 09:58:49.680 DEBUG 7450 --- [ main] o.s.a.r.l.SimpleMessageListenerContainer : Shutting down Rabbit listener container
2017-12-04 09:58:49.814 DEBUG 7450 --- [pool-1-thread-5] o.s.a.r.listener.BlockingQueueConsumer : Storing delivery for Consumer#7c52fc81: tags=[{}], channel=Cached Rabbit Channel: AMQChannel(amqp://admin#xxxx:5672/,2), conn: Proxy#26f1249d Shared Rabbit Connection: SimpleConnection#680bddf5 [delegate=amqp://admin#xxxx:5672/, localPort= 65035], acknowledgeMode=AUTO local queue size=0
2017-12-04 09:58:49.814 DEBUG 7450 --- [enerContainer-1] o.s.a.r.listener.BlockingQueueConsumer : Received message: (Body:'[B#358a5358(byte[618])' MessageProperties [headers={__TypeId__=com.art.service.product.bulkimportproduct.data.model.Vendor}, timestamp=null, messageId=null, userId=null, receivedUserId=null, appId=null, clusterId=null, type=null, correlationId=null, correlationIdString=null, replyTo=null, contentType=json, contentEncoding=UTF-8, contentLength=0, deliveryMode=null, receivedDeliveryMode=NON_PERSISTENT, expiration=null, priority=0, redelivered=false, receivedExchange=ProductStore, receivedRoutingKey=, receivedDelay=null, deliveryTag=2, messageCount=0, consumerTag=amq.ctag-nWGbRxjFiaeTEoZylv6Hrg, consumerQueue=null])
2017-12-04 09:58:49.815 DEBUG 7450 --- [enerContainer-1] s.i.m.AbstractHeaderMapper$HeaderMatcher : headerName=[amqp_receivedDeliveryMode] WILL be mapped, matched pattern=*
2017-12-04 09:58:49.815 DEBUG 7450 --- [enerContainer-1] s.i.m.AbstractHeaderMapper$HeaderMatcher : headerName=[amqp_contentEncoding] WILL be mapped, matched pattern=*
2017-12-04 09:58:49.815 DEBUG 7450 --- [enerContainer-1] s.i.m.AbstractHeaderMapper$HeaderMatcher : headerName=[amqp_receivedExchange] WILL be mapped, matched pattern=*
2017-12-04 09:58:49.815 DEBUG 7450 --- [enerContainer-1] s.i.m.AbstractHeaderMapper$HeaderMatcher : headerName=[amqp_deliveryTag] WILL be mapped, matched pattern=*
2017-12-04 09:58:49.815 DEBUG 7450 --- [enerContainer-1] s.i.m.AbstractHeaderMapper$HeaderMatcher : headerName=[json__TypeId__] WILL be mapped, matched pattern=*
2017-12-04 09:58:49.815 DEBUG 7450 --- [enerContainer-1] s.i.m.AbstractHeaderMapper$HeaderMatcher : headerName=[amqp_redelivered] WILL be mapped, matched pattern=*
2017-12-04 09:58:49.815 DEBUG 7450 --- [enerContainer-1] s.i.m.AbstractHeaderMapper$HeaderMatcher : headerName=[contentType] WILL be mapped, matched pattern=*
2017-12-04 09:58:49.815 DEBUG 7450 --- [enerContainer-1] s.i.m.AbstractHeaderMapper$HeaderMatcher : headerName=[__TypeId__] WILL be mapped, matched pattern=*
2017-12-04 09:58:49.815 DEBUG 7450 --- [enerContainer-1] o.s.integration.channel.QueueChannel : preSend on channel 'inboundQueueChannel', message: GenericMessage [payload=byte[618], headers={amqp_receivedDeliveryMode=NON_PERSISTENT, amqp_contentEncoding=UTF-8, amqp_receivedExchange=ProductStore, amqp_deliveryTag=2, json__TypeId__=com.art.service.product.bulkimportproduct.data.model.Vendor, amqp_redelivered=false, id=a4868670-240f-ddf2-8a8c-ac4b8d234cdd, amqp_consumerTag=amq.ctag-nWGbRxjFiaeTEoZylv6Hrg, contentType=json, __TypeId__=com.art.service.product.bulkimportproduct.data.model.Vendor, timestamp=1512410329815}]
2017-12-04 09:58:49.815 DEBUG 7450 --- [enerContainer-1] o.s.integration.channel.QueueChannel : postSend (sent=true) on channel 'inboundQueueChannel', message: GenericMessage [payload=byte[618], headers={amqp_receivedDeliveryMode=NON_PERSISTENT, amqp_contentEncoding=UTF-8, amqp_receivedExchange=ProductStore, amqp_deliveryTag=2, json__TypeId__=com.art.service.product.bulkimportproduct.data.model.Vendor, amqp_redelivered=false, id=a4868670-240f-ddf2-8a8c-ac4b8d234cdd, amqp_consumerTag=amq.ctag-nWGbRxjFiaeTEoZylv6Hrg, contentType=json, __TypeId__=com.art.service.product.bulkimportproduct.data.model.Vendor, timestamp=1512410329815}]
2017-12-04 09:58:49.853 INFO 7450 --- [ main] o.s.a.r.l.SimpleMessageListenerContainer : Waiting for workers to finish.
2017-12-04 09:58:49.853 DEBUG 7450 --- [pool-1-thread-6] o.s.a.r.listener.BlockingQueueConsumer : Received cancelOk for tag amq.ctag-nWGbRxjFiaeTEoZylv6Hrg (null); Consumer#7c52fc81: tags=[{}], channel=Cached Rabbit Channel: AMQChannel(amqp://admin#xxxx:5672/,2), conn: Proxy#26f1249d Shared Rabbit Connection: SimpleConnection#680bddf5 [delegate=amqp://admin#xxxx:5672/, localPort= 65035], acknowledgeMode=AUTO local queue size=0
2017-12-04 09:58:49.853 DEBUG 7450 --- [enerContainer-1] o.s.a.r.l.SimpleMessageListenerContainer : Cancelling Consumer#7c52fc81: tags=[{}], channel=Cached Rabbit Channel: AMQChannel(amqp://admin#xxxx:5672/,2), conn: Proxy#26f1249d Shared Rabbit Connection: SimpleConnection#680bddf5 [delegate=amqp://admin#xxxx:5672/, localPort= 65035], acknowledgeMode=AUTO local queue size=0
2017-12-04 09:58:49.853 DEBUG 7450 --- [enerContainer-1] o.s.a.r.listener.BlockingQueueConsumer : Closing Rabbit Channel: Cached Rabbit Channel: AMQChannel(amqp://admin#xxxx:5672/,2), conn: Proxy#26f1249d Shared Rabbit Connection: SimpleConnection#680bddf5 [delegate=amqp://admin#xxxx:5672/, localPort= 65035]
2017-12-04 09:58:49.853 DEBUG 7450 --- [enerContainer-1] o.s.a.r.c.CachingConnectionFactory : Closing cached Channel: AMQChannel(amqp://admin#xxxx:5672/,2)
2017-12-04 09:58:50.027 INFO 7450 --- [ main] o.s.a.r.l.SimpleMessageListenerContainer : Successfully waited for workers to finish.
2017-12-04 09:58:50.027 DEBUG 7450 --- [ main] o.s.c.support.DefaultLifecycleProcessor : Bean 'inboundChannelAdapter' completed its stop procedure
What am I missing here? Why is my message getting processed? Please correct me if I'm missing out something here? Also feel free to ask any other configuration that you feel would help analyze the situation here.
EDIT: After removing the code that closes the application context manually( app.run(args).close() ), I was able to receive the messages, but looks like they are lost after a successful retrieve. sharing the debug log for this behavior.
2017-12-04 14:39:11.297 DEBUG 1498 --- [pool-1-thread-5] o.s.a.r.listener.BlockingQueueConsumer : Storing delivery for Consumer#7219ac49: tags=[{amq.ctag-Z8siptJMdxGU6sXdOHkVCA=ProductStore_Partial}], channel=Cached Rabbit Channel: AMQChannel(amqp://admin#xxxx:5672/,2), conn: Proxy#6df20ade Shared Rabbit Connection: SimpleConnection#7ba63fe5 [delegate=amqp://admin#xxxx:5672/, localPort= 51172], acknowledgeMode=AUTO local queue size=0
2017-12-04 14:39:11.297 DEBUG 1498 --- [enerContainer-1] o.s.a.r.listener.BlockingQueueConsumer : Received message: (Body:'[B#347c8f87(byte[624])' MessageProperties [headers={__TypeId__=com.art.service.product.bulkimportproduct.data.model.Vendor}, timestamp=null, messageId=null, userId=null, receivedUserId=null, appId=null, clusterId=null, type=null, correlationId=null, correlationIdString=null, replyTo=null, contentType=json, contentEncoding=UTF-8, contentLength=0, deliveryMode=null, receivedDeliveryMode=NON_PERSISTENT, expiration=null, priority=0, redelivered=false, receivedExchange=ProductStore, receivedRoutingKey=, receivedDelay=null, deliveryTag=2, messageCount=0, consumerTag=amq.ctag-Z8siptJMdxGU6sXdOHkVCA, consumerQueue=ProductStore_Partial])
2017-12-04 14:39:11.297 DEBUG 1498 --- [enerContainer-1] s.i.m.AbstractHeaderMapper$HeaderMatcher : headerName=[amqp_receivedDeliveryMode] WILL be mapped, matched pattern=*
2017-12-04 14:39:11.297 DEBUG 1498 --- [enerContainer-1] s.i.m.AbstractHeaderMapper$HeaderMatcher : headerName=[amqp_contentEncoding] WILL be mapped, matched pattern=*
2017-12-04 14:39:11.297 DEBUG 1498 --- [enerContainer-1] s.i.m.AbstractHeaderMapper$HeaderMatcher : headerName=[amqp_receivedExchange] WILL be mapped, matched pattern=*
2017-12-04 14:39:11.297 DEBUG 1498 --- [enerContainer-1] s.i.m.AbstractHeaderMapper$HeaderMatcher : headerName=[amqp_deliveryTag] WILL be mapped, matched pattern=*
2017-12-04 14:39:11.297 DEBUG 1498 --- [enerContainer-1] s.i.m.AbstractHeaderMapper$HeaderMatcher : headerName=[json__TypeId__] WILL be mapped, matched pattern=*
2017-12-04 14:39:11.297 DEBUG 1498 --- [enerContainer-1] s.i.m.AbstractHeaderMapper$HeaderMatcher : headerName=[amqp_redelivered] WILL be mapped, matched pattern=*
2017-12-04 14:39:11.297 DEBUG 1498 --- [enerContainer-1] s.i.m.AbstractHeaderMapper$HeaderMatcher : headerName=[contentType] WILL be mapped, matched pattern=*
2017-12-04 14:39:11.297 DEBUG 1498 --- [enerContainer-1] s.i.m.AbstractHeaderMapper$HeaderMatcher : headerName=[__TypeId__] WILL be mapped, matched pattern=*
2017-12-04 14:39:11.297 DEBUG 1498 --- [enerContainer-1] o.s.integration.channel.QueueChannel : preSend on channel 'inboundQueueChannel', message: GenericMessage [payload=byte[624], headers={amqp_receivedDeliveryMode=NON_PERSISTENT, amqp_contentEncoding=UTF-8, amqp_receivedExchange=ProductStore, amqp_deliveryTag=2, json__TypeId__=com.art.service.product.bulkimportproduct.data.model.Vendor, amqp_consumerQueue=ProductStore_Partial, amqp_redelivered=false, id=540399a5-62a6-7178-2524-e274bad4ed13, amqp_consumerTag=amq.ctag-Z8siptJMdxGU6sXdOHkVCA, contentType=json, __TypeId__=com.art.service.product.bulkimportproduct.data.model.Vendor, timestamp=1512427151297}]
2017-12-04 14:39:11.297 DEBUG 1498 --- [enerContainer-1] o.s.integration.channel.QueueChannel : postSend (sent=true) on channel 'inboundQueueChannel', message: GenericMessage [payload=byte[624], headers={amqp_receivedDeliveryMode=NON_PERSISTENT, amqp_contentEncoding=UTF-8, amqp_receivedExchange=ProductStore, amqp_deliveryTag=2, json__TypeId__=com.art.service.product.bulkimportproduct.data.model.Vendor, amqp_consumerQueue=ProductStore_Partial, amqp_redelivered=false, id=540399a5-62a6-7178-2524-e274bad4ed13, amqp_consumerTag=amq.ctag-Z8siptJMdxGU6sXdOHkVCA, contentType=json, __TypeId__=com.art.service.product.bulkimportproduct.data.model.Vendor, timestamp=1512427151297}]
2017-12-04 14:39:11.297 DEBUG 1498 --- [enerContainer-1] o.s.a.r.listener.BlockingQueueConsumer : Retrieving delivery for Consumer#7219ac49: tags=[{amq.ctag-Z8siptJMdxGU6sXdOHkVCA=ProductStore_Partial}], channel=Cached Rabbit Channel: AMQChannel(amqp://admin#xxxx:5672/,2), conn: Proxy#6df20ade Shared Rabbit Connection: SimpleConnection#7ba63fe5 [delegate=amqp://admin#xxxx:5672/, localPort= 51172], acknowledgeMode=AUTO local queue size=0
This goes on repeating and new messages are consumed, but the messages are not getting processed and written to the data-store using the itemWriter provided. Now come to think of it, since I have not provided the tasklet/step bean reference anywhere in this code, is that something I am missing out here?
app.run(args).close();
You are explicitly closing the application context, which shuts everything down.
Closing org.springframework.context.annotation.AnnotationConfigApplicationContext.
In continuation to the post - http://forum.spring.io/forum/spring-projects/integration/119697-unable-to-rename-file-in-sftp-remote-directory-please-help
I am using the sftp:outbound-channel-adapter to upload a file into one of the sftp server. Once file is uploaded, API is unable to rename the temporary file sample_test.pgp.writing to sample_test.pgp.
Before uploading the file I verified in the sftp remote folder for file exist and remote folder was completely empty.
When I looked at the debug level log, I could see below message and it is failing at the end with the invalid path error message.
[main] DEBUG: com.ftp.util.FileUploadUtil - Upload for file /sample_test.pgp triggered
[main] DEBUG: org.springframework.integration.channel.DirectChannel - preSend on channel 'ftp.uploadgateway.request.channel', message: [Payload=/sample_test.pgp][Headers={timestamp=1406654118428, id=bbba360d-492d-4348-b2e7-566aec7f4209}]
[main] DEBUG: org.springframework.integration.filter.MessageFilter - org.springframework.integration.filter.MessageFilter#3970ae0 received message: [Payload=/sample_test.pgp][Headers={timestamp=1406654118428, id=bbba360d-492d-4348-b2e7-566aec7f4209}]
[main] DEBUG: org.springframework.integration.channel.DirectChannel - preSend on channel 'upload.file.to.sftp', message: [Payload=/sample_test.pgp][Headers={timestamp=1406654118428, id=bbba360d-492d-4348-b2e7-566aec7f4209}]
[main] DEBUG: org.springframework.integration.channel.DirectChannel - preSend on channel 'logger', message: [Payload=/sample_test.pgp][Headers={timestamp=1406654118428, id=bbba360d-492d-4348-b2e7-566aec7f4209}]
[main] DEBUG: org.springframework.integration.handler.LoggingHandler - org.springframework.integration.handler.LoggingHandler#0 received message: [Payload=/sample_test.pgp][Headers={timestamp=1406654118428, id=bbba360d-492d-4348-b2e7-566aec7f4209}]
[main] INFO : org.springframework.integration.handler.LoggingHandler - [Payload=/sample_test.pgp][Headers={timestamp=1406654118428, id=bbba360d-492d-4348-b2e7-566aec7f4209}]
[main] DEBUG: org.springframework.integration.channel.DirectChannel - postSend (sent=true) on channel 'logger', message: [Payload=/sample_test.pgp][Headers={timestamp=1406654118428, id=bbba360d-492d-4348-b2e7-566aec7f4209}]
[main] DEBUG: org.springframework.integration.file.remote.handler.FileTransferringMessageHandler - org.springframework.integration.file.remote.handler.FileTransferringMessageHandler#0 received message: [Payload=/sample_test.pgp][Headers={timestamp=1406654118428, id=bbba360d-492d-4348-b2e7-566aec7f4209}]
[main] INFO : com.jcraft.jsch - Connecting to remote.sever.com port 10022
[main] INFO : com.jcraft.jsch - Connection established
[main] INFO : com.jcraft.jsch - Remote version string: SSH-2.0-SSHD
[main] INFO : com.jcraft.jsch - Local version string: SSH-2.0-JSCH-0.1.49
[main] INFO : com.jcraft.jsch - CheckCiphers: aes256-ctr,aes192-ctr,aes128-ctr,aes256-cbc,aes192-cbc,aes128-cbc,3des-ctr,arcfour,arcfour128,arcfour256
[main] INFO : com.jcraft.jsch - CheckKexes: diffie-hellman-group14-sha1
[main] INFO : com.jcraft.jsch - diffie-hellman-group14-sha1 is not available.
[main] INFO : com.jcraft.jsch - SSH_MSG_KEXINIT sent
[main] INFO : com.jcraft.jsch - SSH_MSG_KEXINIT received
[main] INFO : com.jcraft.jsch - kex: server: diffie-hellman-group14-sha1,diffie-hellman-group1-sha1,diffie-hellman-group-exchange-sha1
[main] INFO : com.jcraft.jsch - kex: server: ssh-rsa
[main] INFO : com.jcraft.jsch - kex: server: aes128-cbc,aes192-cbc,aes256-cbc,3des-cbc,blowfish-cbc
[main] INFO : com.jcraft.jsch - kex: server: aes128-cbc,blowfish-cbc,aes192-cbc,aes256-cbc,arcfour,arcfour128,arcfour256
[main] INFO : com.jcraft.jsch - kex: server: hmac-sha1,hmac-md5,hmac-sha1-96,hmac-md5-96,hmac-sha256,hmac-sha256#ssh.com
[main] INFO : com.jcraft.jsch - kex: server: hmac-sha1,hmac-md5,hmac-sha1-96,hmac-md5-96,hmac-sha256,hmac-sha256#ssh.com
[main] INFO : com.jcraft.jsch - kex: server: none,zlib
[main] INFO : com.jcraft.jsch - kex: server: none,zlib
[main] INFO : com.jcraft.jsch - kex: server:
[main] INFO : com.jcraft.jsch - kex: server:
[main] INFO : com.jcraft.jsch - kex: client: diffie-hellman-group1-sha1,diffie-hellman-group-exchange-sha1
[main] INFO : com.jcraft.jsch - kex: client: ssh-rsa,ssh-dss
[main] INFO : com.jcraft.jsch - kex: client: aes128-ctr,aes128-cbc,3des-ctr,3des-cbc,blowfish-cbc,aes192-cbc,aes256-cbc
[main] INFO : com.jcraft.jsch - kex: client: aes128-ctr,aes128-cbc,3des-ctr,3des-cbc,blowfish-cbc,aes192-cbc,aes256-cbc
[main] INFO : com.jcraft.jsch - kex: client: hmac-md5,hmac-sha1,hmac-sha2-256,hmac-sha1-96,hmac-md5-96
[main] INFO : com.jcraft.jsch - kex: client: hmac-md5,hmac-sha1,hmac-sha2-256,hmac-sha1-96,hmac-md5-96
[main] INFO : com.jcraft.jsch - kex: client: none
[main] INFO : com.jcraft.jsch - kex: client: none
[main] INFO : com.jcraft.jsch - kex: client:
[main] INFO : com.jcraft.jsch - kex: client:
[main] INFO : com.jcraft.jsch - kex: server->client aes128-cbc hmac-md5 none
[main] INFO : com.jcraft.jsch - kex: client->server aes128-cbc hmac-md5 none
[main] INFO : com.jcraft.jsch - SSH_MSG_KEXDH_INIT sent
[main] INFO : com.jcraft.jsch - expecting SSH_MSG_KEXDH_REPLY
[main] INFO : com.jcraft.jsch - ssh_rsa_verify: signature true
[main] WARN : com.jcraft.jsch - Permanently added 'remote.sever.com' (RSA) to the list of known hosts.
[main] INFO : com.jcraft.jsch - SSH_MSG_NEWKEYS sent
[main] INFO : com.jcraft.jsch - SSH_MSG_NEWKEYS received
[main] INFO : com.jcraft.jsch - SSH_MSG_SERVICE_REQUEST sent
[main] INFO : com.jcraft.jsch - SSH_MSG_SERVICE_ACCEPT received
[main] INFO : com.jcraft.jsch - Authentications that can continue: publickey,keyboard-interactive,password
[main] INFO : com.jcraft.jsch - Next authentication method: publickey
[main] INFO : com.jcraft.jsch - Authentications that can continue: keyboard-interactive,password
[main] INFO : com.jcraft.jsch - Next authentication method: keyboard-interactive
[main] INFO : com.jcraft.jsch - Authentication succeeded (keyboard-interactive).
[main] DEBUG: org.springframework.integration.util.SimplePool - Obtained new org.springframework.integration.sftp.session.SftpSession#6e75d758.
[main] DEBUG: org.springframework.integration.sftp.session.SftpSession - Initial File rename failed, possibly because file already exists. Will attempt to delete file: /inbox/sample_test.pgp and execute rename again.
[main] DEBUG: org.springframework.integration.file.remote.session.CachingSessionFactory - Releasing Session back to the pool.
[main] DEBUG: org.springframework.integration.util.SimplePool - Releasing org.springframework.integration.sftp.session.SftpSession#6e75d758 back to the pool
[main] DEBUG: com.ftp.service.CtrlMPOJO - ERROR UPLOADING FILES EXCEPTION IS
org.springframework.integration.MessageDeliveryException: Error handling message for file [/sample_test.pgp]
at org.springframework.integration.file.remote.handler.FileTransferringMessageHandler.handleMessageInternal(FileTransferringMessageHandler.java:183)
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:73)
at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:115)
at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:102)
at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:77)
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:157)
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:128)
at org.springframework.integration.core.MessagingTemplate.doSend(MessagingTemplate.java:288)
at org.springframework.integration.core.MessagingTemplate.send(MessagingTemplate.java:149)
at org.springframework.integration.filter.MessageFilter.handleRequestMessage(MessageFilter.java:107)
at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:134)
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:73)
at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:115)
at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:102)
at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:77)
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:157)
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:128)
at org.springframework.integration.core.MessagingTemplate.doSend(MessagingTemplate.java:288)
at org.springframework.integration.core.MessagingTemplate.send(MessagingTemplate.java:149)
at org.springframework.integration.core.MessagingTemplate.convertAndSend(MessagingTemplate.java:189)
at org.springframework.integration.gateway.MessagingGatewaySupport.send(MessagingGatewaySupport.java:183)
at org.springframework.integration.gateway.GatewayProxyFactoryBean.invokeGatewayMethod(GatewayProxyFactoryBean.java:309)
at org.springframework.integration.gateway.GatewayProxyFactoryBean.doInvoke(GatewayProxyFactoryBean.java:269)
at org.springframework.integration.gateway.GatewayProxyFactoryBean.invoke(GatewayProxyFactoryBean.java:260)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at $Proxy5.uploadFilesToFTP(Unknown Source)
at com.ftp.util.FileUploadUtil.scanDirectoryAndUpload(FileUploadUtil.java:123)
at com.ftp.service.CtrlMPOJO.main(CtrlMPOJO.java:160)
Caused by: org.springframework.integration.MessagingException: Failed to write to '/inbox/sample_test.pgp.writing' while uploading the file
at org.springframework.integration.file.remote.handler.FileTransferringMessageHandler.sendFileToRemoteDirectory(FileTransferringMessageHandler.java:266)
at org.springframework.integration.file.remote.handler.FileTransferringMessageHandler.handleMessageInternal(FileTransferringMessageHandler.java:172)
... 28 more
Caused by: org.springframework.core.NestedIOException: Failed to delete file /inbox/sample_test.pgp; nested exception is org.springframework.core.NestedIOException: Failed to remove file: 2: Specified file path is invalid.
at org.springframework.integration.sftp.session.SftpSession.rename(SftpSession.java:157)
at org.springframework.integration.file.remote.session.CachingSessionFactory$CachedSession.rename(CachingSessionFactory.java:137)
at org.springframework.integration.file.remote.handler.FileTransferringMessageHandler.sendFileToRemoteDirectory(FileTransferringMessageHandler.java:262)
... 29 more
Caused by: org.springframework.core.NestedIOException: Failed to remove file: 2: Specified file path is invalid.
at org.springframework.integration.sftp.session.SftpSession.remove(SftpSession.java:71)
at org.springframework.integration.sftp.session.SftpSession.rename(SftpSession.java:151)
... 31 more
It works if I set the use-temporary-file-name=false but I do not want to set this flag incase if there is any file watcher job which may pick up incomplete file loaded at the remote server end.
Here is the configuration i have
<int:gateway id="file.upload.gateway"
service-interface="ftp.outbound.FTPUploadGateway"
default-request-channel="ftp.uploadgateway.request.channel"
default-reply-channel="ftp.uploadgateway.response.channel" />
<int:filter
input-channel="ftp.uploadgateway.request.channel"
output-channel="ftp.file.exist.outbound.channel"
discard-channel="upload.file.to.sftp"
expression="${ftp.outbound.remote.file.check.flag:false}">
</int:filter>
<sftp:outbound-channel-adapter id="sftpOutboundAdapter"
session-factory="sftpSessionFactory"
channel="upload.file.to.sftp"
charset="UTF-8"
remote-directory="${ftp.outbound.remote.directory}"
use-temporary-file-name="${ftp.outbound.use.temporary.filename:true}"
remote-filename-generator-expression="${ftp.outbound.remote.filename.expression}"/>
Here are the property values
ftp.outbound.remote.file.check.flag=false
ftp.outbound.remote.directory=/inbox/
ftp.outbound.use.temporary.filename=true
ftp.outbound.remote.filename.expression=payload.getName()
Please show the complete configuration.
Is your sftp user chrooted? If not /inbox/... is trying to manipulate the root directory on the server, and he likely doesn't have permissions for that.
Try removing the leading / from the remote dir.