We have created an IntegrationFlow for an inbound TCP server. Whenever we test this the integration is fine until the client disconnects, then we get a continuous flow of empty messages.
Spring Integration 5.2.5.RELEASE
Flow:
#Bean
public IntegrationFlow inboundFlow(MyService myService) {
return IntegrationFlows.from(
Tcp.inboundAdapter(
Tcp.netServer(12345)
.deserializer(TcpCodecs.lengthHeader4())))
.transform(Transformers.objectToString())
.log(INFO) // Logging only for testing
.handle(myService, "process") // Do some processing of the message
.get();
}
Test method:
public void sendPackets() throws Exception {
List<Path> files = getAllFilesSorted("/some/location");
try (Socket socket = new Socket("localhost", 12345)) {
log.info("local port: {}", socket.getLocalPort());
try (OutputStream outputStream = socket.getOutputStream()) {
for (int i = 0; i < 10; i++) {
Path path = files.get(i);
log.info("{} ({} of {})", path.toString(), i, files.size());
byte[] bytes = Files.readAllBytes(path);
outputStream.write(bytes);
outputStream.flush();
Thread.sleep(200);
}
}
}
}
Partial Log after test completes:
2020-05-05 11:00:52.963 INFO 31793 --- [pool-1-thread-2] o.s.integration.handler.LoggingHandler : GenericMessage [payload=, headers={ip_tcp_remotePort=59407, ip_connectionId=localhost:59407:44745:df23388b-b8b0-4250-b926-4199030b2ff6, ip_localInetAddress=/127.0.0.1, ip_address=127.0.0.1, id=813ccf6b-9eb7-2ff3-0f8d-f2810966d8d7, ip_hostname=localhost, timestamp=1588672852963}]
2020-05-05 11:00:52.963 INFO 31793 --- [pool-1-thread-2] o.s.integration.handler.LoggingHandler : GenericMessage [payload=, headers={ip_tcp_remotePort=59407, ip_connectionId=localhost:59407:44745:df23388b-b8b0-4250-b926-4199030b2ff6, ip_localInetAddress=/127.0.0.1, ip_address=127.0.0.1, id=1fd5c4c7-b2e0-6a89-7de6-894c8e42e242, ip_hostname=localhost, timestamp=1588672852963}]
2020-05-05 11:00:52.963 INFO 31793 --- [pool-1-thread-2] o.s.integration.handler.LoggingHandler : GenericMessage [payload=, headers={ip_tcp_remotePort=59407, ip_connectionId=localhost:59407:44745:df23388b-b8b0-4250-b926-4199030b2ff6, ip_localInetAddress=/127.0.0.1, ip_address=127.0.0.1, id=78b852e8-0094-f8b9-8d05-37bc4912d096, ip_hostname=localhost, timestamp=1588672852963}]
2020-05-05 11:00:52.963 INFO 31793 --- [pool-1-thread-2] o.s.integration.handler.LoggingHandler : GenericMessage [payload=, headers={ip_tcp_remotePort=59407, ip_connectionId=localhost:59407:44745:df23388b-b8b0-4250-b926-4199030b2ff6, ip_localInetAddress=/127.0.0.1, ip_address=127.0.0.1, id=4e4a7ee0-c66e-9501-9ef7-4d25143531b3, ip_hostname=localhost, timestamp=1588672852963}]
2020-05-05 11:00:52.963 INFO 31793 --- [pool-1-thread-2] o.s.integration.handler.LoggingHandler : GenericMessage [payload=, headers={ip_tcp_remotePort=59407, ip_connectionId=localhost:59407:44745:df23388b-b8b0-4250-b926-4199030b2ff6, ip_localInetAddress=/127.0.0.1, ip_address=127.0.0.1, id=c90abe17-7036-94cc-af59-3b0cac5e75e4, ip_hostname=localhost, timestamp=1588672852963}]
2020-05-05 11:00:52.963 INFO 31793 --- [pool-1-thread-2] o.s.integration.handler.LoggingHandler : GenericMessage [payload=, headers={ip_tcp_remotePort=59407, ip_connectionId=localhost:59407:44745:df23388b-b8b0-4250-b926-4199030b2ff6, ip_localInetAddress=/127.0.0.1, ip_address=127.0.0.1, id=425d2e2a-75c5-0184-06da-6d27eb546931, ip_hostname=localhost, timestamp=1588672852963}]
2020-05-05 11:00:52.965 INFO 31793 --- [pool-1-thread-2] o.s.integration.handler.LoggingHandler : GenericMessage [payload=, headers={ip_tcp_remotePort=59407, ip_connectionId=localhost:59407:44745:df23388b-b8b0-4250-b926-4199030b2ff6, ip_localInetAddress=/127.0.0.1, ip_address=127.0.0.1, id=4c55d1d0-7421-1832-23ca-03744419c847, ip_hostname=localhost, timestamp=1588672852965}]
2020-05-05 11:00:52.965 INFO 31793 --- [pool-1-thread-2] o.s.integration.handler.LoggingHandler : GenericMessage [payload=, headers={ip_tcp_remotePort=59407, ip_connectionId=localhost:59407:44745:df23388b-b8b0-4250-b926-4199030b2ff6, ip_localInetAddress=/127.0.0.1, ip_address=127.0.0.1, id=05b6fea6-8f76-5c74-e4cc-33bd8099d8e8, ip_hostname=localhost, timestamp=1588672852965}]
2020-05-05 11:00:52.965 INFO 31793 --- [pool-1-thread-2] o.s.integration.handler.LoggingHandler : GenericMessage [payload=, headers={ip_tcp_remotePort=59407, ip_connectionId=localhost:59407:44745:df23388b-b8b0-4250-b926-4199030b2ff6, ip_localInetAddress=/127.0.0.1, ip_address=127.0.0.1, id=9b8fe16c-07f4-b64b-faec-794eb743559c, ip_hostname=localhost, timestamp=1588672852965}]
2020-05-05 11:00:52.965 INFO 31793 --- [pool-1-thread-2] o.s.integration.handler.LoggingHandler : GenericMessage [payload=, headers={ip_tcp_remotePort=59407, ip_connectionId=localhost:59407:44745:df23388b-b8b0-4250-b926-4199030b2ff6, ip_localInetAddress=/127.0.0.1, ip_address=127.0.0.1, id=222725e8-19c6-62b6-8ba7-103892191c96, ip_hostname=localhost, timestamp=1588672852965}]
2020-05-05 11:00:52.965 INFO 31793 --- [pool-1-thread-2] o.s.integration.handler.LoggingHandler : GenericMessage [payload=, headers={ip_tcp_remotePort=59407, ip_connectionId=localhost:59407:44745:df23388b-b8b0-4250-b926-4199030b2ff6, ip_localInetAddress=/127.0.0.1, ip_address=127.0.0.1, id=951403b6-18a9-ffe9-29b7-67b8c01ea311, ip_hostname=localhost, timestamp=1588672852965}]
2020-05-05 11:00:52.965 INFO 31793 --- [pool-1-thread-2] o.s.integration.handler.LoggingHandler : GenericMessage [payload=, headers={ip_tcp_remotePort=59407, ip_connectionId=localhost:59407:44745:df23388b-b8b0-4250-b926-4199030b2ff6, ip_localInetAddress=/127.0.0.1, ip_address=127.0.0.1, id=2ac3002b-ccd1-9506-f3aa-60f4b63e26b8, ip_hostname=localhost, timestamp=1588672852965}]
2020-05-05 11:00:52.965 INFO 31793 --- [pool-1-thread-2] o.s.integration.handler.LoggingHandler : GenericMessage [payload=, headers={ip_tcp_remotePort=59407, ip_connectionId=localhost:59407:44745:df23388b-b8b0-4250-b926-4199030b2ff6, ip_localInetAddress=/127.0.0.1, ip_address=127.0.0.1, id=dbbf12cf-0a27-407b-0a96-5b27ab0539db, ip_hostname=localhost, timestamp=1588672852965}]
2020-05-05 11:00:52.965 INFO 31793 --- [pool-1-thread-2] o.s.integration.handler.LoggingHandler : GenericMessage [payload=, headers={ip_tcp_remotePort=59407, ip_connectionId=localhost:59407:44745:df23388b-b8b0-4250-b926-4199030b2ff6, ip_localInetAddress=/127.0.0.1, ip_address=127.0.0.1, id=3282d09e-9508-b053-3763-cc23d3c817a8, ip_hostname=localhost, timestamp=1588672852965}]
2020-05-05 11:00:52.966 INFO 31793 --- [pool-1-thread-2] o.s.integration.handler.LoggingHandler : GenericMessage [payload=, headers={ip_tcp_remotePort=59407, ip_connectionId=localhost:59407:44745:df23388b-b8b0-4250-b926-4199030b2ff6, ip_localInetAddress=/127.0.0.1, ip_address=127.0.0.1, id=549b2326-341f-19c0-db98-6b31da1901d8, ip_hostname=localhost, timestamp=1588672852966}]
There is nothing in the log about the client disconnecting. We would see socket close activity in the log.
The log implies we are receiving packets containing 0x00000000.
Use wireshark or similar to look at the activity on the wire.
EDIT
Your custom deserializer is causing the problem.
if (read < 0) {
return 0;
}
You need to throw a SoftEndOfStreamException when you detect a socket close.
This is explained in the documentation.
When the deserializer detects a closed input stream between messages, it must throw a SoftEndOfStreamException; this is a signal to the framework to indicate that the close was "normal". If the stream is closed while decoding a message, some other exception should be thrown instead.
Related
I have flow that I want to implement JdbcMetadataStore in the filers, I used SimpleMetadataStore() but this is causing issues as it an in memory store, I need to use a shared metadata store, so I have Postgres DB installed and I can see that Jdbc supports it, I declared a bean as per documents to return a JdbcMetadataStore but I'm not sure how I can use this in the filters, tried to search a lot for any example but could not find a one, note that I'm using FileSystemPersistentAcceptOnceFileListFilter as well as my datasource for Postgres is all setup in my application properties. I have pasted my code here anyone can please guide me on how to move forward?
private DataSource dataSource;
public IntegrationFlow localToFtpFlow(Branch myBranch){
return IntegrationFlows.from(Files.inboundAdapter(new File(myBranch.getBranchCode()))
.filter(new ChainFileListFilter<File>()
.addFilter(new RegexPatternFileListFilter("final" + myBranch.getBranchCode() +".csv"))
.addFilter(new FileSystemPersistentAcceptOnceFileListFilter(new SimpleMetadataStore(), "foo"))),
e -> e.poller(Pollers.fixedDelay(10_000)))
.transform( p ->{
LOG1.info("Sending file " + p + " to FTP branch " + myBranch.getBranchCode());
return p;
})
.log()
.handle(Ftp.outboundAdapter(createNewFtpSessionFactory(myBranch),FileExistsMode.REPLACE)
.useTemporaryFileName(true)
.autoCreateDirectory(false)
.remoteDirectory(myBranch.getFolderPath()))
.get();
}
public DefaultFtpSessionFactory createNewFtpSessionFactory(Branch branch){
final DefaultFtpSessionFactory factory = new DefaultFtpSessionFactory();
factory.setHost(branch.getHost());
factory.setUsername(branch.getUsern());
factory.setPort(branch.getFtpPort());
factory.setPassword(branch.getPassword());
return factory;
}
#Bean
public MetadataStore metadataStore(final DataSource dataSource) {
return new JdbcMetadataStore(dataSource);
}
error arising before I created the table manually, but shouldn't it be created automatically in Postgres as the DB is supported:
org.springframework.messaging.MessagingException: nested exception is org.springframework.jdbc.BadSqlGrammarException: PreparedStatementCallback; bad SQL grammar [INSERT INTO INT_METADATA_STORE(METADATA_KEY, METADATA_VALUE, REGION) SELECT ?, ?, ? FROM INT_METADATA_STORE WHERE METADATA_KEY=? AND REGION=? HAVING COUNT(*)=0]; nested exception is org.postgresql.util.PSQLException: ERROR: relation "int_metadata_store" does not exist
Logging info regarding the issue when adding another flow for the second server, it will trigger the first flow handling method and sending the data to ftp server for both:
Saved Branch : BEY
Hibernate: select branch0_._id as _id1_0_0_, branch0_.branch_code as branch_c2_0_0_, branch0_.folder_path as folder_p3_0_0_, branch0_.ftp_port as ftp_port4_0_0_, branch0_.host as host5_0_0_, branch0_.password as password6_0_0_, branch0_.usern as usern7_0_0_ from branch branch0_ where branch0_._id=?
BEY
2019-01-07 15:11:25.816 INFO 12940 --- [nio-8081-exec-5] o.s.integration.channel.DirectChannel : Channel 'application.intermediateChannel' has 2 subscriber(s).
2019-01-07 15:11:25.817 INFO 12940 --- [nio-8081-exec-5] o.s.i.endpoint.EventDrivenConsumer : started 1.org.springframework.integration.config.ConsumerEndpointFactoryBean#1
2019-01-07 15:11:25.817 INFO 12940 --- [nio-8081-exec-5] o.s.i.endpoint.EventDrivenConsumer : Adding {transformer} as a subscriber to the '1.channel#0' channel
2019-01-07 15:11:25.817 INFO 12940 --- [nio-8081-exec-5] o.s.integration.channel.DirectChannel : Channel 'application.1.channel#0' has 1 subscriber(s).
2019-01-07 15:11:25.817 INFO 12940 --- [nio-8081-exec-5] o.s.i.endpoint.EventDrivenConsumer : started 1.org.springframework.integration.config.ConsumerEndpointFactoryBean#0
2019-01-07 15:11:25.829 INFO 12940 --- [nio-8081-exec-5] o.s.i.e.SourcePollingChannelAdapter : started stockInboundPoller
BEY
2019-01-07 15:11:25.984 INFO 12940 --- [nio-8081-exec-5] o.s.i.endpoint.EventDrivenConsumer : Adding {message-handler} as a subscriber to the '1o.channel#2' channel
2019-01-07 15:11:25.984 INFO 12940 --- [nio-8081-exec-5] o.s.integration.channel.DirectChannel : Channel 'application.1o.channel#2' has 1 subscriber(s).
2019-01-07 15:11:25.984 INFO 12940 --- [nio-8081-exec-5] o.s.i.endpoint.EventDrivenConsumer : started 1o.org.springframework.integration.config.ConsumerEndpointFactoryBean#1
2019-01-07 15:11:25.984 INFO 12940 --- [nio-8081-exec-5] o.s.i.endpoint.EventDrivenConsumer : Adding {transformer} as a subscriber to the '1o.channel#0' channel
2019-01-07 15:11:25.984 INFO 12940 --- [nio-8081-exec-5] o.s.integration.channel.DirectChannel : Channel 'application.1o.channel#0' has 1 subscriber(s).
2019-01-07 15:11:25.984 INFO 12940 --- [nio-8081-exec-5] o.s.i.endpoint.EventDrivenConsumer : started 1o.org.springframework.integration.config.ConsumerEndpointFactoryBean#0
2019-01-07 15:11:25.984 INFO 12940 --- [nio-8081-exec-5] o.s.i.e.SourcePollingChannelAdapter : started 1o.org.springframework.integration.config.SourcePollingChannelAdapterFactoryBean#0
2019-01-07 15:11:42.655 INFO 12940 --- [ask-scheduler-4] o.s.integration.ftp.session.FtpSession : File has been successfully transferred from: /ftp/erbranch/EDMS/FEFO/FEFOexportBEY.csv
Hibernate: select branch0_._id as _id1_0_, branch0_.branch_code as branch_c2_0_, branch0_.folder_path as folder_p3_0_, branch0_.ftp_port as ftp_port4_0_, branch0_.host as host5_0_, branch0_.password as password6_0_, branch0_.usern as usern7_0_ from branch branch0_
Hibernate: insert into branch (branch_code, folder_path, ftp_port, host, password, usern) values (?, ?, ?, ?, ?, ?)
Hibernate: select currval('branch__id_seq')
Saved Branch : JNB
Hibernate: select branch0_._id as _id1_0_0_, branch0_.branch_code as branch_c2_0_0_, branch0_.folder_path as folder_p3_0_0_, branch0_.ftp_port as ftp_port4_0_0_, branch0_.host as host5_0_0_, branch0_.password as password6_0_0_, branch0_.usern as usern7_0_0_ from branch branch0_ where branch0_._id=?
JNB
2019-01-07 15:13:36.099 INFO 12940 --- [nio-8081-exec-7] o.s.integration.channel.DirectChannel : Channel 'application.intermediateChannel' has 3 subscriber(s).
2019-01-07 15:13:36.099 INFO 12940 --- [nio-8081-exec-7] o.s.i.endpoint.EventDrivenConsumer : started 2.org.springframework.integration.config.ConsumerEndpointFactoryBean#1
2019-01-07 15:13:36.099 INFO 12940 --- [nio-8081-exec-7] o.s.i.endpoint.EventDrivenConsumer : Adding {transformer} as a subscriber to the '2.channel#0' channel
2019-01-07 15:13:36.099 INFO 12940 --- [nio-8081-exec-7] o.s.integration.channel.DirectChannel : Channel 'application.2.channel#0' has 1 subscriber(s).
2019-01-07 15:13:36.099 INFO 12940 --- [nio-8081-exec-7] o.s.i.endpoint.EventDrivenConsumer : started 2.org.springframework.integration.config.ConsumerEndpointFactoryBean#0
2019-01-07 15:13:36.099 INFO 12940 --- [nio-8081-exec-7] o.s.i.e.SourcePollingChannelAdapter : started stockInboundPoller
JNB
2019-01-07 15:13:36.130 INFO 12940 --- [nio-8081-exec-7] o.s.i.endpoint.EventDrivenConsumer : Adding {message-handler} as a subscriber to the '2o.channel#2' channel
2019-01-07 15:13:36.135 INFO 12940 --- [nio-8081-exec-7] o.s.integration.channel.DirectChannel : Channel 'application.2o.channel#2' has 1 subscriber(s).
2019-01-07 15:13:36.135 INFO 12940 --- [nio-8081-exec-7] o.s.i.endpoint.EventDrivenConsumer : started 2o.org.springframework.integration.config.ConsumerEndpointFactoryBean#1
2019-01-07 15:13:36.135 INFO 12940 --- [nio-8081-exec-7] o.s.i.endpoint.EventDrivenConsumer : Adding {transformer} as a subscriber to the '2o.channel#0' channel
2019-01-07 15:13:36.135 INFO 12940 --- [nio-8081-exec-7] o.s.integration.channel.DirectChannel : Channel 'application.2o.channel#0' has 1 subscriber(s).
2019-01-07 15:13:36.135 INFO 12940 --- [nio-8081-exec-7] o.s.i.endpoint.EventDrivenConsumer : started 2o.org.springframework.integration.config.ConsumerEndpointFactoryBean#0
2019-01-07 15:13:36.135 INFO 12940 --- [nio-8081-exec-7] o.s.i.e.SourcePollingChannelAdapter : started 2o.org.springframework.integration.config.SourcePollingChannelAdapterFactoryBean#0
2019-01-07 15:13:40.981 INFO 12940 --- [ask-scheduler-1] o.s.integration.ftp.session.FtpSession : File has been successfully transferred from: /ftp/erbranch/EDMS/FEFO/FEFOexportJNB.csv
2019-01-07 15:13:46.085 INFO 12940 --- [ask-scheduler-7] o.s.i.file.FileReadingMessageSource : Created message: [GenericMessage [payload=BEY\finalBEY.csv, headers={file_originalFile=BEY\finalBEY.csv, id=42a97889-7bfb-8f77-75d8-4e7988a368f9, file_name=finalBEY.csv, file_relativePath=finalBEY.csv, timestamp=1546866826085}]]
2019-01-07 15:13:46.086 INFO 12940 --- [ask-scheduler-7] o.s.integration.handler.LoggingHandler : GenericMessage [payload=BEY\finalBEY.csv, headers={file_originalFile=BEY\finalBEY.csv, id=108a92b0-db42-620e-1c46-90652a071220, file_name=finalBEY.csv, file_relativePath=finalBEY.csv, timestamp=1546866826086}]
2019-01-07 15:13:46.160 INFO 12940 --- [ask-scheduler-8] o.s.i.file.FileReadingMessageSource : Created message: [GenericMessage [payload=JNB\finalJNB.csv, headers={file_originalFile=JNB\finalJNB.csv, id=d3b2c6a0-2e9c-42a8-c224-0ed9cbbfaabb, file_name=finalJNB.csv, file_relativePath=finalJNB.csv, timestamp=1546866826160}]]
2019-01-07 15:13:46.161 INFO 12940 --- [ask-scheduler-8] o.s.integration.handler.LoggingHandler : GenericMessage [payload=JNB\finalJNB.csv, headers={file_originalFile=JNB\finalJNB.csv, id=e34070c2-e6ff-e5e1-8c64-4af697ab1032, file_name=finalJNB.csv, file_relativePath=finalJNB.csv, timestamp=1546866826161}]
2019-01-07 15:13:47.129 INFO 12940 --- [ask-scheduler-7] o.s.integration.ftp.session.FtpSession : File has been successfully transferred to: /ftp/erbranch/EDMS/FEFO/finalBEY.csv.writing
2019-01-07 15:13:47.534 INFO 12940 --- [ask-scheduler-7] o.s.integration.ftp.session.FtpSession : File has been successfully renamed from: /ftp/erbranch/EDMS/FEFO/finalBEY.csv.writing to /ftp/erbranch/EDMS/FEFO/finalBEY.csv
2019-01-07 15:13:49.772 INFO 12940 --- [ask-scheduler-8] o.s.integration.ftp.session.FtpSession : File has been successfully transferred to: /ftp/erbranch/EDMS/FEFO/finalJNB.csv.writing
2019-01-07 15:13:50.757 INFO 12940 --- [ask-scheduler-8] o.s.integration.ftp.session.FtpSession : File has been successfully renamed from: /ftp/erbranch/EDMS/FEFO/finalJNB.csv.writing to /ftp/erbranch/EDMS/FEFO/finalJNB.csv
You can find my app here
https://github.com/EliasKhattar/Spring-Integration-Project/tree/master/spring4ftpappftp
Just use metadataSource(dataSource) instead of new SimpleMetadataStore() in the filter constructor.
EDIT
I just copied your flow into a new app (made a few changes, but not to the filter) and everything works fine for me...
#SpringBootApplication
public class So54039852Application {
public static void main(String[] args) {
SpringApplication.run(So54039852Application.class, args);
}
#Bean
public IntegrationFlow localToFtpFlow(DataSource dataSource) {
return IntegrationFlows.from(Files.inboundAdapter(new File("/tmp/foo"))
.filter(new ChainFileListFilter<File>()
.addFilter(new RegexPatternFileListFilter(".*\\.csv"))
.addFilter(new FileSystemPersistentAcceptOnceFileListFilter(metadataStore(dataSource), "foo"))),
e -> e.poller(Pollers.fixedDelay(10_000)))
.log()
.get();
}
#Bean
public ConcurrentMetadataStore metadataStore(final DataSource dataSource) {
return new JdbcMetadataStore(dataSource);
}
}
and
$ touch /tmp/foo/foo.csv
...
$ touch /tmp/foo/bar.csv
and
2019-01-09 12:46:26.332 INFO 43329 --- [ask-scheduler-2] o.s.integration.handler.LoggingHandler : GenericMessage [payload=/tmp/foo/foo.csv, headers={file_originalFile=/tmp/foo/foo.csv, id=e0613529-a657-fbd3-5e67-8bb53a58b5ca, file_name=foo.csv, file_relativePath=foo.csv, timestamp=1547055986330}]
2019-01-09 12:47:26.487 INFO 43329 --- [ask-scheduler-5] o.s.integration.handler.LoggingHandler : GenericMessage [payload=/tmp/foo/bar.csv, headers={file_originalFile=/tmp/foo/bar.csv, id=4feb74b6-d711-f028-70c7-83cdfcd0aeec, file_name=bar.csv, file_relativePath=bar.csv, timestamp=1547056046487}]
and
mysql> select * from INT_METADATA_STORE;
+---------------------+----------------+---------+
| METADATA_KEY | METADATA_VALUE | REGION |
+---------------------+----------------+---------+
| foo/tmp/foo/bar.csv | 1547056039000 | DEFAULT |
| foo/tmp/foo/foo.csv | 1547055980000 | DEFAULT |
+---------------------+----------------+---------+
2 rows in set (0.00 sec)
I don't see the file again even after a restart, but if I change the date on one of the files...
$ touch /tmp/foo/bar.csv
and
2019-01-09 12:51:58.534 INFO 44430 --- [ask-scheduler-2] o.s.integration.handler.LoggingHandler : GenericMessage [payload=/tmp/foo/bar.csv, headers={file_originalFile=/tmp/foo/bar.csv, id=f92d6b36-c948-37cc-ca56-9ef28de336f2, file_name=bar.csv, file_relativePath=bar.csv, timestamp=1547056318532}]
I have created an application that uses Spring MVC to add ftp servers through a web page. When I add a server with details, it will start a flow polling a directory on the sever for specific csv file, once that file is there it will pull it to my local folder then a method is created to read that csv file and generate a new csv that will be sent back to the server, now this is accomplished, my issue is that when I add a new ftp server to poll and it finds a csv file on the second server, the application is pulling that file to my local as it should do but the handling method is being triggered again for both csv files while it should only do so for the second server since it is already triggered for the first server, so when I add the second server then again another file for server 1 generated and sent back to ftp and new one for the second server is send to the second ftp, at the end I have two files sent to first serve and one file to the second server, below is the coding and the output consol of the app, please assist on pinpointing my issue.
#Configuration
#EnableIntegration
#ComponentScan
public class FTIntegration {
public static final String TIMEZONE_UTC = "UTC";
public static final String TIMESTAMP_FORMAT_OF_FILES = "yyyyMMddHHmmssSSS";
public static final String TEMPORARY_FILE_SUFFIX = ".part";
public static final int POLLER_FIXED_PERIOD_DELAY = 5000;
public static final int MAX_MESSAGES_PER_POLL = 100;
//private static final Logger LOG = LoggerFactory.getLogger(FTIntegration.class);
private static final Logger LOG1 = Logger.getLogger(FTIntegration.class);
private static final String CHANNEL_INTERMEDIATE_STAGE = "intermediateChannel";
private static final String OUTBOUND_CHANNEL = "outboundChannel";
/* pulling the server config from postgres DB*/
private final BranchRepository branchRepository;
#Autowired
private CSVToCSVNoQ csvToCSVNoQ;
#Value("${app.temp-dir}")
private String localTempPath;
public FTIntegration(BranchRepository branchRepository) {
this.branchRepository = branchRepository;
}
#Bean
public Branch myBranch(){
return new Branch();
}
/**
* The default poller with 5s, 100 messages, RotatingServerAdvice and transaction.
*
* #return default poller.
*/
#Bean(name = PollerMetadata.DEFAULT_POLLER)
public PollerMetadata poller(){
return Pollers
.fixedDelay(POLLER_FIXED_PERIOD_DELAY)
.maxMessagesPerPoll(MAX_MESSAGES_PER_POLL)
.transactional()
.get();
}
/**
* The direct channel for the flow.
*
* #return MessageChannel
*/
#Bean
public MessageChannel stockIntermediateChannel() {
return new DirectChannel();
}
/**
* Get the files from a remote directory. Add a timestamp to the filename
* and write them to a local temporary folder.
*
* #return IntegrationFlow
*/
#Bean
public PropertiesPersistingMetadataStore store() {
PropertiesPersistingMetadataStore store = new PropertiesPersistingMetadataStore();
return store;
}
public IntegrationFlow fileInboundFlowFromFTPServer(Branch myBranch) throws IOException {
final FtpInboundChannelAdapterSpec sourceSpecFtp = Ftp.inboundAdapter(createNewFtpSessionFactory(myBranch))
.preserveTimestamp(true)
//.patternFilter("*.csv")
.maxFetchSize(MAX_MESSAGES_PER_POLL)
.remoteDirectory(myBranch.getFolderPath())
.regexFilter("FEFOexport"+myBranch.getBranchCode()+".csv")
.deleteRemoteFiles(true)
.localDirectory(new File(myBranch.getBranchCode()))
.temporaryFileSuffix(TEMPORARY_FILE_SUFFIX)
/*.localFilenameExpression(new FunctionExpression<String>(s -> {
final int fileTypeSepPos = s.lastIndexOf('.');
return DateTimeFormatter
.ofPattern(TIMESTAMP_FORMAT_OF_FILES)
.withZone(ZoneId.of(TIMEZONE_UTC))
.format(Instant.now())
+ "_"
+ s.substring(0,fileTypeSepPos)
+ s.substring(fileTypeSepPos);
}))*/;
// Poller definition
final Consumer<SourcePollingChannelAdapterSpec> stockInboundPoller = endpointConfigurer -> endpointConfigurer
.id("stockInboundPoller")
.autoStartup(true)
.poller(poller());
IntegrationFlow flow = IntegrationFlows
.from(sourceSpecFtp, stockInboundPoller)
.transform(File.class, p ->{
// log step
LOG1.info("flow=stockInboundFlowFromAFT, message=incoming file: " + p);
return p;
})
.channel(CHANNEL_INTERMEDIATE_STAGE)
.handle(m -> {
try {
this.csvToCSVNoQ.writeCSVfinal("test", myBranch.getBranchCode() + "/final" + myBranch.getBranchCode() + ".csv", myBranch.getBranchCode() + "/FEFOexport" + myBranch.getBranchCode() + ".csv");
LOG1.info("Writing final file .csv " + m);
} catch (IOException e) {
e.printStackTrace();
}
})
//.handle(m -> this.csvToCSVNoQ.writeCSVfinal(m.getPayload(),m.getHeaders().get("csv", "FEFOexport"+myBranch.getBranchCode()+".csv")))
.get();
return flow;
}
#Bean
public IntegrationFlow stockIntermediateStageChannel() {
IntegrationFlow flow = IntegrationFlows
.from(CHANNEL_INTERMEDIATE_STAGE)
.transform(p -> {
//log step
LOG1.info("flow=stockIntermediateStageChannel, message=rename file: " + p);
return p;
})
//TODO
.channel(new NullChannel())
.get();
return flow;
}
/*
* Creating the outbound adaptor to send files from local to FTP server
*
* */
public IntegrationFlow localToFtpFlow(Branch myBranch){
return IntegrationFlows.from(Files.inboundAdapter(new File(myBranch.getBranchCode()))
.filter(new ChainFileListFilter<File>()
.addFilter(new RegexPatternFileListFilter("final" + myBranch.getBranchCode() +".csv"))
.addFilter(new FileSystemPersistentAcceptOnceFileListFilter(new SimpleMetadataStore(), "foo"))),
e -> e.poller(Pollers.fixedDelay(10_000)))
.transform( p ->{
LOG1.info("Sending file " + p + " to FTP branch " + myBranch.getBranchCode());
return p;
})
.log()
.handle(Ftp.outboundAdapter(createNewFtpSessionFactory(myBranch),FileExistsMode.REPLACE)
.useTemporaryFileName(true)
.autoCreateDirectory(false)
.remoteDirectory(myBranch.getFolderPath()))
.get();
}
public DefaultFtpSessionFactory createNewFtpSessionFactory(Branch branch){
final DefaultFtpSessionFactory factory = new DefaultFtpSessionFactory();
factory.setHost(branch.getHost());
factory.setUsername(branch.getUsern());
factory.setPort(branch.getFtpPort());
factory.setPassword(branch.getPassword());
return factory;
}
Console Output:
Saved Branch : BEY
Hibernate: select branch0_._id as _id1_0_0_, branch0_.branch_code as branch_c2_0_0_, branch0_.folder_path as folder_p3_0_0_, branch0_.ftp_port as ftp_port4_0_0_, branch0_.host as host5_0_0_, branch0_.password as password6_0_0_, branch0_.usern as usern7_0_0_ from branch branch0_ where branch0_._id=?
2018-12-20 07:58:39.218 INFO 6668 --- [nio-8081-exec-5] o.s.integration.channel.DirectChannel : Channel 'application.intermediateChannel' has 2 subscriber(s).
2018-12-20 07:58:39.218 INFO 6668 --- [nio-8081-exec-5] o.s.i.endpoint.EventDrivenConsumer : started 1.org.springframework.integration.config.ConsumerEndpointFactoryBean#1
2018-12-20 07:58:39.218 INFO 6668 --- [nio-8081-exec-5] o.s.i.endpoint.EventDrivenConsumer : Adding {transformer} as a subscriber to the '1.channel#0' channel
2018-12-20 07:58:39.218 INFO 6668 --- [nio-8081-exec-5] o.s.integration.channel.DirectChannel : Channel 'application.1.channel#0' has 1 subscriber(s).
2018-12-20 07:58:39.218 INFO 6668 --- [nio-8081-exec-5] o.s.i.endpoint.EventDrivenConsumer : started 1.org.springframework.integration.config.ConsumerEndpointFactoryBean#0
2018-12-20 07:58:39.229 INFO 6668 --- [nio-8081-exec-5] o.s.i.e.SourcePollingChannelAdapter : started stockInboundPoller
2018-12-20 07:58:39.417 INFO 6668 --- [nio-8081-exec-5] o.s.i.endpoint.EventDrivenConsumer : Adding {message-handler} as a subscriber to the '1o.channel#2' channel
2018-12-20 07:58:39.417 INFO 6668 --- [nio-8081-exec-5] o.s.integration.channel.DirectChannel : Channel 'application.1o.channel#2' has 1 subscriber(s).
2018-12-20 07:58:39.418 INFO 6668 --- [nio-8081-exec-5] o.s.i.endpoint.EventDrivenConsumer : started 1o.org.springframework.integration.config.ConsumerEndpointFactoryBean#1
2018-12-20 07:58:39.418 INFO 6668 --- [nio-8081-exec-5] o.s.i.endpoint.EventDrivenConsumer : Adding {transformer} as a subscriber to the '1o.channel#0' channel
2018-12-20 07:58:39.418 INFO 6668 --- [nio-8081-exec-5] o.s.integration.channel.DirectChannel : Channel 'application.1o.channel#0' has 1 subscriber(s).
2018-12-20 07:58:39.418 INFO 6668 --- [nio-8081-exec-5] o.s.i.endpoint.EventDrivenConsumer : started 1o.org.springframework.integration.config.ConsumerEndpointFactoryBean#0
2018-12-20 07:58:39.419 INFO 6668 --- [nio-8081-exec-5] o.s.i.e.SourcePollingChannelAdapter : started 1o.org.springframework.integration.config.SourcePollingChannelAdapterFactoryBean#0
2018-12-20 07:59:41.165 INFO 6668 --- [sk-scheduler-10] o.s.integration.ftp.session.FtpSession : File has been successfully transferred from: /ftp/erbranch/EDMS/FEFO/FEFOexportBEY.csv
2018-12-20 07:59:49.446 INFO 6668 --- [ask-scheduler-1] o.s.i.file.FileReadingMessageSource : Created message: [GenericMessage [payload=BEY\finalBEY.csv, headers={file_originalFile=BEY\finalBEY.csv, id=7f0cccb7-a070-bd4c-d468-977a265ceb2e, file_name=finalBEY.csv, file_relativePath=finalBEY.csv, timestamp=1545285589446}]]
2018-12-20 07:59:49.448 INFO 6668 --- [ask-scheduler-1] o.s.integration.handler.LoggingHandler : GenericMessage [payload=BEY\finalBEY.csv, headers={file_originalFile=BEY\finalBEY.csv, id=d857130a-d4a0-eaeb-19ea-f819924d94e2, file_name=finalBEY.csv, file_relativePath=finalBEY.csv, timestamp=1545285589447}]
2018-12-20 07:59:50.488 INFO 6668 --- [ask-scheduler-1] o.s.integration.ftp.session.FtpSession : File has been successfully transferred to: /ftp/erbranch/EDMS/FEFO/finalBEY.csv.writing
2018-12-20 07:59:50.899 INFO 6668 --- [ask-scheduler-1] o.s.integration.ftp.session.FtpSession : File has been successfully renamed from: /ftp/erbranch/EDMS/FEFO/finalBEY.csv.writing to /ftp/erbranch/EDMS/FEFO/finalBEY.csv
Hibernate: select branch0_._id as _id1_0_, branch0_.branch_code as branch_c2_0_, branch0_.folder_path as folder_p3_0_, branch0_.ftp_port as ftp_port4_0_, branch0_.host as host5_0_, branch0_.password as password6_0_, branch0_.usern as usern7_0_ from branch branch0_
Hibernate: insert into branch (branch_code, folder_path, ftp_port, host, password, usern) values (?, ?, ?, ?, ?, ?)
Hibernate: select currval('branch__id_seq')
Saved Branch : JNB
Hibernate: select branch0_._id as _id1_0_0_, branch0_.branch_code as branch_c2_0_0_, branch0_.folder_path as folder_p3_0_0_, branch0_.ftp_port as ftp_port4_0_0_, branch0_.host as host5_0_0_, branch0_.password as password6_0_0_, branch0_.usern as usern7_0_0_ from branch branch0_ where branch0_._id=?
2018-12-20 08:02:24.966 INFO 6668 --- [nio-8081-exec-8] o.s.integration.channel.DirectChannel : Channel 'application.intermediateChannel' has 3 subscriber(s).
2018-12-20 08:02:24.966 INFO 6668 --- [nio-8081-exec-8] o.s.i.endpoint.EventDrivenConsumer : started 2.org.springframework.integration.config.ConsumerEndpointFactoryBean#1
2018-12-20 08:02:24.966 INFO 6668 --- [nio-8081-exec-8] o.s.i.endpoint.EventDrivenConsumer : Adding {transformer} as a subscriber to the '2.channel#0' channel
2018-12-20 08:02:24.966 INFO 6668 --- [nio-8081-exec-8] o.s.integration.channel.DirectChannel : Channel 'application.2.channel#0' has 1 subscriber(s).
2018-12-20 08:02:24.966 INFO 6668 --- [nio-8081-exec-8] o.s.i.endpoint.EventDrivenConsumer : started 2.org.springframework.integration.config.ConsumerEndpointFactoryBean#0
2018-12-20 08:02:24.966 INFO 6668 --- [nio-8081-exec-8] o.s.i.e.SourcePollingChannelAdapter : started stockInboundPoller
2018-12-20 08:02:24.992 INFO 6668 --- [nio-8081-exec-8] o.s.i.endpoint.EventDrivenConsumer : Adding {message-handler} as a subscriber to the '2o.channel#2' channel
2018-12-20 08:02:24.992 INFO 6668 --- [nio-8081-exec-8] o.s.integration.channel.DirectChannel : Channel 'application.2o.channel#2' has 1 subscriber(s).
2018-12-20 08:02:24.992 INFO 6668 --- [nio-8081-exec-8] o.s.i.endpoint.EventDrivenConsumer : started 2o.org.springframework.integration.config.ConsumerEndpointFactoryBean#1
2018-12-20 08:02:24.992 INFO 6668 --- [nio-8081-exec-8] o.s.i.endpoint.EventDrivenConsumer : Adding {transformer} as a subscriber to the '2o.channel#0' channel
2018-12-20 08:02:24.992 INFO 6668 --- [nio-8081-exec-8] o.s.integration.channel.DirectChannel : Channel 'application.2o.channel#0' has 1 subscriber(s).
2018-12-20 08:02:24.992 INFO 6668 --- [nio-8081-exec-8] o.s.i.endpoint.EventDrivenConsumer : started 2o.org.springframework.integration.config.ConsumerEndpointFactoryBean#0
2018-12-20 08:02:24.992 INFO 6668 --- [nio-8081-exec-8] o.s.i.e.SourcePollingChannelAdapter : started 2o.org.springframework.integration.config.SourcePollingChannelAdapterFactoryBean#0
Hibernate: select branch0_._id as _id1_0_, branch0_.branch_code as branch_c2_0_, branch0_.folder_path as folder_p3_0_, branch0_.ftp_port as ftp_port4_0_, branch0_.host as host5_0_, branch0_.password as password6_0_, branch0_.usern as usern7_0_ from branch branch0_
2018-12-20 08:03:00.225 INFO 6668 --- [ask-scheduler-8] o.s.integration.ftp.session.FtpSession : File has been successfully transferred from: /ftp/erbranch/EDMS/FEFO/FEFOexportJNB.csv
2018-12-20 08:03:00.929 INFO 6668 --- [ask-scheduler-5] o.s.i.file.FileReadingMessageSource : Created message: [GenericMessage [payload=BEY\finalBEY.csv, headers={file_originalFile=BEY\finalBEY.csv, id=6ed554eb-f553-0293-f042-d633155357c0, file_name=finalBEY.csv, file_relativePath=finalBEY.csv, timestamp=1545285780929}]]
2018-12-20 08:03:00.930 INFO 6668 --- [ask-scheduler-5] o.s.integration.handler.LoggingHandler : GenericMessage [payload=BEY\finalBEY.csv, headers={file_originalFile=BEY\finalBEY.csv, id=b2d76ac5-fb85-1313-a37e-8849714a545e, file_name=finalBEY.csv, file_relativePath=finalBEY.csv, timestamp=1545285780930}]
2018-12-20 08:03:01.958 INFO 6668 --- [ask-scheduler-5] o.s.integration.ftp.session.FtpSession : File has been successfully transferred to: /ftp/erbranch/EDMS/FEFO/finalBEY.csv.writing
2018-12-20 08:03:02.373 INFO 6668 --- [ask-scheduler-5] o.s.integration.ftp.session.FtpSession : File has been successfully renamed from: /ftp/erbranch/EDMS/FEFO/finalBEY.csv.writing to /ftp/erbranch/EDMS/FEFO/finalBEY.csv
2018-12-20 08:03:05.033 INFO 6668 --- [ask-scheduler-7] o.s.i.file.FileReadingMessageSource : Created message: [GenericMessage [payload=JNB\finalJNB.csv, headers={file_originalFile=JNB\finalJNB.csv, id=4514e132-5684-9e82-28e7-f75c5c3dcf91, file_name=finalJNB.csv, file_relativePath=finalJNB.csv, timestamp=1545285785033}]]
2018-12-20 08:03:05.034 INFO 6668 --- [ask-scheduler-7] o.s.integration.handler.LoggingHandler : GenericMessage [payload=JNB\finalJNB.csv, headers={file_originalFile=JNB\finalJNB.csv, id=59e62375-f1da-461d-ee61-d105ac3159a0, file_name=finalJNB.csv, file_relativePath=finalJNB.csv, timestamp=1545285785034}]
2018-12-20 08:03:07.530 INFO 6668 --- [ask-scheduler-7] o.s.integration.ftp.session.FtpSession : File has been successfully transferred to: /ftp/erbranch/EDMS/FEFO/finalJNB.csv.writing
2018-12-20 08:03:08.539 INFO 6668 --- [ask-scheduler-7] o.s.integration.ftp.session.FtpSession : File has been successfully renamed from: /ftp/erbranch/EDMS/FEFO/finalJNB.csv.writing to /ftp/erbranch/EDMS/FEFO/finalJNB.csv
.addFilter(new FileSystemPersistentAcceptOnceFileListFilter(new SimpleMetadataStore(), "foo"))),
The SimpleMetadataStore() is in-memory - using a new store for each flow means it starts empty each time - you need to use a shared metadata store across all the flows.
Also, if you need to prevent duplicates across server restarts, you need to use a persistent shared store, e.g. Redis, JDBC etc.
Can I have a job with just the slaves and no master and listen to a rabbitmq queue? I want to listen to a queue and process the messages in chunk oriented manner using spring batch and spring integration in a spring boot app.
I want to use the chunkProcessorChunkHandler configuration explained in the RemoteChunking example for Spring batch by Michael Minella (https://www.youtube.com/watch?v=30Tdp1mfR0g), but without a master configuration.
Below is my configuration for the job.
#Configuration
#EnableIntegration
public class QueueIntegrationConfiguration {
#Autowired
private CassandraItemWriter cassandraItemWriter;
#Autowired
private VendorProcessor vendorProcessor;
#Autowired
ConnectionFactory connectionFactory;
#Bean
public AmqpInboundChannelAdapter inboundChannelAdapter(
SimpleMessageListenerContainer listenerContainer) {
AmqpInboundChannelAdapter adapter = new AmqpInboundChannelAdapter(listenerContainer);
adapter.setOutputChannel(inboundQueueChannel());
adapter.setAutoStartup(true);
return adapter;
}
#Bean
public SimpleMessageListenerContainer listenerContainer(ConnectionFactory connectionFactory, MessageConverter jsonMessageConverter) {
SimpleMessageListenerContainer container = new SimpleMessageListenerContainer(
connectionFactory);
container.setQueueNames("ProductStore_Partial");
container.setAutoStartup(true);
container.setMessageConverter(jsonMessageConverter);
return container;
}
#Bean
#ServiceActivator(inputChannel = "ProductStore_Partial")
public ChunkProcessorChunkHandler chunkProcessorChunkHandler()
throws Exception {
SimpleChunkProcessor chunkProcessor = new SimpleChunkProcessor(vendorProcessor,
cassandraItemWriter);
chunkProcessor.afterPropertiesSet();
ChunkProcessorChunkHandler<Vendor> chunkHandler = new ChunkProcessorChunkHandler<>();
chunkHandler.setChunkProcessor(chunkProcessor);
chunkHandler.afterPropertiesSet();
return chunkHandler;
}
#Bean
public QueueChannel inboundQueueChannel() {
return new QueueChannel().;
}
}
Below is my Application.java class for spring boot.
#SpringBootApplication
#EnableBatchProcessing
public class BulkImportProductApplication {
public static void main(String[] args) {
SpringApplication app = new SpringApplication(BulkImportProductApplication.class);
app.setWebEnvironment(false);
app.run(args).close();
}
}
From what I understand from spring integration, I have an AmqpInboundChannelAdapter for listening to messages from the queue. A ServiceActivator, an inboundQueueChannel, autowired ItemProcessor and ItemWriter. I am not sure what am I missing here.
The batch job starts, consumes one message from the queue and get a cancelOk and my job terminates without processing the message.
I am also sharing my debug logging if that would help.
2017-12-04 09:58:49.679 INFO 7450 --- [ main] c.a.s.p.b.BulkImportProductApplication : Started BulkImportProductApplication in 9.412 seconds (JVM running for 10.39)
2017-12-04 09:58:49.679 INFO 7450 --- [ main] s.c.a.AnnotationConfigApplicationContext : Closing org.springframework.context.annotation.AnnotationConfigApplicationContext#31c88ec8: startup date [Mon Dec 04 09:58:40 PST 2017]; root of context hierarchy
2017-12-04 09:58:49.679 DEBUG 7450 --- [ main] o.s.b.f.s.DefaultListableBeanFactory : Returning cached instance of singleton bean 'org.springframework.integration.config.IdGeneratorConfigurer#0'
2017-12-04 09:58:49.680 DEBUG 7450 --- [ main] o.s.b.f.s.DefaultListableBeanFactory : Returning cached instance of singleton bean 'inboundChannelAdapter'
2017-12-04 09:58:49.680 DEBUG 7450 --- [ main] o.s.b.f.s.DefaultListableBeanFactory : Returning cached instance of singleton bean 'listenerContainer'
2017-12-04 09:58:49.680 DEBUG 7450 --- [ main] o.s.b.f.s.DefaultListableBeanFactory : Returning cached instance of singleton bean 'integrationHeaderChannelRegistry'
2017-12-04 09:58:49.680 DEBUG 7450 --- [ main] o.s.b.f.s.DefaultListableBeanFactory : Returning cached instance of singleton bean 'org.springframework.amqp.rabbit.config.internalRabbitListenerEndpointRegistry'
2017-12-04 09:58:49.680 DEBUG 7450 --- [ main] o.s.b.f.s.DefaultListableBeanFactory : Returning cached instance of singleton bean '_org.springframework.integration.errorLogger'
2017-12-04 09:58:49.680 DEBUG 7450 --- [ main] o.s.b.f.s.DefaultListableBeanFactory : Returning cached instance of singleton bean 'queueIntegrationConfiguration.chunkProcessorChunkHandler.serviceActivator.handler'
2017-12-04 09:58:49.680 DEBUG 7450 --- [ main] o.s.b.f.s.DefaultListableBeanFactory : Returning cached instance of singleton bean 'queueIntegrationConfiguration.chunkProcessorChunkHandler.serviceActivator'
2017-12-04 09:58:49.680 DEBUG 7450 --- [ main] o.s.b.f.s.DefaultListableBeanFactory : Returning cached instance of singleton bean 'lifecycleProcessor'
2017-12-04 09:58:49.680 INFO 7450 --- [ main] o.s.c.support.DefaultLifecycleProcessor : Stopping beans in phase 2147483647
2017-12-04 09:58:49.680 DEBUG 7450 --- [ main] o.s.c.support.DefaultLifecycleProcessor : Asking bean 'inboundChannelAdapter' of type [class org.springframework.integration.amqp.inbound.AmqpInboundChannelAdapter] to stop
2017-12-04 09:58:49.680 DEBUG 7450 --- [ main] o.s.a.r.l.SimpleMessageListenerContainer : Shutting down Rabbit listener container
2017-12-04 09:58:49.814 DEBUG 7450 --- [pool-1-thread-5] o.s.a.r.listener.BlockingQueueConsumer : Storing delivery for Consumer#7c52fc81: tags=[{}], channel=Cached Rabbit Channel: AMQChannel(amqp://admin#xxxx:5672/,2), conn: Proxy#26f1249d Shared Rabbit Connection: SimpleConnection#680bddf5 [delegate=amqp://admin#xxxx:5672/, localPort= 65035], acknowledgeMode=AUTO local queue size=0
2017-12-04 09:58:49.814 DEBUG 7450 --- [enerContainer-1] o.s.a.r.listener.BlockingQueueConsumer : Received message: (Body:'[B#358a5358(byte[618])' MessageProperties [headers={__TypeId__=com.art.service.product.bulkimportproduct.data.model.Vendor}, timestamp=null, messageId=null, userId=null, receivedUserId=null, appId=null, clusterId=null, type=null, correlationId=null, correlationIdString=null, replyTo=null, contentType=json, contentEncoding=UTF-8, contentLength=0, deliveryMode=null, receivedDeliveryMode=NON_PERSISTENT, expiration=null, priority=0, redelivered=false, receivedExchange=ProductStore, receivedRoutingKey=, receivedDelay=null, deliveryTag=2, messageCount=0, consumerTag=amq.ctag-nWGbRxjFiaeTEoZylv6Hrg, consumerQueue=null])
2017-12-04 09:58:49.815 DEBUG 7450 --- [enerContainer-1] s.i.m.AbstractHeaderMapper$HeaderMatcher : headerName=[amqp_receivedDeliveryMode] WILL be mapped, matched pattern=*
2017-12-04 09:58:49.815 DEBUG 7450 --- [enerContainer-1] s.i.m.AbstractHeaderMapper$HeaderMatcher : headerName=[amqp_contentEncoding] WILL be mapped, matched pattern=*
2017-12-04 09:58:49.815 DEBUG 7450 --- [enerContainer-1] s.i.m.AbstractHeaderMapper$HeaderMatcher : headerName=[amqp_receivedExchange] WILL be mapped, matched pattern=*
2017-12-04 09:58:49.815 DEBUG 7450 --- [enerContainer-1] s.i.m.AbstractHeaderMapper$HeaderMatcher : headerName=[amqp_deliveryTag] WILL be mapped, matched pattern=*
2017-12-04 09:58:49.815 DEBUG 7450 --- [enerContainer-1] s.i.m.AbstractHeaderMapper$HeaderMatcher : headerName=[json__TypeId__] WILL be mapped, matched pattern=*
2017-12-04 09:58:49.815 DEBUG 7450 --- [enerContainer-1] s.i.m.AbstractHeaderMapper$HeaderMatcher : headerName=[amqp_redelivered] WILL be mapped, matched pattern=*
2017-12-04 09:58:49.815 DEBUG 7450 --- [enerContainer-1] s.i.m.AbstractHeaderMapper$HeaderMatcher : headerName=[contentType] WILL be mapped, matched pattern=*
2017-12-04 09:58:49.815 DEBUG 7450 --- [enerContainer-1] s.i.m.AbstractHeaderMapper$HeaderMatcher : headerName=[__TypeId__] WILL be mapped, matched pattern=*
2017-12-04 09:58:49.815 DEBUG 7450 --- [enerContainer-1] o.s.integration.channel.QueueChannel : preSend on channel 'inboundQueueChannel', message: GenericMessage [payload=byte[618], headers={amqp_receivedDeliveryMode=NON_PERSISTENT, amqp_contentEncoding=UTF-8, amqp_receivedExchange=ProductStore, amqp_deliveryTag=2, json__TypeId__=com.art.service.product.bulkimportproduct.data.model.Vendor, amqp_redelivered=false, id=a4868670-240f-ddf2-8a8c-ac4b8d234cdd, amqp_consumerTag=amq.ctag-nWGbRxjFiaeTEoZylv6Hrg, contentType=json, __TypeId__=com.art.service.product.bulkimportproduct.data.model.Vendor, timestamp=1512410329815}]
2017-12-04 09:58:49.815 DEBUG 7450 --- [enerContainer-1] o.s.integration.channel.QueueChannel : postSend (sent=true) on channel 'inboundQueueChannel', message: GenericMessage [payload=byte[618], headers={amqp_receivedDeliveryMode=NON_PERSISTENT, amqp_contentEncoding=UTF-8, amqp_receivedExchange=ProductStore, amqp_deliveryTag=2, json__TypeId__=com.art.service.product.bulkimportproduct.data.model.Vendor, amqp_redelivered=false, id=a4868670-240f-ddf2-8a8c-ac4b8d234cdd, amqp_consumerTag=amq.ctag-nWGbRxjFiaeTEoZylv6Hrg, contentType=json, __TypeId__=com.art.service.product.bulkimportproduct.data.model.Vendor, timestamp=1512410329815}]
2017-12-04 09:58:49.853 INFO 7450 --- [ main] o.s.a.r.l.SimpleMessageListenerContainer : Waiting for workers to finish.
2017-12-04 09:58:49.853 DEBUG 7450 --- [pool-1-thread-6] o.s.a.r.listener.BlockingQueueConsumer : Received cancelOk for tag amq.ctag-nWGbRxjFiaeTEoZylv6Hrg (null); Consumer#7c52fc81: tags=[{}], channel=Cached Rabbit Channel: AMQChannel(amqp://admin#xxxx:5672/,2), conn: Proxy#26f1249d Shared Rabbit Connection: SimpleConnection#680bddf5 [delegate=amqp://admin#xxxx:5672/, localPort= 65035], acknowledgeMode=AUTO local queue size=0
2017-12-04 09:58:49.853 DEBUG 7450 --- [enerContainer-1] o.s.a.r.l.SimpleMessageListenerContainer : Cancelling Consumer#7c52fc81: tags=[{}], channel=Cached Rabbit Channel: AMQChannel(amqp://admin#xxxx:5672/,2), conn: Proxy#26f1249d Shared Rabbit Connection: SimpleConnection#680bddf5 [delegate=amqp://admin#xxxx:5672/, localPort= 65035], acknowledgeMode=AUTO local queue size=0
2017-12-04 09:58:49.853 DEBUG 7450 --- [enerContainer-1] o.s.a.r.listener.BlockingQueueConsumer : Closing Rabbit Channel: Cached Rabbit Channel: AMQChannel(amqp://admin#xxxx:5672/,2), conn: Proxy#26f1249d Shared Rabbit Connection: SimpleConnection#680bddf5 [delegate=amqp://admin#xxxx:5672/, localPort= 65035]
2017-12-04 09:58:49.853 DEBUG 7450 --- [enerContainer-1] o.s.a.r.c.CachingConnectionFactory : Closing cached Channel: AMQChannel(amqp://admin#xxxx:5672/,2)
2017-12-04 09:58:50.027 INFO 7450 --- [ main] o.s.a.r.l.SimpleMessageListenerContainer : Successfully waited for workers to finish.
2017-12-04 09:58:50.027 DEBUG 7450 --- [ main] o.s.c.support.DefaultLifecycleProcessor : Bean 'inboundChannelAdapter' completed its stop procedure
What am I missing here? Why is my message getting processed? Please correct me if I'm missing out something here? Also feel free to ask any other configuration that you feel would help analyze the situation here.
EDIT: After removing the code that closes the application context manually( app.run(args).close() ), I was able to receive the messages, but looks like they are lost after a successful retrieve. sharing the debug log for this behavior.
2017-12-04 14:39:11.297 DEBUG 1498 --- [pool-1-thread-5] o.s.a.r.listener.BlockingQueueConsumer : Storing delivery for Consumer#7219ac49: tags=[{amq.ctag-Z8siptJMdxGU6sXdOHkVCA=ProductStore_Partial}], channel=Cached Rabbit Channel: AMQChannel(amqp://admin#xxxx:5672/,2), conn: Proxy#6df20ade Shared Rabbit Connection: SimpleConnection#7ba63fe5 [delegate=amqp://admin#xxxx:5672/, localPort= 51172], acknowledgeMode=AUTO local queue size=0
2017-12-04 14:39:11.297 DEBUG 1498 --- [enerContainer-1] o.s.a.r.listener.BlockingQueueConsumer : Received message: (Body:'[B#347c8f87(byte[624])' MessageProperties [headers={__TypeId__=com.art.service.product.bulkimportproduct.data.model.Vendor}, timestamp=null, messageId=null, userId=null, receivedUserId=null, appId=null, clusterId=null, type=null, correlationId=null, correlationIdString=null, replyTo=null, contentType=json, contentEncoding=UTF-8, contentLength=0, deliveryMode=null, receivedDeliveryMode=NON_PERSISTENT, expiration=null, priority=0, redelivered=false, receivedExchange=ProductStore, receivedRoutingKey=, receivedDelay=null, deliveryTag=2, messageCount=0, consumerTag=amq.ctag-Z8siptJMdxGU6sXdOHkVCA, consumerQueue=ProductStore_Partial])
2017-12-04 14:39:11.297 DEBUG 1498 --- [enerContainer-1] s.i.m.AbstractHeaderMapper$HeaderMatcher : headerName=[amqp_receivedDeliveryMode] WILL be mapped, matched pattern=*
2017-12-04 14:39:11.297 DEBUG 1498 --- [enerContainer-1] s.i.m.AbstractHeaderMapper$HeaderMatcher : headerName=[amqp_contentEncoding] WILL be mapped, matched pattern=*
2017-12-04 14:39:11.297 DEBUG 1498 --- [enerContainer-1] s.i.m.AbstractHeaderMapper$HeaderMatcher : headerName=[amqp_receivedExchange] WILL be mapped, matched pattern=*
2017-12-04 14:39:11.297 DEBUG 1498 --- [enerContainer-1] s.i.m.AbstractHeaderMapper$HeaderMatcher : headerName=[amqp_deliveryTag] WILL be mapped, matched pattern=*
2017-12-04 14:39:11.297 DEBUG 1498 --- [enerContainer-1] s.i.m.AbstractHeaderMapper$HeaderMatcher : headerName=[json__TypeId__] WILL be mapped, matched pattern=*
2017-12-04 14:39:11.297 DEBUG 1498 --- [enerContainer-1] s.i.m.AbstractHeaderMapper$HeaderMatcher : headerName=[amqp_redelivered] WILL be mapped, matched pattern=*
2017-12-04 14:39:11.297 DEBUG 1498 --- [enerContainer-1] s.i.m.AbstractHeaderMapper$HeaderMatcher : headerName=[contentType] WILL be mapped, matched pattern=*
2017-12-04 14:39:11.297 DEBUG 1498 --- [enerContainer-1] s.i.m.AbstractHeaderMapper$HeaderMatcher : headerName=[__TypeId__] WILL be mapped, matched pattern=*
2017-12-04 14:39:11.297 DEBUG 1498 --- [enerContainer-1] o.s.integration.channel.QueueChannel : preSend on channel 'inboundQueueChannel', message: GenericMessage [payload=byte[624], headers={amqp_receivedDeliveryMode=NON_PERSISTENT, amqp_contentEncoding=UTF-8, amqp_receivedExchange=ProductStore, amqp_deliveryTag=2, json__TypeId__=com.art.service.product.bulkimportproduct.data.model.Vendor, amqp_consumerQueue=ProductStore_Partial, amqp_redelivered=false, id=540399a5-62a6-7178-2524-e274bad4ed13, amqp_consumerTag=amq.ctag-Z8siptJMdxGU6sXdOHkVCA, contentType=json, __TypeId__=com.art.service.product.bulkimportproduct.data.model.Vendor, timestamp=1512427151297}]
2017-12-04 14:39:11.297 DEBUG 1498 --- [enerContainer-1] o.s.integration.channel.QueueChannel : postSend (sent=true) on channel 'inboundQueueChannel', message: GenericMessage [payload=byte[624], headers={amqp_receivedDeliveryMode=NON_PERSISTENT, amqp_contentEncoding=UTF-8, amqp_receivedExchange=ProductStore, amqp_deliveryTag=2, json__TypeId__=com.art.service.product.bulkimportproduct.data.model.Vendor, amqp_consumerQueue=ProductStore_Partial, amqp_redelivered=false, id=540399a5-62a6-7178-2524-e274bad4ed13, amqp_consumerTag=amq.ctag-Z8siptJMdxGU6sXdOHkVCA, contentType=json, __TypeId__=com.art.service.product.bulkimportproduct.data.model.Vendor, timestamp=1512427151297}]
2017-12-04 14:39:11.297 DEBUG 1498 --- [enerContainer-1] o.s.a.r.listener.BlockingQueueConsumer : Retrieving delivery for Consumer#7219ac49: tags=[{amq.ctag-Z8siptJMdxGU6sXdOHkVCA=ProductStore_Partial}], channel=Cached Rabbit Channel: AMQChannel(amqp://admin#xxxx:5672/,2), conn: Proxy#6df20ade Shared Rabbit Connection: SimpleConnection#7ba63fe5 [delegate=amqp://admin#xxxx:5672/, localPort= 51172], acknowledgeMode=AUTO local queue size=0
This goes on repeating and new messages are consumed, but the messages are not getting processed and written to the data-store using the itemWriter provided. Now come to think of it, since I have not provided the tasklet/step bean reference anywhere in this code, is that something I am missing out here?
app.run(args).close();
You are explicitly closing the application context, which shuts everything down.
Closing org.springframework.context.annotation.AnnotationConfigApplicationContext.
I'm trying to get all the files from a ftp directory. My configuration:
<bean id="ftpSessionFactory"
class="org.springframework.integration.ftp.session.DefaultFtpSessionFactory">
<property name="host" value="ftp.aaaaaa.bb"/>
<property name="port" value="21"/>
<property name="clientMode" value="2"/>
<property name="username" value="XXXXXXX"/>
<property name="password" value="YYYYYYY"/>
</bean>
<int:channel id="inbound"/>
<int:inbound-channel-adapter channel="inbound" expression="'GB2AV'">
<int:poller fixed-rate="10000"/>
</int:inbound-channel-adapter>
<int-ftp:outbound-gateway id="gatewayLS"
session-factory="ftpSessionFactory"
request-channel="inbound"
command="ls"
command-options="-1"
filename-pattern="*"
remote-file-separator="\"
expression="payload"
reply-channel="outbound">
</int-ftp:outbound-gateway>
<int:channel id="outbound"/>
<int-stream:stdout-channel-adapter channel="outbound" append-newline="true"/>
It works great against my own test server (Ubuntu / vsftp). Against the production server (unknown OS and ftp server) I'm not able to get the file names. Debug output:
2016-11-01 16:58:17.017 DEBUG 3957 --- [ask-scheduler-1] o.s.integration.channel.DirectChannel : postSend (sent=true) on channel 'outbound', message: GenericMessage [payload=[], headers={file_remoteDirectory=GB2AV\, id=a1a43b85-f086-fd0a-cbbb-a29548578820, timestamp=1478015897017}]
2016-11-01 16:58:17.017 DEBUG 3957 --- [ask-scheduler-1] o.s.integration.channel.DirectChannel : postSend (sent=true) on channel 'inbound', message: GenericMessage [payload=GB2AV, headers={id=e4e4e425-f25d-e4de-7cd8-8912ecd5151b, timestamp=1478015895998}]
2016-11-01 16:58:17.130 DEBUG 3957 --- [-ClientPoller-0] org.apache.tomcat.util.net.NioEndpoint : timeout completed: keys processed=0; now=1478015897130; nextExpiration=1478015897125; keyCount=0; hasEvents=false; eval=false
2016-11-01 16:58:17.130 DEBUG 3957 --- [-ClientPoller-1] org.apache.tomcat.util.net.NioEndpoint : timeout completed: keys processed=0; now=1478015897130; nextExpiration=1478015897125; keyCount=0; hasEvents=false; eval=false
2016-11-01 16:58:18.135 DEBUG 3957 --- [-ClientPoller-1] org.apache.tomcat.util.net.NioEndpoint : timeout completed: keys processed=0; now=1478015898134; nextExpiration=1478015898130; keyCount=0; hasEvents=false; eval=false
2016-11-01 16:58:18.135 DEBUG 3957 --- [-ClientPoller-0] org.apache.tomcat.util.net.NioEndpoint : timeout completed: keys processed=0; now=1478015898134; nextExpiration=1478015898130; keyCount=0; hasEvents=false; eval=false
2016-11-01 16:58:19.136 DEBUG 3957 --- [-ClientPoller-0] org.apache.tomcat.util.net.NioEndpoint : timeout completed: keys processed=0; now=1478015899136; nextExpiration=1478015899134; keyCount=0; hasEvents=false; eval=false
2016-11-01 16:58:19.136 DEBUG 3957 --- [-ClientPoller-1] org.apache.tomcat.util.net.NioEndpoint : timeout completed: keys processed=0; now=1478015899136; nextExpiration=1478015899134; keyCount=0; hasEvents=false; eval=false
2016-11-01 16:58:20.137 DEBUG 3957 --- [-ClientPoller-1] org.apache.tomcat.util.net.NioEndpoint : timeout completed: keys processed=0; now=1478015900137; nextExpiration=1478015900136; keyCount=0; hasEvents=false; eval=false
2016-11-01 16:58:20.137 DEBUG 3957 --- [-ClientPoller-0] org.apache.tomcat.util.net.NioEndpoint : timeout completed: keys processed=0; now=1478015900137; nextExpiration=1478015900136; keyCount=0; hasEvents=false; eval=false
2016-11-01 16:58:21.142 DEBUG 3957 --- [-ClientPoller-0] org.apache.tomcat.util.net.NioEndpoint : timeout completed: keys processed=0; now=1478015901142; nextExpiration=1478015901137; keyCount=0; hasEvents=false; eval=false
2016-11-01 16:58:21.142 DEBUG 3957 --- [-ClientPoller-1] org.apache.tomcat.util.net.NioEndpoint : timeout completed: keys processed=0; now=1478015901142; nextExpiration=1478015901137; keyCount=0; hasEvents=false; eval=false
2016-11-01 16:58:22.147 DEBUG 3957 --- [-ClientPoller-0] org.apache.tomcat.util.net.NioEndpoint : timeout completed: keys processed=0; now=1478015902146; nextExpiration=1478015902142; keyCount=0; hasEvents=false; eval=false
2016-11-01 16:58:22.147 DEBUG 3957 --- [-ClientPoller-1] org.apache.tomcat.util.net.NioEndpoint : timeout completed: keys processed=0; now=1478015902146; nextExpiration=1478015902142; keyCount=0; hasEvents=false; eval=false
2016-11-01 16:58:23.151 DEBUG 3957 --- [-ClientPoller-1] org.apache.tomcat.util.net.NioEndpoint : timeout completed: keys processed=0; now=1478015903151; nextExpiration=1478015903146; keyCount=0; hasEvents=false; eval=false
2016-11-01 16:58:23.151 DEBUG 3957 --- [-ClientPoller-0] org.apache.tomcat.util.net.NioEndpoint : timeout completed: keys processed=0; now=1478015903151; nextExpiration=1478015903146; keyCount=0; hasEvents=false; eval=false
2016-11-01 16:58:24.154 DEBUG 3957 --- [-ClientPoller-0] org.apache.tomcat.util.net.NioEndpoint : timeout completed: keys processed=0; now=1478015904154; nextExpiration=1478015904151; keyCount=0; hasEvents=false; eval=false
2016-11-01 16:58:24.154 DEBUG 3957 --- [-ClientPoller-1] org.apache.tomcat.util.net.NioEndpoint : timeout completed: keys processed=0; now=1478015904154; nextExpiration=1478015904151; keyCount=0; hasEvents=false; eval=false
2016-11-01 16:58:25.159 DEBUG 3957 --- [-ClientPoller-0] org.apache.tomcat.util.net.NioEndpoint : timeout completed: keys processed=0; now=1478015905159; nextExpiration=1478015905154; keyCount=0; hasEvents=false; eval=false
2016-11-01 16:58:25.159 DEBUG 3957 --- [-ClientPoller-1] org.apache.tomcat.util.net.NioEndpoint : timeout completed: keys processed=0; now=1478015905159; nextExpiration=1478015905154; keyCount=0; hasEvents=false; eval=false
2016-11-01 16:58:25.995 DEBUG 3957 --- [ask-scheduler-3] o.s.i.e.SourcePollingChannelAdapter : Poll resulted in Message: GenericMessage [payload=GB2AV, headers={id=ec5a8b67-1b8e-d675-ad3e-e0ee5f4f955d, timestamp=1478015905995}]
2016-11-01 16:58:25.995 DEBUG 3957 --- [ask-scheduler-3] o.s.integration.channel.DirectChannel : preSend on channel 'inbound', message: GenericMessage [payload=GB2AV, headers={id=ec5a8b67-1b8e-d675-ad3e-e0ee5f4f955d, timestamp=1478015905995}]
2016-11-01 16:58:25.995 DEBUG 3957 --- [ask-scheduler-3] o.s.i.ftp.gateway.FtpOutboundGateway : org.springframework.integration.ftp.gateway.FtpOutboundGateway#0 received message: GenericMessage [payload=GB2AV, headers={id=ec5a8b67-1b8e-d675-ad3e-e0ee5f4f955d, timestamp=1478015905995}]
2016-11-01 16:58:26.032 DEBUG 3957 --- [ask-scheduler-3] o.s.i.f.s.DefaultFtpSessionFactory : Connected to server [ftp.aaaaaaa.bb:21]
2016-11-01 16:58:26.159 DEBUG 3957 --- [-ClientPoller-0] org.apache.tomcat.util.net.NioEndpoint : timeout completed: keys processed=0; now=1478015906159; nextExpiration=1478015906159; keyCount=0; hasEvents=false; eval=false
2016-11-01 16:58:26.159 DEBUG 3957 --- [-ClientPoller-1] org.apache.tomcat.util.net.NioEndpoint : timeout completed: keys processed=0; now=1478015906159; nextExpiration=1478015906159; keyCount=0; hasEvents=false; eval=false
2016-11-01 16:58:26.828 DEBUG 3957 --- [ask-scheduler-3] o.s.integration.channel.DirectChannel : preSend on channel 'outbound', message: GenericMessage [payload=[], headers={file_remoteDirectory=GB2AV\, id=08ff9eb6-31e2-a575-a3ef-91ad4616f5f0, timestamp=1478015906828}]
2016-11-01 16:58:26.828 DEBUG 3957 --- [ask-scheduler-3] i.s.CharacterStreamWritingMessageHandler : org.springframework.integration.stream.CharacterStreamWritingMessageHandler#0 received message: GenericMessage [payload=[], headers={file_remoteDirectory=GB2AV\, id=08ff9eb6-31e2-a575-a3ef-91ad4616f5f0, timestamp=1478015906828}]
When connecting to the ftp server manually I can change the directory with cd GB2AV and show all files in this directory with ls. I also tried all kind of combinations of file separators etc. etc.
When working with org.apache.commons.net.ftp.FTPClient I have to use .listNames() which returns String[] instead of .listFiles() which returns an empty object in my case.
Is there anything I can do in the gateway configuration or does it sound like some missing feature support on the server side?
I am using spring integration in on of my project. And following the is the flow with in my application. When ever I am publishing a messaged to "domainEventChannel" , it goes through the transformer and then messages is sent to inputFromTransformer channel and from this as message is in chain it goes through splitter , transformer and filter. I have configured output-channel="domainEventQueueChannel" ,which is a queue channel.
I am able to track messages till the filter, but they never reach the QueueChannel.
Can I not sent a message to queue channel from a chain ?
<int:transformer ref="partitionlessTransformer" method="transform"
input-channel="domainEventChannel" output-channel="inputFromTransformer" />
<!-- Chain which has the output channel as queue channel-->
<int:chain input-channel="inputFromTransformer"
output-channel="domainEventQueueChannel">
<int:splitter ref="messageSplitter" method="split" />
<int:transformer ref="jsonToObjectTransformer" />
<int:filter ref="autopayModelProcessTransFilter"></int:filter>
</int:chain>
<!-- Queue Channel-->
<int:channel id="domainEventQueueChannel">
<int:queue capacity="10" />
</int:channel>
<!-- service activator which polls queueChannel -->
<int:service-activator id="domainEventReconProcessServiceActivator"
input-channel="domainEventQueueChannel" ref="domainEventReconProcessServiceActivator" method="intiateAutopayProcessTransRecon">
<int:poller task-executor="domainEventReconProcessServicetaskExecutor" fixed-rate="10" >
</int:poller>
</int:service-activator>
<task:executor id="domainEventReconProcessServicetaskExecutor"
pool-size="10" queue-capacity="10" />
Update 2#:
When I add the following piece of config everything works fine. Message flows from queue channel to service activator:
<int:chain input-channel="domainEventQueueChannel"
output-channel="nullChannel">
<int:service-activator id="domainEventReconProcessServiceActivator"
ref="domainEventReconProcessServiceActivator" method="intiateAutopayProcessTransRecon" />
<int:poller task-executor="domainEventReconProcessServicetaskExecutor" fixed-rate="10"></int:poller>
</int:chain>
Update 3#:
With the chain in place I see the service activator's poller polling the queueChannel. But without the chain , the poller is not starting at all I don't see poller logs at all.
With Chain in place:
2015-09-08 09:29:31.438 DEBUG 12817 --- [ask-scheduler-8] o.s.integration.channel.QueueChannel : preSend on channel 'domainEventQueueChannel', message: GenericMessage [payload=com.autopayprocesstransrecon.resource.DomainEvent#6ac4fcc0, headers={sequenceNumber=1, correlationId=7a91b5fb-b1aa-464a-2e1b-5309652a0520, id=25ae85aa-e0df-6c72-9a54-2e8dac3c5ddc, sequenceSize=1, timestamp=1441718971437}]
2015-09-08 09:29:31.438 DEBUG 12817 --- [ask-scheduler-8] o.s.integration.channel.QueueChannel : postSend (sent=true) on channel 'domainEventQueueChannel', message: GenericMessage [payload=com.autopayprocesstransrecon.resource.DomainEvent#6ac4fcc0, headers={sequenceNumber=1, correlationId=7a91b5fb-b1aa-464a-2e1b-5309652a0520, id=25ae85aa-e0df-6c72-9a54-2e8dac3c5ddc, sequenceSize=1, timestamp=1441718971437}]
2015-09-08 09:29:31.438 DEBUG 12817 --- [ask-scheduler-8] o.s.integration.channel.DirectChannel : postSend (sent=true) on channel 'inputFromTransformer', message: GenericMessage [payload=[{"eventInfo":{"eventId":"4535345353","eventName":"AutopayModelsExtracted","parentEventId":"4535345353"},"numOfAutopayModels":20,"message":"Published autopaymodels needed to be processed"}], headers={id=7a91b5fb-b1aa-464a-2e1b-5309652a0520, timestamp=1441718971410}]
2015-09-08 09:29:31.438 DEBUG 12817 --- [ask-scheduler-8] o.s.integration.channel.DirectChannel : postSend (sent=true) on channel 'domainEventChannel', message: GenericMessage [payload={domainevents={0=[{"eventInfo":{"eventId":"4535345353","eventName":"AutopayModelsExtracted","parentEventId":"4535345353"},"numOfAutopayModels":20,"message":"Published autopaymodels needed to be processed"}]}}, headers={id=438c63c8-1ed1-6516-110e-c3478aca35c8, timestamp=1441718971409}]
2015-09-08 09:29:31.438 DEBUG 12817 --- [etaskExecutor-1] o.s.integration.channel.QueueChannel : postReceive on channel 'domainEventQueueChannel', message: GenericMessage [payload=com.autopayprocesstransrecon.resource.DomainEvent#6ac4fcc0, headers={sequenceNumber=1, correlationId=7a91b5fb-b1aa-464a-2e1b-5309652a0520, id=25ae85aa-e0df-6c72-9a54-2e8dac3c5ddc, sequenceSize=1, timestamp=1441718971437}]
2015-09-08 09:29:31.438 DEBUG 12817 --- [etaskExecutor-1] o.s.i.endpoint.PollingConsumer : Poll resulted in Message: GenericMessage [payload=com.autopayprocesstransrecon.resource.DomainEvent#6ac4fcc0, headers={sequenceNumber=1, correlationId=7a91b5fb-b1aa-464a-2e1b-5309652a0520, id=25ae85aa-e0df-6c72-9a54-2e8dac3c5ddc, sequenceSize=1, timestamp=1441718971437}]
2015-09-08 09:29:31.438 DEBUG 12817 --- [etaskExecutor-1] o.s.i.handler.MessageHandlerChain : org.springframework.integration.handler.MessageHandlerChain#1 received message: GenericMessage [payload=com.autopayprocesstransrecon.resource.DomainEvent#6ac4fcc0, headers={sequenceNumber=1, correlationId=7a91b5fb-b1aa-464a-2e1b-5309652a0520, id=25ae85aa-e0df-6c72-9a54-2e8dac3c5ddc, sequenceSize=1, timestamp=1441718971437}]
2015-09-08 09:29:31.438 DEBUG 12817 --- [etaskExecutor-1] o.s.i.handler.ServiceActivatingHandler : ServiceActivator for [org.springframework.integration.handler.MethodInvokingMessageProcessor#72193fb5] (org.springframework.integration.handler.MessageHandlerChain#1$child.domainEventReconProcessServiceActivator) received message: GenericMessage [payload=com.autopayprocesstransrecon.resource.DomainEvent#6ac4fcc0, headers={sequenceNumber=1, correlationId=7a91b5fb-b1aa-464a-2e1b-5309652a0520, id=25ae85aa-e0df-6c72-9a54-2e8dac3c5ddc, sequenceSize=1, timestamp=1441718971437}]`
Without Chain:
2015-09-08 09:48:02.572 DEBUG 13043 --- [ask-scheduler-1] o.s.i.splitter.MethodInvokingSplitter : org.springframework.integration.splitter.MethodInvokingSplitter#53abfc07 received message: GenericMessage [payload=[{"eventInfo":{"eventId":"4535345353","eventName":"AutopayModelsExtracted","parentEventId":"4535345353"},"numOfAutopayModels":20,"message":"Published autopaymodels needed to be processed"}], headers={id=f477f025-e92f-f896-9ac4-2ce1b91fb895, timestamp=1441720082572}]
2015-09-08 09:48:02.574 DEBUG 13043 --- [ask-scheduler-1] o.s.i.t.MessageTransformingHandler : org.springframework.integration.transformer.MessageTransformingHandler#2c8c16c0 received message: GenericMessage [payload={"eventInfo":{"eventId":"4535345353","eventName":"AutopayModelsExtracted","parentEventId":"4535345353"},"numOfAutopayModels":20,"message":"Published autopaymodels needed to be processed"}, headers={sequenceNumber=1, correlationId=f477f025-e92f-f896-9ac4-2ce1b91fb895, id=1364077a-6d5c-87f9-7d7c-3d144e45d661, sequenceSize=1, timestamp=1441720082574}]
2015-09-08 09:48:02.599 DEBUG 13043 --- [ask-scheduler-1] o.s.integration.filter.MessageFilter : org.springframework.integration.filter.MessageFilter#80bfa9d received message: GenericMessage [payload=com.autopayprocesstransrecon.resource.DomainEvent#fd57501, headers={sequenceNumber=1, correlationId=f477f025-e92f-f896-9ac4-2ce1b91fb895, id=c2a324c8-9eb0-66e0-73a1-5595ed87b3ae, sequenceSize=1, timestamp=1441720082599}]
hey I am in filter and my result would betrue
2015-09-08 09:48:02.599 DEBUG 13043 --- [ask-scheduler-1] o.s.integration.channel.QueueChannel : preSend on channel 'domainEventQueueChannel', message: GenericMessage [payload=com.autopayprocesstransrecon.resource.DomainEvent#fd57501, headers={sequenceNumber=1, correlationId=f477f025-e92f-f896-9ac4-2ce1b91fb895, id=c2a324c8-9eb0-66e0-73a1-5595ed87b3ae, sequenceSize=1, timestamp=1441720082599}]
2015-09-08 09:48:02.599 DEBUG 13043 --- [ask-scheduler-1] o.s.integration.channel.QueueChannel : postSend (sent=true) on channel 'domainEventQueueChannel', message: GenericMessage [payload=com.autopayprocesstransrecon.resource.DomainEvent#fd57501, headers={sequenceNumber=1, correlationId=f477f025-e92f-f896-9ac4-2ce1b91fb895, id=c2a324c8-9eb0-66e0-73a1-5595ed87b3ae, sequenceSize=1, timestamp=1441720082599}]
2015-09-08 09:48:02.599 DEBUG 13043 --- [ask-scheduler-1] o.s.integration.channel.DirectChannel : postSend (sent=true) on channel 'inputFromTransformer', message: GenericMessage [payload=[{"eventInfo":{"eventId":"4535345353","eventName":"AutopayModelsExtracted","parentEventId":"4535345353"},"numOfAutopayModels":20,"message":"Published autopaymodels needed to be processed"}], headers={id=f477f025-e92f-f896-9ac4-2ce1b91fb895, timestamp=1441720082572}]
2015-09-08 09:48:02.599 DEBUG 13043 --- [ask-scheduler-1] o.s.integration.channel.DirectChannel : postSend (sent=true) on channel 'domainEventChannel', message: GenericMessage [payload={domainevents={0=[{"eventInfo":{"eventId":"4535345353","eventName":"AutopayModelsExtracted","parentEventId":"4535345353"},"numOfAutopayModels":20,"message":"Published autopaymodels needed to be processed"}]}}, headers={id=bf6f4f20-52bb-27e4-df97-4112ad4be3b5, timestamp=1441720082571}]
2015-09-08 09:48:03.616 DEBUG 13043 --- [ask-scheduler-3] o.s.i.e.SourcePollingChannelAdapter : Received no Message during the poll, returning 'false'
2015-09-08 09:48:04.634 DEBUG 13043 --- [ask-scheduler-4] o.s.i.e.SourcePollingChannelAdapter : Received no Message during the poll, returning 'false'
That simply means the message is not passing the filter.
(The filter is dropping the message).