dynamic delay configuration in spring poller - spring-integration

Polling dynamically
I am using poller component from integration to poll files from s3.The fixed delay is 15 min and max message rate is 1 .The reason i did this was in down stream messages in xd were clogging since I am using http.Now this is file for 100k records file but when file size is small i still wait 15 min though i can process fast.Now is there any way to dynamically set delay depending in size of file.Because we don't know which files will be polled also to know it?Depending on file size i will pick up or number of records can we change dynamically the fixed delay or fixed rate?
<int:poller fixed-delay="${fixedDelay}" default="true" max-messages-per-poll="${maxMessageRate}">
<int:advice-chain>
<ref bean="pollAdvise"/>
</int:advice-chain>
</int:poller>
<bean id="pollAdvise" class="org.springframework.integration.scheduling.PollSkipAdvice">
<constructor-arg ref="healthCheckStrategy"/>
</bean>
<bean id="healthCheckStrategy" class="test.ServiceHealthCheckPollSkipStrategy">
<property name="url" value="${url}"/>
<property name="doHealthCheck" value="${doHealthCheck}"/>
</bean>
<bean id="credentials" class="org.springframework.integration.aws.core.BasicAWSCredentials">
<property name="accessKey" value="${accessKey}"/>
<property name="secretKey" value="${secretKey}"/>
</bean>
<bean id="clientConfiguration" class="com.amazonaws.ClientConfiguration">
<property name="proxyHost" value="${proxyHost}"/>
<property name="proxyPort" value="${proxyPort}"/>
<property name="preemptiveBasicProxyAuth" value="false"/>
</bean>
<bean id="s3Operations" class="org.springframework.integration.aws.s3.core.CustomC1AmazonS3Operations">
<constructor-arg index="0" ref="credentials"/>
<constructor-arg index="1" ref="clientConfiguration"/>
<property name="awsEndpoint" value="s3.amazonaws.com"/>
<property name="temporaryDirectory" value="${temporaryDirectory}"/>
<property name="awsSecurityKey" value="${awsSecurityKey}"/>
</bean>
<!-- aws-endpoint="https://s3.amazonaws.com" -->
<int-aws:s3-inbound-channel-adapter aws-endpoint="s3.amazonaws.com"
bucket="${bucket}"
s3-operations="s3Operations"
credentials-ref="credentials"
file-name-wildcard="${fileNameWildcard}"
remote-directory="${remoteDirectory}"
channel="splitChannel"
local-directory="${localDirectory}"
accept-sub-folders="false"
delete-source-files="true"
archive-bucket="${archiveBucket}"
archive-directory="${archiveDirectory}">
</int-aws:s3-inbound-channel-adapter>
<int-file:splitter id="s3splitter" input-channel="splitChannel" output-channel="bridge" markers="false" charset="UTF-8">
<int-file:request-handler-advice-chain>
<bean class="org.springframework.integration.handler.advice.ExpressionEvaluatingRequestHandlerAdvice">
<property name="onSuccessExpression" value="payload.delete()"/>
</bean>
</int-file:request-handler-advice-chain>
</int-file:splitter>

Starting with Spring Integration 4.2 the AbstractMessageSourceAdvice has been introduced:
This method is called after the receive() method; again, you can reconfigure the source, or take any action perhaps depending on the result (which can be null if there was no message created by the source). You can even return a different message!
Starting with version 4.3 we introduce CompoundTriggerAdvice: http://docs.spring.io/spring-integration/docs/4.3.0.BUILD-SNAPSHOT/reference/html/messaging-channels-section.html#_compoundtriggeradvice
Which you can use for your use-case based on the payload size.

Related

File getting deleted before processing

I have spring xd module with rabbit transport which pulls files from s3 and split line by line and delete it after processing(ExpressionAdvice) .I have around 1 million messages(lines) in my file which is in s3.The file gets downloaded to xd container box and i checked md5sum and its same and has same lines . I see only 260k odd message are coming to output channel which is processor.I am loosing around 740 messages. sometimes it random once i see all messages like 1 million in my output channel and sometimes only 250k .I am measuring this using counter for my stream.File is downloaded but i feel its getting deleted before processing all records/lines in 10 seconds, my file size is around 700Mb.Please let me know if expression advice is deleting before processing.
module.aws-s3-source.count=1 and module.aws-s3-source.concurrency=70
stream1 as-s3-source |processor|sink
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:int="http://www.springframework.org/schema/integration"
xmlns:context="http://www.springframework.org/schema/context"
xmlns:int-aws="http://www.springframework.org/schema/integration/aws"
xmlns:int-file="http://www.springframework.org/schema/integration/file"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/integration/file
http://www.springframework.org/schema/integration/file/spring-integration-file.xsd
http://www.springframework.org/schema/context
http://www.springframework.org/schema/context/spring-context.xsd
http://www.springframework.org/schema/integration http://www.springframework.org/schema/integration/spring-integration.xsd
http://www.springframework.org/schema/integration/aws http://www.springframework.org/schema/integration/aws/spring-integration-aws-1.0.xsd">
<context:property-placeholder location="classpath*:test-${region}.properties" />
<int:poller fixed-delay="${fixedDelay}" default="true">
<int:advice-chain>
<ref bean="pollAdvise"/>
</int:advice-chain>
</int:poller>
<bean id="pollAdvise" class="org.springframework.integration.scheduling.PollSkipAdvice">
<constructor-arg ref="healthCheckStrategy"/>
</bean>
<bean id="healthCheckStrategy" class="test.ServiceHealthCheckPollSkipStrategy">
<property name="url" value="${url}"/>
<property name="doHealthCheck" value="${doHealthCheck}"/>
<property name="restTemplate" ref="restTemplate"/>
</bean>
<bean id="restTemplate"
class="org.springframework.web.client.RestTemplate">
<constructor-arg ref="requestFactory"/>
</bean>
<bean id="requestFactory"
class="test.BatchClientHttpRequestFactory">
<constructor-arg ref="verifier"/>
</bean>
<bean id="verifier"
class="test.NullHostnameVerifier">
</bean>
<bean id="encryptedDatum" class="test.EncryptedSecuredDatum"/>
<bean id="clientConfiguration" class="com.amazonaws.ClientConfiguration">
<property name="proxyHost" value="${proxyHost}"/>
<property name="proxyPort" value="${proxyPort}"/>
<property name="preemptiveBasicProxyAuth" value="false"/>
</bean>
<bean id="s3Operations" class="test.CustomC1AmazonS3Operations">
<constructor-arg index="0" ref="clientConfiguration"/>
<property name="awsEndpoint" value="s3.amazonaws.com"/>
<property name="temporaryDirectory" value="${temporaryDirectory}"/>
<property name="awsSecurityKey" value=""/>
</bean>
<bean id="credentials" class="org.springframework.integration.aws.core.BasicAWSCredentials">
</bean>
<int-aws:s3-inbound-channel-adapter aws-endpoint="s3.amazonaws.com"
bucket="${bucket}"
s3-operations="s3Operations"
credentials-ref="credentials"
file-name-wildcard="${fileNameWildcard}"
remote-directory="${prefix}"
channel="splitChannel"
local-directory="${localDirectory}"
accept-sub-folders="false"
delete-source-files="true"
archive-bucket="${archiveBucket}"
archive-directory="${archiveDirectory}">
</int-aws:s3-inbound-channel-adapter>
<int-file:splitter input-channel="splitChannel" output-channel="output" markers="false" charset="UTF-8">
<int-file:request-handler-advice-chain>
<bean class="org.springframework.integration.handler.advice.ExpressionEvaluatingRequestHandlerAdvice">
<property name="onSuccessExpression" value="payload.delete()"/>
</bean>
</int-file:request-handler-advice-chain>
</int-file:splitter>
<int:channel-interceptor pattern="*" order="3">
<bean class="org.springframework.integration.channel.interceptor.WireTap">
<constructor-arg ref="loggingChannel" />
</bean>
</int:channel-interceptor>
<int:logging-channel-adapter id="loggingChannel" log-full-message="true" level="INFO"/>
<int:channel id="output"/>
</beans>
Update 2 :
My stream is like below
aws-s3-source|processor|http-client| processor> queue:testQueue
1)Now I split the stream like below:
aws-s3-source> queue:s3Queue
I was able to read all my 1 million messages very fast.
2)Now i added one more stream like below i see issue again was s3 stops pulling file and message are lost everytime
queue:s3Queue>processor|http-client| processor> queue:testQueue
3)Observation is when i add http-client this issue happens again ,i.e. some message from input source is missing.
4)Now I split the file into 125 Mb 5 files instead of 660mb one file .ie 200 k records 5 files.I don't see the issue i get all my messages
I also see lot of messages clogging in queue before http-client .
I am thinking is it something to do with memory or threading inside xd?
Please let me know if expression advice is deleting before processing.
No; the advice is an around advice around the message handler; it can't execute (evaluate the expression), until the splitter has emitted all the lines.
Is it possible the file is pulled from s3 before it's completely written?
To debug this problem, I would suggest changing the advice to send the file to another subflow and do some analysis/logging there before deleting.

duplicate message processed when polling files from s3

I am using s3 module to poll files from s3.It downloads the file to local system and starts processing it.I am running this on 3 node cluster with module count as 1.Now lets assume the file is downloaded to local system from s3 and xd is processing it.If xd node goes down it would have processed half the message.When the server comes up it will start processing file again hence I will get duplicate message.I am trying to change to idempotent pattern with message store to change the module count to 3 but still this duplicate message issues will be there.
<int:poller fixed-delay="${fixedDelay}" default="true">
<int:advice-chain>
<ref bean="pollAdvise"/>
</int:advice-chain>
</int:poller>
<bean id="pollAdvise" class="org.springframework.integration.scheduling.PollSkipAdvice">
<constructor-arg ref="healthCheckStrategy"/>
</bean>
<bean id="healthCheckStrategy" class="ServiceHealthCheckPollSkipStrategy">
<property name="url" value="${url}"/>
<property name="doHealthCheck" value="${doHealthCheck}"/>
</bean>
<bean id="credentials" class="org.springframework.integration.aws.core.BasicAWSCredentials">
<property name="accessKey" value="${accessKey}"/>
<property name="secretKey" value="${secretKey}"/>
</bean>
<bean id="clientConfiguration" class="com.amazonaws.ClientConfiguration">
<property name="proxyHost" value="${proxyHost}"/>
<property name="proxyPort" value="${proxyPort}"/>
<property name="preemptiveBasicProxyAuth" value="false"/>
</bean>
<bean id="s3Operations" class="org.springframework.integration.aws.s3.core.CustomC1AmazonS3Operations">
<constructor-arg index="0" ref="credentials"/>
<constructor-arg index="1" ref="clientConfiguration"/>
<property name="awsEndpoint" value="s3.amazonaws.com"/>
<property name="temporaryDirectory" value="${temporaryDirectory}"/>
<property name="awsSecurityKey" value=""/>
</bean>
<!-- aws-endpoint="https://s3.amazonaws.com" -->
<int-aws:s3-inbound-channel-adapter aws-endpoint="s3.amazonaws.com"
bucket="${bucket}"
s3-operations="s3Operations"
credentials-ref="credentials"
file-name-wildcard="${fileNameWildcard}"
remote-directory="${remoteDirectory}"
channel="splitChannel"
local-directory="${localDirectory}"
accept-sub-folders="false"
delete-source-files="true"
archive-bucket="${archiveBucket}"
archive-directory="${archiveDirectory}">
</int-aws:s3-inbound-channel-adapter>
<int-file:splitter input-channel="splitChannel" output-channel="output" markers="false" charset="UTF-8">
<int-file:request-handler-advice-chain>
<bean class="org.springframework.integration.handler.advice.ExpressionEvaluatingRequestHandlerAdvice">
<property name="onSuccessExpression" value="payload.delete()"/>
</bean>
</int-file:request-handler-advice-chain>
</int-file:splitter>
<int:idempotent-receiver id="expressionInterceptor" endpoint="output"
metadata-store="redisMessageStore"
discard-channel="nullChannel"
throw-exception-on-rejection="false"
key-expression="payload"/>
<bean id="redisMessageStore" class="o.s.i.redis.store.RedisChannelMessageStore">
<constructor-arg ref="redisConnectionFactory"/>
</bean>
<bean id="redisConnectionFactory"
class="o.s.data.redis.connection.jedis.JedisConnectionFactory">
<property name="port" value="7379" />
</bean>
<int:channel id="output"/>
Update 2
This configuration worked for me Thanks for your help.
<int:idempotent-receiver id="s3Interceptor" endpoint="s3splitter"
metadata-store="redisMessageStore"
discard-channel="nullChannel"
throw-exception-on-rejection="false"
key-expression="payload.name"/>
<bean id="redisMessageStore" class="org.springframework.integration.redis.metadata.RedisMetadataStore">
<constructor-arg ref="redisConnectionFactory"/>
</bean>
<bean id="redisConnectionFactory"
class="org.springframework.data.redis.connection.jedis.JedisConnectionFactory">
<property name="port" value="6379" />
</bean>
<int:bridge id="batchBridge" input-channel="bridge" output-channel="output">
</int:bridge>
<int:idempotent-receiver id="splitterInterceptor" endpoint="batchBridge"
metadata-store="redisMessageStore"
discard-channel="nullChannel"
throw-exception-on-rejection="false"
key-expression="payload"/>
<int:channel id="output"/>
I had few doubts wanted to clarify If i am doing right.
1)As you can see I have ExpressionEvaluatingRequestHandlerAdvice to delete the file.Will the file get deleted after i read the file into redis or after last record is read?
2)I explored redis using desktop manager I see this I have a MetaData as man Key
Both (file and payload) metadatastore key and value are going to same table is this fine?or should it be different metadatastore?
Can i use hash of payload instead of payload as key?Is there something like payload.hash!
Looks like it is continuation of the Multiple message processed, but unfortunately we don't see <idempotent-receiver> configuration in your case.
According to your comment there looks like you continue to use SimpleMetadataStore or clean the shared one (Redis/Mongo) very often.
You should share more info where to dig. Some logs and DEBUG investigation would be good, too.
UPDATE
The Idempotent Receiver is exactly for endpoint. In your config it is for the MessageChannel. That's why you don't achieve any proper work, because the MessageChannel is just ignored from the IdempotentReceiverInterceptor.
you should add an id for your <int-file:splitter> and use that id from the endpoint attribute. Not should if that would be good idea to use File as a key for idempotency. The name sounds better.
UPDATE 2
If a node goes down and lets assume a file is dowloaded(file size with million records may be gb) to xd node and I would have processed half the records and node crashes .When server comes up I think we will process same records again?
OK. I got your point finally! you have an issue with splitted lines from the file already.
Also I'd use Idempotent Receiver for the <splitter> as well to avoid duplicate files from S3.
To fix your use-case you should place in between <splitter> and output channel one more endpoint - <bridge> to skip duplicate lines with the Idempotent Receiver.

Multiple message processed

I have a spring xd source module which pulls file from s3 and splits line by line.I have my spring config as below.But I have 3 container and 1 admin server.Now I see duplicate message being processed by each container as each of them is downloading there own copy.
I can solve with making source s3 module deployment count as 1 but my processing of message is getting slow.?Any inputs to solve this?
<int:poller fixed-delay="${fixedDelay}" default="true">
<int:advice-chain>
<ref bean="pollAdvise"/>
</int:advice-chain>
</int:poller>
<bean id="pollAdvise"
</bean>
<bean id="credentials" class="org.springframework.integration.aws.core.BasicAWSCredentials">
<property name="accessKey" value="#{encryptedDatum.decryptBase64Encoded('${accessKey}')}"/>
<property name="secretKey" value="${secretKey}"/>
</bean>
<bean id="clientConfiguration" class="com.amazonaws.ClientConfiguration">
<property name="proxyHost" value="${proxyHost}"/>
<property name="proxyPort" value="${proxyPort}"/>
<property name="preemptiveBasicProxyAuth" value="false"/>
</bean>
<bean id="s3Operations" class="org.springframework.integration.aws.s3.core.CustomC1AmazonS3Operations">
<constructor-arg index="0" ref="credentials"/>
<constructor-arg index="1" ref="clientConfiguration"/>
<property name="awsEndpoint" value="s3.amazonaws.com"/>
<property name="temporaryDirectory" value="${temporaryDirectory}"/>
<property name="awsSecurityKey" value="${awsSecurityKey}"/>
</bean>
<bean id="encryptedDatum" class="abc"/>
<!-- aws-endpoint="https://s3.amazonaws.com" -->
<int-aws:s3-inbound-channel-adapter aws-endpoint="s3.amazonaws.com"
bucket="${bucket}"
s3-operations="s3Operations"
credentials-ref="credentials"
file-name-wildcard="${fileNameWildcard}"
remote-directory="${remoteDirectory}"
channel="splitChannel"
local-directory="${localDirectory}"
accept-sub-folders="false"
delete-source-files="true"
archive-bucket="${archiveBucket}"
archive-directory="${archiveDirectory}">
</int-aws:s3-inbound-channel-adapter>
<int-file:splitter input-channel="splitChannel" output-channel="output" markers="false" charset="UTF-8">
<int-file:request-handler-advice-chain>
<bean class="org.springframework.integration.handler.advice.ExpressionEvaluatingRequestHandlerAdvice">
<property name="onSuccessExpression" value="payload.delete()"/>
</bean>
</int-file:request-handler-advice-chain>
</int-file:splitter>
<int:channel id="output"/>
[Updated]
I added the idempotency as suggested by you with a metadata store.But since my xd is running in 3 container cluster with rabbit will simple metadatastore work?I think I should use reds/mongo metadata source.If I use mongo/redis metadatastore howcan i evict/remove the messages because messages will pile up over time?
<int:idempotent-receiver id="expressionInterceptor" endpoint="output"
metadata-store="store"
discard-channel="nullChannel"
throw-exception-on-rejection="false"
key-expression="payload"/>
<bean id="store" class="org.springframework.integration.metadata.SimpleMetadataStore"/>
I can suggest you to take a look to the Idempotent Receiver.
With that you can use shared MetadataStore and don't accept duplicate files.
The <idempotent-receiver> should be configured for that your <int-file:splitter>. And yes: with the discard logic to avoid duplicate message.
UPDATE
.But since my xd is running in 3 container cluster with rabbit will simple metadatastore work?
That doesn't matter because you start the stream from the S3 MessageSource, so you should filter files already there. Therefore you need external shared MetadataStore.
.If I use mongo/redis metadatastore howcan i evict/remove the messages because messages will pile up over time?
That's correct. It is a side affect of the Idempotent Receiver logic. Not sure how it is a problem for you if you use a DataBase...
You can clean the collection/keys by some periodic task. Maybe once a week...

spring integration aws s3 delete local file

I am using spring integration to read file from s3 however this works but my local directory is getting full I want to delete files from local directory after files are processed from s3?
<bean id="credentials" class="org.springframework.integration.aws.core.BasicAWSCredentials">
<property name="accessKey" value="${accessKey}"/>
<property name="secretKey" value="${secretKey}"/>
</bean>
<bean id="clientConfiguration" class="com.amazonaws.ClientConfiguration">
<property name="proxyHost" value="${proxyHost}"/>
<property name="proxyPort" value="${proxyPort}"/>
<property name="preemptiveBasicProxyAuth" value="false"/>
</bean>
<bean id="s3Operations" class="org.springframework.integration.aws.s3.core.CustomC1AmazonS3Operations">
<constructor-arg index="0" ref="credentials"/>
<constructor-arg index="1" ref="clientConfiguration"/>
<property name="awsEndpoint" value="s3.amazonaws.com"/>
<property name="temporaryDirectory" value="${temporaryDirectory}"/>
<property name="awsSecurityKey" value="${awsSecurityKey}"/>
</bean>
<!-- aws-endpoint="https://s3.amazonaws.com" -->
<int-aws:s3-inbound-channel-adapter aws-endpoint="s3.amazonaws.com"
bucket="${bucket}"
s3-operations="s3Operations"
credentials-ref="credentials"
file-name-wildcard="${fileNameWildcard}"
remote-directory="${remoteDirectory}"
channel="splitChannel"
local-directory="${localDirectory}"
accept-sub-folders="false"
delete-source-files="true"
archive-bucket="${archiveBucket}"
archive-directory="${archiveDirectory}">
</int-aws:s3-inbound-channel-adapter>
Looks like you are looking for ExpressionEvaluatingRequestHandlerAdvice.
Please, find Retry and More. There is something like <property name="onSuccessExpression" value="payload.delete()" /> in the expression-advice-context.xml config to take care about the local file after the proper finish of the process.

Dispatcher has no subscribers error for ftp

I have a file dropped at the ftp location which should be picked up by ftp-inbound-adapter. This file is saved to a local-directory. This local-directory is in turn polled by spring file-inbound-adapter. The filenamegenerator bean is used in the file-inbound-adapter and decides the destination dynamically. I have also posted another question about the file in the local-directory not being deleted. This is the problem I am facing.
This is a my entire configuration
<util:properties id="someid" location="classpath:config/config.properties"/>
<mvc:annotation-driven />
<context:component-scan base-package="com.dms" />
<bean
class="org.springframework.web.servlet.view.InternalResourceViewResolver">
<property name="prefix">
<value>/WEB-INF/</value>
</property>
<property name="suffix">
<value>.jsp</value>
</property>
</bean>
<context:property-placeholder location="classpath:config/jdbc.properties,classpath:config/config.properties,classpath:config/ftp.properties"/>
<bean id="dataSource"
class="org.springframework.jdbc.datasource.DriverManagerDataSource"
>
<property name="driverClassName" value="${jdbc.driverClassName}"/>
<property name="url" value="${jdbc.url}"/>
<property name="username" value="${jdbc.username}"/>
<property name="password" value="${jdbc.password}"/>
</bean>
<bean id="sessionFactory"
class="org.springframework.orm.hibernate4.LocalSessionFactoryBean">
<property name="dataSource">
<ref bean="dataSource" />
</property>
<property name="hibernateProperties">
<props>
<prop key="hibernate.dialect">${hibernate.dialect}</prop>
<prop key="hibernate.show_sql">${hibernate.show_sql}</prop>
<prop key="hibernate.format_sql">${hibernate.format_sql}</prop>
<prop key="hibernate.generate_statistics">${hibernate.generate_statistics}</prop>
</props>
</property>
<property name="packagesToScan">
<list>
<value>com.dms.entity</value>
</list>
</property>
</bean>
<tx:annotation-driven />
<bean id="transactionManager"
class="org.springframework.orm.hibernate4.HibernateTransactionManager">
<property name="sessionFactory" ref="sessionFactory" />
</bean>
<bean id="multipartResolver"
class="org.springframework.web.multipart.commons.CommonsMultipartResolver">
<!-- setting maximum upload size -->
<property name="maxUploadSize" value="10485760" />
</bean>
<!-- scheduler to pickup temp folder files to permanent location -->
<bean class="org.springframework.scheduling.quartz.SchedulerFactoryBean">
<property name="triggers">
<list>
<ref bean="simpleTrigger" />
</list>
</property>
</bean>
<bean id="dmsFilesDetectionJob" class="com.dms.scheduler.job.DMSFilesDetectionJob">
</bean>
<bean id="dmsFilesDetectionJobDetail" class="org.springframework.scheduling.quartz.MethodInvokingJobDetailFactoryBean">
<property name="targetObject" ref="dmsFilesDetectionJob" />
<property name="targetMethod" value="pollTempFolder" />
<property name="concurrent" value="false" />
</bean>
<bean id="simpleTrigger" class="org.springframework.scheduling.quartz.CronTriggerBean">
<property name="jobDetail" ref="dmsFilesDetectionJobDetail" />
<!-- <property name="cronExpression" value="1 * * * * ?" /> -->
<property name="cronExpression" value="0 0/1 * * * ?" />
</bean>
<bean id="fileNameGenerator" class="com.dms.util.FileNameGenerator"/>
<int-file:inbound-channel-adapter id="filesIn" directory="file:${paths.root}" channel="abc" filter="compositeFilter" >
<int:poller id="poller" fixed-delay="5000" />
</int-file:inbound-channel-adapter>
<int:channel id="abc"/>
<bean id="compositeFilter" class="org.springframework.integration.file.filters.CompositeFileListFilter">
<constructor-arg>
<list>
<!-- Ensures that the file is whole before processing it -->
<bean class="com.dms.util.CustomFileFilter"/>
<!-- Ensures files are picked up only once from the directory -->
<bean class="org.springframework.integration.file.filters.AcceptOnceFileListFilter" />
</list>
</constructor-arg>
</bean>
<int-file:outbound-channel-adapter channel="abc" id="filesOut"
directory-expression="#outPathBean.getPath()"
delete-source-files="true" filename-generator="fileNameGenerator" />
<bean class="org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter">
<property name="messageConverters">
<list>
<ref bean="jsonMessageConverter"/>
</list>
</property>
</bean>
<bean id="jsonMessageConverter" class="org.springframework.http.converter.json.MappingJackson2HttpMessageConverter">
<!-- <property name="prefixJson" value="false"/> -->
<!-- <property name="objectMapper">
<bean class="com.dms.util.HibernateAwareObjectMapper" />
</property> -->
<property name="supportedMediaTypes" value="application/json"/>
</bean>
<bean id="ftpClientFactory"
class="org.springframework.integration.ftp.session.DefaultFtpSessionFactory">
<property name="host" value="${ftp.ip}"/>
<property name="port" value="${ftp.port}"/>
<property name="username" value="${ftp.username}"/>
<property name="password" value="${ftp.password}"/>
<property name="clientMode" value="0"/>
<property name="fileType" value="2"/>
<property name="bufferSize" value="100000"/>
</bean>
<int-ftp:outbound-channel-adapter id="ftpOutbound"
channel="ftpChannel"
session-factory="ftpClientFactory"
charset="UTF-8"
remote-file-separator="/"
auto-create-directory="true"
remote-directory="."
use-temporary-file-name="true"
auto-startup="true"
/>
<int-ftp:inbound-channel-adapter id="ftpInbound"
channel="ftpChannel"
session-factory="ftpClientFactory"
charset="UTF-8"
local-directory="file:${paths.root}"
delete-remote-files="true"
temporary-file-suffix=".writing"
remote-directory="."
filename-pattern="${file.char}*${file.char}"
preserve-timestamp="true"
auto-startup="true">
<int:poller fixed-rate="1000"/>
</int-ftp:inbound-channel-adapter>
<int:channel id="ftpChannel" />
This is the error I am getting
18:02:34.655 E|LoggingHandler |org.springframework.messaging.MessageDeliveryException: Dispatcher has no subscribers for channel 'org.springframework.web.context.WebApplicationContext:/DMS/DMS-dispatcher.ftpChannel'.
This exception does not appear everytime.
As you can see I have added auto-startup="true". Have used unique id's for both the channels as well as adapters. Please let me know what is wrong here!
Thanks
I just had to deal with this but with a file inbound-channel-adapter. The issue is intermittent, and only at startup. I think adapters with pollers can start pulling in messages before Spring Integration has fully initialized.
My fix is to disable the adapter at startup. The details of the adapter are not so important, other than it has an id and is set to not autostart:
<!--
Read files from an "inbox" directory, placing them on an "inbox" channel...
-->
<int-file:inbound-channel-adapter id="inboxScanner"
directory="$import{inbox}"
auto-create-directory="true"
channel="fileInbox"
prevent-duplicates="false"
auto-startup="false">
<int:poller fixed-rate="$import{inbox.scan.rate.seconds}"
time-unit="SECONDS"
max-messages-per-poll="$import{inbox.max.imports.per.scan}"/>
</int-file:inbound-channel-adapter>
I then tap into Spring's application lifecycle events, and once the application context is finished being created (or refreshed), I tell the adapter to start:
<!--
Only start the scanner after the application has finished initializing...
-->
<int-event:inbound-channel-adapter event-types="org.springframework.context.event.ContextRefreshedEvent"
channel="contextRefreshEvents"/>
<int:publish-subscribe-channel id="contextRefreshEvents"/>
<int:outbound-channel-adapter channel="contextRefreshEvents"
expression="#inboxScanner.start()" />
The "event" components are from spring-integration-event.
First of all your integration flow ins't clear. But I see that your issue is for this
<int-ftp:inbound-channel-adapter id="ftpInbound" channel="ftpChannel"
Where there is really no one who is subscribed to that ftpChannel SubscribableChannel.
So, that adapter starts his work and sends message to that channel, but ... Dispatcher has no subscribers.
Try to fix that issue and figure out how to go ahead.
EDIT
Not sure what you have found in my answer so bad that it forced you to downvote it. Anyway.
There was some phase problem before, but as you see it has been fixed in the version 4.1.
So, to reach the immediate fix right now you should do this:
phase="0x7fffffff" // Integer.MAX_VALUE
by default phase is 0, so the inbound channel adapter may start before outbound channel adapter.
Or just upgrade to the latest Spring Integration!

Resources