Poller stops working after 1 cycle - spring-integration

At the beginning of my flow I have a file inbound adapter which reads a directory periodically:
<int-file:inbound-channel-adapter id="filteredFiles"
directory="${controller.cycle.lists.input.dir}"
channel="semaphoreChannel" filename-pattern="*.xml">
<int:poller fixed-delay="3000"/>**
</int-file:inbound-channel-adapter>
When the SI workflow ends it never happens again. It seems the poller is dead and stops working.
There aren't any error messages in the log nor any warnings.
Channel configuration:
<int:channel id="semaphoreChannel" datatype="java.io.File"/>
Second configuration:
<int-file:inbound-channel-adapter id="filteredFiles"
directory="${controller.cycle.lists.input.dir}"
channel="semaphoreChannel" filename-pattern="*.xml">
<int:poller cron="0 * * * * *" />
</int-file:inbound-channel-adapter>

It does not make sense.
Since you use default settings for other <poller> options, you end up with:
public static final int MAX_MESSAGES_UNBOUNDED = Integer.MIN_VALUE;
private volatile long maxMessagesPerPoll = MAX_MESSAGES_UNBOUNDED;
That means the FileReadingMessageSource reads all files by provided pattern during the single poller cycle.
The poller doesn't stop to work, but there is nothing more in the directory to read.
Change to this max-messages-per-poll="1" and let us know how it is.
From other side you can switch on DEBUG logging level for the org.springframework.integration.endpoint.SourcePollingChannelAdapter and there will be a message in logs:
Received no Message during the poll, returning 'false'

Related

Issues in polling a file using Spring Integration

My req. is to poll a directory for a specified time interval say 10 mins. If a file of a particular extension say *.xml is found in the directory then the it just consumes (i.e. picks and deletes) the file and prints the name else after the specified time (say 10 mins.) interval it sends out a mail that the file has not been picked (i.e. consumed) or the file has not come.
There are 2 options either I do it through Spring integration OR WatchService of Core Java. Following is the code in Spring Integration which I have written till now:
<int:channel id="fileChannel" />
<int:channel id="processedFileChannel" />
<context:property-placeholder location="localProps.properties" />
<int:poller default="true" fixed-rate="10000" id="poller"></int:poller>
<int-file:inbound-channel-adapter
directory="file:${inbound.folder}" channel="fileChannel"
filename-pattern="*.xml" />
<int:service-activator input-channel="fileChannel"
ref="fileHandlerService" method="processFile" output-channel="processedFileChannel"/>
<bean id="fileHandlerService" class="com.practice.cmrs.springintegration.Poll" />
The above code is successfully polling the folder for a particular file pattern. Now I have 2 things to do:
1) Stop polling after a particular time interval (configurable) say 10 mins.
2) Check whether a file with a particular extension is there in the folder ... if the file is there (it consumes and then deletes) else it sends an email to a group of people (email part is done.)
Please help me in the above 2 points.
You can use a Smart Poller to do things like that.
You can adjust the poller and/or take different actions if/when the poll results in a message.
Version 4.2 introduced the AbstractMessageSourceAdvice. Any Advice objects in the advice-chain that subclass this class, are applied to just the receive operation. Such classes implement the following methods:
beforeReceive(MessageSource<?> source)
This method is called before the MessageSource.receive() method. It enables you to examine and or reconfigure the source at this time. Returning false cancels this poll (similar to the PollSkipAdvice mentioned above).
Message<?> afterReceive(Message<?> result, MessageSource<?> source)
This method is called after the receive() method; again, you can reconfigure the source, or take any action perhaps depending on the result (which can be null if there was no message created by the source). You can even return a different message!

How do I use the sqs-message-driven-channel-adapter in spring-integration-aws

EDIT: Here is a gist showing my log. It appears that there is ReceiveMessage and then a preSend on inputChannel:
https://gist.github.com/louisalexander/04e7d95835521efdd15455c98075e2ea
Apologies for being so dense, but I can't seem to figure out how to properly make use of the sqs-message-driven-channel-adapter
In my context file, I am configuring it as such:
<int-aws:sqs-message-driven-channel-adapter
id="my-message-driven-adapter" sqs="sqs" queues="some-queue-of-mine"
max-number-of-messages="5" visibility-timeout="200" wait-time-out="10"
send-timeout="2000" channel="inputChannel" />
I observe that messages are properly making it into some-queue-of-mine (by removing the above bit of code and sending messages to the queue). I then restart my server, enabling the message driven adapter and I observe that all the messages are consumed from the queue, but where did they go? :-/
My expectation was that the messages would be funneled into a DirectChannel named inputChannel:
<int:channel id="inputChannel"/>
That I have a service-activator consuming from as follows:
<int:service-activator ref="myConsumer"
method='execute' input-channel="inputChannel" output-channel="outputChannel">
<int:request-handler-advice-chain>
...
</int:request-handler-advice-chain>
</int:service-activator>
But of course, I am never seeing myConsumer get invoked. I imagine my understanding of how the MessageProducer mechanism works is inadequate. Can someone please correct my thinking by providing a trivial example of XML wiring?
According to the Logs the message is consumed by the handler.AbstractMessageHandler (AbstractMessageHandler.java:115) - ServiceActivator for [org.springframework.integration.handler.MethodInvokingMessageProcessor#493f49cd]. Although that might be a fully different story.
SqsMessageDrivenChannelAdapter can be supplied with the errorChannel to handle downstream exceptions. By default it is only logged.
The message sent from that adapter is like:
return new GenericMessage<>(message.getBody(), new SqsMessageHeaders(messageHeaders));
Where that message.getBody() is String. See QueueMessageUtils.createMessage().
So, be sure that your service-activator accepts that String as a paylaod and not any other type.

Flow not exiting like expected

new to SI. I have this SI Flow below. I would expect the Java program to exit when done, after the outbound channel, but it continues waiting, and I'm confused as to why. Thanks in advance!
<int-file:inbound-channel-adapter id="filesIn"
directory="file:${com.cld.creditexpenses.inputdirectory}"
filename-pattern="*.313" auto-startup="true">
<int:poller id="poller" fixed-delay="10000"/>
</int-file:inbound-channel-adapter>
<int:service-activator input-channel="filesIn" ref="delimitedFileProcessor"
method="processFile" output-channel="outChannel" />
<int:outbound-channel-adapter id="outChannel" ref="fileArchiver" method="archiveFile" >
</int:outbound-channel-adapter>
You have to close the application context when processing is complete, or replace the default taskScheduler with one that uses daemon threads - see the documentation.
But doing that means the program may exit before processing is complete - so you need to keep it running some other way.

Spring Integration File Polling. If moving the file does a AcceptOnceFileListFilter need to be used?

I'm writing a file polling implementation and am trying to determine if I need to use a AcceptOnceFileListFilter.
The first step the FileProcessor will perform is to move the file to another directory.
Does the poller "batchFilePoller" use multiple threads when polling? Can a race condition occur where a file will be read by multiple threads? In this case I assume I need to use the AcceptOnceFileListFilter.
However if the poller is only using one thread from the pool.
Then if the file is moved before the next poll time and it succeeds I assume there is no posability of the file been processed twice?
<int-file:inbound-channel-adapter id="batchFileInAdapter" directory="/somefolder" auto-create-directory="true" auto-startup="false" channel="batchFileInChannel" >
<int:poller id="batchFilePoller" fixed-rate="6000" task-executor="batchTaskExecutor" max-messages-per-poll="1" error-channel="batchPollingErrorChannel" />
</int-file:inbound-channel-adapter>
<int:channel id="batchFileInChannel"/>
<int:service-activator input-channel="batchFileInChannel" >
<bean class="com.foo.FileProcessor" />
</int:service-activator>
<task:executor id="batchTaskExecutor" pool-size="5" queue-capacity="20"/>
The <int-file:inbound-channel-adapter> has prevent-duplicates option which is true by default and it is your case since you don't provide any other options which prevent prevent-duplicates to be true.
And yes: any polling adapter is multi-threaded, if you use fixed-rate. In this case the new polling task can be run before a finish of previous one.
Even if it will be a single-threaded (using fixed-delay), the AcceptOnceFileListFilter must be there, because a new polling task doesn't know if file has been processed or not. And it reads the same file again.
AcceptOnceFileListFilter is exactly for those cases when you don't like to read the same file one more time. You can overcome that with <int:transactional synchronization-factory=""/> for the <poller> of the <int-file:inbound-channel-adapter>:
<int:transaction-synchronization-factory id="txSyncFactory">
<int:after-commit expression="payload.delete()"/>
</int:transaction-synchronization-factory>
and PseudoTransactionManager.
More info you can find in the Spring Integration Reference Manual.

How to specify output-channel or order of execution when we use inbound-channel-adaptor

I am using spring integration to download files and to process them.
<int-sftp:inbound-channel-adapter channel="FileDownloadChannel"
session-factory="SftpSessionFactory"
remote-directory="/home/sshaji/from_disney/files"
filter = "modifiedFileListFilter"
local-directory="/home/sshaji/to_disney/downloads"
auto-create-local-directory="true" >
<integration:poller cron="*/10 * * * * *" default="true"/>
</int-sftp:inbound-channel-adapter>
<integration:transformer input-channel="FileDownloadChannel"
ref="ErrorTransformer"
output-channel="EndChannel"/>
The execution is started by the poller.
It calls the "FileDownloadChannel" and then tries to download files from the sftp server.
I want to specify an output channel for this inbound-channel-adaptor but it doesnot have any output-channel attribute.
So i named the transformer with the same name as that of inbound-channel-adaptor so that it will also be called once poller starts.
My issue is that the transformer gets called before the download happens and hence transformer wont get any inputs to process and causes error.
Is there any way we can specify "order" attribute for this two tasks. or is there any workaround for the output-channel for the inbound-channel adaptor?.
I would really appreciate any help on this.
You need to read the Spring Integration Reference Manual and work through some sample applications.
Channel adapters don't have input and output channels, they have channels. Channel adapters either produce or consume a message (inbound Vs outbound) on their channel. Elements such as transformers, service activators etc, that receive a message and produce a reply, have input and output channels.
"My issue is that the transformer gets called before the download happens and hence transformer wont get any inputs to process and causes error."
This statement makes no sense to me; if there's no file yet, there's nothing to "call" the transformer with.
"attribute for this two tasks."
There are not two "tasks".
The poller thread invokes the inbound adapter; then, when a file arrives, it is sent as a message to the configured channel which, with your configuration, means the poller thread invokes the transformer with the message.
A Channel Adapter is a Message Endpoint that enables connecting a single sender or receiver to a Message Channel.
In your case, output channel is 'FileDownloadChannel'
<integration:channel id="FileDownloadChannel"/>
<int-sftp:inbound-channel-adapter channel="FileDownloadChannel" ...>
<integration:poller fixed-rate="10000"/>
</int-sftp:inbound-channel-adapter>
In order to execute your tasks in order, you can use the Message Handler chain as follows:
<integration:channel id="outputChannel"/>
<chain input-channel="FileDownloadChannel" output-channel="outputChannel">
<filter ref="someSelector" throw-exception-on-rejection="true"/>
<transformer ref="ErrorTransformer"/>
<service-activator ref="someService" method="someMethod"/>
</chain>

Resources