Spring Integration - Not all files are getting uploaded to destination - clearThreadKey issue - spring-integration

setThreadKey is getting invoked for every file, but clearThreadKey is getting invoked for alternative files. When ever a flow invokes clearThreadKey the file is not getting uploaded to SFTP destination path. Out of 10 files, only 5 files are getting uploaded. I have customized the DelegatingSessionFactory class to decide the threadKey. I guess i am doing something wrong. I think, clearThreadKey is getting invoked even before uploading the file to destination, due to which the flow is not able to upload the file to destination. But it is supposed to invoke only after uploading the file. At the same time, strangely clearThreadKey is not invoked for all the files.
public Message<?> setThreadKey(Message<?> message, Object key) {
String keyStr = String.valueOf(key).split("-")[0];
this.threadKey.set(keyStr);
return message;
}
public Message<?> clearThreadKey(Message<?> message, Object key) {
this.threadKey.remove();
return message;
}
integration.xml
<int-file:inbound-channel-adapter directory="myowndirectorypath" id="fileInbound" channel="sftpChannel">
<int:poller fixed-rate="1000" max-messages-per-poll="100"/>
</int-file:inbound-channel-adapter>
<int:channel id="sftpChannel"/>
<int:service-activator input-channel="sftpChannel" output-channel="outsftpChannel"
expression="#dsf.setThreadKey(#root, headers['file_name'])"/>
<int:channel id="outsftpChannel"/>
<int-sftp:outbound-channel-adapter id="sftpOutboundAdapter" session-factory="dsf"
channel="outsftpChannel" charset="UTF-8"
remote-directory-expression="#sftpConfig.determineRemoteDirectory(headers['file_name'])"/>
<int:service-activator input-channel="outsftpChannel" output-channel="nullChannel"
expression="#dsf.clearThreadKey(#root, headers['file_name'])" requires-reply="true"/>

You have two subscribers on outsftpChannel; they will receive alternate messages in round-robin fashion because it is a DirectChannel.
You need a publish-subscribe-channel so that both recipients will receive the message.

Related

Issues in polling a file using Spring Integration

My req. is to poll a directory for a specified time interval say 10 mins. If a file of a particular extension say *.xml is found in the directory then the it just consumes (i.e. picks and deletes) the file and prints the name else after the specified time (say 10 mins.) interval it sends out a mail that the file has not been picked (i.e. consumed) or the file has not come.
There are 2 options either I do it through Spring integration OR WatchService of Core Java. Following is the code in Spring Integration which I have written till now:
<int:channel id="fileChannel" />
<int:channel id="processedFileChannel" />
<context:property-placeholder location="localProps.properties" />
<int:poller default="true" fixed-rate="10000" id="poller"></int:poller>
<int-file:inbound-channel-adapter
directory="file:${inbound.folder}" channel="fileChannel"
filename-pattern="*.xml" />
<int:service-activator input-channel="fileChannel"
ref="fileHandlerService" method="processFile" output-channel="processedFileChannel"/>
<bean id="fileHandlerService" class="com.practice.cmrs.springintegration.Poll" />
The above code is successfully polling the folder for a particular file pattern. Now I have 2 things to do:
1) Stop polling after a particular time interval (configurable) say 10 mins.
2) Check whether a file with a particular extension is there in the folder ... if the file is there (it consumes and then deletes) else it sends an email to a group of people (email part is done.)
Please help me in the above 2 points.
You can use a Smart Poller to do things like that.
You can adjust the poller and/or take different actions if/when the poll results in a message.
Version 4.2 introduced the AbstractMessageSourceAdvice. Any Advice objects in the advice-chain that subclass this class, are applied to just the receive operation. Such classes implement the following methods:
beforeReceive(MessageSource<?> source)
This method is called before the MessageSource.receive() method. It enables you to examine and or reconfigure the source at this time. Returning false cancels this poll (similar to the PollSkipAdvice mentioned above).
Message<?> afterReceive(Message<?> result, MessageSource<?> source)
This method is called after the receive() method; again, you can reconfigure the source, or take any action perhaps depending on the result (which can be null if there was no message created by the source). You can even return a different message!

How do I use the sqs-message-driven-channel-adapter in spring-integration-aws

EDIT: Here is a gist showing my log. It appears that there is ReceiveMessage and then a preSend on inputChannel:
https://gist.github.com/louisalexander/04e7d95835521efdd15455c98075e2ea
Apologies for being so dense, but I can't seem to figure out how to properly make use of the sqs-message-driven-channel-adapter
In my context file, I am configuring it as such:
<int-aws:sqs-message-driven-channel-adapter
id="my-message-driven-adapter" sqs="sqs" queues="some-queue-of-mine"
max-number-of-messages="5" visibility-timeout="200" wait-time-out="10"
send-timeout="2000" channel="inputChannel" />
I observe that messages are properly making it into some-queue-of-mine (by removing the above bit of code and sending messages to the queue). I then restart my server, enabling the message driven adapter and I observe that all the messages are consumed from the queue, but where did they go? :-/
My expectation was that the messages would be funneled into a DirectChannel named inputChannel:
<int:channel id="inputChannel"/>
That I have a service-activator consuming from as follows:
<int:service-activator ref="myConsumer"
method='execute' input-channel="inputChannel" output-channel="outputChannel">
<int:request-handler-advice-chain>
...
</int:request-handler-advice-chain>
</int:service-activator>
But of course, I am never seeing myConsumer get invoked. I imagine my understanding of how the MessageProducer mechanism works is inadequate. Can someone please correct my thinking by providing a trivial example of XML wiring?
According to the Logs the message is consumed by the handler.AbstractMessageHandler (AbstractMessageHandler.java:115) - ServiceActivator for [org.springframework.integration.handler.MethodInvokingMessageProcessor#493f49cd]. Although that might be a fully different story.
SqsMessageDrivenChannelAdapter can be supplied with the errorChannel to handle downstream exceptions. By default it is only logged.
The message sent from that adapter is like:
return new GenericMessage<>(message.getBody(), new SqsMessageHeaders(messageHeaders));
Where that message.getBody() is String. See QueueMessageUtils.createMessage().
So, be sure that your service-activator accepts that String as a paylaod and not any other type.

Error channel for executor channel

I am using an executor channel since I want to switch threads to finish the transaction at this point. The only other way to do this is a poller and I think the executor channel is a much nicer solution. The only problem is that I cannot find a way to define an error channel for this dispatcher. Errors are always published on the global errorChannel.
Here is my config:
<task:executor id="routingExec" pool-size="10"/>
<int:channel id="baseFlow.route">
<int:dispatcher failover="false" task-executor="routingExec"/>
</int:channel>
And I want to have something like this (like in the poller):
<task:executor id="routingExec" pool-size="10"/>
<int:channel id="baseFlow.route">
<int:dispatcher error-channel="myErrorChannel" failover="false" task-executor="routingExec"/>
</int:channel>
Errors are handled in the ExecutorChannel via ErrorHandlingTaskExecutor:
if (!(this.executor instanceof ErrorHandlingTaskExecutor)) {
ErrorHandler errorHandler = new MessagePublishingErrorHandler(
new BeanFactoryChannelResolver(this.getBeanFactory()));
this.executor = new ErrorHandlingTaskExecutor(this.executor, errorHandler);
}
Where defaultErrorChannel is really like IntegrationContextUtils.ERROR_CHANNEL_BEAN_NAME.
So, to use your own channel for error from that ExecutorChannel, you should use errorChannel header for each message to send, or just inject your own ErrorHandlingTaskExecutor with MessagePublishingErrorHandler and defaultErrorChannel configured to your expectations.

Download multiple files using SourcePollingChannelAdapter- start/stop issue?

My requirement is to donwload multiple files from a remote directory in a file server in my spring-batch application. One of the example here suggests the usage of SourcePollingChannelAdapter. In this example the adapter is started adapter.start() but its not stopped. So how does the lifecycle of this work ? My requirement is to first download & Only when all files are downloaded, then proceed to read the files. But this seems this is kind of async process to download the files. So how would I get notified when all the files are downloaded & ready to proceed further ? I dont see any other methods to inject any handlers to the adapter/pollablechannel.
Sample config used :
<int-sftp:inbound-channel-adapter id="sftpInbondAdapterBean"
auto-startup="false" channel="receiveChannelBean" session-factory="sftpSessionFactory"
local-directory="${sftp.download.localDirResource}"
remote-directory="${sftp.download.remoteFilePath}"
auto-create-local-directory="true" delete-remote-files="false"
filename-regex=".*\.csv$">
<int:poller fixed-rate="5000" max-messages-per-poll="3" />
</int-sftp:inbound-channel-adapter>
<int:channel id="receiveChannelBean">
<int:queue/>
</int:channel>
It throws InterruptedException if adapter.stop() is called explicitly. If stop is not called , the files are downloaded in async way without blocking & hence the next step doesn't know if the download is completed or is in-progress.
EDIT : Code Snippet of DownloadingTasklet
#Override
public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext) throws Exception {
try {
sftpInbondAdapter.start();
Thread.sleep(15000); // ---Is this mandatory ?
} catch (Exception e) {
throw new BatchClientException("Batch File Download was un-successful !", e);
} finally {
// sftpInbondAdapter.stop();
Logger.logDebug("BATCH_INPUT_FILE_DOWNLOAD", "COMPLETED");
}
return RepeatStatus.FINISHED;
}
The SftpInboundFileSynchronizingMessageSource works in two phases:
First of all it does synchronization from the remote directory to the local one:
Message<File> message = this.fileSource.receive();
if (message == null) {
this.synchronizer.synchronizeToLocalDirectory(this.localDirectory);
message = this.fileSource.receive();
}
And as you see only after that it starts to emit files as a messages. But already only local files. So, download from the FTP has been done already.
The next download attempt happens only when all local files are cleared during sending process.
So, what you want is a built-in functionality.
EDIT
Some DEBUG logs from our tests:
19:29:32,005 DEBUG task-scheduler-1 util.SimplePool:190 - Obtained new org.springframework.integration.sftp.session.SftpSession#26f16625.
19:29:32,036 DEBUG task-scheduler-1 inbound.SftpInboundFileSynchronizer:287 - cannot copy, not a file: /sftpSource/subSftpSource
19:29:32,036 DEBUG task-scheduler-1 session.CachingSessionFactory:187 - Releasing Session org.springframework.integration.sftp.session.SftpSession#26f16625 back to the pool.
19:29:32,036 DEBUG task-scheduler-1 util.SimplePool:221 - Releasing org.springframework.integration.sftp.session.SftpSession#26f16625 back to the pool
19:29:32,037 DEBUG task-scheduler-1 inbound.SftpInboundFileSynchronizer:270 - 3 files transferred
19:29:32,037 DEBUG task-scheduler-1 file.FileReadingMessageSource:380 - Added to queue: [local-test-dir\rollback\ sftpSource1.txt, local-test-dir\rollback\sftpSource2.txt]
19:29:32,043 INFO task-scheduler-1 file.FileReadingMessageSource:368 - Created message: [GenericMessage [payload=local-test-dir\rollback\ sftpSource1.txt, headers={id=40f75bd1-2150-72a4-76f0-f5c619a246da, timestamp=1465514972043}]]
It is with config like:
<int-sftp:inbound-channel-adapter
session-factory="sftpSessionFactory"
channel="requestChannel"
remote-directory="/sftpSource"
local-directory="local-test-dir/rollback"
auto-create-local-directory="true"
local-filter="acceptOnceFilter">
<int:poller fixed-delay="1" max-messages-per-poll="0"/>
</int-sftp:inbound-channel-adapter>
Pay attention how it shows 3 files transferred, where one of them is directory, so skipped.
How it says Added to queue:
And only after that it starts to emit them as messages.
And everything is just because I have fixed-delay="1". Even if it is only 1 millisecond.
The max-messages-per-poll="0" means processes everything with one polling task.

Poller stops working after 1 cycle

At the beginning of my flow I have a file inbound adapter which reads a directory periodically:
<int-file:inbound-channel-adapter id="filteredFiles"
directory="${controller.cycle.lists.input.dir}"
channel="semaphoreChannel" filename-pattern="*.xml">
<int:poller fixed-delay="3000"/>**
</int-file:inbound-channel-adapter>
When the SI workflow ends it never happens again. It seems the poller is dead and stops working.
There aren't any error messages in the log nor any warnings.
Channel configuration:
<int:channel id="semaphoreChannel" datatype="java.io.File"/>
Second configuration:
<int-file:inbound-channel-adapter id="filteredFiles"
directory="${controller.cycle.lists.input.dir}"
channel="semaphoreChannel" filename-pattern="*.xml">
<int:poller cron="0 * * * * *" />
</int-file:inbound-channel-adapter>
It does not make sense.
Since you use default settings for other <poller> options, you end up with:
public static final int MAX_MESSAGES_UNBOUNDED = Integer.MIN_VALUE;
private volatile long maxMessagesPerPoll = MAX_MESSAGES_UNBOUNDED;
That means the FileReadingMessageSource reads all files by provided pattern during the single poller cycle.
The poller doesn't stop to work, but there is nothing more in the directory to read.
Change to this max-messages-per-poll="1" and let us know how it is.
From other side you can switch on DEBUG logging level for the org.springframework.integration.endpoint.SourcePollingChannelAdapter and there will be a message in logs:
Received no Message during the poll, returning 'false'

Resources