I have have the next error:
ERROR com...ErrorHandler - receiving error response with null parametter org.springframework.integration.MessagingException: Problem occurred while synchronizing remote to local directory
with spring integration 2.2.6, I am using an inbound-channel-adapter with the next configuration:
<int-sftp:inbound-channel-adapter id="sftpInbound"
channel="sftpInboundChannel" session-factory="sftpClientFactory"
auto-startup="true" filename-pattern="*.csv"
auto-create-local-directory="true" delete-remote-files="true"
remote-directory="${ftp.remoteInputDirectory}" local-directory="${ftp.localTempFolder}">
<int:poller fixed-rate="1000" />
</int-sftp:inbound-channel-adapter>
This error occurs randomly. If someone know why this could be happening or maybe if is something in the configuration above.
Related
After we upgrade spring integration version from 4.3.12 to 5.2.5, the new exception occurred as follows：
org.springframework.messaging.MessageHandlingException: error occurred in message handler [bean 'ollSrFeedbackDeliver$child.ollSrFeedbackOutboundGateway.handler' for component 'ollSrFeedbackDeliver$child.ollSrFeedbackOutboundGateway'; defined in: 'class path resource [config/olltms/olltms_srFeedback_context.xml]'; from source: ''int-sftp:outbound-gateway' with id='ollSrFeedbackOutboundGateway'']; nested exception is org.springframework.messaging.MessagingException: Failed to execute on session; nested exception is org.springframework.core.NestedIOException: failed to rename from /out/DCS_SR_FEEDBACK/SVR3oBweZKH7sf5Lf1Szlef-20200618211747550.xml to /out/DCS_SR_FEEDBACK/backup/SVR3oBweZKH7sf5Lf1Szlef-20200618211747550.xml; nested exception is 2: No such file
our config as follows:
<int-sftp:inbound-streaming-channel-adapter id="ollSrFeedbackInboundAdapter"
comparator="sftpFileModifiedTimeComparator"
session-factory="cachingSftpSessionFactory"
channel="ollSrFeedbackInputChannel"
filename-pattern="${filename-pattern}"
remote-file-separator="/"
remote-directory="${sr-feedback-remote-directory}">
<int:poller fixed-delay="${sftp-in-fixed-delay}"/>
</int-sftp:inbound-streaming-channel-adapter>
use outbound gateway to move file to backup folder:
<int-sftp:outbound-gateway session-factory="cachingSftpSessionFactory"
id="ollSrFeedbackOutboundGateway"
command="mv"
expression="headers.file_remoteDirectory+'/'+headers.file_remoteFile"
rename-expression="headers.file_remoteDirectory+'/backup/'+headers.file_remoteFile"
/>
Before the upgrade, everything is running normally, and problems occurred after the upgrade. After check jsch log, find:
DEBUG [task-scheduler-2] SftpSession: - File: /out/DCS_SR_FEEDBACK/SVR3oBweZKH7sf5Lf1Szlef-20200618211747550.xml was successfully renamed to /out/DCS_SR_FEEDBACK/backup/SVR3oBweZKH7sf5Lf1Szlef-20200618211747550.xml
DEBUG [task-scheduler-3] SftpSession: - Initial File rename failed, possibly because file already exists. Will attempt to delete file: /out/DCS_SR_FEEDBACK/backup/SVR3oBweZKH7sf5Lf1Szlef-20200618211747550.xml and execute rename again.
DEBUG [task-scheduler-3] SftpSession: - Delete file: /out/DCS_SR_FEEDBACK/backup/SVR3oBweZKH7sf5Lf1Szlef-20200618211747550.xml succeeded. Will attempt rename again
is there anything change after upgrade ?
After we upgraded the version of Spring Integration from 4.2.13 to 5.3.1, SFTP Outbound Gateway would often execute the MV command for more than 30 seconds.
We use inbound-stream-channel-adapter to fetch the file and then use outbound-gateway to move it to the Backup folder, below is our xml code snippet
<int:channel id="input">
<int:queue />
</int:channel>
<int:channel id="output">
<int:queue />
<int:interceptors>
<int:wire-tap channel="successHistory"/>
</int:interceptors>
</int:channel>
<int-sftp:inbound-streaming-channel-adapter id="sftInboundAdapter"
session-factory="cachingSftpSessionFactory"
channel="input"
remote-file-separator="/"
remote-directory="/home/box">
<int:poller fixed-delay="2000" max-messages-per-poll="1"/>
</int-sftp:inbound-streaming-channel-adapter>
<int:chain id="chain1" input-channel=" input" output-channel="output">
<int:poller fixed-delay="1000"/>
<int:stream-transformer charset="UTF-8"/>
<int:header-enricher>
<int:error-channel ref="error" overwrite="true"/>
<int:header name="originalPayload" expression="payload"/>
</int:header-enricher>
<int-sftp:outbound-gateway session-factory="cachingSftpSessionFactory"
id="sftpOutboundGateway"
command="mv"
expression="headers.file_remoteDirectory+'/'+headers.file_remoteFile"
rename-expression="headers.file_remoteDirectory+'/backup/'+headers.file_remoteFile"
>
<int-sftp:request-handler-advice-chain>
<ref bean="gatewayLogger"/>
</int-sftp:request-handler-advice-chain>
</int-sftp:outbound-gateway>
<int:transformer expression="headers.originalPayload"/>
</int:chain>
<jms:outbound-channel-adapter channel="output" connection-factory="tibcoEmsConnectionFactory" destination="topic"/>
<bean id="sftpSessionFactory"
class="org.springframework.integration.sftp.session.DefaultSftpSessionFactory">
<property name="host" value="${sftp.host}"/>
<property name="port" value="${sftp.port}"/>
<property name="user" value="${sftp.user}"/>
<property name="password" value="${sftp.password}"/>
<property name="allowUnknownKeys" value="true"/>
<property name="timeout" value="300000"/>
</bean>
<bean id="cachingSftpSessionFactory"
class="org.springframework.integration.file.remote.session.CachingSessionFactory">
<constructor-arg ref="sftpSessionFactory"/>
<constructor-arg value="2"/>
<property name="sessionWaitTimeout" value="300000"/>
</bean>
The Gateway Advice generated logs are as follows, the rename(MV) operation took more than 30 seconds
2020-07-07 12:20:16 INFO [task-scheduler-8] gatewayLogger - ''int-sftp:outbound-gateway' with id='sftpOutboundGateway''#1346093219 - before: {file_remoteHostPort=0.0.0.0, fileName=20200707115747609.xml, errorChannel=bean 'error', file_remoteDirectory=/home/box, originalPayload=<?xml version="1.0" encoding="UTF-8"?>
2020-07-07 12:20:48 INFO [task-scheduler-8] gatewayLogger - ''int-sftp:outbound-gateway' with id='sftpOutboundGateway''#1346093219 - after: org.springframework.integration.support.MessageBuilder#153944c0
As we use a chain for message processing, and session will be released by Stream transformer, if the gateway runs too long , then messages will be pend in queue and related session can’t be released, that will cause message stuck and the adapter will use up all sessions in cache.
It caused by org.springframework.integration.file.remote.RemoteFileUtils#makeDirectories, which is synchronized static method, when there are lots of (S)ftp move operation concurrently and slow speed of network, all requests of AbstractRemoteFileOutboundGateway#mv are queued and observed as very slow speed.
The method signature is as below:
public static synchronized <F> void makeDirectories(String path, Session<F> session, String remoteFileSeparator,
Log logger) throws IOException {
I think the problem is really how you use a CachingSessionFactory. Your cache with that <constructor-arg value="2"/> is too low, therefore there is a high chance of race condition for cached sessions.
You use this session factory in the <int-sftp:inbound-streaming-channel-adapter> which opens a session and keeps it out of the cache until the <int:stream-transformer>. But that happens already on the other thread because your input channel is a QueueChannel. This way you let a thread for <int-sftp:inbound-streaming-channel-adapter> to go and this one is able to take a new session (if any) from the cache. So, when the <int-sftp:outbound-gateway> turn comes, there probably no sessions in the cache to deal with.
Explain, please, why your cache is so low and why do you use QueueChannel just after an inbound polling channel adapter? Not related, but why do you use the QueueChannel for output destination as well?
I think SpringIntegration-5.3.1 has a bug in int-sftp:outbound-gateway as we can easily reproduce sftp gateway execute mv command with long time on certain machine (Our production)
however after we replaced gateway with our own activator, the mv command executed very very fast.
we replaced:
<int-sftp:outbound-gateway session-factory="cachingSftpSessionFactory"
id="sftpOutboundGateway"
command="mv"
expression="headers.file_remoteDirectory+'/'+headers.file_remoteFile"
rename-expression="headers.file_remoteDirectory+'/backup/'+headers.file_remoteFile"
>
with:
<int:header-enricher>
<int:header name="PATH_FROM" expression="headers.file_remoteDirectory+'/'+headers.file_remoteFile"/>
<int:header name="PATH_TO" expression="headers.file_remoteDirectory+'/backup/'+headers.file_remoteFile"/>
</int:header-enricher>
<int:service-activator ref="remoteFileRenameActivator"/>
and here is the source code of our remoteFileRenameActivator
#ServiceActivator
public Message moveFile(Message message, #Header("PATH_FROM") String pathFrom, #Header("PATH_TO") String pathTo) throws IOException {
try (Session session = sessionFactory.getSession();) {
LOGGER.debug(contextName + " " + session.toString() + " is moving file from " + pathFrom + " to " + pathTo);
session.rename(pathFrom, pathTo);
}
return message;
}
The reason of why we think this is a bug because:
We upgraded Spring integration from 4.2.13 to 5.3.1, we didn't meet
such problem in 4.2.13
After we replaced gateway's mv command with
our own implemented mv command, mv command execution was not a
bottleneck anymore.
The issue was still there after we changed
QueueChannel to DirectChannel and increased the session quantity.
We run rename command by command line in client, it is also very fast
While playing with the code from spring-integration-samples project and trying to implement following flow:
read_file->backup_file->transform_file->write_file
'read_file' is file:inbound-channel-adapter
'backup_file' is 'service-activator' calling java service (doing copy file)
'transform_file' are 2 transformers
'write_file' is file:outbound-channel-adapter that is writing the processed file
Everything is working good until the moment when I did try to implement 'backup_file' step as a file:outbound-gateway
I got the error:
Error while loading fileCopyDemo-file due to No qualifying bean of type 'org.springframework.integration.file.FileWritingMessageHandler' available: expected single matching bean but found 2: org.springframework.integration.file.config.FileWritingMessageHandlerFactoryBean#0,org.springframework.integration.file.config.FileWritingMessageHandlerFactoryBean#1
Here is my config:
<!--read_file-->
<int-file:inbound-channel-adapter auto-startup="true" id="filesIn" channel="filesBackup"
directory="${int.uvrp.original}"
filter="compositeFileFilter" >
<int:poller id="poller" max-messages-per-poll="10" cron="*/10 * * * * *" default="true" />
</int-file:inbound-channel-adapter>
<!--backup_file-->
<int-file:outbound-gateway id="backupGate" request-channel="filesBackup" reply-channel="filesProcessing" directory="file:${int.archive}" />
<!--transform_file-->
<int-file:file-to-bytes-transformer input-channel="filesProcessing" output-channel="bytes" />
<int:transformer input-channel="bytes" output-channel="filesStore" ref="myTransformer1" method="transform" />
<!--write_file-->
<int-file:outbound-channel-adapter id="filesOut" channel="filesStore" directory="${int.processed}" delete-source-files="true"/>
Any ideas?
Thanks for the hint. I have created the project from scratch, copied the configuration file and everything is working as expected.
I am maintaining an existing Spring Integration application which is polling a third-party SFTP server for files. It occasionally throws permission or 'not found' errors, which I suspect are caused by transient problems at the remote end. I would like the application to retry on getting these errors, as it will probably resolve the issue. (I also have a requirement to "retry on any problems", which should cover this case.)
e.g.
org.springframework.messaging.MessagingException: Problem occurred while synchronizing remote to local directory; nested exception is org.springframework.messaging.MessagingException: Failure occurred while copying from remote to local directory; nested exception is org.springframework.core.NestedIOException: failed to read file mypath/myfile.csv; nested exception is 3: Permission denied
at [snip]
Caused by: org.springframework.messaging.MessagingException: Failure occurred while copying from remote to local directory; nested exception is org.springframework.core.NestedIOException: failed to read file mypath/myfile.csv; nested exception is 3: Permission denied
at [snip]
Caused by: 3: Permission denied
at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2846) [snip]
After extensive googling and going round in circles, I am still unable to figure out how to do this with Spring Integration. Here is the existing config:
<bean id="myAcceptOnceFilter" class="org.springframework.integration.sftp.filters.SftpPersistentAcceptOnceFileListFilter">
<constructor-arg index="0" ref="myLocalFileStore"/>
<constructor-arg index="1" name="prefix" value="myprefix_"/>
<property name="flushOnUpdate" value="true"/>
</bean>
<bean id="myCompositeFilter" class="org.springframework.integration.file.filters.CompositeFileListFilter">
<constructor-arg>
<list>
<bean class="org.springframework.integration.sftp.filters.SftpSimplePatternFileListFilter">
<constructor-arg value="myprefix" />
</bean>
<ref bean="myAcceptOnceFilter"/>
</list>
</constructor-arg>
</bean>
<int-sftp:inbound-channel-adapter id="myInboundChannel"
session-factory="mySftpSessionFactory"
channel="myDownstreamChannel"
remote-directory="blah"
filter="myCompositeFilter"
local-directory="blah"
auto-create-local-directory="true"
>
<int:poller fixed-rate="10000" max-messages-per-poll="-1">
<int:transactional transaction-manager="transactionManager" synchronization-factory="syncFactory" />
</int:poller>
</int-sftp:inbound-channel-adapter>
EDIT: I think the problem lies in myCompositeFilter. It doesn't look like rollback() is being called inside myAcceptOnceFilter when the exception is thrown. If I simply use myAcceptOnceFilter without the composite then the code works as intended (i.e. rollback() is called). Question is now: how do I continue to use a CompositeFilter which calls rollback on all its children?
I've looked into putting a retry adapter inside the poller (EDIT: I now know this is irrelevant):
<bean id="retryAdvice" class="org.springframework.integration.handler.advice.RequestHandlerRetryAdvice"/>
<int:poller fixed-rate="10000" max-messages-per-poll="-1">
<int:advice-chain>
<tx:advice transaction-manager="transactionManager"/>
<int:ref bean="retryAdvice"/>
</int:advice-chain>
</int:poller>
...but this throws a warning that
This advice org.springframework.integration.handler.advice.RequestHandlerRetryAdvice can only be used for MessageHandlers
In short, I'm stuck. Any help on getting it to retry on this kind of sftp exception would be very gratefully received. Thanks!
EDIT: Added in mention of SftpPersistentAcceptOnceFileListFilter.
EDIT: Added discussion of CompositeFileLIstFilter, which now looks like the location of the problem.
The retry advice is for consuming endpoints (push-retrying).
It's not clear why you need to add retry here - the poller will inherently retry on the the next poll.
I have an inbound gateway which is listening to MQ for messages. Whenever the MQ server is bought down. I loose the connection and in the logs I see that my inbound gateway tries to refresh the connection every 5 seconds. But once the MQ server is up I still see the same error in the log. Can you please let me know if I'm missing any info from the config or is there anything to be done on the MQ server.
Spring Config
<bean id="cachingConnectionFactory1" class="org.springframework.jms.connection.CachingConnectionFactory">
<property name="targetConnectionFactory" ref="mqConnectionFactory1"/>
<property name="reconnectOnException" value="true"/>
</bean>
<bean id="mqConnectionFactory1" class="org.springframework.jms.connection.UserCredentialsConnectionFactoryAdapter">
<property name="targetConnectionFactory">
<bean class="com.ibm.mq.jms.MQQueueConnectionFactory">
<property name="hostName" value="${mq.hostname.1}"/>
<property name="port" value="${mq.port}"/>
<property name="queueManager" value="${mq.queuemanager.1}"/>
<property name="transportType" value="${mq.transporttype}"/>
<property name="channel" value="${mq.channel}"/>
</bean>
</property>
<property name="username" value="${mq.username}"/>
</bean>
<int-jms:message-driven-channel-adapter
id="mqEnterpriseRequestAdapter1"
connection-factory="cachingConnectionFactory1"
destination="xyzQueue"
concurrent-consumers="2"
max-concurrent-consumers="5"
cache-level="5"
idle-consumer-limit="5"
max-messages-per-task="2"
channel="xyzReceive"/>
Error Log:
[2015-09-19 01:52:56,267] WARN [org.springframework.jms.listener.DefaultMessageListenerContainer#4-494492] (DefaultMessageListenerContainer.java:842) - Setup of JMS message listener invoker failed for destination 'queue:///queuename' - trying to recover. Cause: MQJMS2002: failed to get message from MQ queue; nested exception is com.ibm.mq.MQException: MQJE001: Completion Code 2, Reason 2019
[2015-09-19 01:52:51,292] WARN [org.springframework.jms.listener.DefaultMessageListenerContainer#4-494037] (DefaultMessageListenerContainer.java:842) - Setup of JMS message listener invoker failed for destination 'queue:///queuename' - trying to recover. Cause: MQJMS2002: failed to get message from MQ queue; nested exception is com.ibm.mq.MQException: MQJE001: Completion Code 2, Reason 2019
[2015-09-19 01:52:51,263] WARN [org.springframework.jms.listener.DefaultMessageListenerContainer#4-494488] (DefaultMessageListenerContainer.java:842) - Setup of JMS message listener invoker failed for destination 'queue:///queuename' - trying to recover. Cause: MQJMS2002: failed to get message from MQ queue; nested exception is com.ibm.mq.MQException: MQJE001: Completion Code 2, Reason 2019
[2015-09-19 01:52:46,291] WARN [org.springframework.jms.listener.DefaultMessageListenerContainer#4-494033] (DefaultMessageListenerContainer.java:842) - Setup of JMS message listener invoker failed for destination 'queue:///queuename' - trying to recover. Cause: MQJMS2002: failed to get message from MQ queue; nested exception is com.ibm.mq.MQException: MQJE001: Completion Code 2, Reason 2019
[2015-09-19 01:52:46,262] WARN [org.springframework.jms.listener.DefaultMessageListenerContainer#4-494485] (DefaultMessageListenerContainer.java:842) - Setup of JMS message listener invoker failed for destination 'queue:///queuename' - trying to recover. Cause: MQJMS2002: failed to get message from MQ queue; nested exception is com.ibm.mq.MQException: MQJE001: Completion Code 2, Reason 2019
I found the answer to your question by googling MQJE001: Completion Code 2, Reason 2019
The answer is on IBM's support site.
Reason code 2019 usually occurs after a connection broken error (reason code 2009) occurs. You would see a JMSException with reason code 2009 preceding reason code 2019 in the SystemOut.log.
Reason code 2009 indicates that the connection to the MQ queue manager is no longer valid, usually due to a network or firewall issue.
Reason code 2019 errors will occur when invalid connections remain in the connection pool after the reason code 2009 error occurs. The next time that the application tries to use one of these connections, the reason code 2019 occurs.
Resolved after following the below post and the config changes after the modification
http://forum.spring.io/forum/spring-projects/integration/jms/89532-defaultmessagelistenercontainer-cachingconnectionfactory-tomcat-and-websphere-mq
<bean id="mqContainer1" class="org.springframework.jms.listener.DefaultMessageListenerContainer">
<property name="connectionFactory" ref="cachingConnectionFactory1" />
<property name="exceptionListener" ref="cachingConnectionFactory1" />
<property name="destinationName" value="${mq.requestqueue}" />
<property name="maxConcurrentConsumers" value="x"/>
<property name="concurrentConsumers" value="x"/>
<property name="maxMessagesPerTask" value="x"/>
<property name="idleConsumerLimit" value="x"/>
</bean>
<int-jms:message-driven-channel-adapter
id="mqEnterpriseRequestAdapter1"
container="cbatsMqContainer1"
channel="mqMessageReceive"/>