What is the wrong in my integration flow and tasklet - spring-integration

This is my flows LS-GET(SFTP outbound gateway: download files from a remote SFTP server.)
and MessagingGateway.
#MessagingGateway
public interface IntegratedRemoteFileProcessMessagingGateway {
#Gateway(requestChannel = "getFlows.input")
void getFlows(final String remoteDirectory);
#Gateway(requestChannel = "moveFlows.input")
void moveFlows(final String remoteDirectory);
}
#Bean
public QueueChannelSpec getOutputChannel() {
return MessageChannels.queue();
}
#Bean
public IntegrationFlow getFlows() {
return f -> f
.enrichHeaders(h -> h
.headerExpression("originalPayload", "payload.toString()")
.headerExpression(FileHeaders.REMOTE_DIRECTORY, "payload.toString()"))
.log(LoggingHandler.Level.INFO, "eu.haee", "'Header originalPayload=' + headers[originalPayload]")
.handle(Sftp.outboundGateway(sessionFactory, Command.LS.getCommand(), "payload")
.autoCreateDirectory(false)
.autoCreateLocalDirectory(false)
.charset("UTF-8")
.filter(new SftpSimplePatternFileListFilter("*.xml"))
.options(Option.NAME_ONLY, Option.RECURSIVE))
.split()
.log(LoggingHandler.Level.INFO, "eu.haee", "'LS Payload= ' + payload.toString()")
.enrichHeaders(h -> h
.headerExpression("originalRemoteFile", "payload.toString()")
.headerExpression(FileHeaders.REMOTE_FILE, "payload.toString()"))
.handle(Sftp.outboundGateway(sessionFactory, Command.GET.getCommand(), "headers['originalPayload'] + headers['file_remoteFile']")
.autoCreateLocalDirectory(false)
.charset("UTF-8")
.fileNameExpression("headers['file_remoteFile']")
.localDirectory(new File(flowsConfiguration.localDirectory()))
.localFilenameExpression(new FunctionExpression<Message<?>>(m -> {
IntegrationMessageHeaderAccessor accessor = new IntegrationMessageHeaderAccessor(m);
final String remoteFileName = (String) accessor.getHeader("file_remoteFile");
final int extensionIndex = remoteFileName.lastIndexOf('.');
DateTimeFormatter formatter = DateTimeFormatter.ofPattern("yyyyMMddHHmmss.SSSSS", Locale.GERMAN);
return String.format("%s_MYC(%s-%d)%s", remoteFileName.substring(0, extensionIndex),
ZonedDateTime.of(LocalDateTime.now(), ZoneId.of("Europe/Berlin")).format(formatter),
(new SecureRandom()).nextInt(99999),
remoteFileName.substring(extensionIndex));
}))
.options(Option.PRESERVE_TIMESTAMP)
.remoteFileSeparator("/"))
.channel("getOutputChannel");
}
This is my spring-batch tasklet and Junit. MessagingGateway injected via tasklet constructor.
#Override
public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext) throws Exception {
for (Endpoint endpoint : endpoints) {
final String remoteDirectory = endpoint.getEpConUri();
logger.info("ProcessRemoteFilesFlowsTasklet {} dealer at {} remote files process starting",
endpoint.getId().getDlrCd(), remoteDirectory);
flowsMessagingGateway.getFlows(remoteDirectory);
}
return RepeatStatus.FINISHED;
}
#Override
public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext) throws Exception {
controlChannel.send(new GenericMessage<>("#getPoller.start()"));
logger.info("GetPollerRemoteFilesFlowsTasklet poller starting...");
return RepeatStatus.FINISHED;
}
#Autowired
private IntegratedRemoteFileProcessMessagingGateway flowsMessagingGateway;
#Autowired
private EndpointRepository endpointRepository;
#Test
public void getFlows() {
flowsMessagingGateway.getFlows("/c07va00011/iris/import/");
Uninterruptibles.sleepUninterruptibly(60, TimeUnit.SECONDS);
}
When I execute getFlows test code. I met exception. but file downloaded to my local computer.
I have no idea. I've tried many variants but didn't get any progress.
org.springframework.messaging.MessageDeliveryException: Dispatcher has no subscribers for channel 'application.getFlows.input'.; nested exception is org.springframework.integration.MessageDispatchingException: Dispatcher has no subscribers, failedMessage=GenericMessage [payload=/c07va00011/iris/import/, headers={replyChannel=org.springframework.messaging.core.GenericMessagingTemplate$TemporaryReplyChannel#2e64ae1a, errorChannel=org.springframework.messaging.core.GenericMessagingTemplate$TemporaryReplyChannel#2e64ae1a, id=bd393cb7-42d0-03b2-674d-40e3cf9211de, timestamp=1609844917799}]
...
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:210)
Caused by: org.springframework.integration.MessageDispatchingException: Dispatcher has no subscribers
at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:139)
at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:106)
at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:72)
... 145 common frames omitted
#EnableIntegration placed every spring-integration related configuration classes. #IntegrationComponentScan also placed my main flow configuration class (with string arrays of package names to scan).
If #EnableIntegration annotation located in multiple classes what happens?
Should I merge all spring-batch and spring-integration configuration classes into one?
Also, I've tested ControlBus(Send messages to the poller in the spring-batch tasklet) and got the same exception.
11:57:36.481 [main] ERROR o.s.batch.core.step.AbstractStep - Encountered an error executing step startGetPollerRemoteFilesStep in job integratedFilesProcessJob2
org.springframework.messaging.MessageDeliveryException: Dispatcher has no subscribers for channel 'application.controlChannel'.; nested exception is org.springframework.integration.MessageDispatchingException: Dispatcher has no subscribers, failedMessage=GenericMessage [payload=#getPoller.start(), headers={id=539a27d0-9bce-062d-8664-53aae14b5680, timestamp=1609930656454}]
#Lazy, #DependsOn also not working. (#Lazy added to the ControlBus, #DependsOn added in the spring service class: Spring-batch jobs also manually starts/stops by rest API calls.)
#Autowired
public BatchFileServiceConfiguration(JobBuilderFactory jobBuilderFactory,
StepBuilderFactory stepBuilderFactory,
PropertyConfiguration propertyConfiguration,
#Qualifier("sourceBatchTransactionManager") PlatformTransactionManager sourceBatchTransactionManager,
#Qualifier("sourceBatchEntityManagerFactory") EntityManagerFactory sourceBatchEntityManagerFactory,
#Qualifier("processFileTaskExecutor") TaskExecutor processFileTaskExecutor,
BatchEndpointRepository batchEndpointRepository,
RemoteFileProcessMessagingGateway remoteFileProcessMessagingGateway,
#Lazy #Qualifier("controlChannel") MessageChannel controlChannel) {
this.jobBuilderFactory = jobBuilderFactory;
this.stepBuilderFactory = stepBuilderFactory;
this.propertyConfiguration = propertyConfiguration;
this.sourceBatchTransactionManager = sourceBatchTransactionManager;
this.sourceBatchEntityManagerFactory = sourceBatchEntityManagerFactory;
this.processFileTaskExecutor = processFileTaskExecutor;
this.batchEndpointRepository = batchEndpointRepository;
this.remoteFileProcessMessagingGateway = remoteFileProcessMessagingGateway;
this.controlChannel = controlChannel;
#Service
#DependsOn({"lsFlows", "getFlows", "moveFlows", "moveFailedFlows", "getPollableFlows"})
public class FileServiceImpl implements FileService {
These exceptions were never occurred in the spring-integration stand-alone applications.

Related

Spring Integration resequencer does not release the last group of messages

I have the following configuration:
#Bean
public IntegrationFlow messageFlow(JdbcMessageStore groupMessageStore, TransactionSynchronizationFactory syncFactory, TaskExecutor te, ThreadPoolTaskScheduler ts, RealTimeProcessor processor) {
return IntegrationFlows
.from("inputChannel")
.handle(processor, "handleInputMessage", consumer -> consumer
.taskScheduler(ts)
.poller(poller -> poller
.fixedDelay(pollerFixedDelay)
.receiveTimeout(pollerReceiveTimeout)
.maxMessagesPerPoll(pollerMaxMessagesPerPoll)
.taskExecutor(te)
.transactional()
.transactionSynchronizationFactory(syncFactory)))
.resequence(s -> s.messageStore(groupMessageStore)
.releaseStrategy(new TimeoutCountSequenceSizeReleaseStrategy(50, 30000)))
.channel("sendingChannel")
.handle(processor, "sendMessage")
.get();
}
If I send a single batch of e.g. 100 messages to the inputChannel it works as expected until there are no messages in the inputChannel. After the inputChannel becomes empty it also stops processing for messages that were waiting for sequencing. As a result there are always a couple of messages left in the groupMessageStore even after the set release timeout.
I'm guessing it's because the poller is configured only for the inputChannel and if there are no messages in there it will never get to the sequencer (so will never call canRelease on the release strategy).
But if I try adding a separate poller for the resequencer I get the following error A poller should not be specified for endpoint since channel x is a SubscribableChannel (not pollable).
Is there a different way to configure it so that the last group of messages is always released?
The release strategy is passive and needs something to trigger it to be called.
Add .groupTimeout(...) to release the partial sequence after the specified time elapses.
EDIT
#SpringBootApplication
public class So67993972Application {
private static final Logger log = LoggerFactory.getLogger(So67993972Application.class);
public static void main(String[] args) {
SpringApplication.run(So67993972Application.class, args);
}
#Bean
IntegrationFlow flow(MessageGroupStore mgs) {
return IntegrationFlows.from(MessageChannels.direct("input"))
.resequence(e -> e.messageStore(mgs)
.groupTimeout(5_000)
.sendPartialResultOnExpiry(true)
.releaseStrategy(new TimeoutCountSequenceSizeReleaseStrategy(50, 2000)))
.channel(MessageChannels.queue("output"))
.get();
}
#Bean
MessageGroupStore mgs() {
return new SimpleMessageStore();
}
#Bean
public ApplicationRunner runner(MessageChannel input, QueueChannel output, MessageGroupStore mgs) {
return args -> {
MessagingTemplate template = new MessagingTemplate(input);
log.info("Sending");
template.send(MessageBuilder.withPayload("foo")
.setHeader(IntegrationMessageHeaderAccessor.CORRELATION_ID, "bar")
.setHeader(IntegrationMessageHeaderAccessor.SEQUENCE_NUMBER, 2)
.setHeader(IntegrationMessageHeaderAccessor.SEQUENCE_SIZE, 2)
.build());
log.info(output.receive(10_000).toString());
Thread.sleep(1000);
log.info(mgs.getMessagesForGroup("bar").toString());
};
}
}

Spring Integration config errror: Dispatcher has no subscribers

Can you help with spring integration config.
After receiving message I got:
org.springframework.messaging.MessageDeliveryException: Dispatcher has no subscribers for channel 'myapp-1.scMyRequestChannel'.; nested exception is org.springframework.integration.MessageDispatchingException: Dispatcher has no subscribers, failedMessage=GenericMessage [payload=input me, headers={JMS_IBM_Character_Set=UTF-8, errorChannel=org.springframework.messaging.core.GenericMessagingTemplate$TemporaryReplyChannel#6c7a1290, JMS_IBM_MsgType=8, jms_destination=queue:///SYN.REQ, JMSXUserID=mqm , JMS_IBM_Encoding=273, priority=4, jms_timestamp=1590129774770, JMSXAppID=hermes.browser.HermesBrowser, JMS_IBM_PutApplType=28, JMS_IBM_Format=MQSTR , replyChannel=org.springframework.messaging.core.GenericMessagingTemplate$TemporaryReplyChannel#6c7a1290, jms_redelivered=true, JMS_IBM_PutDate=20200522, JMSXDeliveryCount=649, JMS_IBM_PutTime=06425480, id=ed13c600-56b5-33ff-7228-0a5d146b618b, jms_messageId=ID:414d5120514d31202020202020202020aa6fc75e004d3e12, timestamp=1590135772781}]
at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:77) ~[spring-integration-core-5.1.5.RELEASE.jar:5.1.5.RELEASE]
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:453) ~[spring-integration-core-5.1.5.RELEASE.jar:5.1.5.RELEASE]
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:187) ~[spring-messaging-5.1.7.RELEASE.jar:5.1.7.RELEASE]
at org.springframework.messaging.core.GenericMessagingTemplate.doSendAndReceive(GenericMessagingTemplate.java:233) ~[spring-messaging-5.1.7.RELEASE.jar:5.1.7.RELEASE]
at org.springframework.messaging.core.GenericMessagingTemplate.doSendAndReceive(GenericMessagingTemplate.java:47) ~[spring-messaging-5.1.7.RELEASE.jar:5.1.7.RELEASE]
at org.springframework.messaging.core.AbstractMessagingTemplate.sendAndReceive(AbstractMessagingTemplate.java:46) ~[spring-messaging-5.1.7.RELEASE.jar:5.1.7.RELEASE]
at org.springframework.integration.core.MessagingTemplate.sendAndReceive(MessagingTemplate.java:97) ~[spring-integration-core-5.1.5.RELEASE.jar:5.1.5.RELEASE]
at org.springframework.integration.gateway.MessagingGatewaySupport.doSendAndReceive(MessagingGatewaySupport.java:499) ~[spring-integration-core-5.1.5.RELEASE.jar:5.1.5.RELEASE]
at org.springframework.integration.gateway.MessagingGatewaySupport.sendAndReceiveMessage(MessagingGatewaySupport.java:470) ~[spring-integration-core-5.1.5.RELEASE.jar:5.1.5.RELEASE]
at org.springframework.integration.jms.ChannelPublishingJmsMessageListener$GatewayDelegate.sendAndReceiveMessage(ChannelPublishingJmsMessageListener.java:511) ~[spring-integration-jms-5.1.2.RELEASE.jar:5.1.2.RELEASE]
at org.springframework.integration.jms.ChannelPublishingJmsMessageListener.onMessage(ChannelPublishingJmsMessageListener.java:344) ~[spring-integration-jms-5.1.2.RELEASE.jar:5.1.2.RELEASE]
at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:736) ~[spring-jms-5.1.7.RELEASE.jar:5.1.7.RELEASE]
at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:696) ~[spring-jms-5.1.7.RELEASE.jar:5.1.7.RELEASE]
at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:674) ~[spring-jms-5.1.7.RELEASE.jar:5.1.7.RELEASE]
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:318) [spring-jms-5.1.7.RELEASE.jar:5.1.7.RELEASE]
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:257) [spring-jms-5.1.7.RELEASE.jar:5.1.7.RELEASE]
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1189) [spring-jms-5.1.7.RELEASE.jar:5.1.7.RELEASE]
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1179) [spring-jms-5.1.7.RELEASE.jar:5.1.7.RELEASE]
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1076) [spring-jms-5.1.7.RELEASE.jar:5.1.7.RELEASE]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_201]
My code:
#Configuration
public class MyEndpointConfiguration {
public static final String SC_My_REQUEST_CHANNEL = "scMyRequestChannel";
public static final String SC_My_RESPONSE_CHANNEL = "scMyResponseChannel";
#Bean
public MessageChannel scMyRequestChannel() {
return new DirectChannel();
}
#Bean
public MessageChannel scMyResponseChannel() {
return new DirectChannel();
}
#Bean
public MessageConverter myMessageConverter() {
return new MyMessageConverter();
}
#Bean
JmsInboundGateway myJmsInboundGateway(ConnectionFactory myConnectionFactory,
MessageConverter myMessageConverter,
MessageChannel scMyRequestChannel,
MessageChannel scMyResponseChannel,
Destination myScRequestDestination,
Destination myScResponseDestination) {
return Jms
.inboundGateway(myConnectionFactory)
.destination(myScRequestDestination)
.defaultReplyDestination(myScResponseDestination)
.requestChannel(scMyRequestChannel)
.replyChannel(scMyResponseChannel)
.jmsMessageConverter(myMessageConverter)
.get();
}
public IntegrationFlow getFromConvertAndPutToSpring(JmsInboundGateway myJmsInboundGateway) {
return IntegrationFlows
.from(myJmsInboundGateway)
.handle((p,h) -> "my answer")
.logAndReply("inbound");
}
// public IntegrationFlow getFromInputChannel(MessageChannel scMyRequestChannel) {
// return IntegrationFlows
// .from(scMyRequestChannel)
// .log("sc input")
// .handle((p,h) -> "my answer")
// .log()
// .nullChannel();
// }
}
I have simillar JmsInboudGateway config for other queue without .requestChannel and .replyChannel and it well.
If instead of injecting request channel bean I declare it with text name got this
org.springframework.messaging.MessagingException: No request channel available. Cannot send request message.
at org.springframework.integration.gateway.MessagingGatewaySupport.doSendAndReceive(MessagingGatewaySupport.java:479) ~[spring-integration-core-5.1.5.RELEASE.jar:5.1.5.RELEASE]
at org.springframework.integration.gateway.MessagingGatewaySupport.sendAndReceiveMessage(MessagingGatewaySupport.java:470) ~[spring-integration-core-5.1.5.RELEASE.jar:5.1.5.RELEASE]
at org.springframework.integration.jms.ChannelPublishingJmsMessageListener$GatewayDelegate.sendAndReceiveMessage(ChannelPublishingJmsMessageListener.java:511) ~[spring-integration-jms-5.1.2.RELEASE.jar:5.1.2.RELEASE]
And more text for publicating question in sof.
sorry, it looks like I miss Bean annotation in process of testing.

Recursively read files with Spring Integration SFTP DSL

I am trying to make Spring Integration SFTP read files (.txt) from a remote server recursively from all subfolders. The remote folder is something like "/tmp/remoteFolder" and all subfolders are date folders like "/tmp/remoteFolder/20180830", "/tmp/remoteFolder/20170902".
This is the code that I have until now
#Bean
#InboundChannelAdapter(value = "sftpMgetInputChannel",
poller = #Poller(fixedDelay = "5000"))
public IntegrationFlow sftpMGetFlow() {
return IntegrationFlows.from("sftpMgetInputChannel")
.handleWithAdapter(h -> h.sftpGateway(this.sftpSessionFactory,
Command.MGET, "'/tmp/remoteDirectory/*'")
.options(Option.RECURSIVE)
.regexFileNameFilter("((\\d{8})|*\\.txt)")
.localDirectoryExpression("sftp-inbound" + "/" + "#remoteDirectory"))
.channel(remoteFileOutputChannel())
.get();
}
#Bean
public MessageChannel sftpMgetInboundChannel(){
return new DirectChannel();
}
#Bean
public PollableChannel remoteFileOutputChannel() {
return new QueueChannel();
}
How do I specify the root remote directory for sftp mget to be /tmp/remoteFolder? Why isn't this working? Why do I need to specifiy the output channel?
Update: Instead of calling channel(remoteFileOutputChannel()) I call a handler like this
#Bean
public MessageHandler messageHandler(){
return new MessageHandler() { ... }
}
Code updated:
#InboundChannelAdapter(value = "sftpMgetInputChannel",
poller = #Poller(fixedDelay = "5000"))
public String filesForMGET(){
return "'/tmp/input/remoteDirectory/*'";
}
#Bean
public IntegrationFlow sftpMGetFlow() {
return IntegrationFlows.from("sftpMgetInputChannel")
.handleWithAdapter(h -> h.sftpGateway(this.sftpSessionFactory,
Command.MGET, "payload")
.options(Option.RECURSIVE)
.regexFileNameFilter("((\\d{8})|*\\.txt)")
.localDirectoryExpression("'sftp-inbound/'" + "#remoteDirectory"))
.handler(messageHandler())
.get();
}
#Bean
public MessageChannel sftpMgetInboundChannel(){
return new DirectChannel();
}
#Bean
public MessageHandler messageHandler(){
return new MessageHandler() { ... }
}
With this updated code, I get the following error:
rg.springframework.core.NestedIOException: failed to read file; nested exception is 2: No such file
at org.springframework.integration.sftp.session.SftpSession.read(SftpSession.java:100)
at org.springframework.integration.file.remote.session.CachingSessionFactory$CachedSession.read(CachingSessionFactory.java:137)
at org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizer.copyFileToLocalDirectory(AbstractInboundFileSynchronizer.java:176)
at org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizer.synchronizeToLocalDirectory(AbstractInboundFileSynchronizer.java:138)
at org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizingMessageSource.receive(AbstractInboundFileSynchronizingMessageSource.java:144)
at org.springframework.integration.endpoint.SourcePollingChannelAdapter.doPoll(SourcePollingChannelAdapter.java:89)
at org.springframework.integration.endpoint.AbstractPollingEndpoint$1.call(AbstractPollingEndpoint.java:146)
at org.springframework.integration.endpoint.AbstractPollingEndpoint$1.call(AbstractPollingEndpoint.java:144)
at org.springframework.integration.endpoint.AbstractPollingEndpoint$Poller$1.run(AbstractPollingEndpoint.java:207)
at org.springframework.integration.util.ErrorHandlingTaskExecutor$1.run(ErrorHandlingTaskExecutor.java:52)
at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:48)
at org.springframework.integration.util.ErrorHandlingTaskExecutor.execute(ErrorHandlingTaskExecutor.java:49)
at org.springframework.integration.endpoint.AbstractPollingEndpoint$Poller.run(AbstractPollingEndpoint.java:202)
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:51)
at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:81)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)
Caused by: 2: No such file
at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289)
at com.jcraft.jsch.ChannelSftp._stat(ChannelSftp.java:1741)
at com.jcraft.jsch.ChannelSftp.get(ChannelSftp.java:1011)
at com.jcraft.jsch.ChannelSftp.get(ChannelSftp.java:986)
at org.springframework.integration.sftp.session.SftpSession.read(SftpSession.java:96)
... 22 more
With the expression set to payload (as was the case in your question before the edit), the message payload sent to the gateway should be /tmp/remoteFolder/* which internally is split into remote directory and remote filename (*).
Why do I need to specifiy the output channel?
The result of the MGET (list of retrieved files) needs to go somewhere.
EDIT
You misunderstond; you can't add the #InboundChannelAdapter annotation to the flow; you need something like this...
#InboundChannelAdapter(value = "sftpMgetInputChannel",
poller = #Poller(fixedDelay = "5000"))
public String filesForMGET() {
return "/tmp/remoteDirectory/";
}
#Bean
public IntegrationFlow sftpMGetFlow() {
return IntegrationFlows.from("sftpMgetInputChannel")
.handleWithAdapter(h -> h.sftpGateway(this.sftpSessionFactory,
Command.MGET, "payload")
.options(Option.RECURSIVE)
.regexFileNameFilter("((\\d{8})|*\\.txt)")
.localDirectoryExpression("sftp-inbound" + "/" + "#remoteDirectory"))
.channel(remoteFileOutputChannel())
.get();
}

How could I transfer message dynamically in Sftp Inbound adapter through Spring DSL in java 1.7

I have a Sftp inbound flow and I got the session information from DefaultSftpSessionFactory. But I need to implement mulitple session information dynamically which I will get from database table. That means I have multiple number of Sftp server details that I need to implement in my integration flow. Now I have done with file transfer from single source to single destination but I need to implement multiple source to multiple destination. So can any one provide some pointer on this.
This is my session Factory...Here I have a single Sftp server information but how to configure multiple server details.
#Autowired
private DefaultSftpSessionFactory sftpSessionFactory;
#Bean
public DefaultSftpSessionFactory sftpSessionFactory() {
DefaultSftpSessionFactory factory = new DefaultSftpSessionFactory(
true);
factory.setHost("111.11.12.143");
factory.setPort(22);
factory.setUser("sftp");
factory.setPassword("*******");
return factory;
}
and this is my Sftp Inbound flow..
#Bean
public IntegrationFlow sftpInboundFlow() {
System.out.println("enter sftpInboundFlow....."
+ sftpSessionFactory.getSession());
return IntegrationFlows
.from(Sftp.inboundAdapter(this.sftpSessionFactory)
.preserveTimestamp(true).remoteDirectory(remDir)
.regexFilter(".*\\.txt$")
.localFilenameExpression("#this.toUpperCase()")
.localDirectory(new File(localDir))
.remoteFileSeparator("/"),
new Consumer<SourcePollingChannelAdapterSpec>() {
#Override
public void accept(SourcePollingChannelAdapterSpec e) {
e.id("sftpInboundAdapter")
.autoStartup(true)
.poller(Pollers.fixedRate(1000)
.maxMessagesPerPoll(1));
}
})
//.channel(MessageChannels.queue("sftpInboundResultChannel"))
.channel(sftpInboundResultChannel())
.get();
}
As suggested by Gary I am editing my post....
Hi Gary,
I am taking the reference from Github dynamic FTP example.
Through the ChannelResolver class I need to call my above DSL class. and set the dynamic value in context property without using XML.
In my ChannelResolver class I want some thing like this
StandardEnvironment env = new StandardEnvironment();
Properties props = new Properties();
props.setProperty("inbound.host", host); //I am getting the value of 'host' from a DB table.
PropertiesPropertySource pps = new PropertiesPropertySource("sftpprop", props);
env.getPropertySources().addLast(pps);
context.setEnvironment(env);
And my DSL class I need to use like this.
#Value("${inbound.host}")
private String host;
So in this way can I set dynamic value for String 'host' ?
I am editing my original post...........
In my Outbound dynamic resolver class I am doing like this
StandardEnvironment env = new StandardEnvironment();
Properties props = new Properties();
props.setProperty("outbound.host", host);
props.setProperty("outbound.port", String.valueOf(port));
props.setProperty("outbound.user", user);
props.setProperty("outbound.password", password);
props.setProperty("outbound.remote.directory", remoteDir);
props.setProperty("outbound.local.directory", localDir);
PropertiesPropertySource pps = new PropertiesPropertySource("ftpprops", props);
env.getPropertySources().addLast(pps);
ctx.setEnvironment(env);
And this is my dsl class....
#Autowired
private DefaultSftpSessionFactory sftpSessionFactory;
#Bean
public DefaultSftpSessionFactory sftpSessionFactory(#Value("${outbound.host}") String host, #Value("${outbound.port}") int port,
#Value("${outbound.user}") String user, #Value("${outbound.password}") String password
) {
DefaultSftpSessionFactory factory = new DefaultSftpSessionFactory(true);
factory.setHost(host);
factory.setPort(port);
factory.setUser(user);
factory.setPassword(password);
return factory;
}
#Bean
public IntegrationFlow fileInboundFlow(#Value("${outbound.local.directory}") String localDir)
{
return IntegrationFlows
.from(Files.inboundAdapter(new File(localDir)),
new Consumer<SourcePollingChannelAdapterSpec>() {
#Override
public void accept(SourcePollingChannelAdapterSpec e) {
e.autoStartup(true).poller(
Pollers.fixedDelay(5000)
.maxMessagesPerPoll(1));
}
})
.channel(sftpSendChannel())
.get();
}
#Bean
public IntegrationFlow sftpOutboundFlow(#Value("${outbound.remote.directory}") String remDir) {
return IntegrationFlows
.from(sftpSendChannel())
.handle(Sftp.outboundAdapter(this.sftpSessionFactory)
.useTemporaryFileName(false)
.remoteDirectory(remDir))
.get();
}
#Bean
public MessageChannel sftpSendChannel() {
return new DirectChannel();
}
#Bean
public static PropertySourcesPlaceholderConfigurer configurer1() {
return new PropertySourcesPlaceholderConfigurer();
}
And this the error log from console...
Aug 03, 2015 7:50:25 PM org.apache.catalina.core.StandardContext listenerStart
SEVERE: Exception sending context initialized event to listener instance of class org.springframework.web.context.ContextLoaderListener
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'sftpOutBoundDsl': Injection of autowired dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not autowire field: private org.springframework.integration.sftp.session.DefaultSftpSessionFactory com.tcs.iux.ieg.sftp.dynamic.SftpOutBoundDsl.sftpSessionFactory; nested exception is java.lang.IllegalArgumentException: Could not resolve placeholder 'outbound.host' in string value "${outbound.host}"
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:334)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1204)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:538)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:476)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:302)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:229)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:298)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:725)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:757)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:480)
at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:403)
at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:306)
at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:106)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4973)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5467)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1559)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1549)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.springframework.beans.factory.BeanCreationException: Could not autowire field: private org.springframework.integration.sftp.session.DefaultSftpSessionFactory com.tcs.iux.ieg.sftp.dynamic.SftpOutBoundDsl.sftpSessionFactory; nested exception is java.lang.IllegalArgumentException: Could not resolve placeholder 'outbound.host' in string value "${outbound.host}"
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:555)
at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:87)
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:331)
... 22 more
Caused by: java.lang.IllegalArgumentException: Could not resolve placeholder 'outbound.host' in string value "${outbound.host}"
at org.springframework.util.PropertyPlaceholderHelper.parseStringValue(PropertyPlaceholderHelper.java:174)
at org.springframework.util.PropertyPlaceholderHelper.replacePlaceholders(PropertyPlaceholderHelper.java:126)
at org.springframework.core.env.AbstractPropertyResolver.doResolvePlaceholders(AbstractPropertyResolver.java:204)
at org.springframework.core.env.AbstractPropertyResolver.resolveRequiredPlaceholders(AbstractPropertyResolver.java:178)
at org.springframework.context.support.PropertySourcesPlaceholderConfigurer$2.resolveStringValue(PropertySourcesPlaceholderConfigurer.java:175)
at org.springframework.beans.factory.support.AbstractBeanFactory.resolveEmbeddedValue(AbstractBeanFactory.java:800)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:917)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:904)
at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:815)
at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:743)
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:466)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1113)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1008)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:505)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:476)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:302)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:229)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:298)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.findAutowireCandidates(DefaultListableBeanFactory.java:1088)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1006)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:904)
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:527)
... 24 more
It is currently not supported.
We have an open JIRA to add support for dynamic server selection but it's unlikely to be done in time for the upcoming 4.2 release.
You could work around it by writing your own custom delegating session factory that uses some criteria (e.g. a ThreadLocal) to determine which delegate factory to use.
EDIT:
As with the XML, you need a PropertySourcesPlaceholderConfigurer bean.
You should also use factory-method injection because the #Configuration class is created too early to have the #Value injected...
#Configuration
public class FooConfig {
#Bean
public DefaultSftpSessionFactory factory(
#Value("${inbound.host}") String host,
#Value("${inbound.port}") int port) {
DefaultSftpSessionFactory sf = new DefaultSftpSessionFactory();
sf.setHost(host);
sf.setPort(port);
return sf;
}
#Bean
public PropertySourcesPlaceholderConfigurer configurer() {
return new PropertySourcesPlaceholderConfigurer();
}
}
.
public class Testing {
#Test
public void test() {
AnnotationConfigApplicationContext context = new AnnotationConfigApplicationContext();
context.register(FooConfig.class);
StandardEnvironment env = new StandardEnvironment();
Properties props = new Properties();
props.setProperty("inbound.host", "bar");
props.setProperty("inbound.port", "23");
PropertiesPropertySource pps = new PropertiesPropertySource("sftpprop", props);
env.getPropertySources().addLast(pps);
context.setEnvironment(env);
context.refresh();
DefaultSftpSessionFactory sessionFactory = context.getBean(DefaultSftpSessionFactory.class);
assertEquals("bar", TestUtils.getPropertyValue(sessionFactory, "host"));
context.close();
}
}
By the way, the delegating session factory will be in 4.2 after all.
EDIT2:
You can avoid the early instantiation of the config class and use global #Value injection, as long as you make the PSPC bean static...
#Configuration
public class FooConfig {
#Value("${foo}")
public String foo;
#Bean
public String earlyFoo() {
return this.foo;
}
#Bean
public String foo(#Value("${foo}") String foo) {
return foo;
}
#Bean
public static PropertySourcesPlaceholderConfigurer configurer() {
return new PropertySourcesPlaceholderConfigurer();
}
}
In this case, earlyFoo is populated as expected.

Large data processing using Spring Batch Multi-threaded Step and RepositoryItemWriter/ RepositoryItemReader

I am trying to write a batch processing application using spring batch with multi-thread step.this is simple application reading data from a table and writing to another table but data is large around 2 million record .
I am using RepositoryItemReader & RepositoryItemWriter for reading and writing data. But after processing some data it failing due to Unable to acquire JDBC Connection.
//Config.Java
#Bean
public TaskExecutor taskExecutor() {
SimpleAsyncTaskExecutor taskExecutor = new SimpleAsyncTaskExecutor();
taskExecutor.setConcurrencyLimit(10);
return taskExecutor;
}
#Bean(name = "personJob")
public Job personKeeperJob() {
Step step = stepBuilderFactory.get("step-1")
.<User, Person> chunk(1000)
.reader(userReader)
.processor(jpaProcessor)
.writer(personWriter)
.taskExecutor(taskExecutor())
.throttleLimit(10)
.build();
Job job = jobBuilderFactory.get("person-job")
.incrementer(new RunIdIncrementer())
.listener(this)
.start(step)
.build();
return job;
}
//Processor.Java
#Override
public Person process(User user) throws Exception {
Optional<User> userFromDb = userRepo.findById(user.getUserId());
Person person = new Person();
if(userFromDb.isPresent()) {
person.setName(userFromDb.get().getName());
person.setUserId(userFromDb.get().getUserId());
person.setDept(userFromDb.get().getDept());
}
return person;
}
//Reader.Java
#Autowired
public UserItemReader(final UserRepository repository) {
super();
this.repository = repository;
}
#PostConstruct
protected void init() {
final Map<String, Sort.Direction> sorts = new HashMap<>();
sorts.put("userId", Direction.ASC);
this.setRepository(this.repository);
this.setSort(sorts);
this.setMethodName("findAll");
}
//Writer.Java
#PostConstruct
protected void init() {
this.setRepository(repository);
}
#Transactional
public void write(List<? extends Person> persons) throws Exception {
repository.saveAll(persons);
}
application.properties
# Datasource
spring.datasource.platform=h2
spring.datasource.url=jdbc:h2:mem:batchdb
spring.main.allow-bean-definition-overriding=true
spring.datasource.hikari.maximum-pool-size=500
Error :
org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection
at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:447)
......................
Caused by: org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection
at org.hibernate.exception.internal.SQLExceptionTypeDelegate.convert(SQLExceptionTypeDelegate.java:48)
............................
Caused by: java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30927ms.
You run out of connections.
Try to set the Hikari Connection Pool to a bigger number:
spring.datasource.hikari.maximum-pool-size=20

Resources