I am creating a sftp inbound flow in spring dsl but when I am trying to test this in junit the mesage is coming null. I think the sftp ibound adapter is not starting properly, but not sure. So I am not able to go further. Can any ony one please provide any pointer to it as I am not able to proceed.
This is my flow....
#Bean
public IntegrationFlow sftpInboundFlow() {
return IntegrationFlows
.from(Sftp.inboundAdapter(this.sftpSessionFactory)
.preserveTimestamp(true).remoteDirectory(remDir)
.regexFilter(".*\\.txt$")
.localFilenameExpression("#this.toUpperCase()")
.localDirectory(new File(localDir))
.remoteFileSeparator("/"),
new Consumer<SourcePollingChannelAdapterSpec>() {
#Override
public void accept(
SourcePollingChannelAdapterSpec e) {
e.id("sftpInboundAdapter")
.autoStartup(true)
.poller(Pollers.fixedRate(1000)
.maxMessagesPerPoll(1));
}
})
.channel(MessageChannels.queue("sftpInboundResultChannel"))
.get();
}
This is my session Factory...
#Autowired
private DefaultSftpSessionFactory sftpSessionFactory;
#Bean
public DefaultSftpSessionFactory sftpSessionFactory() {
DefaultSftpSessionFactory factory = new DefaultSftpSessionFactory(
true);
factory.setHost("111.11.12.143");
factory.setPort(22);
factory.setUser("sftp");
factory.setPassword("*******");
return factory;
}
This is my Pollable channel..
#Autowired
private PollableChannel sftpInboundResultChannel;
#Bean
public PollableChannel sftpChannel() {
return new QueueChannel();
}
And this is my Test method..
#Test
public void testSftpInboundFlow() {
Message<?> message = ((PollableChannel) sftpInboundResultChannel).receive(1000); //Not receiving any message
System.out.println("message====" + message);
Object payload = message.getPayload();
File file = (File) payload;
message = this.sftpInboundResultChannel.receive(1000);
file = (File) message.getPayload();
}
Related
I have a SpringBoot 2.2.6 application and I would like to set an endpoint with spring-integration therefore I have the follow configuration:
#Bean
public MessageChannel reply() {
return new DirectChannel();
}
#Bean
public IntegrationFlow inbound(TestTransformer testTransformer) {
return IntegrationFlows.from(Http.inboundGateway("/foo")
.requestMapping(m -> m.methods(HttpMethod.GET))
.replyChannel("reply")
.requestPayloadType(String.class))
.channel("httpRequest")
.get();
}
#Bean
#ServiceActivator(inputChannel = "httpRequest", outputChannel = "reply")
public Function<Message<?>, String> handler() {
return new Function<Message<?>, String>() {
public String apply(Message<?> message) throws MessagingException {
log.info("myHandler: " + message.getPayload());
log.info("myHandler: " + message.getHeaders());
return "ok";
}
};
}
Now if I call the enpoint passing params as http://localhost:8080/MyApp/foo?q=test&q1=test2 I receive the parameters in a JSON form.
Is possible to receive something like #PathVariable in the MVC for example writing:
return IntegrationFlows.from(Http.inboundGateway("/foo/{name}")
I have tried it but doesn't work and I canno't find any docs talking about that (at least with java bean configuration)
Thanks
Maybe:
#Bean
public IntegrationFlow test(TestTransformer testTransformer, Jackson2JsonObjectMapper obj) {
return IntegrationFlows.from(Http.inboundGateway("/foo/{name}")
.requestMapping(m -> m.methods(HttpMethod.GET))
.payloadExpression("#pathVariables.name")
.replyChannel("reply")
.requestPayloadType(String.class))
.transform(testTransformer)
.transform(new ObjectToJsonTransformer(obj))
.channel("httpRequest")
.get();
}
is more usefull... The docs is full of xml config and I read it but I'm not able to trasnform them all into java DSL
I have an app hosted multiple hosts listening to single remote SFTP location. How should i make sure same file is not picked by an host which is already picked up by other? I am pretty new to spring integration. Appreciate someone can share examples
}
EDIT:
Here is my integration flow getting file from sftp and placing in local directory and performing business logic in transformer and returning file and send it to remote sftp.
#Bean
public SessionFactory<ChannelSftp.LsEntry> sftpSessionFactory() {
LOGGER.debug(" Creating SFTP Session Factory -Start");
DefaultSftpSessionFactory factory = new DefaultSftpSessionFactory(true);
factory.setHost(sftpHost);
factory.setUser(sftpUser);
factory.setPort(port);
factory.setPassword(sftpPassword);
factory.setAllowUnknownKeys(true);
return new CachingSessionFactory<>(factory);
}
#Bean
public SftpInboundFileSynchronizer sftpInboundFileSynchronizer() {
SftpInboundFileSynchronizer fileSynchronizer = new SftpInboundFileSynchronizer(sftpSessionFactory());
fileSynchronizer.setDeleteRemoteFiles(true);
fileSynchronizer.setRemoteDirectory(sftpInboundDirectory);
fileSynchronizer.setFilter(new SftpPersistentAcceptOnceFileListFilter(store(), "*.json"));
return fileSynchronizer;
}
#Bean(name = PollerMetadata.DEFAULT_POLLER)
public PollerMetadata defaultPoller() {
PollerMetadata pollerMetadata = new PollerMetadata();
pollerMetadata.setTrigger(new PeriodicTrigger(5000));
return pollerMetadata;
}
#Bean
#InboundChannelAdapter(channel = "fileInputChannel", poller = #Poller(fixedDelay = "5000"))
public MessageSource<File> sftpMessageSource() {
SftpInboundFileSynchronizingMessageSource source =
new SftpInboundFileSynchronizingMessageSource(sftpInboundFileSynchronizer());
source.setLocalDirectory(localDirectory);
source.setAutoCreateLocalDirectory(true);
source.setLocalFilter(new AcceptOnceFileListFilter<File>());
source.setMaxFetchSize(1);
return source;
}
#Bean
IntegrationFlow integrationFlow() {
return IntegrationFlows.from(this.sftpMessageSource()).channel(fileInputChannel()).
transform(this::messageTransformer).channel(fileOutputChannel()).
handle(orderOutMessageHandler()).get();
}
#Bean
#ServiceActivator(inputChannel = "fileOutputChannel")
public SftpMessageHandler orderOutMessageHandler() {
SftpMessageHandler handler = new SftpMessageHandler(sftpSessionFactory());
LOGGER.debug(" Creating SFTP MessageHandler - Start ");
handler.setRemoteDirectoryExpression(new LiteralExpression(sftpOutboundDirectory));
handler.setFileNameGenerator(new FileNameGenerator() {
#Override
public String generateFileName(Message<?> message) {
if (message.getPayload() instanceof File) {
return ((File) message.getPayload()).getName();
} else {
throw new IllegalArgumentException("Expected Input is File.");
}
}
});
LOGGER.debug(" Creating SFTP MessageHandler - End ");
return handler;
}
#Bean
#org.springframework.integration.annotation.Transformer(inputChannel = "fileInputChannel", outputChannel = "fileOutputChannel")
public Transformer messageTransformer() {
return message -> {
File file=orderTransformer.transformInboundMessage(message);
return (Message<?>) file;
};
}
#Bean
public ConcurrentMetadataStore store() {
return new SimpleMetadataStore(hazelcastInstance().getMap("idempotentReceiverMetadataStore"));
}
#Bean
public HazelcastInstance hazelcastInstance() {
return Hazelcast.newHazelcastInstance(new Config().setProperty("hazelcast.logging.type", "slf4j"));
See SftpPersistentAcceptOnceFileListFilter to be injected into the SFTP Inbound Channel Adapter. This one has to be supplied with a MetadataStore based on the shared database.
See more info in the docs:
https://docs.spring.io/spring-integration/docs/current/reference/html/system-management.html#metadata-store
https://docs.spring.io/spring-integration/docs/current/reference/html/sftp.html#sftp-inbound
ServiceActivator does not receive messages from ImapIdleChannelAdapter...
JavaMail logs successful FETCH, but MIME messages do not get delivered to SA endpoint... I want to understand what is wrong in my code.
A7 FETCH 1:35 (ENVELOPE INTERNALDATE RFC822.SIZE FLAGS BODYSTRUCTURE)
* 1 FETCH (ENVELOPE ("Fri....
Code snippet below:
`
#Autowired
EmailConfig emailCfg;
#Bean
public SubscribableChannel mailChannel() {
return MessageChannels.direct().get();
}
#Bean
public ImapIdleChannelAdapter getMailAdapter() {
ImapMailReceiver mailReceiver = new ImapMailReceiver(emailCfg.getImapUrl());
mailReceiver.setJavaMailProperties(javaMailProperties());
mailReceiver.setShouldDeleteMessages(false);
mailReceiver.setShouldMarkMessagesAsRead(true);
ImapIdleChannelAdapter imapIdleChannelAdapter = new ImapIdleChannelAdapter(mailReceiver);
imapIdleChannelAdapter.setOutputChannel(mailChannel());
imapIdleChannelAdapter.setAutoStartup(true);
imapIdleChannelAdapter.afterPropertiesSet();
return imapIdleChannelAdapter;
}
#ServiceActivator(inputChannel = "mailChannel")
public void receive(String mail) {
log.warn(mail);
}
private Properties javaMailProperties() {
Properties javaMailProperties = new Properties();
javaMailProperties.setProperty("mail.imap.socketFactory.class", "javax.net.ssl.SSLSocketFactory");
javaMailProperties.setProperty("mail.imap.socketFactory.fallback", "false");
javaMailProperties.setProperty("mail.store.protocol", "imaps");
javaMailProperties.setProperty("mail.debug", "true");
javaMailProperties.setProperty("mail.imap.ssl", "true");
return javaMailProperties;
}
`
The problem was due to wrong bean initialization. Full version that works OK:
#Slf4j
#Configuration
#EnableIntegration
public class MyMailAdapter {
#Autowired
EmailConfig emailCfg;
#Bean
public SubscribableChannel mailChannel() {
log.info("Channel ready");
return MessageChannels.direct().get();
}
#Bean
public ImapMailReceiver receiver() {
ImapMailReceiver mailReceiver = new ImapMailReceiver(emailCfg.getImapUrl());
mailReceiver.setJavaMailProperties(javaMailProperties());
mailReceiver.setShouldDeleteMessages(false);
mailReceiver.setShouldMarkMessagesAsRead(true);
return mailReceiver;
}
#Bean
public ImapIdleChannelAdapter adapter() {
ImapIdleChannelAdapter imapIdleChannelAdapter = new ImapIdleChannelAdapter(receiver());
imapIdleChannelAdapter.setOutputChannel(mailChannel());
imapIdleChannelAdapter.afterPropertiesSet();
return imapIdleChannelAdapter;
}
#ServiceActivator(inputChannel = "mailChannel")
public void receive(Message<MimeMessage> mail) throws MessagingException {
log.info(mail.getPayload().toString());
}
private Properties javaMailProperties() {
Properties javaMailProperties = new Properties();
javaMailProperties.setProperty("mail.imap.socketFactory.class", "javax.net.ssl.SSLSocketFactory");
javaMailProperties.setProperty("mail.imap.socketFactory.fallback", "false");
javaMailProperties.setProperty("mail.store.protocol", "imaps");
javaMailProperties.setProperty("mail.debug", "true");
javaMailProperties.setProperty("mail.imap.ssl", "true");
return javaMailProperties;
}
}
I don't know what's exactly wrong with your code but I will suggest you few approches that could help you.
Firstly I suggest you to use java DSL in java based configuration. It will provide you nice way to directly specific flow of your integration application (and avoid simply mistakes). For example for spliiter and service activator:
#Bean
public IntegrationFlow yourFlow(AbstractMessageSplitter splitter,
MessageHandler handler) {
return
IntegrationFlows
.from(CHANNEL)
.split(splitter)
.handle(handler).get();
}
Secondly it's generally bad idea to directly specify message type to String. Try something like this (why String?):
#ServiceActivator(inputChannel = "mailChannel")
public void receive(Message<?> message) {
/* (String) message.getPayload() */
}
Maybe it's not a case but let's check it.
I have implemented the following scenario:
A queueChannel holding Messages in form of byte[]
A MessageHandler, polling the queue channel and uploading files over sftp
A Transformer, listening to errorChannel and sending extracted payload from the failed message back to the queueChannel (thought of as an error handler to handle failed messages so nothing gets lost)
If the sftp server is online, everything works as expected.
If the sftp server is down, the errormessage, that arrives as the transformer is:
org.springframework.messaging.MessagingException: Failed to obtain pooled item; nested exception is java.lang.IllegalStateException: failed to create SFTP Session
The transformer cannot do anything with this, since the payload's failedMessage is null and throws an exception itself. The transformer looses the message.
How can I configure my flow to make the tranformer get the right message with the corresponding payload of the unsucsesfully uploaded file?
My Configuration:
#Bean
public MessageChannel toSftpChannel() {
final QueueChannel channel = new QueueChannel();
channel.setLoggingEnabled(true);
return new QueueChannel();
}
#Bean
public MessageChannel toSplitter() {
return new PublishSubscribeChannel();
}
#Bean
#ServiceActivator(inputChannel = "toSftpChannel", poller = #Poller(fixedDelay = "10000", maxMessagesPerPoll = "1"))
public MessageHandler handler() {
final SftpMessageHandler handler = new SftpMessageHandler(sftpSessionFactory());
handler.setRemoteDirectoryExpression(new LiteralExpression(sftpRemoteDirectory));
handler.setFileNameGenerator(message -> {
if (message.getPayload() instanceof byte[]) {
return (String) message.getHeaders().get("name");
} else {
throw new IllegalArgumentException("byte[] expected in Payload!");
}
});
return handler;
}
#Bean
public SessionFactory<LsEntry> sftpSessionFactory() {
final DefaultSftpSessionFactory factory = new DefaultSftpSessionFactory(true);
final Properties jschProps = new Properties();
jschProps.put("StrictHostKeyChecking", "no");
jschProps.put("PreferredAuthentications", "publickey,password");
factory.setSessionConfig(jschProps);
factory.setHost(sftpHost);
factory.setPort(sftpPort);
factory.setUser(sftpUser);
if (sftpPrivateKey != null) {
factory.setPrivateKey(sftpPrivateKey);
factory.setPrivateKeyPassphrase(sftpPrivateKeyPassphrase);
} else {
factory.setPassword(sftpPasword);
}
factory.setAllowUnknownKeys(true);
return new CachingSessionFactory<>(factory);
}
#Bean
#Splitter(inputChannel = "toSplitter")
public DmsDocumentMessageSplitter splitter() {
final DmsDocumentMessageSplitter splitter = new DmsDocumentMessageSplitter();
splitter.setOutputChannelName("toSftpChannel");
return splitter;
}
#Transformer(inputChannel = "errorChannel", outputChannel = "toSftpChannel")
public Message<?> errorChannelHandler(ErrorMessage errorMessage) throws RuntimeException {
Message<?> failedMessage = ((MessagingException) errorMessage.getPayload())
.getFailedMessage();
return MessageBuilder.withPayload(failedMessage)
.copyHeadersIfAbsent(failedMessage.getHeaders())
.build();
}
#MessagingGateway
public interface UploadGateway {
#Gateway(requestChannel = "toSplitter")
void upload(#Payload List<byte[]> payload, #Header("header") DmsDocumentUploadRequestHeader header);
}
Thanks..
Update
#Bean(PollerMetadata.DEFAULT_POLLER)
#Transactional(propagation = Propagation.REQUIRED, isolation = Isolation.READ_COMMITTED)
PollerMetadata poller() {
return Pollers
.fixedRate(5000)
.maxMessagesPerPoll(1)
.receiveTimeout(500)
.taskExecutor(taskExecutor())
.transactionSynchronizationFactory(transactionSynchronizationFactory())
.get();
}
#Bean
#ServiceActivator(inputChannel = "toMessageStore", poller = #Poller(PollerMetadata.DEFAULT_POLLER))
public BridgeHandler bridge() {
BridgeHandler bridgeHandler = new BridgeHandler();
bridgeHandler.setOutputChannelName("toSftpChannel");
return bridgeHandler;
}
The null failedMessage is a bug; reproduced INT-4421.
I would not recommend using a QueueChannel for this scenario. If you use a direct channel, you can configure a retry advice to attempt redeliveries. when the retries are exhausted (if so configured), the exception will be thrown back to the calling thread.
Add the advice to the SftpMessageHandler's adviceChain property.
EDIT
You can work around the "missing" failedMessage by inserting a bridge between the pollable channel and the sftp adapter:
#Bean
#ServiceActivator(inputChannel = "toSftpChannel", poller = #Poller(fixedDelay = "5000", maxMessagesPerPoll = "1"))
public BridgeHandler bridge() {
BridgeHandler bridgeHandler = new BridgeHandler();
bridgeHandler.setOutputChannelName("toRealSftpChannel");
return bridgeHandler;
}
#Bean
#ServiceActivator(inputChannel = "toRealSftpChannel")
public MessageHandler handler() {
final SftpMessageHandler handler = new SftpMessageHandler(sftpSessionFactory());
handler.setRemoteDirectoryExpression(new LiteralExpression("foo"));
handler.setFileNameGenerator(message -> {
if (message.getPayload() instanceof byte[]) {
return (String) message.getHeaders().get("name");
}
else {
throw new IllegalArgumentException("byte[] expected in Payload!");
}
});
return handler;
}
I'm trying to use the outbound gateway to download files from sftp server,
my config:
#IntegrationComponentScan
#EnableIntegration
#Configuration
public class FtpConfig {
#Bean(name = "myGateway")
#ServiceActivator(inputChannel = "sftpChannel")
public MessageHandler handlerLs() {
SftpOutboundGateway sftpOutboundGateway = new SftpOutboundGateway(sftpSessionFactory(), "mget", "payload");
sftpOutboundGateway.setLocalDirectory(new File("/Users/xxx/Documents/"));
return sftpOutboundGateway;
}
#MessagingGateway
public interface OutboundGatewayOption {
#Gateway(requestChannel = "sftpChannel")
List<File> mget(String dir);
}
#Bean
public MessageChannel sftpChannel() {
return new DirectChannel();
}
}
and the execute bean:
#Service
public class DownloadService implements InitializingBean{
#Autowired
FtpConfig.OutboundGatewayOption gatewayOption;
#Override
public void afterPropertiesSet() throws Exception {
List<File> files = gatewayOption.mget("/sftp/server/path");
}
}
and I got this exception:org.springframework.messaging.MessageDeliveryException: Dispatcher has no subscribers for channel 'application.sftpChannel'.;
Qestion :how can I add the 'subscribers' ?
You can't perform messaging operator in the afterPropertiesSet(). That's too early : some beans might not be initialized yet. And that exception confirms the problem.
You have to implement SmartLifecicle instead and do the same in the start().