I want to get all file under a particular remote directory in a periodic manner. I am able to get the file under that directory only once after application startup. Not sure why the Poller is not working. This is registered in a spring boot project and the version is 2.2.1
#InboundChannelAdapter(value = "sftpReportChannel",
poller = #Poller(fixedDelay = "5000"))
public String filesForGET(){
return "/etl/biq/autoscore/output/report-data/";
}
#Bean
public IntegrationFlow sftpGetFlow(SessionFactory<ChannelSftp.LsEntry> csf) {
return IntegrationFlows.from("sftpReportChannel")
.handle(Sftp.outboundGateway(csf,
AbstractRemoteFileOutboundGateway.Command.LS, "payload")
.options(AbstractRemoteFileOutboundGateway.Option.RECURSIVE, AbstractRemoteFileOutboundGateway.Option.NAME_ONLY)
//Persistent file list filter using the server's file timestamp to detect if we've already 'seen' this file.
.filter(new SftpPersistentAcceptOnceFileListFilter(new SimpleMetadataStore(), "autoscore-meta-data")))
.split()
.log(message -> "file path -> "+message.getPayload())
.handle(Sftp.outboundGateway(csf, AbstractRemoteFileOutboundGateway.Command.GET, "'/etl/biq/autoscore/output/report-data/' + payload")
.options(AbstractRemoteFileOutboundGateway.Option.STREAM))
.handle(new ReportHandler()) //get the payload and create email content and send eamil to recipients
.get();
}
The .filter(new SftpPersistentAcceptOnceFileListFilter(new SimpleMetadataStore(), "autoscore-meta-data"))) makes it working the way that it doesn't pick up the same file again and again on the subsequent poll activities.
Make sure you add new files in that remote dir at runtime or modify already processed file. The SftpPersistentAcceptOnceFileListFilter logic relies on the mtime property of the LsEntry to determine that the file has been changed therefore it is good for processing again.
Related
How/where can I compute md5 digest for a file I need to transfer to a samba location in spring-integration in order to validate it against the digest I receive at the beginning of the flow. I get the file from a rest service and I have to make sure file is safely landing to samba location. The middle flow looks like this: (the digest to be compared against is stored somewhere in the messages)
GenericHandler smbUploader;
HttpRequestExecutingMessageHandler httpDownloader;
from(inbound()) //here I receive a notification with url where to download file + a checksum to be validated against
...
.handle(httpDownloader) //here I get file effectively
.handle(smbUploader) //here I upload the file to samba
...
and httpDownloader is defined like this:
public HttpRequestExecutingMessageHandler httpDownloader(){
HttpRequestExecutingMessageHandler h = new HttpRequestExecutingMessageHandler ("payload.url");
h.setExpectedResponseType(String.class);
h.setHttpMethod(GET);
return h;
}
and smbUploader is defined like this:
public GenericHandler smbUploader (MessageHandler smbMessageHandler){
return new GenericHandler<Message>(){
#Override
public Message handle(Message m, MessageHeaders h){
smbMessageHandler.handleMessage(m);
return m;
}
}
and smbMessageHandler is defined like this:
public MessageHandler smbMessageHandler (SmbRemoteFileTemplate template, FileNameGenerator g){
SmbMessageHandler h = new smbMessageHandler (template, REPLACE);
h.setAutoCreateDirectory(true);
h.setRemoteDirectoryExpression(getExpression("headers['msg'].smbFolder"));
h.setFileNameGenerator(g);
return h;
}
the inbound (starting the flow) is defined like this:
public HttpRequestHandlerEndpointSpec inbound(){
return Http.inboundChannelAdapter ("/notification")
.requestMapping(m->m.methods(POST))
.requestPayloadType(String.class)
.validator(notificationValidator);
}
First of all you should store a digest in the message headers in the beginning of the flow.
Then you need to write a service method to calculate a checksum of the file you got downloaded. And insert a new handle() in between:
.handle(httpDownloader) //here I get file effectively
.handle(smbUploader) //here I upload the file to samba
to call your service method. The input for that method must be a whole Message, so you got access to the downloaded file in the payload and digest in the headers. The result of this method could be just your file to proceed into an SMB handler for uploading.
How to calculate a checksum you can find in this SO thread: Getting a File's MD5 Checksum in Java
I've been trying to use SftpInboundFileSynchronizer with a remote directory that contains a subdir, say /myfiles/mysubdir/lefile.txt, I have set a filter to grab the files inside the dirs:
mysync.setRemoteDirectory("myfiles/");
mysync.setFilter(new SftpRegexPatternFileListFilter(".*\\.txt$"));
And then a SftpInboundFileSynchronizingMessageSource as my InboundChannelAdapter
I have set on the SftpInboundFileSynchronizingMessageSource a RecursiveDirectoryScanner as scanner and i have no set limit to the depth or the amount of files to retrieve. I also set a FOLLOW_LINKS fileVisitOption on the scanner for good measure.
I am only able to pull files into the local directory from the myfiles path, but anything deeper is not copied to the local dir.
I can't for the life of me figure out if there is something I'm not doing.
EDIT:
What would the InboundChannelAdapter contain if I'm only going to send "/" as the directory to check with mget -R?
#Bean
#InboundChannelAdapter(value = "sftpChannel", poller = #Poller(fixedDelay = "10"))
public MessageSource<?> myMessageSource() {
}
#Bean(name = "myGateway")
#ServiceActivator(inputChannel = "sftpChannel")
public MessageHandler handler() {
SftpOutboundGateway gateway =
new SftpOutboundGateway(sftpSessionFactory(), "mget", "'myfiles/*'");
gateway.setOutputChannelName("listSplitter");
gateway.setOptions("-R");
gateway.setAutoCreateLocalDirectory(true);
myLocalPath = Paths.get(myLocalParentDir).toRealPath().toString();
gateway.setLocalDirectory(new File(myLocalPath));
SftpRegexPatternFileListFilter regexFilter = new regexFilter("^.*\\.txt");
regexFilter.setAlwaysAcceptDirectories(true);
regexFilter.setFilter(sftpRegexPatternFileListFilter);
return gateway;
}
Recursion of the remote file system is not supported by the inbound synchronizer; use an SftpOutboundGateway (request/reply) instead, with a recursive mget command.
By default, files existing in the local directory are not re-fetched; you can control that with the FileExistsMode.
<int-sftp:outbound-gateway id="sftpOutBound"
session-factory="sftpSessionFactory" expression="payload" command="put" request-channel="outboundFtpChannel"
remote-directory="/tmp/tsiftp" reply-channel="sftpReplyChannel"/>
with the above xml, i can send files and get reply . In java, how to set the remote directory in SftpOutboundGateway .If I use SftpMessageHandler,is there any possibility to get reply.Commented code is transferring files but no reply.
#Bean
#ServiceActivator(inputChannel = "outboundFtpChannel")
public MessageHandler transfertoPeopleSoft(){
/* SftpMessageHandler handler = new SftpMessageHandler(sftpSessionFactory());
handler.setRemoteDirectoryExpression(new LiteralExpression("/tmp/tsiftp"));
return handler;*/
SftpOutboundGateway sftpOutboundGateway = new SftpOutboundGateway( sftpSessionFactory(), "put", "/tmp/tsiftp");
sftpOutboundGateway.setOutputChannelName("sftpReplyChannel");
return sftpOutboundGateway;
}
Exception I am getting is
exception is org.springframework.expression.spel.SpelParseException: Expression [/tmp/tsiftp] #0: EL1070E: Problem parsing left operand
Thanks for your help.
The remote directory for the SftpOutboundGateway can be configured by the SftpRemoteFileTemplate and its:
/**
* Set the remote directory expression used to determine the remote directory to which
* files will be sent.
* #param remoteDirectoryExpression the remote directory expression.
*/
public void setRemoteDirectoryExpression(Expression remoteDirectoryExpression) {
https://docs.spring.io/spring-integration/docs/5.0.0.RELEASE/reference/html/sftp.html#sftp-rft
Feel free to raise a JIRA for improvements on the matter.
I know that Expression variant isn't so useful because you need to use SpelExpressionParser or just LiteralExpression just for simple dir variant.
I've managed to send all local files to the target ftp server folder with following config:
#Bean
#ServiceActivator(inputChannel = FtpDef.FTP_OUTBOUND_CHANNEL)
public MessageHandler handler() {
FtpMessageHandler handler = new FtpMessageHandler(ftpSessionFactory());
handler.setRemoteDirectoryExpression(
// only one path can be set here
new LiteralExpression("/path/on/ftp/"));
return handler;
}
now I need each file saved in a directory structure as the local.
e.g.
/base/a/a.txt => /path/on/ftp/a/a.txt
/base/a/aa.txt => /path/on/ftp/a/aa.txt
/base/b/b.txt => /path/on/ftp/b/b.txt
/base/b/bb.txt => /path/on/ftp/b/bb.txt
how can I accomplish that, I
new LiteralExpression("/path/on/ftp/")
Don't use a LiteralExpression, which is, er... literal.
Instead, use:
new SpelExpressionParser().parseExpression(rdExpression)
Where rdExpression is something like...
"'/path/on/ftp/' + payload.absolutePath"
I have two IntegrationFlows defined that both uses this component. One reads from ftp and one read files from disk.
#Bean
public IntegrationFlow csvLineFlowDefinition() {
return IntegrationFlows.from(CHANNEL_NAME)
.filter(String.class, m -> {
// filter to remove column definition csv line
return !m.startsWith("ID");
})
.<String, MyPrettyObject>transform(csvLinePayload -> {
String[] array = csvLinePayload.split(",");
MyPrettyObject myPrettyObject = new MyPrettyObject();
myPrettyObject.setId(array[0]);
myPrettyObject.setType(array[1]);
return myPrettyObject;
})
.<MyPrettyObject, String>route(myPrettyObject -> myPrettyObject.getType(),
routeResult -> routeResult
.channelMapping("AA", "AA_CHANNEL")
.channelMapping("BB", "BB_CHANNEL")
.channelMapping("CC", "CC_CHANNEL"))
.get();
}
I would like these two IntegrationFlows only to fail if something is wrong with reading from ftp or reading files from the disk.
They have their own error channel defined.
I do not want an error in the transform of a csv line to MyPrettyObject to reach these two IntegrationFlows.
I have thought about dispatching the raw csv lines to a message queue and then i can define a specific error channel on the inbound consumer of this message queue.
However this seems a bit overkill.
I have tried to insert a ExpressionEvaluatingRequestHandlerAdvice for the transformer, but i'm not sure how to use it properly, and the messages does not reach the router or ERROR_CHANNEL_NAME
#Bean
public ExpressionEvaluatingRequestHandlerAdvice csvLineTransformerAdvice() {
ExpressionEvaluatingRequestHandlerAdvice expressionEvaluatingRequestHandlerAdvice = new ExpressionEvaluatingRequestHandlerAdvice();
expressionEvaluatingRequestHandlerAdvice.setFailureChannelName(ERROR_CHANNEL_NAME);
expressionEvaluatingRequestHandlerAdvice.setTrapException(true);
return expressionEvaluatingRequestHandlerAdvice;
}
.<String, MyPrettyObject>transform(csvLinePayload -> {
String[] array = csvLinePayload.split(",");
MyPrettyObject myPrettyObject = new MyPrettyObject();
myPrettyObject.setId(array[0]);
myPrettyObject.setType(array[1]);
return myPrettyObject;
}, t -> t.advice(csvLineTransformerAdvice()))
I'm afraid that "something wrong with reading" doesn't reach the error-channel because there is no message yet to deal with. So, isolating inbound channel adapter from the rest of the flow might not be a good idea. That is pretty normal for any downstream error to be propagated to the error-channel on the inbound channel adapter.
The ExpressionEvaluatingRequestHandlerAdvice is right way to go, but you should keep in mind that it works only for the transformer. The downstream flow isn't involved in that advice already.
In case of error the flow stops and it really can't reach the next endpoint because of error. Not sure what is your concerns there...