I am attempting to write a simple test of renaming files in-place using a FileWritingMessageHandler, however I can't seem to figure out how to properly specify the target destination directory.
Since I am recursively scanning a directory tree I would ultimately like to simply read the parent path from the file payload and rename it using the FileNameGenerator, but that doesn't appear to work.
The 'payload.name' in the DefaultFileNameGenerator resolves properly, but 'payload.path' does not.
How do I properly determine the source file's location and use that in the handler?
Edit
Here is the channel adapter that scans for files. I had to use .setUseWatchService(true) to achieve recursive scanning.
#Bean
#InboundChannelAdapter(channel = "sourceFileChannel", poller = #Poller(fixedRate = "5000", maxMessagesPerPoll = "-1"))
public MessageSource<File> sourceFiles() {
CompositeFileListFilter<File> filters = new CompositeFileListFilter<>();
filters.addFilter(new SimplePatternFileListFilter(sourceFilenamePattern));
filters.addFilter(persistentFilter());
FileReadingMessageSource source = new FileReadingMessageSource();
source.setAutoCreateDirectory(true);
source.setDirectory(new File(sourceDirectory));
source.setFilter(filters);
source.setUseWatchService(true);
return source;
}
UPDATE
Artem helped me understand my mistake.
I was able to achieve the desired result by using the SpelExpressionParser as Artem outlined.
The key piece being:
new SpelExpressionParser().parseExpression("payload.parent")
Where "payload.parent" resolves to the file parent path properly.
#Bean
#ServiceActivator(inputChannel = "processingFileChannel")
public MessageHandler copyFileForProcessingOutboundChannelAdapter() {
FileWritingMessageHandler adapter = new FileWritingMessageHandler(new SpelExpressionParser().parseExpression("payload.parent"));
adapter.setDeleteSourceFiles(false);
adapter.setAutoCreateDirectory(true);
adapter.setExpectReply(false);
adapter.setFileNameGenerator(processingFileNameGenerator());
return adapter;
}
#Bean
public DefaultFileNameGenerator processingFileNameGenerator() {
DefaultFileNameGenerator defaultFileNameGenerator = new DefaultFileNameGenerator();
defaultFileNameGenerator.setExpression("'p_' + payload.name");
return defaultFileNameGenerator;
}
The 'payload.name' in the DefaultFileNameGenerator resolves properly, but 'payload.path' does not.
Well, I'm not sure what should be in your case, but for me that always returns the full, absolute path for source file, including the root directory to scan.
Since you have it there as a sourceDirectory, how about to use it in your processingFileNameGenerator to sever the root dir from the target path to use? For example if my root dir is /root and I get a file from the subdir /root/foo/my_file.txt, I could do payload.path.replaceFirst('/root', ''). So in the end I have just /foo/my_file.txt.
At least that is what I'm going to do in that JIRA to populate FileHeaders.FILENAME with the relative path from the provided directory to scan.
UPDATE
Oh! I see. No, that isn't going to work that way. See FileWritingMessageHandler ctors. The String once accepts static target directory. Root for our case. The code like new LiteralExpression("payload.path") isn't going to work a desired way, too. See LiteralExpression JavaDocs. It is just an Expression variant to always return the same static value. In your case it is payload.path.
If you are really going to evaluate against an incoming Message, you should use new SpeLexpressionParser().parseExpression("payload.path"). But as I said before, it returns the absolute path for any file in the sub-directories.
Related
We have a remote FTP server in which we have a folder "test/" which contains certain text files.
The "test/" folder has another subdirectory "archive/" inside it.
FTPserver->
-test/
---abc.txt
---xyz.txt
---archive/
We are able to download all the text files via Spring Integration flows in our local directory.
Now we are looking into ways to move the remote text files inside the folder "archive" within the FTP Server itself once it has been downloaded into the local.
We are trying to do it in the handle() method like this ->
#Bean
public IntegrationFlow integrationFlow() {
File localDirectory = new File("tmp/");
FtpInboundChannelAdapterSpec ftpInboundChannelAdapterSpec = Ftp.inboundAdapter(gimmeFactory())
.remoteDirectory("test/")
.autoCreateLocalDirectory(true)
.regexFilter(".*\\.txt$")
.localDirectory(localDirectory)
.preserveTimestamp(true)
.remoteFileSeparator("/");
return IntegrationFlows.from(ftpInboundChannelAdapterSpec, pc -> pc.poller(pm -> pm.fixedRate(1000, TimeUnit.MILLISECONDS)))
.handle((file, messageHeaders) -> {
messageHeaders.forEach((k, v) -> System.out.println(k + ':' + v));
return null;
})
.handle(Ftp.outboundGateway(gimmeFactory(), AbstractRemoteFileOutboundGateway.Command.MV, "'test/archive'"))
.get();
}
But it is not moving into the remote "archive" folder location.
We are quite not sure how to handle this operation in any other way.
Is there anything we can do to fix the above code snippet or do something differently in order to achieve what we want ?
Please advise.
Update
Thank you Gary for the pointers.
I was able to solve the problem by doing as given in below code snippet->
#Bean
public IntegrationFlow integrationFlow() {
File localDirectory = new File("tmp/");
FtpInboundChannelAdapterSpec ftpInboundChannelAdapterSpec = Ftp.inboundAdapter(gimmeFactory())
.remoteDirectory("test/")
.autoCreateLocalDirectory(true)
.regexFilter(".*\\.txt$")
.localDirectory(localDirectory)
.preserveTimestamp(true)
.remoteFileSeparator("/");
return IntegrationFlows
.from(ftpInboundChannelAdapterSpec, e -> e.poller(Pollers.fixedDelay(Duration.ofSeconds(5))))
.handle(Ftp.outboundGateway(gimmeFactory(), AbstractRemoteFileOutboundGateway.Command.LS, "'test/'")
.options(AbstractRemoteFileOutboundGateway.Option.NAME_ONLY))
.split()
.handle(Ftp.outboundGateway(gimmeFactory(), AbstractRemoteFileOutboundGateway.Command.MV, "'test/' +payload").renameExpression("'test/archive/' +payload"))
.channel("nullChannel")
.get();
}
Since you are returning null from the first .handle(), the flow stops there; you need to return a payload containing the from path and the rename-expression is needed to specify the to path.
https://docs.spring.io/spring-integration/docs/current/reference/html/ftp.html#using-the-mv-command
The mv command moves files.
The mv command has no options.
The expression attribute defines the “from” path and the rename-expression attribute defines the “to” path. By default, the rename-expression is headers['file_renameTo']. This expression must not evaluate to null or an empty String. If necessary, any necessary remote directories are created. The payload of the result message is Boolean.TRUE. The file_remoteDirectory header provides the original remote directory, and file_remoteFile header provides the file name. The new path is in the file_renameTo header.
Starting with version 5.5.6, the remoteDirectoryExpression can be used in the mv command for convenience. If the “from” file is not a full file path, the result of remoteDirectoryExpression is used as the remote directory. The same applies for the “to” file, for example, if the task is just to rename a remote file in some directory.
What I want: I'd like to move content from one ContentStore (regular) to another ContentStore (e.g. an archive) with Spring-Content version 1.2.7.
What I did is this (and it does work at least with DefaultFilesystemStoreImpls):
Creating two ContentStores like this:
#Bean(name = "mytmpfsstore1")
public ContentStore<File, String> getFileContentStore1() {
FileSystemResourceLoader loader = new FileSystemResourceLoader(".\\tmpstore1");
PlacementService placementService = new PlacementServiceImpl();
return new DefaultFilesystemStoreImpl<File, String>(loader, placementService, new FileServiceImpl());
}
Moving content from one ContentStore to another like this:
Optional<File> fileEntity = filesRepo.findById(id);
if (fileEntity.isPresent()) {
Resource resource = regularContentStore.getResource(fileEntity.get());
archiveContentStore.setContent(fileEntity.get(), resource);
filesRepo.save(fileEntity.get());
if (resource instanceof DeletableResource) {
((DeletableResource) resource).delete();
}
}
Question: Is this the intended way of moving (/archiving) content with Spring-Content or is there a more elegant / more convenient / more intended way of moving/archiving files (especially from filesystem to S3 and back again)?
Spring Content doesn't provide any magic here. I try to keep the API fairly low-level and this use case, whilst valid, is a little too granular.
So, ultimately you have to build the "archiving" yourself and at the end of the day copy the content from one store to another as you are doing.
A couple of comments/pointers:
Not sure why you instantiated mytmpfsstore1 yourself. If you have file storage and s3 storage you can just make your storage interfaces extend FileSystemContentStore and S3ContentStore respectively and let the framework instantiate them for you; i.e.
public interface MyTmpFsStore extends FileSystemContentStore {}
public interface MyS3store extends S3ContentStore()
You can then wire these two beans into your 'archive' code. Assuming that is a controller, something like:
#RequestMapping(...) public void archive(MyTmpFsStore fsStore, S3ContentStore s3Store { ... }
where you do your copy operation.
ContentStore extends AssociativeStore and there are a few different APIs for setting/unsetting content. Resource-based or InputStream. They all achieve the same objective at the end of the day. You could just have easily used getContent instead of getResource and unsetContent instead of delete.
The 'archive' code probably needs to be in a #Transactional so the archive operation is atomic.
I am very new to Spring Integration and my project is using File Support to read a file and load into data base.
I have XML config , trying to understand it's content.
<int-file:inbound-channel-adapter auto-startup= true channel="channelOne" directory="${xx}" filename-regex="${xx}" id="id" prevent-duplicates="false">
<int:poller fixed-delay="1000" receive-timeout="5000"/>
</int-file:inbound-channel-adapter>
<int:channel id="channelOne"/>
From the above piece, my understanding is :
We define a channel and
Then define inbound-channel-adapter - this will look into directory for the file and create a message with file as a payload.
I was able to convert this in JavaConfig as below :
#Bean
public MessageChannel fileInputChannel() {
return new DirectChannel();
}
#Bean
#InboundChannelAdapter(value = "fileInputChannel", poller = #Poller(fixedDelay = "1000"))
public MessageSource<File> fileReadingMessageSource() {
FileReadingMessageSource sourceReader= new FileReadingMessageSource();
RegexPatternFileListFilter regexPatternFileListFilter = new RegexPatternFileListFilter(
file-regex);
//List<FileListFilter<File>> fileListFilter = new ArrayList<FileListFilter<File>>();
fileListFilter.add(regexPatternFileListFilter);
//CompositeFileListFilter compositeFileListFilter = new CompositeFileListFilter<File>(
fileListFilter);
sourceReader.setDirectory(new File(inputDirectorywhereFileComes));
sourceReader.setFilter(regexPatternFileListFilter );
return sourceReader;
}
Then the next piece of code , which literally I am struggling to understand and moreover to convert to JavaConfig.
Here is the next piece:
<int-file:outbound-gateway
delete-source-files="true"
directory="file:${pp}"
id="id"
reply-channel="channelTwo"
request-channel="channelOne"
temporary-file-suffix=".tmp"/>
<int:channel id="channelTwo"/>
<int:outbound-channel-adapter channel="channelTwo" id="id" method="load" ref="beanClass"/>
So from this piece , my understanding :
1: Define an output channel.
2: Define an outbound-gateway, which will write that message as a file again in directory(other one), also remove file from source directory. And finally it will call the method Load of Bean Class. This is our class and has load method which takes file as input and load it to DB.
I tried to covert it into Java Config. Here is my code:
#Bean
#ServiceActivator(inputChannel= "fileInputChannel")
public MessageHandler fileWritingMessageHandler() throws IOException, ParseException {
FileWritingMessageHandler handler = new FileWritingMessageHandler(new File(path to output directory));
handler.setFileExistsMode(FileExistsMode.REPLACE);
beaObject.load(new File(path to output directory or input directory:: Nothing Worked));
handler.setDeleteSourceFiles(true);
handler.setOutputChannel(fileOutputChannel());
return handler;
}
I am able to write this file to output folder also was able to delete from source. After that I am totally lost. I have to call method Load of my BeanClass(ref=class in XML ).
I tried a lot, but not able to get it. Read multiple times the integration File Support doc, but couldn't make it.
Note: When I tried , I got one error saying , the File Not Found Exception. I believe , I am able to call my method , but can not get the file.
This XML config is working perfectly fine.
Spring Integration with DSL also anyone can suggest, if possible.
Please help me to understand the basic flow and get this thing done. Any help and comments is really appreciable.
Thanks in advance.
First of all you need to understand that #Bean method is exactly for configuration and components definitions which are going to be used later at runtime. You definitely must not call a business logic in the #Bean. I mean that your beaObject.load() is totally wrong.
So, please, go first to Spring Framework Docs to understand what is #Bean and its parent #Configuration: https://docs.spring.io/spring/docs/5.1.2.RELEASE/spring-framework-reference/core.html#beans-java
Your #ServiceActivator for the FileWritingMessageHandler is really correct (when you remove that beaObject.load()). What you just need is to declare one more #ServiceActivator for calling your beaObject.load() at runtime when message appears in the fileOutputChannel:
#ServiceActivator(inputChannel= "fileOutputChannel")
public void loadFileIntoDb(File payload) {
this.beaObject.load(payload);
}
See https://docs.spring.io/spring-integration/docs/5.1.1.BUILD-SNAPSHOT/reference/html/configuration.html#annotations for more info.
I am Spring Integration 4.3.13 and trying to pass patterns when configuring #GlobalChannelInterceptor
Here is the example
#Configuration
public class IntegrationConfig{
#Bean
#GlobalChannelInterceptor(patterns = "${spring.channel.interceptor.patterns:*}")
public ChannelInterceptor channelInterceptor(){
return new ChannelInterceptorImpl();
}
}
properties file has following values:
spring.channel.interceptor.patterns=*intchannel, *event
I am using direct channels with names that end with these two string
springintchannel
registrationevent
With the above config, both the channels should have interceptor configured but it is not getting configured.
The comma-separate value isn't support there currently.
I agree that we need to fix it, so feel free to raise a JIRA on the matter and we will file a solution from some other place.
Meanwhile you can do this as a workaround:
#Bean
public GlobalChannelInterceptorWrapper channelInterceptorWrapper(#Value("${spring.channel.interceptor.patterns:*}") String[] patterns) {
GlobalChannelInterceptorWrapper globalChannelInterceptorWrapper = new GlobalChannelInterceptorWrapper(channelInterceptor());
globalChannelInterceptorWrapper.setPatterns(patterns);
return globalChannelInterceptorWrapper;
}
I would like to implement a method that can get the svn revision number from the path where a SVN repository has been checked out. The method declaration would look something like this:
long getRevisionNumber(String localPath) { ... }
I'm trying to use SVNKit for this, but it seems to require an SVN URL to start with. Is there any way to start with a local path?
public static long getRevisionNumber(String localPath) throws SVNException {
final SVNStatus status = SVNClientManager.newInstance().getStatusClient().doStatus(new File(localPath), false);
return status != null ? status.getRevision().getNumber() : -1;
}
Also you can use getCommittedRevision() method. It returns revision for a particular file when it was last commited.
clientManager.getStatusClient().doStatus(destination, false).getCommittedRevision().getNumber();