Zipping files with apache camel - zip

i am saving files on path C:\Cdr\core\year\month\date\fileName
and i want to zip all the files created on previous day and before that.I am unable to find a way to provide dynamic file path and file name to file Consumer for zipping files. Even when using filename=${beans:utility.generateFileName}, i am only able to provide file name not the file path.Is there a way to do this using apache Zip file data format.

I found the way for same by using filter
<routeContext id="zipFileRoute" xmlns="http://camel.apache.org/schema/spring">
<route id="zipFile">
<from uri="file://C:/CdrJson?recursive=true&delete=true&filter=#myFilter/>
<log message="reading from ${in.header.CamelFileName} and file path is ${file:path}"/>
<setHeader headerName="CamelFileName">
<simple>${bean:utility?method=processFileName}</simple>
</setHeader>
<marshal>
<zipFile/>
</marshal>
<to uri="file://C:/CdrJson"/>
<log message="This route finished zipping files"/>
</route>
</routeContext>
Code for myFilter:
public class MyFileFilter<T> implements GenericFileFilter<T> {
public boolean accept(GenericFile<T> file) {
// we want all directories
if (file.isDirectory()) {
return true;
}
Calendar date = new GregorianCalendar();
String fileName = new SimpleDateFormat("yyyy-MM-dd").format(new Date());
if(file.getFileNameOnly().startsWith(fileName)){
return false;
}
return !file.getFileName().endsWith(".zip");
}
}

Use GenericFileFilter to filter the file based on created date. Using this at run time you can skip the files not matched with desired criteria.
Ref: http://camel.apache.org/file2
Dynamic path: I am not sure about what level of dynamism you are expecting. File consumer is a polling consumer, one that explicitly makes a call when it wants to receive a message. So you can either hard-code the file path or set dynamic path resolved at run time from property file (Ref:Proertyplaceholder) as below,
<from uri="file:{{consumer.file.path}}"/>
To Zip files: You have to use aggregate pattern as below,
from("file:{{consumer.file.path}}")
.aggregate(new ZipAggregationStrategy())
.constant(true)
.completionFromBatchConsumer()
.eagerCheckCompletion()
.setHeader(Exchange.FILE_NAME, constant("<DesiredFileName>.zip"))
.to("file:output/directory");
Hope it helps!!

Related

Spring Integration Ftp | how to move remote files into another directory in remote server after FTP fetch complete

We have a remote FTP server in which we have a folder "test/" which contains certain text files.
The "test/" folder has another subdirectory "archive/" inside it.
FTPserver->
-test/
---abc.txt
---xyz.txt
---archive/
We are able to download all the text files via Spring Integration flows in our local directory.
Now we are looking into ways to move the remote text files inside the folder "archive" within the FTP Server itself once it has been downloaded into the local.
We are trying to do it in the handle() method like this ->
#Bean
public IntegrationFlow integrationFlow() {
File localDirectory = new File("tmp/");
FtpInboundChannelAdapterSpec ftpInboundChannelAdapterSpec = Ftp.inboundAdapter(gimmeFactory())
.remoteDirectory("test/")
.autoCreateLocalDirectory(true)
.regexFilter(".*\\.txt$")
.localDirectory(localDirectory)
.preserveTimestamp(true)
.remoteFileSeparator("/");
return IntegrationFlows.from(ftpInboundChannelAdapterSpec, pc -> pc.poller(pm -> pm.fixedRate(1000, TimeUnit.MILLISECONDS)))
.handle((file, messageHeaders) -> {
messageHeaders.forEach((k, v) -> System.out.println(k + ':' + v));
return null;
})
.handle(Ftp.outboundGateway(gimmeFactory(), AbstractRemoteFileOutboundGateway.Command.MV, "'test/archive'"))
.get();
}
But it is not moving into the remote "archive" folder location.
We are quite not sure how to handle this operation in any other way.
Is there anything we can do to fix the above code snippet or do something differently in order to achieve what we want ?
Please advise.
Update
Thank you Gary for the pointers.
I was able to solve the problem by doing as given in below code snippet->
#Bean
public IntegrationFlow integrationFlow() {
File localDirectory = new File("tmp/");
FtpInboundChannelAdapterSpec ftpInboundChannelAdapterSpec = Ftp.inboundAdapter(gimmeFactory())
.remoteDirectory("test/")
.autoCreateLocalDirectory(true)
.regexFilter(".*\\.txt$")
.localDirectory(localDirectory)
.preserveTimestamp(true)
.remoteFileSeparator("/");
return IntegrationFlows
.from(ftpInboundChannelAdapterSpec, e -> e.poller(Pollers.fixedDelay(Duration.ofSeconds(5))))
.handle(Ftp.outboundGateway(gimmeFactory(), AbstractRemoteFileOutboundGateway.Command.LS, "'test/'")
.options(AbstractRemoteFileOutboundGateway.Option.NAME_ONLY))
.split()
.handle(Ftp.outboundGateway(gimmeFactory(), AbstractRemoteFileOutboundGateway.Command.MV, "'test/' +payload").renameExpression("'test/archive/' +payload"))
.channel("nullChannel")
.get();
}
Since you are returning null from the first .handle(), the flow stops there; you need to return a payload containing the from path and the rename-expression is needed to specify the to path.
https://docs.spring.io/spring-integration/docs/current/reference/html/ftp.html#using-the-mv-command
The mv command moves files.
The mv command has no options.
The expression attribute defines the “from” path and the rename-expression attribute defines the “to” path. By default, the rename-expression is headers['file_renameTo']. This expression must not evaluate to null or an empty String. If necessary, any necessary remote directories are created. The payload of the result message is Boolean.TRUE. The file_remoteDirectory header provides the original remote directory, and file_remoteFile header provides the file name. The new path is in the file_renameTo header.
Starting with version 5.5.6, the remoteDirectoryExpression can be used in the mv command for convenience. If the “from” file is not a full file path, the result of remoteDirectoryExpression is used as the remote directory. The same applies for the “to” file, for example, if the task is just to rename a remote file in some directory.

How can I implement a file download function via OCC?

I need an API that will let me download a (e.g. CSV file). I tried to do something like this:
#RequestMapping(value="/{cartId}/export", method = RequestMethod.GET, produces = { "application/octet-stream" })
#ApiOperation(hidden = false, value = "Exports the contents of a cart as a CSV file.", notes = "Exports the contents of a cart as a CSV file.")
#ApiBaseSiteIdUserIdAndCartIdParam
public #ResponseBody ResponseEntity export() {
File file = new File("test.csv");
return ResponseEntity.ok()
.header("Content-Disposition", "attachment; filename=" + "test" + ".csv")
.contentLength(file.length())
.contentType(MediaType.parseMediaType("application/octet-stream"))
.body(new FileSystemResource(file));
}
However, I would get errors like this:
0203_17:48:01,878 INFO [hybrisHTTP21]
[de.hybris.platform.webservicescommons.resolver.RestHandlerExceptionResolver.doResolveException:73]
Translating exception
[org.springframework.web.HttpMediaTypeNotAcceptableException]: Could
not find acceptable representation 0203_17:48:01,879 WARN
[hybrisHTTP21]
[de.hybris.platform.webservicescommons.resolver.AbstractRestHandlerExceptionResolver.writeWithMessageConverters:72]
Could not find HttpMessageConverter that supports return type [class
de.hybris.platform.webservicescommons.dto.error.ErrorListWsDTO] and
[application/octet-stream] 0203_17:48:01,879 WARN [hybrisHTTP21]
[org.springframework.web.servlet.handler.AbstractHandlerExceptionResolver.logException:197]
Resolved [org.springframework.web.HttpMediaTypeNotAcceptableException:
Could not find acceptable representation]
I understand this is because springmvc-servlet.xml only has resolverXStreamJSONConverter and resolverXStreamXmlConverter.
If I were to do it the MVC way, I'd write the file to a HttpServletResponse, but that doesn't seem ideal for OCC. So, how should I implement the file download? (Code sample also appreciated)
NOTE: The API/method will be used with Spartacus storefront.
A workaround is to add org.springframework.http.converter.ResourceHttpMessageConverter in messageConvertersV2 in web/webroot/WEB-INF/config/v2/jaxb-converters-spring.xml:
<util:list id="messageConvertersV2">
<bean class="org.springframework.http.converter.ResourceHttpMessageConverter"/>
<ref bean="customJsonHttpMessageConverter"/>
<ref bean="customXmlHttpMessageConverter"/>
</util:list>
However, not sure if this is correct or a good idea.
I needed a similar implementation to offer a PDF download. The controller now returns ResponseEntity<byte[]> and I appended the corresponding MessageConverter. I also didnt find a 'better' way, but it works.
<bean id="myByteArrayHttpMessageConverter" class="org.springframework.http.converter.ByteArrayHttpMessageConverter"/>
<bean depends-on="messageConvertersV2" parent="listMergeDirective">
<property name="add" ref="myByteArrayHttpMessageConverter"/>
</bean>

Rename file in-place using Spring Integration FileWritingMessageHandler

I am attempting to write a simple test of renaming files in-place using a FileWritingMessageHandler, however I can't seem to figure out how to properly specify the target destination directory.
Since I am recursively scanning a directory tree I would ultimately like to simply read the parent path from the file payload and rename it using the FileNameGenerator, but that doesn't appear to work.
The 'payload.name' in the DefaultFileNameGenerator resolves properly, but 'payload.path' does not.
How do I properly determine the source file's location and use that in the handler?
Edit
Here is the channel adapter that scans for files. I had to use .setUseWatchService(true) to achieve recursive scanning.
#Bean
#InboundChannelAdapter(channel = "sourceFileChannel", poller = #Poller(fixedRate = "5000", maxMessagesPerPoll = "-1"))
public MessageSource<File> sourceFiles() {
CompositeFileListFilter<File> filters = new CompositeFileListFilter<>();
filters.addFilter(new SimplePatternFileListFilter(sourceFilenamePattern));
filters.addFilter(persistentFilter());
FileReadingMessageSource source = new FileReadingMessageSource();
source.setAutoCreateDirectory(true);
source.setDirectory(new File(sourceDirectory));
source.setFilter(filters);
source.setUseWatchService(true);
return source;
}
UPDATE
Artem helped me understand my mistake.
I was able to achieve the desired result by using the SpelExpressionParser as Artem outlined.
The key piece being:
new SpelExpressionParser().parseExpression("payload.parent")
Where "payload.parent" resolves to the file parent path properly.
#Bean
#ServiceActivator(inputChannel = "processingFileChannel")
public MessageHandler copyFileForProcessingOutboundChannelAdapter() {
FileWritingMessageHandler adapter = new FileWritingMessageHandler(new SpelExpressionParser().parseExpression("payload.parent"));
adapter.setDeleteSourceFiles(false);
adapter.setAutoCreateDirectory(true);
adapter.setExpectReply(false);
adapter.setFileNameGenerator(processingFileNameGenerator());
return adapter;
}
#Bean
public DefaultFileNameGenerator processingFileNameGenerator() {
DefaultFileNameGenerator defaultFileNameGenerator = new DefaultFileNameGenerator();
defaultFileNameGenerator.setExpression("'p_' + payload.name");
return defaultFileNameGenerator;
}
The 'payload.name' in the DefaultFileNameGenerator resolves properly, but 'payload.path' does not.
Well, I'm not sure what should be in your case, but for me that always returns the full, absolute path for source file, including the root directory to scan.
Since you have it there as a sourceDirectory, how about to use it in your processingFileNameGenerator to sever the root dir from the target path to use? For example if my root dir is /root and I get a file from the subdir /root/foo/my_file.txt, I could do payload.path.replaceFirst('/root', ''). So in the end I have just /foo/my_file.txt.
At least that is what I'm going to do in that JIRA to populate FileHeaders.FILENAME with the relative path from the provided directory to scan.
UPDATE
Oh! I see. No, that isn't going to work that way. See FileWritingMessageHandler ctors. The String once accepts static target directory. Root for our case. The code like new LiteralExpression("payload.path") isn't going to work a desired way, too. See LiteralExpression JavaDocs. It is just an Expression variant to always return the same static value. In your case it is payload.path.
If you are really going to evaluate against an incoming Message, you should use new SpeLexpressionParser().parseExpression("payload.path"). But as I said before, it returns the absolute path for any file in the sub-directories.

gradle get relative resource path

When I iterate over source repository I do like this
def resourceDir = proj.sourceSets.main.output.resourcesDir
resourceDir.eachFileRecurse(groovy.io.FileType.FILES) { // only files will be recognized
file ->
def path = FilenameUtils.separatorsToUnix(file.toString())
if (FilenameUtils.getExtension(file.toString()) in supportedResourceExt) {
proj.logger.lifecycle("Reading file {}.", file)
//.....
}
}
In log it writes this
Reading file D:\PROJECT_FOLDER\project\subproject\subsubproject\build\resources\main\com\package\something\file.txt
How to get only the part starting with com\package\something\file.txt without explicitly reading it like file.substring(file.indexOf)?
Maybe it's posible to relativize it with project path somehow?
It seems that:
proj.logger.lifecycle("Reading file {}.", file.absolutePath - resourceDir.absolutePath)
should work. Can't check it right now.

Flatten first directory of a FileTree in Gradle

I'm writing a task to extract a tarball into a directory. I don't control this tarball's contents.
The tarball contains a single directory which contains all the files I actually care about. I want to pull everything out of that directory and copy that into my destination.
Example:
/root/subdir
/root/subdir/file1
/root/file2
Desired:
/subdir
/subdir/file1
/file2
Here's what I tried so far, but this seems like a really goofy way of doing it:
copy {
eachFile {
def segments = it.getRelativePath().getSegments() as List
it.setPath(segments.tail().join("/"))
return it
}
from tarTree(resources.gzip('mytarfile.tar.gz'))
into destinationDir
}
For each file, I get the elements of its path, remove the first, join that with /, then set that as the file's path. And this works...sort of. The problem is that this creates the following structure as a result:
/root/subdir
/root/subdir/file1
/root/file2
/subdir
/subdir/file1
/file2
I'm fine with just removing the root directory myself as a final action of the task, but I feel like there should be a much simpler way of doing this.
AFAIK, the only way is to unpack the zip, tar, tgz file :(
There is an open issue here
Please go vote for it!
Until then, the solution isn't very pretty, but not that hard either. In the example below, I am assuming that you want to remove the 'apache-tomcat-XYZ' root-level directory from a 'tomcat' configuration that only includes the apache-tomcat zip file.
def unpackDir = "$buildDir/tmp/apache.tomcat.unpack"
task unpack(type: Copy) {
from configurations.tomcat.collect {
zipTree(it).matching {
// these would be global items I might want to exclude
exclude '**/EMPTY.txt'
exclude '**/examples/**', '**/work/**'
}
}
into unpackDir
}
def mainFiles = copySpec {
from {
// use of a closure here defers evaluation until execution time
// It might not be clear, but this next line "moves down"
// one directory and makes everything work
"${unpackDir}/apache-tomcat-7.0.59"
}
// these excludes are only made up for an example
// you would only use/need these here if you were going to have
// multiple such copySpec's. Otherwise, define everything in the
// global unpack above.
exclude '**/webapps/**'
exclude '**/lib/**'
}
task createBetterPackage(type: Zip) {
baseName 'apache-tomcat'
with mainFiles
}
createBetterPackage.dependsOn(unpack)
Using groovy's syntax, we can use a regex to eliminate the first path segment:
task myCopyTask(type: Copy) {
eachFile {
path -= ~/^.+?\//
}
from tarTree(resources.gzip('mytarfile.tar.gz'))
into destinationDir
includeEmptyDirs = false // ignore empty directories
}

Resources