I'm trying to write files to smb folder, it works fine for the first write but when the same file is written again it throws error
Caused by: org.springframework.core.NestedIOException: Failed to rename [/remote-target-dir/smbTest.test.writing] to [/remote-target-dir/smbTest.test].; nested exception is jcifs.smb.SmbException: Cannot create a file when that file already exists.
Im' using the FileTransferringMessageHandler which has FileExistsMode.REPLACE as default file exists mode but still it fails to replace.
I debugged into the SmbSession class, when flow reaches to this code in rename(String _pathFrom, String _pathTo) method
if (this.smbShare.isReplaceFile() && smbFileTo.exists()) {
smbFileTo.delete();
}
I see that the smbShare's isReplaceFile() is false, which is why I assume it is unable to delete the previous file with the same name
Here is the full code:
Connection:
#Bean
public SmbSessionFactory smbSessionFactory() {
SmbSessionFactory smbSession = new SmbSessionFactory();
smbSession.setHost("localhost");
smbSession.setPort(445);
smbSession.setDomain("DESKTOP-07O79IT");
smbSession.setUsername("Dell");
smbSession.setPassword("changeIt");
smbSession.setShareAndDir("Users\\DELL\\Desktop\\Shared");
smbSession.setSmbMinVersion(DialectVersion.SMB210);
smbSession.setSmbMaxVersion(DialectVersion.SMB311);
return smbSession;
}
Outbound to send files to smb:
#ServiceActivator(inputChannel = "storeToSmb")
#Bean
public MessageHandler smbMessageHandler(SmbSessionFactory smbSessionFactory) {
FileTransferringMessageHandler<SmbFile> handler =
new FileTransferringMessageHandler<>(smbSessionFactory);
handler.setRemoteDirectoryExpression(
new LiteralExpression("/remote-target-dir"));
handler.setFileNameGenerator(m -> "smbTest.test");
handler.setAutoCreateDirectory(true);
return handler;
}
Gateway:
#Override
public void run(ApplicationArguments args) throws Exception {
File file = new File("smbFile.txt");
try(FileWriter writer = new FileWriter(file)){
writer.write("This is sample smb write");
messageGateway.storeToSmb(file);
}catch (IOException e){
System.out.println("Error: "+e);
}
}
Can you please help with this issue?
The SmbSessionFactory must come with the setReplaceFile(true). The FileTransferringMessageHandler does nothing about its dependency and can't mutate it although it has its own logic about that FileExistsMode.REPLACE.
Related
I am getting the MessageRemovedException on parsing the email after receiving the email in the EmailReceiveChannel, if I have configure ShouldMarkAsDelete flag as "true".
Everything works fine if the flag is set as false
#Bean
public ImapMailReceiver imapMailReceiver() {
StringBuilder impaURI = new StringBuilder();
impaURI.append(MAIL_PROTOCOL).append("://").append(MAIL_USERNAME).append(":").append(MAIL_PASSWORD)
.append("#").append(MAIL_HOST).append(":").append(MAIL_PORT).append("/").append(MAIL_FOLDER);
ImapMailReceiver mailReceiver = new ImapMailReceiver(impaURI.toString());
mailReceiver.setShouldDeleteMessages(true);
mailReceiver.setShouldMarkMessagesAsRead(true);
mailReceiver.setJavaMailProperties(javaMailProperties());
mailReceiver.setAutoCloseFolder(false);
return mailReceiver;
}
On debugging the flow, expunged flag for MimeMessage is coming as "true". This stopped working recently.
Any idea why is this issue?
The logic there in the AbstractMailReceiver is like this:
private void postProcessFilteredMessages(Message[] filteredMessages) throws MessagingException {
setMessageFlags(filteredMessages);
if (shouldDeleteMessages()) {
deleteMessages(filteredMessages);
}
where that deleteMessages() does only this:
message.setFlag(Flags.Flag.DELETED, true);
The expunge functionality is performed over here:
protected void closeFolder() {
this.folderReadLock.lock();
try {
MailTransportUtils.closeFolder(this.folder, this.shouldDeleteMessages);
}
finally {
this.folderReadLock.unlock();
}
}
So, since you have that mailReceiver.setAutoCloseFolder(false);, it means you close folder yourself and only after that you try to get access to the message content. Must be opposite: you have to close folder only after you have already parsed the content of the message.
Hi I have configured the default filter as below and if some emails have a certain subject or from address I try to move them to specific folders using java mail api as below:
Filter implementation:
#Bean(name = ImapAdaptersUtil.DEFAULT_FILTER_BEAN_NAME)
#Scope(WebApplicationContext.SCOPE_APPLICATION)
public MessageSelector defaultFilter() {
return message -> {
if (message.getPayload() instanceof MimeMessage) {
try {
String from = Optional.ofNullable(((InternetAddress) ((MimeMessage) message.getPayload()).getFrom()[0]).getAddress()).orElse(GeneralConst.EMPTY_STRING);
String subject = Optional.ofNullable(((MimeMessage) message.getPayload()).getSubject()).orElse(GeneralConst.EMPTY_STRING);
if (!from.matches(DELIVERY_ERROR_FROM)
&& !from.matches(SPAM_FROM)
&& !subject.matches(DELIVERY_ERROR_SUBJECT)
&& !subject.matches(OUT_OF_OFFICE)
&& !subject.matches(SPAM_SUBJECT)) {
return true;
}
} catch (MessagingException me) {
throw new ApplicationBusinessException(ApplicationBusinessException.ApplicationBusinessExceptionType.FUNCTIONAL_VIOLATION,
"Could not filter incoming email: " + me.getMessage());
}
}
try {
this.moveMessage(((MimeMessage) message.getPayload()));
} catch (MessagingException me) {
throw new ApplicationBusinessException(ApplicationBusinessException.ApplicationBusinessExceptionType.FUNCTIONAL_VIOLATION,
"Could not move incoming email: " + me.getMessage());
}
return false;
};
}
Move to folder implementation:
private void moveMessage(MimeMessage message) throws MessagingException {
Folder folder = message.getFolder();
Store store = folder.getStore();
Folder[] folders = store.getDefaultFolder().list("*");
for (Folder folder1 : folders) {
LOGGER.info("folder name {}", folder1.getName());
}
Folder deliveryErrorFolder = store.getFolder("Delivery error");
if (!deliveryErrorFolder.exists()) {
if (deliveryErrorFolder.create(Folder.HOLDS_MESSAGES)) {
deliveryErrorFolder.setSubscribed(true);
move(message, folder, deliveryErrorFolder);
LOGGER.info("Delivery error created");
}
} else {
move(message, folder, deliveryErrorFolder);
}
}
private void move(MimeMessage message, Folder folder, Folder spamFolder) throws MessagingException {
List<javax.mail.Message> tempList = new ArrayList<>();
tempList.add(message);
javax.mail.Message[] tempMessageArray = tempList.toArray(new javax.mail.Message[0]);
folder.copyMessages(tempMessageArray, spamFolder);
LOGGER.info("message moved");
}
ImapMailReceiver configured as an Integration flow :
public static IntegrationFlow getImapAdapterIntegrationFlow(String imapsUrl, MessageSelector filter, QueueChannelSpec channelSpec) {
return IntegrationFlows
.from(Mail.imapInboundAdapter(imapsUrl)
.userFlag("testSIUserFlag")
.simpleContent(false)
.autoCloseFolder(false)
.shouldMarkMessagesAsRead(true)
.javaMailProperties(getPropertiesBuilderConsumer()),
e -> e.autoStartup(true)
.poller(p -> p.fixedDelay(1000)))
.filter(filter)
.channel(channelSpec)
.get();
}
I get this exception :
Caused by: java.lang.ClassCastException: class org.springframework.integration.mail.AbstractMailReceiver$IntegrationMimeMessage cannot be cast to class com.sun.mail.imap.IMAPMessage (org.springframework.integration.mail.AbstractMailReceiver$IntegrationMimeMessage and com.sun.mail.imap.IMAPMessage are in unnamed module of loader 'app')
at com.sun.mail.imap.Utility.toMessageSet(Utility.java:85)
Yeah... What you are looking for is available starting with version 5.4: https://docs.spring.io/spring-integration/docs/current/reference/html/mail.html#mail-inbound
Starting with version 5.4, it is possible now to return a MimeMessage as is without any conversion or eager content loading. This functionality is enabled with this combination of options: no headerMapper provided, the simpleContent property is false and the autoCloseFolder property is false.
So, all good in your config - only what you need to to upgrade your project to the latest Spring Integration. Directly with the 5.4.5 or via respective latest Spring Boot.
I am reading the file from remote directory using SFTP. I am able to get file by stream using outbound gateway, and move it to archive folder even.
I am processing the data in file but if there is some issue in data then I am throwing an error. I do not want to rename the file if there is any error thrown while processing the data, how can I achieve that. It will be very helpful if I can get some good practices for having error handler while using spring integration.
.handle(Sftp.outboundGateway(sftpSessionFactory(), GET, "payload.remoteDirectory + payload.filename").options(STREAM).temporaryFileSuffix("_reading"))
.handle(readData(),c->c.advice(afterReading()))
.enrichHeaders(h -> h
.headerExpression(FileHeaders.RENAME_TO, "headers[file_remoteDirectory] + 'archive/' + headers[file_remoteFile]")
.headerExpression(FileHeaders.REMOTE_FILE, "headers[file_remoteFile]")
.header(FileHeaders.REMOTE_DIRECTORY, "headers[file_remoteDirectory]"))
.handle(Sftp.outboundGateway(sftpSessionFactory(), MV, "headers[file_remoteDirectory]+headers[file_remoteFile]").renameExpression("headers['file_renameTo']"))
.get();
#Bean
public ExpressionEvaluatingRequestHandlerAdvice afterReading() {
ExpressionEvaluatingRequestHandlerAdvice advice = new ExpressionEvaluatingRequestHandlerAdvice();
advice.setSuccessChannelName("successReading.input");
advice.setOnSuccessExpressionString("payload + ' was successful streamed'");
advice.setFailureChannelName("failureReading.input");
advice.setOnFailureExpressionString("payload + ' was bad, with reason: ' + #exception.cause.message");
advice.setTrapException(true);
advice.setPropagateEvaluationFailures(true);
return advice;
}
#Bean
public IntegrationFlow successReading() {
return f -> f.log();
}
#Bean
public IntegrationFlow failureReading() {
return f -> f.log(ERROR);
}
public GenericHandler readData() {
return new GenericHandler() {
#Override
public Object handle(Object o, Map map) {
InputStream file = (InputStream) o;
String fileName = (String) map.get(REMOTE_FILE);
try {
// processing data
} catch (Exception e) {
return new SftpException(500, String.format("Error while processing the file %s because of Error: %s and reason %s", fileName, e.getMessage(), e.getCause()));
}
Closeable closeable = (Closeable) map.get(CLOSEABLE_RESOURCE);
if (closeable != null) {
try {
closeable.close();
file.close();
} catch (Exception e) {
logger.error(String.format("Session didn`t get closed after reading the stream data for file %s and error %s"), fileName, e.getMessage());
}
}
return map;
}
};
}
Updated
Add an ExpressionEvaluatingRequestHandlerAdvice to the .handler() endpoint .handle(readData(), e -> e.advice(...)).
The final supplied advice class is the o.s.i.handler.advice.ExpressionEvaluatingRequestHandlerAdvice. This advice is more general than the other two advices. It provides a mechanism to evaluate an expression on the original inbound message sent to the endpoint. Separate expressions are available to be evaluated, after either success or failure. Optionally, a message containing the evaluation result, together with the input message, can be sent to a message channel.
I want to add a watch on the remote machine for newly added CSV files or unread. Once files are identified read it according to their timestamp which will be there in the file name. The file will be read using streaming rather coping to the local machine. While the file is getting read, append _reading to the filename and append _read once the file is read. The file will be read over SFTP protocol and I am planning to use spring integration sftp. In case of error while reading file or data in the file is not as per expectation I want to move that file in sub-directory.
I have tried to poll the remote directory and reading once CSV file. Once read I am removing the file from the directory.
<dependency>
<groupId>org.springframework.integration</groupId>
<artifactId>spring-integration-sftp</artifactId>
<version>5.1.0.RELEASE</version>
</dependency>
<dependency>
<groupId>org.springframework.integration</groupId>
<artifactId>spring-integration-core</artifactId>
<version>5.0.6.RELEASE</version>
</dependency>
Spring boot version 2.0.3.RELEASE
#Bean
public SessionFactory<ChannelSftp.LsEntry> sftpSessionFactory() {
DefaultSftpSessionFactory factory = new DefaultSftpSessionFactory(true);
factory.setHost(hostname);
factory.setPort(22);
factory.setUser(username);
factory.setPassword(password);
factory.setAllowUnknownKeys(true);
return new CachingSessionFactory<ChannelSftp.LsEntry>(factory);
}
#Bean
public MessageSource<InputStream> sftpMessageSource() {
SftpStreamingMessageSource messageSource = new SftpStreamingMessageSource(template());
messageSource.setRemoteDirectory(path);
messageSource.setFilter(compositeFilters());
return messageSource;
}
public CompositeFileListFilter compositeFilters() {
return new CompositeFileListFilter()
.addFilter(new SftpRegexPatternFileListFilter(".*csv"));
}
#Bean
public SftpRemoteFileTemplate template() {
return new SftpRemoteFileTemplate(sftpSessionFactory());
}
#Bean
public IntegrationFlow sftpOutboundListFlow() {
return IntegrationFlows.from(this.sftpMessageSource(), e -> e.poller(Pollers.fixedDelay(5, TimeUnit.SECONDS)))
.handle(Sftp.outboundGateway(template(), NLST, path).options(Option.RECURSIVE)))
.filter(compositeFilters())
.transform(sorter())
.split()
.handle(Sftp.outboundGateway(template(), GET, "headers['file_remoteDirectory'] + headers['file_remoteFile']").options(STREAM))
.transform(csvToPojoTransformer())
.handle(service())
.handle(Sftp.outboundGateway(template(), MV, "headers['file_remoteDirectory'] + headers['file_remoteFile'] + _read"))
.handle(after())
.get();
}
#Bean
public MessageHandler sorter() {
return new MessageHandler() {
#Override
public void handleMessage(Message<?> message) throws MessagingException {
List<String> fileNames = (List<String>) message.getPayload();
Collections.sort(fileNames);
}
};
}
#Bean
public MessageHandler csvToPojoTransformer() {
return new MessageHandler() {
#Override
public void handleMessage(Message<?> message) throws MessagingException {
InputStream streamData = (InputStream) message.getPayload();
convertStreamtoObject(streamData, Class.class);
}
};
}
public List<?> convertStreamtoObject(InputStream inputStream, Class clazz) {
HeaderColumnNameMappingStrategy ms = new HeaderColumnNameMappingStrategy();
ms.setType(clazz);
Reader reader = new InputStreamReader(inputStream);
CsvToBean cb = new CsvToBeanBuilder(reader)
.withType(clazz)
.withMappingStrategy(ms)
.withSkipLines(0)
.withSeparator('|')
.withThrowExceptions(true)
.build();
return cb.parse();
}
#Bean
public MessageHandler service() {
return new MessageHandler() {
#Override
public void handleMessage(Message<?> message) throws MessagingException {
List<Class> csvDataAsListOfPojo = List < Class > message.getPayload();
// use this
}
};
}
#Bean
public ExpressionEvaluatingRequestHandlerAdvice after() {
ExpressionEvaluatingRequestHandlerAdvice advice = new ExpressionEvaluatingRequestHandlerAdvice();
advice.setSuccessChannelName("success.input");
advice.setOnSuccessExpressionString("payload + ' was successful'");
advice.setFailureChannelName("failure.input");
advice.setOnFailureExpressionString("payload + ' was bad, with reason: ' + #exception.cause.message");
advice.setTrapException(true);
return advice;
}
#Bean
public IntegrationFlow success() {
return f -> f.handle(System.out::println);
}
#Bean
public IntegrationFlow failure() {
return f -> f.handle(System.out::println);
}
Updated Code
For complex scenarios (list, move, fetch, remove, etc), you should use SFTP remote file gateways instead.
The SFTP outbound gateway provides a limited set of commands that let you interact with a remote SFTP server:
ls (list files)
nlst (list file names)
get (retrieve a file)
mget (retrieve multiple files)
rm (remove file(s))
mv (move and rename file)
put (send a file)
mput (send multiple files)
Or use the SftpRemoteFileTemplate directly from your code.
EDIT
In response to your comments; you need something like this
Inbound Channel Adapter (with poller) - returns directory name
LS Gateway
Filter (remove any files already fetched)
Transformer (sort the list)
Splitter
GET Gateway(stream option)
Transformer (csv to POJO)
Service (process POJO)
If you add
RM Gateway
after your service (to remove the remote file), you don't need the filter step.
You might find the Java DSL simpler to assemble this flow...
#Bean
public IntegrationFlow flow() {
return IntegrationFlows.from(() -> "some/dir", e -> e.poller(...))
.handle(...) // LS Gateway
.filter(...)
.transform(sorter())
.split
.handle(...) // GET Gateway
.transform(csvToPojoTransformer())
.handle(myService())
.get()
}
i am writing my own image import for my product catalog. I want to read the images from the local filesystem and store them in the configured assets folder. The import is very simple for now. Its one controller in the admin project and i trigger it by calling an url.
It is creating the files along with the folder structure and the files seem to have the same filesize, but somehow they get messed up along the way and they are not readable as images anymore (picture viewers wont open them). Any ideas why its being messed up ?
here the code:
#Controller("blImageImportController")
#RequestMapping("/imageimport")
public class ImageImportController extends AdminAbstractController {
#Value("${image.import.folder.location}")
private String importFolderLocation;
#Resource(name = "blStaticAssetService")
protected StaticAssetService staticAssetService;
#Resource(name = "blStaticAssetStorageService")
protected StaticAssetStorageService staticAssetStorageService;
#RequestMapping(method = {RequestMethod.GET})
public String chooseMediaForMapKey(HttpServletRequest request,
HttpServletResponse response,
Model model
) throws Exception {
File imageImportFolder = new File(importFolderLocation);
if (imageImportFolder.isDirectory()) {
Arrays.stream(imageImportFolder.listFiles()).forEach(directory ->
{
if (directory.isDirectory()) {
Arrays.stream(directory.listFiles()).forEach(this::processFile);
}
});
}
return "";
}
private void processFile(File file) {
FileInputStream fis = null;
try {
HashMap properties = new HashMap();
properties.put("entityType", "product");
properties.put("entityId", file.getParentFile().getName());
fis = new FileInputStream(file);
StaticAsset staticAsset = this.staticAssetService.createStaticAsset(fis, file.getName(), file.length(), properties);
this.staticAssetStorageService.createStaticAssetStorage(fis, staticAsset);
fis.close();
} catch (Exception e) {
} finally {
try {
if (fis != null)
fis.close();
} catch (IOException ex) {
ex.printStackTrace();
}
}
}
}
There is a check in the StaticAssetService to try to detect this as an image (see https://github.com/BroadleafCommerce/BroadleafCommerce/blob/b55848f/admin/broadleaf-contentmanagement-module/src/main/java/org/broadleafcommerce/cms/file/service/StaticAssetServiceImpl.java#L217-L220). If it detected this correctly, you should get back an ImageStaticAssetImpl in the result to that call.
The flipside of this is the controller that actually reads the file (the StaticAssetViewController that renders a StaticAssetView). One of the things that the StaticAssetView does is set a response header for mimeType which the browser uses to render. This is set by this piece in the StaticAssetStorageService: https://github.com/BroadleafCommerce/BroadleafCommerce/blob/b55848f837f26022a620f0c2c143eed7902ba3f1/admin/broadleaf-contentmanagement-module/src/main/java/org/broadleafcommerce/cms/file/service/StaticAssetStorageServiceImpl.java#L213. I suspect that is the root of your problem.
Also just a note, sending those properties is not necessary when you are uploading the file yourself. That is mainly used in the admin when you are uploading an image for a specific entity (like a product or a category).