Using FTPClient How do you add the directory names to an ArrayList<String>()? - ftp-client

I want to:
In Java on Netbeans:
Create a program that lists names of all files on an ftp site. Then check on a regular basis whether anything has been updated and produce a new list.
My thought process is:
return Files and directories in root via ftp connection FTPClient() ftp.listFiles(dir) (name and last modified date)
If file add to fileArray
2.1 If file exists in fileArray compare dates
2.2 if scanned file date newer than old file in fileArray replace
2.22 else update scanned date to fileArray
2.11 Else add
If Directory add to dirArray
3.11 If scanned directory exists in dirArray compare last date modified
3.21 If scanned directory date is newer than old directory date in dirArray replace in arraylist
3.22 Else remove
3.12 Else add directory to arraylist
Loop through dirArray grabbing the name and inputting it into dir and adding +1 to counter
run through from step 1 again until counter is > directory arraylist length;
Remove any files from file arraylist that are older than the scanDate
7.1 Loop though fileArray
7.2 compare returned file Date with scanDate
7.21 If scanned date != scanDate remove file.
List files in fileArray
Here is my code so far:
public class DanLister {
private Date scannedDate;
private String dir;
private ArrayList<String> dirArray = new ArrayList<String>();
private ArrayList<String> fileArray = new ArrayList<String>();
public void listFTP(String FTPdirectoryName,String server, String un, String pw, String dir, Calendar start, Calendar end){
try {
/// Instantiate new FTP Client
FTPClient ftp = new FTPClient();
/// Connect to FTP
ftp.connect(server);
ftp.login(un, pw);
System.out.println("Connected to " + server + "!");
System.out.println(ftp.getReplyString());
FTPFile[] filelist = ftp.listFiles(dir);
System.out.println(Arrays.toString(filelist));
ftp.logout();
ftp.disconnect();
} catch (IOException ex) {
Logger.getLogger(FTPListFiles.class.getName()).log(Level.SEVERE,
null, ex);
}
}
}
My problem is this returns FTPFile[] which won't go into my dirArray I'm unsure how to proceed with this part? I don't know how to convert the FTPFile[] or just grab the name and date and put it into the dirArray?

Solved with FTPListParseEngine
I've created this class:
package com.tableview.ftpmonitor;
import java.io.IOException;
import java.io.InputStream;
import org.apache.commons.net.ftp.FTPFile;
import org.apache.commons.net.ftp.FTPFileEntryParser;
import org.apache.commons.net.ftp.FTPListParseEngine;
public class parseFileList {
private String encoding;
public FTPFile[] parseFileList(InputStream listStream) throws
IOException {
FTPListParseEngine engine = new
FTPListParseEngine((FTPFileEntryParser) this);
engine.readServerList(listStream, encoding);
return engine.getFiles();
}
}
And used this code to put file details in ArrayList:
public void listFTP(String FTPdirectoryName,String server, String un, String pw, String dir, Calendar start, Calendar end){
try {
/// Instantiate new FTP Client
FTPClient ftp = new FTPClient();
FTPFile ftpFile = new FTPFile();
parseFileList pfl = new parseFileList();
/// Connect to FTP
ftp.connect(server);
ftp.login(un, pw);
System.out.println("Connected to " + server + "!");
System.out.println(ftp.getReplyString());
FTPListParseEngine engine = ftp.initiateListParsing(dir);
int files = engine.getFiles().length;
System.out.println("This is number of files:" + files);
if(engine.hasNext()){
String ln = Arrays.toString(engine.getFiles());
dirArray.add(ln);
System.out.println(dirArray);
}

Related

Spring Integration- FTP should synchronize with local folder

I have ftp location files and have local folder, on first time the files are copied to local and on restarting the server(Currently it is copying already copied files to the local folder) it should not look for the files which are already exist in the local and it should lookup for new files only. Please let me know is it possible to achieve it using Spring-Integration ftp?
I have added Filter also but still it is not working, please let me know where I am going wrong,
#Bean
#InboundChannelAdapter(value = "inputChannel", poller = #Poller(fixedDelay = "1000", maxMessagesPerPoll = "1"))
public MessageSource<?> receive() {
FtpInboundFileSynchronizingMessageSource messageSource = new FtpInboundFileSynchronizingMessageSource(synchronizer());
PropertiesPersistingMetadataStore metadataStore = new PropertiesPersistingMetadataStore();
FileSystemPersistentAcceptOnceFileListFilter acceptOnceFilter = new FileSystemPersistentAcceptOnceFileListFilter(metadataStore,"*.xml");
File Temp = new File(TEMP_FOLDER);
metadataStore.setBaseDirectory(TEMP_FOLDER);
messageSource.setLocalDirectory(Temp);
messageSource.setAutoCreateLocalDirectory(false);
messageSource.setLocalFilter(acceptOnceFilter);
return messageSource;
}
private AbstractInboundFileSynchronizer<FTPFile> synchronizer() {
folderCleanUp();
AbstractInboundFileSynchronizer<FTPFile> fileSynchronizer = new FtpInboundFileSynchronizer(sessionFactory());
fileSynchronizer.setRemoteDirectory(ftpFileLocation);
fileSynchronizer.setDeleteRemoteFiles(false);
Pattern pattern = Pattern.compile(".*\\.xml$");
FtpRegexPatternFileListFilter ftpRegexPatternFileListFilter = new FtpRegexPatternFileListFilter(pattern);
fileSynchronizer.setFilter(ftpRegexPatternFileListFilter);
return fileSynchronizer;
}
To clarify Artem's advice about implementing your custom FileListFilter, here is an example of such filter (aimed to filter out files older than given moment):
#Component
public class OldFilesFilter extends AbstractFileListFilter<FTPFile> {
// (oldFilesTimestamp field declaration and its source)
#Override
protected boolean accept(FTPFile file) {
String fileName = file.getName();
long fileTimestamp = file.getTimestamp().getTimeInMillis();
ZonedDateTime fileModTimestamp = ZonedDateTime.ofInstant(Instant.ofEpochMilli(fileTimestamp), ZoneId.systemDefault());
boolean isFileAcceptable = fileModTimestamp.isAfter(oldFilesTimestamp);
if (log.isTraceEnabled()) {
log.trace("File {}:\n" +
"file timestamp : {};\n" +
"given timestamp: {};\n" +
"file is new : {}",
fileName, fileModTimestamp, oldFilesTimestamp, isFileAcceptable);
}
return isFileAcceptable;
}
}
Also note that Spring Integration allows multiple filters to be applied to single file source at the same time. This can be achieved with CompositeFileListFilter:
private CompositeFileListFilter<FTPFile> remoteFileFilter() {
FtpPersistentAcceptOnceFileListFilter persistentFilter =
new FtpPersistentAcceptOnceFileListFilter(metadataStore, "remoteProcessedFiles.");
return new CompositeFileListFilter<>(Arrays.asList(new FtpSimplePatternFileListFilter("*.zip"),
persistentFilter,
oldFilesFilter /*known from previous example*/));
}
Yes, it is. Take a look to the local-filter property and FileSystemPersistentAcceptOnceFileListFilter is for you to track local files via external MetadataStore, e.g. Redis, MongoDb or any other which keeps the data over system restarts.

How to download excel file from web application?

I'm using Wicket (not sure if it matters) but I'm using Workbook to create an excel file for a user to download. But I'm not sure how exactly to do this. What I would like to happen is the user clicks the button, a log is created and a prompt is given to the user to open (and save to temp files) or to save to their computer. The file is then deleted from the server side, or maybe it is stored in the User's session and deleted at end of session.
Can someone point me in the right direction? If I can have the file not saved in the session at all, that'd be create and have it just have it sent to the client using FileOutputStream somehow..
here is my current code:
private void excelCreator()
{
Workbook workbook = new HSSFWorkbook();
Sheet sheet = workbook.createSheet(WorkbookUtil.createSafeSheetName("SSA User ID " + currentSSAIDSelection2.getSsaUserId()));
Iterator<AuditLogEntry> auditLogEntrys = logList.iterator();
int i = 0;
while (auditLogEntrys.hasNext())
{
final AuditLogEntry auditLogEntry = auditLogEntrys.next();
Row row = sheet.createRow(i);
row.createCell(0).setCellValue(auditLogEntry.getTimeStamp());
row.createCell(1).setCellValue(auditLogEntry.getSourceName());
row.createCell(2).setCellValue(auditLogEntry.getCategory());
row.createCell(3).setCellValue(auditLogEntry.getSsaAdmin());
row.createCell(4).setCellValue(auditLogEntry.getAction());
i++;
}
try
{
FileOutputStream output = new FileOutputStream("ssaUserIDAccess.xls");
workbook.write(output);
output.close();
}catch(Exception e)
{
e.printStackTrace();
}
}
You would have to create a DownloadLink with the temporary file as input. The temporary File must be deleted after download (file.delete())).
Alternatively you can try this:
IResourceStream stream = new ByteArrayResourceStream(data, "application/vnd.ms-excel");
RequestCycle.get().scheduleRequestHandlerAfterCurrent(new ResourceStreamRequestHandler(stream, filename).setContentDisposition(ContentDisposition.ATTACHMENT));
In this case data is the byte[] content of your workbook which can be for example retrieved with output.toByteArray().
In case anyone runs into this problem here is my solution. There wasn't a lot of straight forward answers on this but this is my solution:
My excelCreator method handles the creation of the excel Sheet, and returns it as a file.
private File excelCreator()
{
Workbook workbook = new HSSFWorkbook();
File excelfile = new File("userIDAccess.xls");
logList = getServer().findAuditLogs(getUserId(), null);
Sheet sheet = workbook.createSheet(WorkbookUtil.createSafeSheetName("User ID " + getUserId()));
Iterator<AuditLogEntry> auditLogEntrys = logList.iterator();
int i = 0;
while (auditLogEntrys.hasNext())
{
final AuditLogEntry auditLogEntry = auditLogEntrys.next();
Row row = sheet.createRow(i);
row.createCell(0).setCellValue(auditLogEntry.getTimeStamp());
row.createCell(1).setCellValue(auditLogEntry.getSourceName());
row.createCell(2).setCellValue(auditLogEntry.getCategory());
row.createCell(3).setCellValue(auditLogEntry.getSsaAdmin());
row.createCell(4).setCellValue(auditLogEntry.getAction());
i++;
}
try
{
FileOutputStream output = new FileOutputStream(excelfile);
workbook.write(output);
output.close();
}catch(Exception e)
{
e.printStackTrace();
}
return excelfile;
}
IModel excelFileModel = new AbstractReadOnlyModel()
{
public Object getObject()
{
return excelCreator();
}
};
I created an IModel to capture the file created inside my excelCreator() method and returned.
auditDownloadlink = new DownloadLink("auditDownloadlink", excelFileModel);
I pass the I.D. of the download link, and then pass the imodel.
finally,
I call,
auditDownloadlink.setDeleteAfterDownload(true);
auditDownloadlink.setCacheDuration(Duration.NONE);
This deletes the file after it is created. And the cache setting is a setting to make sure it is compatible with all browsers (That's how I interpreted it, but you may not need it).
The Imodel creates the File on the fly so it doesn't have to be stored anywhere, and then the file is deleted once it is downloaded.
Hope this helps someone!
You could create a Resource to do this, and make a ResourceLink.
public class ExcelProducerResource extends AbstractResource
{
public ExcelProducerResource()
{
}
#Override
protected ResourceResponse newResourceResponse( Attributes attributes )
{
final String fileName = getFileName();
ResourceResponse resourceResponse = new ResourceResponse();
resourceResponse.setContentType( "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet" );
resourceResponse.setCacheDuration( Duration.NONE );
resourceResponse.setFileName( fileName );
resourceResponse.setWriteCallback( new WriteCallback()
{
#Override
public void writeData( Attributes attributes ) throws IOException
{
OutputStream outputStream = attributes.getResponse().getOutputStream();
writeToStream( outputStream );
outputStream.close();
}
} );
return resourceResponse;
}
void writeToStream(OutputStream outputStream) throws IOException
{
//.. do stuff here :)
}
String getFileName()
{
//.. do stuff here :)
}
}

SharePoint 2010 folder woes

I've put together a function that creates a sharepoint folder in a document library based on the url that's past in as an argument. The code works and the folder shows up in sharepoint from the webapplication.
However, when I query the SPWeb object for the folder afterward, it says the folder doesnt exist. Which makes no sense to me. Stranger still, is that this very same code worked no too long ago. I had been using it to create tree structures in sharepoint.
Even if the query folder fails, the GetFolder still returns a the folder, but when I add files to the returned folder, I get a runtime exception indicating that the file doesn't exist...which I assume means the folder I am trying to add it to doesn't exist since the file I am adding, doesn't exist yet. Which is why I am adding it.
So my question is, why am I getting this error, and why does FolderExists return false when the folder actually exists? We know it exists because GetFolder actually returns it...
I've included some actual code from the app to make things clear.
If someone could have a look at the code and see and anything jumps out at them, that would be fantabulous...Thanks
Code to build folders:
public void CreateFolder(SPUriBuilder url)
{
try
{
Log.Instance.WriteToLog("CreateFolder({0})", url);
var library = GetLibrary(url.Library);
if (library != null)
{
// parse out string data
//
var parent = library.RootFolder.ServerRelativeUrl;
var segments = url.Account.Split(new char[] { '/' }, StringSplitOptions.RemoveEmptyEntries);
var path = parent;
// get default folder collection
//
SPFolderCollection subFolders = _web.GetFolder(parent).SubFolders;
// check for sub-folders to create
//
if (segments.Length > 0)
{
int i = 0;
do
{
// check for folder and create if non-existant
//
var buildPath = String.Format("{0}/{1}", path, segments[i]);
if (_web.GetFolder(buildPath).Exists == false)
_web.GetFolder(path).SubFolders.Add(segments[i]);
// retrieve new sub-folder collection
//
subFolders = _web.GetFolder(buildPath).SubFolders;
path = buildPath;
// next folder in path
//
i++;
}
while (i < segments.Length);
}
// finally, add folder of interest
//
subFolders.Add(url.Folder);
}
}
catch (Exception e)
{
throw new SPImportException("Exception: {0}, creating folder: {1} in Library: {2}", e.Message, url.Folder, url.Library);
}
}
Code to Query folder:
public bool FolderExists(SPUriBuilder url)
{
return _web.GetFolder(url.Uri.LocalPath).Exists;
}
Code to Get Folder:
private SPFolder GetFolder(SPUriBuilder url)
{
return _web.GetFolder(url.Uri.LocalPath);
}
The SPUriBuilder is a custom class I created to assemble the Uri:
public class SPUriBuilder
{
public string SiteUrl { get; private set; }
public string Library { get; private set; }
public string Parent { get; private set; }
public string Folder { get; private set; }
public string File { get; private set; }
public string Account { get; private set; }
public Uri Uri { get; private set; }
public SPUriBuilder(string siteUrl, string library, string account, string parent, string folder)
{
this.SiteUrl = siteUrl;
this.Library = library;
this.Account = account;
this.Parent = parent.Replace("\\", "/");
this.Parent = this.Parent.StartsWith("/") ? this.Parent.Substring(1) : this.Parent;
this.Folder = folder;
StringBuilder url = new StringBuilder();
url.AppendFormat("{0}/{1}/{2}", SiteUrl, Library, Account);
if (String.IsNullOrEmpty(Parent) == false)
url.AppendFormat("/{0}", Parent);
url.AppendFormat("/{0}", Folder);
this.Uri = new Uri(url.ToString());
}
public SPUriBuilder(SPUriBuilder uri, string file)
: this(uri.SiteUrl, uri.Library, uri.Account, uri.Parent, uri.Folder)
{
this.File = file;
StringBuilder url = new StringBuilder();
url.AppendFormat("{0}/{1}", this.Uri.ToString(), this.File);
this.Uri = new Uri(url.ToString());
}
public override string ToString()
{
return Uri.ToString();
}
}
I found the answer this to this myself. The problem was in the code used to create the folder.
var parent = library.RootFolder.ServerRelativeUrl;
// This line of code is incorrect, so it returned the wrong data, thus building the folder path incorrectly.
//
var segments = url.Account.Split(new char[] { '/' }, StringSplitOptions.RemoveEmptyEntries);
var path = parent;
// This is the replacement line of code that fixed the issue.
//
var segments = url.Uri.LocalPath.Substring(parent.Length+1).Split(new char[] { '/' }, StringSplitOptions.RemoveEmptyEntries);
// as well, this line had to be removed since it was no longer needed
//
// finally, add folder of interest
//
subFolders.Add(url.Folder);
Ultimately the issue turned out be that the folder structure did not exist that I was attempting to create the file in. One or more segments in the path were missing.
So if you ever see this error, make sure you're the folder exists that you are adding the file to. If it isn't, you will certainly experience this error.

Copy folders from Sharepoint by modified date

I need to create a simple program, which goes through a user-given directory on Sharepoint and finds all the folders which are older than 1 month and then it copies them to some local hard drive.
Perhaps it creates some log in a way that this folder was moved to.......
Thanks
Jakub
I wrote this sample code which you can use to understand how it can be done, or you can just use it, because it seems to work fine.
class Program
{
static void Main(string[] args)
{
MoveFolders("your_web_url", "your_doclib_url");
}
public static void MoveFolders(string webUrl, string listUrl)
{
using (SPSite site = new SPSite(webUrl))
{
using (SPWeb web = site.OpenWeb())
{
SPList targetList = web.GetList(web.Url + "/" + listUrl);
MoveFolders(targetList.RootFolder, #"C:\test"); // path to your local storage folder
}
}
}
public static void MoveFolders(SPFolder targetFolder, string rootLocalPath)
{
string currentPath = Path.Combine(rootLocalPath, targetFolder.Name);
if (!Directory.Exists(currentPath))
Directory.CreateDirectory(currentPath);
DateTime lastModified = (DateTime)targetFolder.Properties["vti_timelastmodified"]; //folder last modified date
if (lastModified < DateTime.Today.AddMonths(-1))
SaveFolderLocal(targetFolder, currentPath);
foreach (SPFolder folder in targetFolder.SubFolders)
{
MoveFolders(folder, currentPath);
}
}
public static void SaveFolderLocal(SPFolder folder, string localStoragePath)
{
foreach (SPFile file in folder.Files)
{
var contents = file.OpenBinary();
using (FileStream fileStream = new FileStream(Path.Combine(localStoragePath, file.Name), FileMode.Create))
{
fileStream.Write(contents, 0, contents.Length);
}
}
}
}
This code will save your doclib folder structure locally with contents of any folder modified more than one month ago. Just be careful of using recursive MoveFolders method, because it can cause a StackOverflowException on libraries with very complex folder structure.

File Read/Write Locks

I have an application where I open a log file for writing. At some point in time (while the application is running), I opened the file with Excel 2003, which said the file should be opened as read-only. That's OK with me.
But then my application threw this exception:
System.IO.IOException: The process cannot access the file because another process has locked a portion of the file.
I don't understand how Excel could lock the file (to which my app has write access), and cause my application to fail to write to it!
Why did this happen?
(Note: I didn't observe this behavior with Excel 2007.)
Here is a logger which will take care of sync locks. (You can modify it to fit to your requirements)
using System;
using System.Collections.Generic;
using System.Text;
using System.IO;
namespace Owf.Logger
{
public class Logger
{
private static object syncContoller = string.Empty;
private static Logger _logger;
public static Logger Default
{
get
{
if (_logger == null)
_logger = new Logger();
return _logger;
}
}
private Dictionary<Guid, DateTime> _starts = new Dictionary<Guid, DateTime>();
private string _fileName = "Log.txt";
public string FileName
{
get { return _fileName; }
set { _fileName = value; }
}
public Guid LogStart(string mesaage)
{
lock (syncContoller)
{
Guid id = Guid.NewGuid();
_starts.Add(id, DateTime.Now);
LogMessage(string.Format("0.00\tStart: {0}", mesaage));
return id;
}
}
public void LogEnd(Guid id, string mesaage)
{
lock (syncContoller)
{
if (_starts.ContainsKey(id))
{
TimeSpan time = (TimeSpan)(DateTime.Now - _starts[id]);
LogMessage(string.Format("{1}\tEnd: {0}", mesaage, time.TotalMilliseconds.ToString()));
}
else
throw new ApplicationException("Logger.LogEnd: Key doesn't exisits.");
}
}
public void LogMessage(string message)
{
lock (syncContoller)
{
string filePath = Environment.GetFolderPath(Environment.SpecialFolder.ApplicationData);
if (!filePath.EndsWith("\\"))
filePath += "\\owf";
else
filePath += "owf";
if (!Directory.Exists(filePath))
Directory.CreateDirectory(filePath);
filePath += "\\Log.txt";
lock (syncContoller)
{
using (StreamWriter sw = new StreamWriter(filePath, true))
{
sw.WriteLine(DateTime.Now.ToString("yyyy-MM-dd HH:mm:ss.sss") + "\t" + message);
}
}
}
}
}
}
How do you write the log? Have your own open/close or use some thirty party product?
I thing that the log is opened and locked only when it writes something. Once the data writing is finished, the code closes the file and, of course, releases the lock
This seems like a .NET issue. (Well; a Bug if you ask me).
Basically I have replicated the problem by using the following multi-threaded code:
Dim FS As System.IO.FileStream
Dim BR As System.IO.BinaryReader
Dim FileBuffer(-1) As Byte
If System.IO.File.Exists(FileName) Then
Try
FS = New System.IO.FileStream(FileName, System.IO.FileMode.Open, IO.FileAccess.Read, IO.FileShare.Read)
BR = New System.IO.BinaryReader(FS)
Do While FS.Position < FS.Length
FileBuffer = BR.ReadBytes(&H10000)
If FileBuffer.Length > 0 Then
... do something with the file here...
End If
Loop
BR.Close()
FS.Close()
Catch
ErrorMessage = "Error(" & Err.Number & ") while reading file:" & Err.Description
End Try
Basically, the bug is that trying to READ the file with all different share-modes (READ, WRITE, READ_WRITE) have absolutely no effect on the file locking, no matter what you try; you would always end up in the same result: The is LOCKED and not available for any other user.
Microsoft won't even admit to this problem.
The solution is to use the internal Kernel32 CreateFile APIs to get the proper access done as this would ensure that the OS LISTENs to your request when requesting to read files with a share-locked or locked access.
I believe I'm having the same type of locking issue, reproduced as follows:
User 1 opens Excel2007 file from network (read-write) (WindowsServer, version unkn).
User 2 opens same Excel file (opens as ReadOnly, of course).
User 1 successfully saves file many times
At some point, User 1 is UNABLE to save the file due to message saying "file is locked".
Close down User 2's ReadOnly version...lock is released, and User 1 can now save again.
How could opening the file in ReadOnly mode put a lock on that file?
So, it seems to be either an Excel2007 issue, or a server issue.

Resources