File Read/Write Locks - excel

I have an application where I open a log file for writing. At some point in time (while the application is running), I opened the file with Excel 2003, which said the file should be opened as read-only. That's OK with me.
But then my application threw this exception:
System.IO.IOException: The process cannot access the file because another process has locked a portion of the file.
I don't understand how Excel could lock the file (to which my app has write access), and cause my application to fail to write to it!
Why did this happen?
(Note: I didn't observe this behavior with Excel 2007.)

Here is a logger which will take care of sync locks. (You can modify it to fit to your requirements)
using System;
using System.Collections.Generic;
using System.Text;
using System.IO;
namespace Owf.Logger
{
public class Logger
{
private static object syncContoller = string.Empty;
private static Logger _logger;
public static Logger Default
{
get
{
if (_logger == null)
_logger = new Logger();
return _logger;
}
}
private Dictionary<Guid, DateTime> _starts = new Dictionary<Guid, DateTime>();
private string _fileName = "Log.txt";
public string FileName
{
get { return _fileName; }
set { _fileName = value; }
}
public Guid LogStart(string mesaage)
{
lock (syncContoller)
{
Guid id = Guid.NewGuid();
_starts.Add(id, DateTime.Now);
LogMessage(string.Format("0.00\tStart: {0}", mesaage));
return id;
}
}
public void LogEnd(Guid id, string mesaage)
{
lock (syncContoller)
{
if (_starts.ContainsKey(id))
{
TimeSpan time = (TimeSpan)(DateTime.Now - _starts[id]);
LogMessage(string.Format("{1}\tEnd: {0}", mesaage, time.TotalMilliseconds.ToString()));
}
else
throw new ApplicationException("Logger.LogEnd: Key doesn't exisits.");
}
}
public void LogMessage(string message)
{
lock (syncContoller)
{
string filePath = Environment.GetFolderPath(Environment.SpecialFolder.ApplicationData);
if (!filePath.EndsWith("\\"))
filePath += "\\owf";
else
filePath += "owf";
if (!Directory.Exists(filePath))
Directory.CreateDirectory(filePath);
filePath += "\\Log.txt";
lock (syncContoller)
{
using (StreamWriter sw = new StreamWriter(filePath, true))
{
sw.WriteLine(DateTime.Now.ToString("yyyy-MM-dd HH:mm:ss.sss") + "\t" + message);
}
}
}
}
}
}

How do you write the log? Have your own open/close or use some thirty party product?
I thing that the log is opened and locked only when it writes something. Once the data writing is finished, the code closes the file and, of course, releases the lock

This seems like a .NET issue. (Well; a Bug if you ask me).
Basically I have replicated the problem by using the following multi-threaded code:
Dim FS As System.IO.FileStream
Dim BR As System.IO.BinaryReader
Dim FileBuffer(-1) As Byte
If System.IO.File.Exists(FileName) Then
Try
FS = New System.IO.FileStream(FileName, System.IO.FileMode.Open, IO.FileAccess.Read, IO.FileShare.Read)
BR = New System.IO.BinaryReader(FS)
Do While FS.Position < FS.Length
FileBuffer = BR.ReadBytes(&H10000)
If FileBuffer.Length > 0 Then
... do something with the file here...
End If
Loop
BR.Close()
FS.Close()
Catch
ErrorMessage = "Error(" & Err.Number & ") while reading file:" & Err.Description
End Try
Basically, the bug is that trying to READ the file with all different share-modes (READ, WRITE, READ_WRITE) have absolutely no effect on the file locking, no matter what you try; you would always end up in the same result: The is LOCKED and not available for any other user.
Microsoft won't even admit to this problem.
The solution is to use the internal Kernel32 CreateFile APIs to get the proper access done as this would ensure that the OS LISTENs to your request when requesting to read files with a share-locked or locked access.

I believe I'm having the same type of locking issue, reproduced as follows:
User 1 opens Excel2007 file from network (read-write) (WindowsServer, version unkn).
User 2 opens same Excel file (opens as ReadOnly, of course).
User 1 successfully saves file many times
At some point, User 1 is UNABLE to save the file due to message saying "file is locked".
Close down User 2's ReadOnly version...lock is released, and User 1 can now save again.
How could opening the file in ReadOnly mode put a lock on that file?
So, it seems to be either an Excel2007 issue, or a server issue.

Related

Is disposing SXSSFWorkbook necessary when used in try with resource

Below is the sample code snippet to create SXSSFWorkbook:
try(SXSSFWorkbook wb = new SXSSFWorkbook()) {
//...
} finally {
wb.dispose(); //wb not accessible over here, so can't use try with resource
}
Here problem is that if I use try with resource then can't dispose() SXSSFWorkbook in finally, as variable wb won't be accessible in finally block.
I wanted know that is disposing of workbook necessary to delete temporary files or since SXSSFWorkbook is AutoCloseable, try with resource will take care of it.
Not sure whether someone of the apache poi programmers will answering this. But apache poi is open source. So every programmer can answering this itself by looking at the code.
State May 2018, apache poiversion 3.17.
SXSSFWorkbook.java:
public class SXSSFWorkbook implements Workbook
So why can this be a resource for using in try with resource? Because
Workbook.java:
public interface Workbook extends Closeable, Iterable<Sheet>
So org.apache.poi.ss.usermodel.Workbook extends java.io.Closeable and so classes which implements this must providing a method close.
SXSSFWorkbook.close
As you see, the single SheetDataWriters will be closed and then the internally XSSFWorkbook _wb will be closed.
SheetDataWriter.close
SheetDataWriter.close only flushes and closes the Writer _out.
So no, nowhere the dispose is called while auto closing until now (May 2018) in apache poiversion 3.17
And only SheetDataWriter.dispose will deleting the TempFile _fd created for each sheet.
This is a forrmal resolution of the problem.
SXSSFWorkbook t_wb = null;
try(SXSSFWorkbook wb = t_wb = new SXSSFWorkbook()) {
//...
} finally {
if(t_wb != null) t_wb.dispose();
}
This question bothers me too, so my solution is to override the close method, like this:
//a utility method somewhere
Workbook createMyCustomWorkbook() {
return new SXSSFWorkbook() {
public void close() throws IOException {
try {
dispose();
} catch (Exception e) {
//some logging
}
super.close();
}
};
}
//use in a simple try catch block
try(Workbook wb = createMyCustomWorkbook())
//do stuff with wb
}

How to debug FileSystemWatcher c#

Can someone explain how I debug this?
I have built a windows service which monitors a folder and when a file is created in that folder, it moves that newly created file to another folder. Pretty simple and works. I am trying to pad it out now with out features and I'm starting to get generic IOExpections thrown in event viewer, so I want to try and debug. But the issue I am having is when to create the new file in the directory during debugging.
This is my code so far
which line should I stop at during stepping through, place my new file in the directory and then continue debugging so that it picks up the created file?
public void OnDebug()
{
OnStart(null);
}
[PermissionSet(SecurityAction.Demand, Name = "FullTrust")]
protected override void OnStart(string[] args)
{
string pathToWatch = ConfigurationManager.AppSettings["DirectoryToWatch"];
FileSystemWatcher watcher = new FileSystemWatcher();
watcher.Path = pathToWatch;
watcher.NotifyFilter = NotifyFilters.LastAccess | NotifyFilters.LastWrite | NotifyFilters.FileName;
watcher.Created += new FileSystemEventHandler(FileCreated);
watcher.EnableRaisingEvents = true;
}
private void FileCreated(object source, FileSystemEventArgs e)
{
try
{
DateTime dt = File.GetCreationTime(e.FullPath);
File.Move(e.FullPath.ToString(), ConfigurationManager.AppSettings["DirectoryToMoveTo"] + e.Name + dt.ToString());
LogEvent($"New file found and moved \n {e.FullPath.ToString()}");
}
catch (IOException ex)
{
LogEvent(ex.ToString());
}
}
You can start by adding a breakpoint inside the function FileCreated.
I suggest you add a breakpoint at
DateTime dt = File.GetCreationTime(e.FullPath);
When you create a new file in the directory being watched, the FileSystemWatcher gets notified and the function FileCreated is called.
I also suggest you to log the destination path for your move. Just to make sure it's a valid path.

UWP apps accessing files from random location on system

in UWP there are files and permissions restrictions, so we can only acces files directly from few folders or we can use filepicker to access from anywhere on system.
how can I use the files picked from filepicker and use them anytime again when the app runs ? tried to use them again by path but it gives permission error. I know about the "futureacceslist" but its limit is 1000 and also it will make the app slow if I am not wrong? .
Is there a better way to do this ? or can we store storage files link somehow in local sqlite database?
If you need to access lots of files, asking the user to select the parent folder and then storing that is probably a better solution (unless you want to store 1,000 individually-picked files from different locations). You can store StorageFolders in the access list as well.
I'm not sure why you think it will make your app slow, but the only real way to know if this will affect your performance is to try it and measure against your goals.
Considering this method..
public async static Task<byte[]> ToByteArray(this StorageFile file)
{
byte[] fileBytes = null;
using (IRandomAccessStreamWithContentType stream = await file.OpenReadAsync())
{
fileBytes = new byte[stream.Size];
using (DataReader reader = new DataReader(stream))
{
await reader.LoadAsync((uint)stream.Size);
reader.ReadBytes(fileBytes);
}
}
return fileBytes;
}
This class..
public class AppFile
{
public string FileName { get; set; }
public byte[] ByteArray { get; set; }
}
And this variable
List<AppFile> _appFiles = new List<AppFile>();
Just..
var fileOpenPicker = new FileOpenPicker();
IReadOnlyList<StorageFile> files = await fileOpenPicker.PickMultipleFilesAsync();
foreach (var file in files)
{
var byteArray = await file.ToByteArray();
_appFiles.Add(new AppFile { FileName = file.DisplayName, ByteArray = byteArray });
}
UPDATE
using Newtonsoft.Json;
using System.Linq;
using Windows.Security.Credentials;
using Windows.Storage;
namespace Your.Namespace
{
public class StateService
{
public void SaveState<T>(string key, T value)
{
var localSettings = ApplicationData.Current.LocalSettings;
localSettings.Values[key] = JsonConvert.SerializeObject(value);
}
public T LoadState<T>(string key)
{
var localSettings = ApplicationData.Current.LocalSettings;
if (localSettings.Values.ContainsKey(key))
return JsonConvert.DeserializeObject<T>(((string) localSettings.Values[key]));
return default(T);
}
public void RemoveState(string key)
{
var localSettings = ApplicationData.Current.LocalSettings;
if (localSettings.Values.ContainsKey(key))
localSettings.Values.Remove((key));
}
public void Clear()
{
ApplicationData.Current.LocalSettings.Values.Clear();
}
}
}
A bit late, but, yes the future access list will slow down your app in that it returns storagfile, storagefolder, or storeageitem objects. These run via the runtime broker which hits a huge performance barrier at about 400 objects regardless of the host capability

Custom FileResult on Azure: Browser Waits forever

I have an action that returns an Excel as a custom FileResult. My solution is based on the ClosedXml library (internaly using OpenXml).
My XlsxResult class uses a read-only .xlsx file on the server as a template. It then passes on the template into a memory stream that gets manipulated and saved back with ClosedXml. In the end the memory stream get written to the response.
This works fine both on Cassini as well as IIS Express but fails when deployed on azure with no error whatsoever. The only effect I am experiencing is the request sent to the server never gets any response. I am still waiting for something to happen after 60 minutes or so...
My action:
[OutputCache(Location= System.Web.UI.OutputCacheLocation.None, Duration=0)]
public FileResult Export(int year, int month, int day) {
var date = new DateTime(year, month, day);
var filename = string.Format("MyTemplate_{0:yyyyMMdd}.xlsx", date);
//return new FilePathResult("~/Content/templates/MyTemplate.xlsx", "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet");
var result = new XlsxExportTemplatedResult("MyTemplate.xlsx", filename, (workbook) => {
var ws = workbook.Worksheets.Worksheet("My Export Sheet");
ws.Cell("B3").Value = date;
// Using a OpenXML's predefined formats (15 stands for date)
ws.Cell("B3").Style.NumberFormat.NumberFormatId = 15;
ws.Columns().AdjustToContents(); // You can also specify the range of columns to adjust, e.g.
return workbook;
});
return result;
}
My FileResult
public class XlsxExportTemplatedResult : FileResult
{
// default buffer size as defined in BufferedStream type
private const int BufferSize = 0x1000;
public static readonly string TEMPLATE_FOLDER_LOCATION = #"~\Content\templates";
public XlsxExportTemplatedResult(string templateName, string fileDownloadName, Func<XLWorkbook, XLWorkbook> generate)
: base("application/vnd.openxmlformats-officedocument.spreadsheetml.sheet") {
this.TempalteName = templateName;
this.FileDownloadName = fileDownloadName;
this.Generate = generate;
}
public string TempalteName { get; protected set; }
public Func<XLWorkbook, XLWorkbook> Generate { get; protected set; }
protected string templatePath = string.Empty;
public override void ExecuteResult(ControllerContext context) {
templatePath = context.HttpContext.Server.MapPath(System.IO.Path.Combine(TEMPLATE_FOLDER_LOCATION, this.TempalteName));
base.ExecuteResult(context);
}
//http://msdn.microsoft.com/en-us/library/office/ee945362(v=office.11).aspx
protected override void WriteFile(System.Web.HttpResponseBase response) {
FileStream fileStream = new FileStream(templatePath, FileMode.Open, FileAccess.Read);
using (MemoryStream memoryStream = new MemoryStream()) {
CopyStream(fileStream, memoryStream);
using (var workbook = new XLWorkbook(memoryStream)) {
Generate(workbook);
workbook.Save();
}
// At this point, the memory stream contains the modified document.
// grab chunks of data and write to the output stream
Stream outputStream = response.OutputStream;
byte[] buffer = new byte[BufferSize];
while (true) {
int bytesRead = memoryStream.Read(buffer, 0, BufferSize);
if (bytesRead == 0) {
// no more data
break;
}
outputStream.Write(buffer, 0, bytesRead);
}
}
fileStream.Dispose();
}
static private void CopyStream(Stream source, Stream destination) {
byte[] buffer = new byte[BufferSize];
int bytesRead;
do {
bytesRead = source.Read(buffer, 0, buffer.Length);
destination.Write(buffer, 0, bytesRead);
} while (bytesRead != 0);
}
}
So am I missing something (apparently I am).
Please Note:
There are no dlls missing from Azure because I checked using RemoteAccess feature of the Windows Azure Tools 1.7
My export is not a heavy long running task.
when I changed the action to just return a FilePathResult with the template xlsx it worked on azure. But I need to process the file before returning it as u might suspect :-)
Tanks.
UPDATE:
After I logged extensively in my code the execution hangs with no error at the ClosedXml "Save" method call. But still no error. Abstract from the WADLogsTable:
Opening template file from path:
E:\sitesroot\0\Content\templates\MyTemplate.xlsx
Opened template from path:
E:\sitesroot\0\Content\templates\MyTemplate.xlsx just
copied template to editable memory stream. Bytes copied: 15955,
Position: 15955
modified the excel document in memory.
here it hangs when a it calls to workbook.Save(); This is a ClosedXml method call.
I was facing the exact same error situation as you. I can't offer a fix in your specific situation, and I know you switched tracks, but after going through the same frustrating steps you had faced, I'd like to "pave the way" for an answer for you (or others).
Drop into your package manager console in Visual Studio and install Elmah with the MVC goodies (routing):
Install-Package elmah.MVC
Now, in your root web.config, update your Elmah entry. It's likely at the end of the file, looking like this:
<elmah></elmah>
Update that bad boy to allow remote access and set up your log path:
<elmah>
<security allowRemoteAccess="1" />
<errorLog type="Elmah.XmlFileErrorLog, Elmah" logPath="~/app_data/elmah" />
</elmah>
Now, push that up to Azure.
Finally, visit your site, force the error then navigate to http://your-site-here.azurewebsites.net/elmah and you'll see the exact cause of the error.
Elmah is so the awesome.
Sheepish confession: The error for me wasn't in the third party code, it turned out to be in my connection string, for which I hadn't set MultipleActiveResultsSets to true. The other fix I had to do was pass my entities in after calling ToList() to one of the internal methods on that library, leaving it as IQueryable borked the method up.

Streamwriter, StringBuilder and Parallel loops

Sorry for big chunk of code, I couldn't explain that with less.Basically I'm trying to write into a file from many tasks.
Can you guys please tell me what I'm doing wrong? _streamWriter.WriteLine() throws the ArgumentOutOfRangeException.
class Program
{
private static LogBuilder _log = new LogBuilder();
static void Main(string[] args)
{
var acts = new List<Func<string>>();
var rnd = new Random();
for (int i = 0; i < 10000; i++)
{
acts.Add(() =>
{
var delay = rnd.Next(300);
Thread.Sleep(delay);
return "act that that lasted "+delay;
});
}
Parallel.ForEach(acts, act =>
{
_log.Log.AppendLine(act.Invoke());
_log.Write();
});
}
}
public class LogBuilder : IDisposable
{
public StringBuilder Log = new StringBuilder();
private FileStream _fileStream;
private StreamWriter _streamWriter;
public LogBuilder()
{
_fileStream = new FileStream("log.txt", FileMode.Create, FileAccess.ReadWrite, FileShare.ReadWrite);
_streamWriter = new StreamWriter(_fileStream) { AutoFlush = true };
}
public void Write()
{
lock (Log)
{
if (Log.Length <= 0) return;
_streamWriter.WriteLine(Log.ToString()); //throws here. Although Log.Length is greater than zero
Log.Clear();
}
}
public void Dispose()
{
_streamWriter.Close(); _streamWriter.Dispose(); _fileStream.Close(); fileStream.Dispose();
}
}
This is not a bug in StringBuilder, it's a bug in your code. And the modification you shown in your followup answer (where you replace Log.String with a loop that extracts characters one at a time) doesn't fix it. It won't throw an exception any more, but it won't work properly either.
The problem is that you're using the StringBuilder in two places in your multithreaded code, and one of them does not attempt to lock it, meaning that reading can occur on one thread simultaneously with writing occurring on another. In particular, the problem is this line:
_log.Log.AppendLine(act.Invoke());
You're doing that inside your Parallel.ForEach. You are not making any attempt at synchronization here, even though this will run on multiple threads at once. So you've got two problems:
Multiple calls to AppendLine may be in progress simultaneously on multiple threads
One thread may attempt to be calling Log.ToString at the same time as one or more other threads are calling AppendLine
You'll only get one read at a time because you are using the lock keyword to synchronize those. The problem is that you're not also acquiring the same lock when calling AppendLine.
Your 'fix' isn't really a fix. You've succeeded only in making the problem harder to see. It will now merely go wrong in different and more subtle ways. For example, I'm assuming that your Write method still goes on to call Log.Clear after your for loop completes its final iteration. Well in between completing that final iteration, and making the call to Log.Clear, it's possible that some other thread will have got in another call to AppendLine because there's no synchronization on those calls to AppendLine.
The upshot is that you will sometimes miss stuff. Code will write things into the string builder that then get cleared out without ever being written to the stream writer.
Also, there's a pretty good chance of concurrent AppendLine calls causing problems. If you're lucky they will crash from time to time. (That's good because it makes it clear you have a problem to fix.) If you're unlucky, you'll just get data corruption from time to time - two threads may end up writing into the same place in the StringBuilder resulting either in a mess, or completely lost data.
Again, this is not a bug in StringBuilder. It is not designed to support being used simultaneously from multiple threads. It's your job to make sure that only one thread at a time does anything to any particular instance of StringBuilder. As the documentation for that class says, "Any instance members are not guaranteed to be thread safe."
Obviously you don't want to hold the lock while you call act.Invoke() because that's presumably the very work you want to parallelize. So I'd guess something like this might work better:
string result = act();
lock(_log.Log)
{
_log.Log.AppendLine(result);
}
However, if I left it there, I wouldn't really be helping you, because this looks very wrong to me.
If you ever find yourself locking a field in someone else's object, it's a sign of a design problem in your code. It would probably make more sense to modify the design, so that the LogBuilder.Write method accepts a string. To be honest, I'm not even sure why you're using a StringBuilder here at all, as you seem to use it just as a holding area for a string that you immediately write to a stream writer. What were you hoping the StringBuilder would add here? The following would be simpler and doesn't seem to lose anything (other than the original concurrency bugs):
public class LogBuilder : IDisposable
{
private readonly object _lock = new object();
private FileStream _fileStream;
private StreamWriter _streamWriter;
public LogBuilder()
{
_fileStream = new FileStream("log.txt", FileMode.Create, FileAccess.ReadWrite, FileShare.ReadWrite);
_streamWriter = new StreamWriter(_fileStream) { AutoFlush = true };
}
public void Write(string logLine)
{
lock (_lock)
{
_streamWriter.WriteLine(logLine);
}
}
public void Dispose()
{
_streamWriter.Dispose(); fileStream.Dispose();
}
}
I think the cause is because you are accessing the stringBuilder in the Parellel bracket
_log.Log.AppendLine(act.Invoke());
_log.Write();
and inside the LogBuilder you perform lock() to disallow memory allocation on stringBuidler. You are changing the streamwriter to handle the log in every character so would give the parellel process to unlock the memory allocation to stringBuilder.
Segregate the parallel process into distinct action would likely reduce the problem
Parallel.ForEach(acts, act =>
{
_log.Write(act.Invoke());
});
in the LogBuilder class
private readonly object _lock = new object();
public void Write(string logLines)
{
lock (_lock)
{
//_wr.WriteLine(logLines);
Console.WriteLine(logLines);
}
}
An alternate approach is to use TextWriter.Synchronized to wrap StreamWriter.
void Main(string[] args)
{
var rnd = new Random();
var writer = new StreamWriter(#"C:\temp\foo.txt");
var syncedWriter = TextWriter.Synchronized(writer);
var tasks = new List<Func<string>>();
for (int i = 0; i < 1000; i++)
{
int local_i = i; // get a local value, not closure-reference to i
tasks.Add(() =>
{
var delay = rnd.Next(5);
Thread.Sleep(delay);
return local_i.ToString() + " act that that lasted " + delay.ToString();
});
}
Parallel.ForEach(tasks, task =>
{
var value = task();
syncedWriter.WriteLine(value);
});
writer.Dispose();
}
Here are some of the synchronization helper classes
http://referencesource.microsoft.com/#q=Synchronized
System.Collections
static ArrayList Synchronized(ArrayList list)
static IList Synchronized(IList list)
static Hashtable Synchronized(Hashtable table)
static Queue Synchronized(Queue queue)
static SortedList Synchronized(SortedList list)
static Stack Synchronized(Stack stack)
System.Collections.Generic
static IList Synchronized(List list)
System.IO
static Stream Synchronized(Stream stream)
static TextReader Synchronized(TextReader reader)
static TextWriter Synchronized(TextWriter writer)
System.Text.RegularExpressions
static Match Synchronized(Match inner)
static Group Synchronized(Group inner)
It is seems that it isn't problem of Parallelism. It's StringBuilder's problem.
I have replaced:
_streamWriter.WriteLine(Log.ToString());
with:
for (int i = 0; i < Log.Length; i++)
{
_streamWriter.Write(Log[i]);
}
And it worked.
For future reference: http://msdn.microsoft.com/en-us/library/system.text.stringbuilder(v=VS.100).aspx
Memory allocation section.

Resources