In one of my Azure Web App Web API application, I am creating temp files using this code in a Get method
string path = Path.GetTempFileName();
// do some writing on this file. then read
var fileStream = File.OpenRead(path);
// then returning this stream as a HttpResponseMessage response
My question is, in a managed environment like this (not in VM), do I need to clear those temporary files by myself?
Shouldn't Azure itself supposed to clear those temp files?
Those files only get cleaned when your site is restarted.
If your site is running in Free or Shared mode, it only gets 300MB for temp files, so you could run out if you don't clean up.
If your site is in Basic or Standard mode, then there is significantly more space (around 200GB!). So you could probably get away with not cleaning up without running into the limit. Eventually, your site will get restarted (e.g. during platform upgrade), so things will get cleaned up.
See this page for some additional detail on this topic.
Maybey if you extend FileStream you can override dispose and remove it when disposed is called? That is how i'm resolving it for now. If i'm wrong let me know.
/// <summary>
/// Create a temporary file and removes its when the stream is closed.
/// </summary>
internal class TemporaryFileStream : FileStream
{
public TemporaryFileStream() : base(Path.GetTempFileName(), FileMode.Open)
{
}
protected override void Dispose(bool disposing)
{
base.Dispose(disposing);
// After the stream is closed, remove the file.
File.Delete(Name);
}
}
Following sample demonstrate how to save temp file in azure, both Path and Bolb.
Doc is here:https://code.msdn.microsoft.com/How-to-store-temp-files-in-d33bbb10
Code click here:https://github.com/Azure-Samples/storage-blob-dotnet-store-temp-files/archive/master.zip
Under part is the core logic of bolb code:
private long TotalLimitSizeOfTempFiles = 100 * 1024 * 1024;
private async Task SaveTempFile(string fileName, long contentLenght, Stream inputStream)
{
try
{
await container.CreateIfNotExistsAsync();
CloudBlockBlob tempFileBlob = container.GetBlockBlobReference(fileName);
tempFileBlob.DeleteIfExists();
await CleanStorageIfReachLimit(contentLenght);
tempFileBlob.UploadFromStream(inputStream);
}
catch (Exception ex)
{
if (ex.InnerException != null)
{
throw ex.InnerException;
}
else
{
throw ex;
}
}
}
private async Task CleanStorageIfReachLimit(long newFileLength)
{
List<CloudBlob> blobs = container.ListBlobs()
.OfType<CloudBlob>()
.OrderBy(m => m.Properties.LastModified)
.ToList();
long totalSize = blobs.Sum(m => m.Properties.Length);
long realLimetSize = TotalLimitSizeOfTempFiles - newFileLength;
foreach (CloudBlob item in blobs)
{
if (totalSize <= realLimetSize)
{
break;
}
await item.DeleteIfExistsAsync();
totalSize -= item.Properties.Length;
}
}
Related
The issue happens when I upload a file to one blob (Blob1) which in turn runs a background compression service. The background service streams the file from Blob1, compresses it, and stores it as a zip file in a separate blob (Blob2) to cache for the user to download.
The process works without issue for files < 2GB, but throws a Micrososft.Azure.Storage.StorageException when the file size is > 2GB.
using Microsoft.Azure.Storage.Blob 11.2.2
Sample Code
public async void DoWork(CancellationToken cancellationToken)
{
while (!cancellationToken.IsCancellationRequested)
{
await _messageSemaphore.WaitAsync(cancellationToken);
MyModel model = await _queue.PeekMessage();
if(model != null)
{
try
{
//Get CloudBlockBlob zip blob reference
var zipBlockBlob = await _storageAccount.GetFileBlobReference(_configuration[ConfigKeys.ContainerName], model.Filename, model.FileRelativePath);
using (var zipStream = zipBlockBlob.OpenWrite()) //Opens zipstream
{
using (var archive = new ZipArchive(zipStream, ZipArchiveMode.Create, false)) // Create new ZipArchive
{
//Add each file to the zip archive
foreach (var fileUri in Files)
{
var file = new Uri(fileUri);
var cloudBlockBlob = blobContainer.GetBlockBlobReference(file);
using (var blobStream = cloudBlockBlob.OpenRead())//Opens read stream
{
//Create new ZipEntry
var zipEntry = archive.CreateEntry(model.Filename, CompressionLevel.Fastest);
using (var zipEntryStream = zipEntry.Open())
{
//Zip file
blobStream.CopyTo(zipEntryStream);
}
}
}
}
}
}
catch (FileNotFoundException e)
{
Console.WriteLine($"Download Error: {e.Message}");
}
catch (Microsoft.Azure.Storage.StorageException e) // "
{
Console.WriteLine($"Storage Exception Error: {e.Message}");
}
}
else
{
_messageSemaphore.Release();
// Wait for 1 minute between polls when idle
await Task.Delay(_sleepTime);
}
}
}
The problem was with concurrency and having multiple application instances writing to the same blob at the same time. To get around this I ended up implementing an Azure blob lease on the blob being created but had to create a temp file to apply the lease to. This is a decent enough work around for now, but this could / should probably be implemented using an Azure event driven service.
I am trying to set the proper content type after uploading almost 2000 images , not realizing I had to set their ContentType property. Fortunately I realized that before I moved from the .png files to some other type.
Here is my method:
private static void ChangeImageTypeInAzureStorage()
{
var client = GetAzureClient();
var blobContainer = client.GetContainerReference("accessibleimages");
var list = blobContainer.ListBlobs().OfType<CloudBlockBlob>().ToList();
if (!list.Any()) return; //log no entries returned
try
{
foreach (var item in list)
{
if (Path.GetExtension(item.Uri.AbsoluteUri) == ".png")
{
item.Properties.ContentType = "image/png";
}
item.SetProperties();
}
}
catch (Exception ex)
{
//log exceptions with your own methods
Console.WriteLine(ex);
}
Console.WriteLine("Done... press a key to end.");
Console.ReadKey();
}
I'm not getting why nothing is returned to list. The client and blobContainer are correct. I had no problem uploading those images to the same client blobContainer. Needless to say it fails because the list always has a count of 0.
Any help appreciated.
Well, I found the answer after a lot of googling. The boolean parameter useFlatBoolListing for the ListBlobs method has to be set to true.
var list = blobContainer.ListBlobs(null, true).OfType<CloudBlockBlob>().ToList();
I have an application written with Xamarin.iOS that causes files to be corrupted after they are downloaded into the application.
I am comparing checksums by download the file within the application, and then uploading the downloaded file to dropbox. I am then comparing the checksum of the dropbox file to the original file on the server.
Normally, the checksums match. Occasionally, the checksums do not match. These files are really videos, and it causes the videos to not be playable.
The Question
How can downloads (using HttpWebRequest) occasionally cause corrupted files, but continue to download successfully?
The Code
public class DownloadService : IDownloadService
{
private const int BufferSize = 1024*1024;
/// <summary>
/// Download a remote file to a local location.
/// </summary>
/// <param name="url">The url of the file to download.</param>
/// <param name="localPath">The local path to save the file.</param>
/// <param name="progress">The progress off the download (percentage|bytesread|totalbytes).</param>
/// <param name="cancel">A reference to a boolean that can be set to true if you want to cancel the download.</param>
/// <returns>
/// Returns the local file that was downloaded.
/// Use this instead of the localPath parameter because you may have passed no extension asking for us to autodetermine the extension.
/// </returns>
public string DownloadFile(string url, string localPath, Action<int, long, long> progress, ref bool cancel)
{
cancel = false;
var didCancel = false;
HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create(url);
webRequest.ReadWriteTimeout = 3000;
var buffer = new byte[BufferSize];
var realPath = localPath;
try
{
using (var response = webRequest.GetResponse())
{
// get the total size of this request
var totalSize = 0;
try { totalSize = int.Parse(response.Headers["MediaItemLength"]);
}catch(Exception ex) { throw new NotImplementedException("Get the total size some other way!"); }
// ensure there is a proper extension
if (string.IsNullOrEmpty(Path.GetExtension(localPath)))
{
// the localpath given didn't have an extension.
// we need to try to determine the exention through custom HTTP headers.
string httpHeaderExtension = null;
try
{
httpHeaderExtension = response.Headers["MediaExtension"];
if (string.IsNullOrEmpty(httpHeaderExtension))
throw new Exception("The header was present, but it was empty.");
}
catch (Exception ex)
{
throw new Exception("localPath didn't provide an extension and we couldn't infer would through custom HTTP headers.");
}
// add the correct extension to our local file path
realPath += httpHeaderExtension;
}
if(File.Exists(realPath))
File.Delete(realPath);
using (var responseStream = response.GetResponseStream())
{
using (var dest = new FileStream(realPath, FileMode.CreateNew, FileAccess.Write))
{
long totalBytes = 0;
int currentBlockSize;
while ((currentBlockSize = responseStream.Read(buffer, 0, buffer.Length)) > 0)
{
totalBytes += currentBlockSize;
var percentage = (int)(totalBytes * 100.0 / totalSize);
dest.Write(buffer, 0, currentBlockSize);
if (progress != null)
progress(percentage, totalBytes, totalSize);
if (cancel)
{
didCancel = true;
break;
}
}
}
}
}
}catch(Exception ex)
{
// delete the file
if (File.Exists(realPath))
File.Delete(realPath);
throw;
}
// delete the file
if (didCancel && File.Exists(realPath))
File.Delete(realPath);
return realPath;
}
}
The code is pretty straight forward. I do include some additional/custom headers for getting the total file size so that I can avoid a HEAD request to get the value. I need the total file size for download progress.
I want to download the files from a sharepoint document library through code as there are thousand of files in the document library.
I am thinking of creating console application, which I will run on sharepoint server and download files. Is this approach correct or, there is some other efficient way to do this.
Any help with code will be highly appreciated.
Like SigarDave said, it's perfectly possible to achieve this without writing a single line of code. But if you really want to code the solution for this, it's something like:
static void Main(string[] args)
{
// Change to the URL of your site
using (var site = new SPSite("http://MySite"))
using (var web = site.OpenWeb())
{
var list = web.Lists["MyDocumentLibrary"]; // Get the library
foreach (SPListItem item in list.Items)
{
if (item.File != null)
{
// Concat strings to get the absolute URL
// to pass to an WebClient object.
var fileUrl = string.Format("{0}/{1}", site.Url, item.File.Url);
var result = DownloadFile(fileUrl, "C:\\FilesFromMyLibrary\\", item.File.Name);
Console.WriteLine(result ? "Downloaded \"{0}\"" : "Error on \"{0}\"", item.File.Name);
}
}
}
Console.ReadKey();
}
private static bool DownloadFile(string url, string dest, string fileName)
{
var client = new WebClient();
// Change the credentials to the user that has the necessary permissions on the
// library
client.Credentials = new NetworkCredential("Username", "Password", "Domain");
var bytes = client.DownloadData(url);
try
{
using (var file = File.Create(dest + fileName))
{
file.Write(bytes, 0, bytes.Length); // Write file to disk
return true;
}
}
catch (Exception)
{
return false;
}
}
another way without using any scripts is by opening the document library using IE then in the ribbon you can click on Open in File Explorer where you can then drag and drop the files right on your desktop!
I'm making a service that among other has the "photo albums" feature that serve photos to users. User has to be "allowed" to see the photo from the album. So sending the direct link to other person shouldn't allow to view photo.
Photos are stored in the folder outside of the context.
What I need to do is to perform some checks when user requests the photo and then if checks are OK - serve the file. I want to avoid making a wheel and just let tomcat serve the image as it usually does for static files. Can you give some advice on that?
Ok, guys.
After struggling hard with this question I think I've finally found out what to do to solve it. First of all it looks like the question actually decomposes into two independent tasks. One of them is securing access to some resources and second one is feeding resources from the folder outside of the context.
First task is trivial and can be solved by writing a simple filter hanged to "/".
Second task is much less trivial but fortunately also can be resolved. Tomcat uses the implementation of javax.naming.directory.DirContext to load all resources of the given web application including class files. It also allows you to provide the custom implementation of this interface and configure it in the context.xml file. The default implementation is org.apache.naming.resources.FileDirContext. Details here: http://tomcat.apache.org/tomcat-6.0-doc/config/resources.html
I've created my own implementation of DirContext by simply extending FileDirContext. Luckily enough there was a single method that had to be overwritten in order to "hook up" file discovery. The method is called file().
I'm posting my test code here. It is far from perfect and does not take into account the corner cases like renaming files but I don't think that these are needed under a normal run of the server.
The basic idea under this code is to check if the path starts with "virtual directory" prefix and if it is - search for file in the other place in the filesystem (I know there is some duplicate code there but I hope you're not that lazy to remove it if you ever want to use it :-). setVirtualName and setVirtualBase are called automatically to inject the configuration params.
/**
* TODO: add javadocs
*
* #author Juriy Bura
*/
public class VirtualFolderDirContext extends FileDirContext {
private String virtualName;
private String realName;
private File virtualBase;
private String absoluteVirtualBase;
public VirtualFolderDirContext() {
super();
}
public VirtualFolderDirContext(Hashtable env) {
super(env);
}
public void setVirtualName(String path) {
virtualName = path;
}
public void setVirtualBase(String base) {
this.realName = base;
virtualBase = new File(realName);
try {
virtualBase = virtualBase.getCanonicalFile();
} catch (IOException e) {
// Ignore
}
this.absoluteVirtualBase = virtualBase.getAbsolutePath();
}
protected File file(String name) {
File file = null;
boolean virtualFile = name.startsWith(virtualName + "/");
if (virtualFile) {
file = new File(virtualBase, name.substring(virtualName.length()));
} else {
file = new File(base, name);
}
if (file.exists() && file.canRead()) {
if (allowLinking)
return file;
// Check that this file belongs to our root path
String canPath = null;
try {
canPath = file.getCanonicalPath();
} catch (IOException e) {
}
if (canPath == null)
return null;
// Check to see if going outside of the web application root
if (!canPath.startsWith(absoluteBase) && !canPath.startsWith(absoluteVirtualBase)) {
return null;
}
// Case sensitivity check
if (caseSensitive) {
String fileAbsPath = file.getAbsolutePath();
if (fileAbsPath.endsWith("."))
fileAbsPath = fileAbsPath + "/";
String absPath = normalize(fileAbsPath);
if (canPath != null)
canPath = normalize(canPath);
if (virtualFile) {
if ((absoluteVirtualBase.length() < absPath.length())
&& (absoluteVirtualBase.length() < canPath.length())) {
absPath = absPath.substring(absoluteVirtualBase.length() + 1);
if ((canPath == null) || (absPath == null))
return null;
if (absPath.equals(""))
absPath = "/";
canPath = canPath.substring(absoluteVirtualBase.length() + 1);
if (canPath.equals(""))
canPath = "/";
if (!canPath.equals(absPath))
return null;
}
} else {
if ((absoluteBase.length() < absPath.length())
&& (absoluteBase.length() < canPath.length())) {
absPath = absPath.substring(absoluteBase.length() + 1);
if ((canPath == null) || (absPath == null))
return null;
if (absPath.equals(""))
absPath = "/";
canPath = canPath.substring(absoluteBase.length() + 1);
if (canPath.equals(""))
canPath = "/";
if (!canPath.equals(absPath))
return null;
}
}
}
} else {
return null;
}
return file;
}
}
After you have this class in place you have to jar it and put that jar into the Tomcat lib folder. For obvious reasons it cannot go together with war file. In your context.xml you should add a config lines like these:
<?xml version="1.0" encoding="UTF-8"?>
<Context antiResourceLocking="true" antiJARLocking="true">
<Resources
className="com.juriy.tomcat.virtualdir.VirtualFolderDirContext"
virtualName="/upload"
virtualBase="c:/temp/up">
</Resources>
...
...
Now any time user asks for /upload/ it will be resolved to c:\temp. With this technique you can implement loading resources from virtually any location: http, shared folder, database, even version control system. So it is pretty cool.
P.S. I've killed the whole day to make this all work together so don't hesitate to give me your vote if you like the answer :-))
Cheers
Juriy