I have a project ASP.Net MVC classics which need to be migrated to Azure and host in AppServices. Currently this project save files in the root folder and file size could be 2GB.
Now the question is should I leave the current logic to store the file in wwwroot folder as:\wwwroot\Files\myfile.txt"; or should I store it in the blob?
I am looking for the best practice and do not want to change the current logic? Can someone give me the idea?
Thanks
Storing files in Azure Blob Storage:
According to Documentation it says,
Azure Blob Storage enables the creation of data lakes for analytics purposes and provides storage for the development of powerful cloud-native and mobile apps. Reduce costs by using tiered storage for long-term data and scalability for high-performance computing and machine learning workloads.
According to Documentation it says,
SAS enables you to securely upload and download files from Azure Blob Storage without having to share the connection string.
While uploading you can split large data in small amounts with which we can decrease uploading time and after uploading then it again combines into a single blob.
Storing file in wwwroot folder:
According to Documentation it says
Static resource files are stored in web root, The default directory is {content root}/wwwroot folder.
According to Documentation we can clearly say that capacity depends on price tier.
Web apps performance can be effected by uploading large files.
In Azure linux web apps, if the size is around 2 Gb then it might lead to timeoutException.
Related
My application till this day was working with local zip files,
meaning I was using a direct return new FileStream()
in the application and the local zip file that was located on the SDD/Network drive path (zip files can be hundreds of GB).
I configured the application to work with Azure Blob Storage, meaning each FileStream that was returned in now return as the Azure Blob SDK method:
GetBlobStreamAsync(ContainerName, BlobName).ConfigureAwait(false).GetAwaiter().GetResult()
I uploaded some zip files to a container in the blob storage and set the connection string in the application to work with that storage account.
The application was deployed and running on a virtual windows machine located in the same region of the Azure Storage Blob.
Note: This is a private cloud network.
When the app is streaming the zip file on Azure blob storage it seems that the performance has decreased by at least 8-9 times (problematic with hundreds of GB).
Speed comparison is between local C: drive on the same windows virtual machine that the application is running on an Azure Storage account which is located in the same region.
Note: NW Bandwidth - is 50 GB on the VM on azure
Solutions that I tried:
Azure blob Premium Performance storage - Didn’t improve performance
.Net Core - advantage of performance enhancements (we work with .Net framework so this is irrelevant).
Network File System (NFS) 3.0 performance considerations in Azure Blob storage - (Does not work with private cloud).
Hot, Cool, and Archive access tiers for blob data - The default is Hot so we already tried this scenario with no improvements.
Solutions I want to try:
Azure Files Share Storage as a cache solution
.Net Framework configuration - lists several quick configuration settings that you can use to make significant performance improvements
Question:
Does anyone have any suggestions on how can I optimize the streaming in front of the Azure Storage Blob?
Azure Files (share) or Storage Blob services are likely not the right services to be utilized for this scenario. There are two possible paths:
Break a single file into multiple files and leverage Storage Blob service that handles throughput better than Azure Files. Azure Files performs better with small(er) files which are typical to user documents (PDFs, Word, Excel, etc.)
Switch over to a more dedicated service that is designed specifically for large-size data transfer if breaking up a single file into multiple blobs is not an option.
The recommendation for each option will highly depend on the implementation details, requirements and constraints of the system.
I want to spike whether azure and the cloud is a good fit for us.
We have a website where users upload documents to our currently hosted website.
Every document has an equivalent record in a database.
I am using terraform to create the azure infrastructure.
What is my best way of migrating the documents from the local file path on the server to azure?
Should I be using file storage or blob storage. I am confused about the difference.
Is there anything in terraform that can help with this?
Based on your comments, I would recommend storing them in Blob Storage. This service is suited for storing and serving unstructured data like files and images. There are many other features like redundancy, archiving etc. that you may find useful in your scenario.
File Storage is more suitable in Lift-and-Shift kind of scenarios where you're moving an on-prem application to the cloud and the application writes data to either local or network attached disk.
You may also find this article useful: https://learn.microsoft.com/en-us/azure/storage/common/storage-decide-blobs-files-disks
UPDATE
Regarding uploading files from local computer to Azure Storage, there are actually many options available:
Use a Storage Explorer like Microsoft's Storage Explorer.
Use AzCopy command-line tool.
Use Azure PowerShell Cmdlets.
Use Azure CLI.
Write your own code using any available Storage Client libraries or directly consuming REST API.
I have over 500 large image files that I need to process in my .NET Core app hosted in an Azure App Service. That said, I need to download all of the images and run them through a machine learning categorization function in my code. I currently use blob storage as my mechanism for storing the images, but downloading all those images via blob rest api is slow. Is there a better architecture in Azure that I should be making use of to greatly increase performance of processing these images? Perhaps a storage mechanism much faster than blob storage?
Yes, I tried at my side. Even the Storage Account is at the same location as my web app, it will take about 3-6 second to download a 30MB file. (In VM, it will only take less than 1 second)
My suggestions:
You can zip your pictures into one archive file, and download it. It would be faster than downloading them one by one.
You can use DownloadToFileParallelAsync method to download a file. It would be a little faster.
You can refer to the official tutorial to Download large amounts of random data from Azure storage
I have an app running on Azure app service, I have created some batch scripts which can take backup of the databases (DB running on some other servers i.e. 3rd party cloud db services - Not azure). Question is what is the best way/place to store these backup files in azure app services. Creating a folder named "Backup" in my source directory would overwrite these backups every time code is deployed. Followings are some of the concerns
Security of backup files
Backup files should be easily downloaded whenever I want to restore it
Backup files Shouldn't be overwritten or lost when the deployment is done or app slots are switched.
I was thinking of storing files in %HOME% directory, is it good idea ?
Also is there any size or storage limit with azure app service plans ?
I would recommend that you store the backups outside the Azure app service. Here's some problems with storing the files in App service:
You can't move the app easily from an App Service to an another.
App service has some storage limitations: Free and Shared sites get 1GB of space, Basic sites get 10GB, and Standard sites get 50GB.
It's not easy to access the backups outside of your app.
Instead, Azure Blob Storage is an ideal place for storing large files.
Regarding your concerns:
1) You can make the Azure Blob Storage container private, so that you can only access it if you know the key.
2) There's multiple ways to access the backups stored in Azure Blob Storage:
Azure Storage Explorer is a GUI for accessing the blob storage.
AZCopy which you can easily use from .BAT-files
Simple C#
3) When storing backups in Blob Storage, deployments slots doesn't affect the backups.
Blob storage also offers "Archive" tier which is ideal for storing the rarely used backups.
I have a WCF service hosted on Windows Azure as a "cloud service." When the service starts, it needs to populate data from files/disk to its memory so it is accessed fast (cached in other words). Right now I'm using like C:\Documents\Filestoprocess folder so that the WCF calls the folder and populates data data in that folder in its memory. I have like 5,000 small files. How do I do this in Azure? Is there a folder path that I can call within the WCF so that the WCF calls these files and opens each files and saves each data in the files? I'm not really looking for complicated Blob access through network using bandwidth. I'm looking for simple disk I/O access to these files from the WCF "cloud service" that is running on its own public web address.
You should try to use a cloud storage service to store data, as if you write to the local file system it can get destroyed on a restart of the service or recycling of the service.
You can look into using the azure drive service, which is like creating a disk dive. It is on top of blob storage.
But if you really want to write and read data on the local file system check out this blog post http://blog.codingoutloud.com/2011/06/12/azure-faq-can-i-write-to-the-file-system-on-windows-azure/
It talks about setting up your service definition to allow writing to the local file system.
Depending on the size of your instances you'll get a non-presistent disk where you can store this kind of temporary data. The minimum is 20GB for an extra small instance. You shouldn't access the disk directly, but you need to use a local resource instead which you can configure in your service definition file or in Visual Studio (double click your Web / Worker Role).
This storage is non-persistent, this means if you delete your deployment, if you decrease the number of instances, in case of hardware problems, ... you loose all data saved here. If you want to persist your files you should use blob storage instead. But in your case, where you need the files as some kind of caching mechanism, local resources are perfect.
And if your goal is to cache data you might want to take a look at the caching features included in Windows Azure: Caching in Windows Azure
Blob access is not complex. In fact, you could do a single download of a zip file from blob storage to local disk, unzip it, then prime your wcf service from those 5,000 small files.
Check out this msdn page documenting DownloadBlobToFile(). The essential parts:
CloudBlobClient blobClient =
new CloudBlobClient(blobEndpoint, new StorageCredentialsAccountAndKey(accountName, accountKey));
// Return a reference to the blob.
CloudBlob blob = blobClient.GetBlobReference("mycontainer/myblob.txt");
// Download the blob to a local file.
blob.DownloadToFile("c:\\mylocalblob.txt");
Now: I don't agree with saving to the root folder on C:. Rather, you should grab some local storage (easily configurable). Once you configure local storage in your role configuration, just ask the role environment for it, and ask for root path:
var localResource = RoleEnvironment.GetLocalResource("mylocalstorage");
var rootPath = localResource.RootPath;
Note: As #KingPancake mentioned, you could use an Azure drive. However: remember that an Azure drive can only be writeable by one instance. You'd need to make additional snapshots for your other instances. I think it's much simpler for you to go with a simple blob, copy your files down (either as single zip or individual files), and go from there.
You mentioned concern with network+bandwidth. You don't pay for bandwidth within the same data center. Also: It's extremely fast: 100Mbps per core. So even with a Small instance, you'll have your files copied down very quickly, moreso when you go to larger instance sizes.
One last thought: The only other ways to gain access to your 5,000 files, without using blob storage or Azure Drives (which are mounted as vhd's in blob storage) would be to either download the files from an external source or bundle them with your Windows Azure package (and then they'd show up in your app's folder, under whatever subfolder you stuck them in). Bundling has two downsides:
Longer time to upload your deployment package due to added size
Inability to change any of the individual files without redeploying the package.
By storing in a blob, you can easily change one (or all) of your small files without redeploying your code - you'd just need to signal it to either re-read from blob storage or restart the instances so they automatically download the new files.