My application till this day was working with local zip files,
meaning I was using a direct return new FileStream()
in the application and the local zip file that was located on the SDD/Network drive path (zip files can be hundreds of GB).
I configured the application to work with Azure Blob Storage, meaning each FileStream that was returned in now return as the Azure Blob SDK method:
GetBlobStreamAsync(ContainerName, BlobName).ConfigureAwait(false).GetAwaiter().GetResult()
I uploaded some zip files to a container in the blob storage and set the connection string in the application to work with that storage account.
The application was deployed and running on a virtual windows machine located in the same region of the Azure Storage Blob.
Note: This is a private cloud network.
When the app is streaming the zip file on Azure blob storage it seems that the performance has decreased by at least 8-9 times (problematic with hundreds of GB).
Speed comparison is between local C: drive on the same windows virtual machine that the application is running on an Azure Storage account which is located in the same region.
Note: NW Bandwidth - is 50 GB on the VM on azure
Solutions that I tried:
Azure blob Premium Performance storage - Didn’t improve performance
.Net Core - advantage of performance enhancements (we work with .Net framework so this is irrelevant).
Network File System (NFS) 3.0 performance considerations in Azure Blob storage - (Does not work with private cloud).
Hot, Cool, and Archive access tiers for blob data - The default is Hot so we already tried this scenario with no improvements.
Solutions I want to try:
Azure Files Share Storage as a cache solution
.Net Framework configuration - lists several quick configuration settings that you can use to make significant performance improvements
Question:
Does anyone have any suggestions on how can I optimize the streaming in front of the Azure Storage Blob?
Azure Files (share) or Storage Blob services are likely not the right services to be utilized for this scenario. There are two possible paths:
Break a single file into multiple files and leverage Storage Blob service that handles throughput better than Azure Files. Azure Files performs better with small(er) files which are typical to user documents (PDFs, Word, Excel, etc.)
Switch over to a more dedicated service that is designed specifically for large-size data transfer if breaking up a single file into multiple blobs is not an option.
The recommendation for each option will highly depend on the implementation details, requirements and constraints of the system.
Related
Could someone please tell some examples where we can use Azure file share in azure instead of Azure Blobs. In the internet whenever I search I get it can be mounted or it follows SMB protocol. But still I am not understanding a single case where we can use Azure File share.
For this I tried to look into When to use Azure blob storage versus Azure file share?
-This is a similar question but doesn't answer my question.
Azure provides a variety of storage tools and services, including Azure Storage. To determine which Azure technology is best suited for your scenario, see Review your storage options in the Azure Cloud Adoption Framework.
For detailed information and examples refer to this article: https://learn.microsoft.com/en-us/azure/storage/common/storage-introduction
It depends mostly on your use-case and how you plan to access the data. If you simply want to mount and access your files Azure Files will be your best fit. If you are looking for the lowest cost and want to access your data programmatically through your application Azure Blob would be a better fit. Both are accessible through the portal or Azure Storage Explorer.
I also recommend this Learn module which covers the difference in data types and solutions.
Additional information: Azure Blob Storage vs Azure File Storage
Cost details of Azure Blob Storage pricing & Azure Files pricing
In short: if you ...
have an application that needs to store or access files in the cloud, use Blob Storage
need a file share that can be used by, for instance, a server, use File Shares
Azure Files shares can be mounted concurrently by cloud or on-premises deployments of Windows, Linux, and macOS. Azure Files shares can also be cached on Windows Servers with Azure File Sync for fast access near where the data is being used.
This means a File Share is, somewhat simplified, similar to a network share you would have in a local environment.
Azure Blob Storage helps you create data lakes for your analytics needs, and provides storage to build powerful cloud-native and mobile apps. Optimize costs with tiered storage for your long-term data, and flexibly scale up for high-performance computing and machine learning workloads.
This means Blob Storage is what you need when you're building powerful cloud-native and mobile apps.
I have a project ASP.Net MVC classics which need to be migrated to Azure and host in AppServices. Currently this project save files in the root folder and file size could be 2GB.
Now the question is should I leave the current logic to store the file in wwwroot folder as:\wwwroot\Files\myfile.txt"; or should I store it in the blob?
I am looking for the best practice and do not want to change the current logic? Can someone give me the idea?
Thanks
Storing files in Azure Blob Storage:
According to Documentation it says,
Azure Blob Storage enables the creation of data lakes for analytics purposes and provides storage for the development of powerful cloud-native and mobile apps. Reduce costs by using tiered storage for long-term data and scalability for high-performance computing and machine learning workloads.
According to Documentation it says,
SAS enables you to securely upload and download files from Azure Blob Storage without having to share the connection string.
While uploading you can split large data in small amounts with which we can decrease uploading time and after uploading then it again combines into a single blob.
Storing file in wwwroot folder:
According to Documentation it says
Static resource files are stored in web root, The default directory is {content root}/wwwroot folder.
According to Documentation we can clearly say that capacity depends on price tier.
Web apps performance can be effected by uploading large files.
In Azure linux web apps, if the size is around 2 Gb then it might lead to timeoutException.
We are migrating from an on-premises virtual machine to Azure cloud. The virtual machine will eventually be decommissioned and we have many files and folders that we don't want to lose, like old websites and databases, scripts, programs etc.
We use an Azure storage account for storing and retrieving images via blob containers for the live websites.
Q: What is the best and most cost effective way to backup large amount of files unused in production, rarely accessed, from an on-premises virtual machine to Azure cloud?
Changing the Access tier to Azure Archive Storage(if storing data in Blobs) would be your best option. A few notes:
The Archive storage tier is only available at the blob level and not at the storage account level.
Archive storage is offline and offers the lowest storage costs but also the highest access costs
Hot, Cool, and Archive tiers can be set at the object level.
Additional info can be found here:https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blob-storage-tiers
recommendation would be to move those unused files to Azure storage archives, which is cost effective and easily accessible when required.
https://azure.microsoft.com/en-us/services/storage/archive/
I have an app running on Azure app service, I have created some batch scripts which can take backup of the databases (DB running on some other servers i.e. 3rd party cloud db services - Not azure). Question is what is the best way/place to store these backup files in azure app services. Creating a folder named "Backup" in my source directory would overwrite these backups every time code is deployed. Followings are some of the concerns
Security of backup files
Backup files should be easily downloaded whenever I want to restore it
Backup files Shouldn't be overwritten or lost when the deployment is done or app slots are switched.
I was thinking of storing files in %HOME% directory, is it good idea ?
Also is there any size or storage limit with azure app service plans ?
I would recommend that you store the backups outside the Azure app service. Here's some problems with storing the files in App service:
You can't move the app easily from an App Service to an another.
App service has some storage limitations: Free and Shared sites get 1GB of space, Basic sites get 10GB, and Standard sites get 50GB.
It's not easy to access the backups outside of your app.
Instead, Azure Blob Storage is an ideal place for storing large files.
Regarding your concerns:
1) You can make the Azure Blob Storage container private, so that you can only access it if you know the key.
2) There's multiple ways to access the backups stored in Azure Blob Storage:
Azure Storage Explorer is a GUI for accessing the blob storage.
AZCopy which you can easily use from .BAT-files
Simple C#
3) When storing backups in Blob Storage, deployments slots doesn't affect the backups.
Blob storage also offers "Archive" tier which is ideal for storing the rarely used backups.
We are new to Windows azure and are developing a web application. In the beginning of the project , we have deployed complete code to different environments which actually publish complete code and uploaded blob objects to azure storage as we linked sitefinity to hold blob objects in azure storage . But now as we are in the middle of development , we are just required to upload any new blob files created which can be quite less in numbers (1 or 2 or maybe few).Now I would like to know best process to sync these blob files to different azure storage environments which is for each cloud service. So ideally we would like to update staging cloud service and staging storage first and then test there and then once no bugs are found, then will be required to update UAT and production storages as well with the changed or new blob objects.
Please help.
You can use the Azure Storage Explorer to manually upload/download blobs from storage accounts very easily. For one or two blobs, this would be an easy solution, otherwise you will need to write a tool that connects to the blob storage via an API and does the copying for you.