What is Azure storage made of, the underlying storage technology which supports the Azure storage we access in azure portal?
Is it object based storage or block storage (persistent/ephemeral) similar to categorization in Ceph?
If there is a mix of block and object based, which storage is used for each of exposed Azure storage service - block blob, append blob, page blob, storage tables, storage queues, Azure files
A few years ago, Azure Storage team made a presentation about internals of Azure Storage at 23rd ACM Symposium on Operating Systems Principles (SOSP).
You can read more about this presentation here: https://azure.microsoft.com/en-in/blog/sosp-paper-windows-azure-storage-a-highly-available-cloud-storage-service-with-strong-consistency/.
Direct links:
Paper: http://sigops.org/sosp/sosp11/current/2011-Cascais/printable/11-calder.pdf
Video of presentation: https://www.youtube.com/watch?v=QnYdbQO0yj4
Powerpoint: http://sigops.org/sosp/sosp11/current/2011-Cascais/11-calder.pptx
Please go through this material. Hopefully it will give an idea about how Azure Storage is designed.
Related
Could someone please tell some examples where we can use Azure file share in azure instead of Azure Blobs. In the internet whenever I search I get it can be mounted or it follows SMB protocol. But still I am not understanding a single case where we can use Azure File share.
For this I tried to look into When to use Azure blob storage versus Azure file share?
-This is a similar question but doesn't answer my question.
Azure provides a variety of storage tools and services, including Azure Storage. To determine which Azure technology is best suited for your scenario, see Review your storage options in the Azure Cloud Adoption Framework.
For detailed information and examples refer to this article: https://learn.microsoft.com/en-us/azure/storage/common/storage-introduction
It depends mostly on your use-case and how you plan to access the data. If you simply want to mount and access your files Azure Files will be your best fit. If you are looking for the lowest cost and want to access your data programmatically through your application Azure Blob would be a better fit. Both are accessible through the portal or Azure Storage Explorer.
I also recommend this Learn module which covers the difference in data types and solutions.
Additional information: Azure Blob Storage vs Azure File Storage
Cost details of Azure Blob Storage pricing & Azure Files pricing
In short: if you ...
have an application that needs to store or access files in the cloud, use Blob Storage
need a file share that can be used by, for instance, a server, use File Shares
Azure Files shares can be mounted concurrently by cloud or on-premises deployments of Windows, Linux, and macOS. Azure Files shares can also be cached on Windows Servers with Azure File Sync for fast access near where the data is being used.
This means a File Share is, somewhat simplified, similar to a network share you would have in a local environment.
Azure Blob Storage helps you create data lakes for your analytics needs, and provides storage to build powerful cloud-native and mobile apps. Optimize costs with tiered storage for your long-term data, and flexibly scale up for high-performance computing and machine learning workloads.
This means Blob Storage is what you need when you're building powerful cloud-native and mobile apps.
I have some e-mail attachments being saved to Azure Blob.
I am now trying to write a Azure Functions App that would connect to that blob storage, run some scripts and re-save the file.
However, when selecting a storage account for the function, I couldn't select my blob storage account.
I went on the website and it said this:
When creating a function app, you must create or link to a general-purpose Azure Storage account that supports Blob, Queue, and Table storage. Some storage accounts don't support queues and tables. These accounts include blob-only storage accounts and Azure Premium Storage.
I'm wondering, is there any workaround this? and if not, perhaps any other suggestions? I'm becoming a little lost in all the options, and which one to actually choose.
Thanks!
EDIT: Might I add I writing the function Python
I think you are overlooking the fact that you can have multiple storage accounts. In order for an Azure Function to work you need a storage account. That storage account is used to store runtime information of the Azure Function for internal purposes like state management. This storage account is subject to restrictions as you already found out. There is no workaround for that.
However, if the function you are writing needs to access another storage account it is free to do so. You just have to provide details to connect to that specific storage account. In that case you also have a clear seperation between the storage account that is used by the azure function for its internal operations and the storage account your application needs to connect and which you have total control about withouth having to worry that you break things by deleting internal used blobs/tables/queues.
You can have a blob triggered function that gets triggered when changes occur on your specific blob storage. That doesn't need to be the storage account that the azure function internally uses, which is created/selected when creating the azure function.
Here is a sample that shows how to add a blob triggered azure function in Python. MyStorageAccountAppSetting refers to an app setting that holds the connection string to the storage account that you use for storage.
The snippet from the website you are quoting is for storing the function app code itself and any related modules. It does not pertain to what your function can access when the code of your function executes.
When your function executes it will need to use the Azure Blob Storage SDK/modules to connect to your blob storage account and read the email attachments. Here's a quickstart guide for using Azure Storage with Python: Quickstart with Azure Storage Blobs SDK for Python
General-purpose v2 storage accounts support the latest Azure Storage features and incorporate all of the functionality of general-purpose v1 and Blob storage accounts here
There are more integration options with GPv2 accounts including Azure Function Triggers. See: Azure Blob storage bindings for Azure Functions
Further refer: Types of storage accounts
If Blob, based on your need, you can choose an access tier based on the frequency of access for the data (e-mail attachments)Access tiers for Azure Blob Storage - hot, cool, and archive. If General purpose storage account, its standard performance tier.
I have a question about the use cases of the different Azure storage services:
Azure Data Lake Storage.
Azure Blob Storage.
Azure File Storage.
what is the difference between these services? and when to use them since they all provide the same functionality (storage) on Azure's cloud platform.
You can take a look at this article: https://learn.microsoft.com/en-us/azure/data-lake-store/data-lake-store-comparison-with-blob-storage
I'd say main differences between Data Lake and Azure Storage Blob is scale and permissions model.
It really makes no sense to paste the whole article here. But you might want to look at Data Lake v2, which (as MS claims) is a mesh or Data Lake v1 and Azure Storage Blob: https://learn.microsoft.com/azure/storage/blobs/data-lake-storage-introduction
As for the Azure File storage its just an SMB share over HTTPS (and it not really fast due to being 1 "stream" only).
In my Azure subscription, I have used both Storage Accounts that are of the type BlobStorage and some that say Storage or StorageV2...
I know the difference that my BlobStorage types do NOT support Tables, Files, etc containers.
But, are there other differences that I should be aware of? Is StorageV2 any faster that Blob only storage?
General-purpose v2 storage accounts support the latest Azure Storage
features and incorporate all of the functionality of general-purpose
v1 and Blob storage accounts. General-purpose v2 accounts deliver the
lowest per-gigabyte capacity prices for Azure Storage, as well as
industry-competitive transaction prices.
From the description, the v2 General Storage account which takes the features of the blob storage accounts and combines then with the general storage account, plus tiering. And I think the most important to the custom is the price. Follow this link, there is an exmaple analysing the difference in price of the two type storages.
No real 'nuances', just feature differences as stated in document, new storage you want on V2, older storage you want to migrate to V2 if possible.
Example:
General-purpose V2
Blob Tier: Hot, Cool, Archive
Replication: LRS, ZRS4, GRS, RA-GRS
Resource Manager
General-purpose V1
Blob Tier:N/A
Replication: LRS, GRS, RA-GRS
Resource Manager, Classic
Hope this help.........
Having the need to store and access blobs which type of storage account is the most appropriate? Both types (general purpose and blob storage) seem to support blobs and in addition to this general purpose accounts allow selecting default or premium performance while blob storage accounts allow only default performance but on the other hand they also allow selecting the access tier (cool or hot).
In the end I find unclear what would be the best option.
A few differences between Blob and General Purpose Storage Accounts:
Blob storage account only support blobs while General purpose storage accounts support blobs, files, queues & tables (some exception apply - please see note about replication below). So if you ever need these additional services, you may want to choose general purpose accounts over blob accounts.
Blob storage account only supports block and append blobs while General purpose storage accounts support block, append & page blobs (some exception apply - please see note about replication below). So if you need to create virtual machines, you would want to choose general purpose accounts over blob accounts.
Blob storage accounts support both Hot and Cool access tier while General purpose storage accounts only support Hot access tier. So if you need to use Cool access tier i.e. use storage primarily for near-shore archiving, you would want to choose blob accounts over general purpose accounts.
You may want to be careful in choosing replication type in general purpose accounts as features offered varies by replication types.
LRS, GRS, RAGRS: Supports everything. Blobs (Block, Append, Page), Files, Queues & Tables.
ZRS: Supports only block blobs and nothing else.
Premium LRS: Supports only page blobs and nothing else.
Microsoft's guidance on this (as of 7/13/2018) recommends using General Purpose v2 (GPv2) Storage Accounts over Blob Storage accounts for two reasons I recently discovered (there might be more):
They offer more flexibility in terms of what you can store and do with them (Queues, Tables, Files and/or Blobs vs just Blobs). Ref: Microsoft Azure Documentation - Azure Storage account options
Microsoft recommends using general-purpose v2 storage accounts over Blob storage accounts for most scenarios.
There are more integration options with GPv2 accounts including Azure Function Triggers via Event Grid. Ref: Microsoft Azure Documentation - Azure Blob storage bindings for Azure Functions
Blob-only storage accounts are supported for blob input and output bindings but not for blob triggers. Blob storage triggers require a general-purpose storage account.