I'm trying to configure an online backup to an Azure Storage account. Some of the files I am backing up are larger than 200GB, so I have to be using page Blob type storage.
I believe that, at the moment, this is the kind of storage I have configured; However, my backup of the files that are larger than this 200GB fails stating that the "block blob maximum size is 200GB."
How can I check what kind of storage my Azure storage is configured as? And, how can i ensure that in the future, I am configuring the correct type of storage?
An Azure Storage account can contain Block, Append and Page blobs in a same container. We do not any configurations on Account level or container level. The difference is we will need to use different APIs in SDK or implement with different REST APIs for the different type of Blobs.
You can refer to https://msdn.microsoft.com/en-us/library/azure/dd135733.aspx for more info.
And according to your requirement, for those blobs will be larger than 200GB. You can divide them into several pieces of block blobs, and you can custom mimetype of the blobs pieces to determine whether they are the piece of a special file.
Any further concern, please feel free to let me know.
It depends on how you upload the files to Azure Storge, you specify what type of blob you want to create, either Page, Blob or Append Blob.
Ex:
CloudPageBlob blob = container.GetPageBlobReference("file name");
blob.Properties.ContentType = "binary/octet-stream";
blob.Create(size)
Then you have to divide your stream into pages and iterate over it and upload it to the blob.
Related
I would like to know if there's a way to create an Azure storage container of a specific size, say 20 gb. I know it can be created without any restriction (I think up to 200 TB?), but can it be created with a specific size? What if I need that kind of set up? Like giving a user 20 gb initially, then at a later time increasing it to, say 50? Is that possible?
Like, how do I create that boundary/limitation for a new user that signs up my app?
Not possible with the service by itself. This should be a feature implemented in your app.
As mentioned in other answer, it is not possible to do with Blob Storage at the service level and you will have to implement your own logic to calculate the size of the blob container.
If restricting container size is the most important feature you are after, you may want to look at Azure File Storage. Equivalent to a blob container is a File Share there and you can set the quota for a File Share and change it dynamically. The quota of a File Share can be any value between 1GB - 5TB (100TB in case of Premium File Storage account) at the time of writing this answer.
Azure File Storage and Blob Storage are somewhat similar but they are meant to serve different purposes. However for simple object storage purposes you can use either of the two (File Storage is more expensive that Blob Storage though).
I'd like to be able to serve up SAS URIs to clients so they may upload large blobs to Azure Blob Storage. As far as I can tell, there's no way to do this while limiting the size of blobs being uploaded. I've thought about introducing an intermediary proxy service that inspects the size of the payload before uploading it to storage (and thus giving up on the SAS URI approach), but I feel like there must be another way. Thoughts?
Per my investigation , sas uri could control permission ,period, access type, IP address etc. However, it could not limit the size of individual upload blob. Please check this feedback.
I think you could create individual container for the application to limit the total size of storage space. You could check the limitation doc.Or you could implement this limit in your application.
Hope it helps you.
I am working with a program which connects to multiple APIs, the logs for each operation (HTML/XML/Json) need to be stored for possible later review. Is it feasible to store each request/reply in an Azure blob? There can be hundreds of requests per second (all of which need storing) which vary in size and have an average size of 100kB.
Because the logs need to be searchable (by metadata) my plan is to store it in Azure Blob and put metadata (with blob locations, custom application-related request and content identifiers, etc) in an easily-searchable database.
You can store logs in the Azure table storage or Blob storage but Microsoft itself recommends using Blob storage. Azure Storage Analytics stores log data in Blob storage.
This 'Azure Storage Table Design Guide' points out several draw backs of using table storage for logs and also provides details on how to use the blob storage to store logs. Read the 'Log data anti-pattern' section in particular for this use case.
Are there ways to check if a container in Azure has a new blob (doesn't matter which blob it is)? LastModifiedUtc does not seem to change if a blob is dropped into the container
You should use a BlobTrigger function in an App Service resource.
Documentation
Windows Azure Blob Storage does not provide this functionality out of the box. You would need to handle this on your end. A few things come to my mind (just thinking out loud):
If the blobs are uploaded using your application (and not through 3rd party tools), after the blob is uploaded, you could just update the container properties (may be add/update a metadata entry with information about the last blob uploaded). You could also make an entry into Azure Table Storage and keep on updating it with the information about last blob uploaded. As I said above, this method will only work if all blobs are uploaded through your application.
You could manually iterate through blobs in the blob container periodically and then sort them by last modified date. This method would work fine for a blob container having lesser number of blobs. If the number of blobs are more (say in tens of thousands), then you would end up fetching a long list because blob storage only sorts the blob by blob name.
I am writing a small web application for Windows Azure, which should use the blob storage for, obviously, storing blobs.
Is there a function or a way to automatically generate a unique name for a blob on insert?
You can use a Guid for that:
string blobName = Guid.NewGuid().ToString();
There is nothing that generates a unique name "on insert"; you need to come up with the name ahead of time.
When choosing the name of your blob, be careful when using any algorithm that generates a sequential number of some kind (either at the beginning or the end of the name of a blob). Azure Storage relies of the name for load balancing; using sequential values can create contention in accessing/writing to Azure Blobs because it can prevent Azure from properly load-balancing its storage. You get 60MB/Sec on each node (i.e. server). So to ensure proper load-balancing and to leverage 60MB/Sec on multiple storage nodes you need to use random names for your blobs. I typically use Guids to avoid this problem, just as Sandrino is recommending.
In addition to what Sandrino said (using GUID which have very low probability of being duplicated) you can consider some third-party libraries which generate conflict-free identifiers example: Flake ID Generator
EDIT
Herve has pointed out very valid Azure Blob feature which should be considered with any blob names, namely, Azure Storage load balancing and blobs partitioning.
Azure keeps all blobs in partition servers. Which partition server should be used to store particular blob is decided on the blob container and the blob file name. Unfortunately I was not able to find and documentation describing algorithm used for blobs partitioning.
More on Azure Blob architecture can be found on Windows Azure Storage Architecture Overview article.
This is an old post, but it was the first hit that showed up for 'azure storage blob unique id' search.
It looks like the 'metadata_storage_path' generated property is unique, but it's not filterable so it may not be useful for some purposes.