I would like the stats of specific folders in Azure Blob Storage. For example I would like to know how many files are present in a folder, whats the size of each file or whats the total size of a folder. Does blob storage provide similar data through an api endpoint?
Edit: I have a very large number of files on Azure Blob so I am looking for a solution where I do not have to iterate over all the files in order to calculate total size of the virtual folder.
Does blob storage provide similar data through an api endpoint?
As such Azure Blob Storage does not provide an API to get storage statistics at the folder level but you can make use of List Blobs REST API operation to get that information.
List Blobs operation will list the blobs insider a container but you can use prefix parameter to get the list of blobs inside a virtual folder where prefix will be the path of the virtual folder. For example, if you wish to list the blobs inside folder1 virtual folder, you would specify prefix as folder1/
Each item in the list is a blob which will have a size attribute which will give you the size of the blob. You will simply add the size of individual blobs to get the total size of the folder.
Related
we have some blob containers in Azure Storage,
I would like to have a Dashboard with links to some specific folders e.g. to see at one glance the latest files in a specific folder of the Blob Container.
At the moment it is only possible with some clicks, navigation down and sorting into the folder.
I already tried to create a Metrics chart on the Dashboard, but it gives me only BLOB count and stats for the whole BLOB not for granular folders.
Any ideas how to whatch specific folders immediately?
Thing is, folders don't exists in Azure Storage Blobs. There are only containers and blobs inside containers. Blobs define virtual folders. Tools like the Azure Portal or Azure Storage Explorer use the / seperator in the blob url as a way to present a virtual folder structure.
So the answer is that it is impossible since there are no physical folders, as stated in the docs as well:
Blob storage offers three types of resources:
The storage account.
A container in the storage account
A blob in a container
I'm trying to configure an online backup to an Azure Storage account. Some of the files I am backing up are larger than 200GB, so I have to be using page Blob type storage.
I believe that, at the moment, this is the kind of storage I have configured; However, my backup of the files that are larger than this 200GB fails stating that the "block blob maximum size is 200GB."
How can I check what kind of storage my Azure storage is configured as? And, how can i ensure that in the future, I am configuring the correct type of storage?
An Azure Storage account can contain Block, Append and Page blobs in a same container. We do not any configurations on Account level or container level. The difference is we will need to use different APIs in SDK or implement with different REST APIs for the different type of Blobs.
You can refer to https://msdn.microsoft.com/en-us/library/azure/dd135733.aspx for more info.
And according to your requirement, for those blobs will be larger than 200GB. You can divide them into several pieces of block blobs, and you can custom mimetype of the blobs pieces to determine whether they are the piece of a special file.
Any further concern, please feel free to let me know.
It depends on how you upload the files to Azure Storge, you specify what type of blob you want to create, either Page, Blob or Append Blob.
Ex:
CloudPageBlob blob = container.GetPageBlobReference("file name");
blob.Properties.ContentType = "binary/octet-stream";
blob.Create(size)
Then you have to divide your stream into pages and iterate over it and upload it to the blob.
Currently i have some ftp where on it i have some deep structure of folders and files within it. It could be even 10 levels down from root folder. As i migrated already with success my local database to azure database, i wonder also whether is there any azure ftp i could use to migrate this as well. I know we have something like Azure storage and i could create Container for it of type File or Blobs - are one of those could be used like particural ftp - could i create folder structure there somehow using container and either File or Blob for that purpose, how it works there? Does either container blob or file for such purposes?
Let me add to what NDJ has written. So both Azure Blobs and Files would serve your purpose.
As mentioned by NDJ, Azure Blob Storage is a 2-level hierarchy system. At the top you have a blob container and the each blob container contains 0 or more files. So it does not support a folder structure per se but as NDJ mentioned, you can create an illusion of a sub folder by using appropriate blob delimiters (usually /). If you were to compare it with local file system, a directory at the root level (C:) is a container in blob storage and then the files would go in there. So imagine you have a folder called images in C:\ of your computer, that would be a container in blob storage. Now imagine that you have 2 sub folders beneath this folder (let's call them hires and lores) and both of them contains some files (say image1.png). When you move them to Azure Blob Storage, the container name would be images but the blob names would be hires/image1.png and lores/image1.png. Some of the storage explorers would take this delimiter (/) and show you that your container contains 2 folders and inside each folder you have an image called image1.png but in reality there are only 2 blobs in that blob container.
Azure File Service is a close match to your local file system. At the top level, you've got a Share and each share will container directories and files. Each directory can again contain many directories and files.
As NDJ mentioned, there's no FTP access to Azure Storage but there are many tools that will allow you to upload files from local computer to Azure Storage and many of them will preserve the file hierarchy. You can always write code to upload the files yourself. If you decide to use Azure Files, you can simply mount a File Storage Share as a network drive on your local computer and then transfer the files from your local computer to Azure Files as if you're transferring files from one drive to another.
UPDATE
Regarding difference between Azure Blob Storage and File Storage, both are used to store files. There are a few differences that I could think of:
A Share in Azure File Storage can be mount as a network drive on your local computer/Azure VM whereas a Blob Container in Azure Blob Storage can't. So if you have an application which writes files to local file system, you can take the application as is and make use of Azure File Storage and write the file to that network drive without making many changes to your code (typical example of Lift-And-Shift kind of application.
You can set ACL on a Blob Container whereas you can't do the same on a Share. This makes Azure Blob Storage ideal for storing static content (images, css, js) for your websites. For exposing files in File Storage, you would need to resort to Shared Access Signature.
You can set the size of a Share (default is 5GB) whereas no such thing exist for a Blob Container. A blob container can go up to the size of a storage account.
To understand Azure Files, I would recommend reading this: https://azure.microsoft.com/en-in/documentation/articles/storage-dotnet-how-to-use-files/.
Azure blob supports 10 levels down (up to 254. Basically the files are stored non hierarchically, but each / separator gives the appearance of directories.
It's relatively trivial to write something to move files to azure, as far as I know there is no ftp functionality yet - but it has been requested. It looks like some people have already created some code for this
You can now use Storage Explorer across all platforms to easily work within any folder structure.
Where would it be appropriate to store structured photos in Azure storage? There a ton (millions) of photos and they are currently sitting in folders locally.
I originally looked at blob storage to hold them, but that is for unstructured data; then I looked at table storage, but I'm not sure if the file size is too large for the entity. Also looked at file storage, but it seems like that's only in preview upon request.
Blob Storage is the way to go. It is meant for that purpose only - storing files in the cloud. Some additional reasons:
Today, each storage account can hold 500 TB of data so if you're storing images only in your storage account, you can store up to 500 TB of data.
3 copies of each item (file in your case) is maintained in the region. If you enable Geo-Replication on your storage (GRS), 3 additional copies are maintained in a secondary region which is at least 400 miles away from the primary reason. So it would be a good strategy for disaster recovery purposes.
As it is cloud storage solution, you only pay for the storage space you occupy. So for example if you are storing only 15 GB data, you will only pay for 15 GB.
Table Storage is mainly intended for storing structured/semi-structured data in key/value pair format. Further, size of each item (known as Entity in table storage lingo) can be of a maximum 1 MB in size. Each item in blob storage can be of a maximum of 200 GB in size.
A few other things to consider:
Blob storage is a 2 level storage: Container and Blob. Think of a container as a folder on your computer and blob as a file. Unlike local storage, you can't have nested folders in blob storage.
Even though blob storage doesn't support nested folder hierarchy, you can create an illusion of nested folder hierarchy by something called blob prefix. To give you an example, let's say you have images folder and inside that folder the image files are grouped by year (2014, 2015 etc.). So in this case, you can create a container called images. Now when it comes to saving files (say C:\images\2014\image1.png), you can prefix the folder path so your image will be saved as 2014/image1.png in the container images.
You can make use of some available storage explorers for uploading purposes. Most of the storage explorers support preserving the folder hierarchy.
I have groups of files using the following structure:
RandomFolderName1 [File1.jpg, File2.jpg, File3.jpg...]
RandomFolderName2 [File1.jpg, File2.jpg, File3.jpg...]
I wonder what will be the bast way to store this in Blob Storage.
Should I use GUID.jpg for every file name and manage the folder structure in the DB
Should I use FolderName+FileName.jpg, but again will have to manage the folder structure in DB
Should I use a Container for a folder and inside have File1.jpg, File2.jpg, File3.jpg...
Should I Store the whole ForderName as a zip and have all the files inside
Is there any other way to define a folder structure in Blob Storage?
Edit: The files will be accessed on a folder basis
So you can use file names in Azure blobs like "randomfoldername1/file1.jpg". It will look like a folder structure and some GUI clients will even let you navigate like it it. But the reality is that the "container" is the only real grouping factor and from there its just a matter of filterng the files in that container based on partial file names.
So to answer your question, you'll likely be fine putting all the files into a single container. The containers help control acces policy an each blob has its own performance target. The aside from acl reasons, the only other reason to split them across blobs in the same container is if you have enough blobs that quering them starts to degrade due to the shere number (or you're exceeding the storage account throughput targets).
You can find out more about Azure Storage abstractions and throughput targets at: http://www.windows-azure.net/windows-azure-storage-abstractions-and-their-scalability-targets/