I am attempting to copy a page blob from one storage account to another. The copy operation starts and the I see the blob in the destination storage account - however progress is always stuck at 0. I am using the java SDK and generating a SAS for the source. The SAS appears valid because I was able to download the src blob from the browser. Here is some of the sample code I am using
CloudStorageAccount account = CloudStorageAccount.parse(storageConnectionString);
CloudBlobClient strClient = account.createCloudBlobClient();
CloudBlobContainer strBlobContainer = strClient.getContainerReference("data");
CloudStorageAccount backupAccount = CloudStorageAccount.parse(backupStorageConnectionString);
CloudBlobClient backupClient = backupAccount.createCloudBlobClient();
CloudBlobContainer backupBlobContainer = backupClient.getContainerReference("data");
String src = "SG-aztest10-703-disk-0-test.vhd";
String target = "Tool-test4.vhd";
com.microsoft.azure.storage.blob.CloudPageBlob srcBlob = strBlobContainer.getPageBlobReference(src, null);
com.microsoft.azure.storage.blob.CloudPageBlob dstBlob = backupBlobContainer.getPageBlobReference(target, null);
System.out.println("Copying src blob " + " over dst blob " + dstBlob.getUri());
String copyID = dstBlob.startCopyFromBlob(new URI(getSharedAccessURI(src, null, strBlobContainer)));
System.out.println("Copy ID is " + copyID);
dstBlob.downloadAttributes();
CopyState state = dstBlob.getProperties().getCopyState();
while(state.getStatus().equals(CopyStatus.PENDING))
{
System.out.println("Progress is " + state.getBytesCopied()/state.getTotalBytes());
Thread.sleep(30000);
dstBlob.downloadAttributes();
state = dstBlob.getProperties().getCopyState();
}
Here is the code I am using to generate the SAS token
public static String getSharedAccessURI(String blobName, String snapshotID, CloudBlobContainer container) throws Exception
{
//Get the srcBlob without the snapshot
CloudPageBlob srcBlob = container.getPageBlobReference(blobName);
SharedAccessBlobPolicy sasPolicy = new SharedAccessBlobPolicy();
//Start Time = Current Time (UTC) - 15 minutes to account for Clock Skew
//Expiry Time = Current Time (UTC) + 1 day
sasPolicy.setPermissions(EnumSet.of(SharedAccessBlobPermissions.READ));
sasPolicy.setSharedAccessStartTime(DateTime.now().minusMinutes(15).toDate());
sasPolicy.setSharedAccessExpiryTime(DateTime.now().plusDays(1).toDate());
String sasToken = srcBlob.generateSharedAccessSignature(sasPolicy, null);
String sasURI = null;
if (snapshotID != null)
{
sasURI = String.format("%s?snapshot=%s&%s", srcBlob.getUri().toString(), snapshotID, sasToken);
}
else
{
sasURI = String.format("%s?%s", srcBlob.getUri().toString(), sasToken);
}
System.out.println("Shared access url is - " + sasURI);
return sasURI;
}
We use spare bandwidth to do async copies so the copy may not start until bandwidth is available. In that case, the copy progress field may remain 0 for quite some time while the copy status is pending. If the blob is small the progress field may go from 0 to complete in one shot as the copy is very fast. Even if the blob is large, your polling may miss the window during which the copy progress is between 0 and the content length.
Related
I am working on stress testing for our IoT Usecase.
For testing, I need to create 100 devices.
So, I have developed one azure function to use IoTHubs Import Device feature as per MSFT docs.
When I used, sample code using Public Storage account for Import/Output blob container SAS token.
It worked as per expectation and created devices on IoTHub.
But when I am using same code with Private Storage account, it is sometimes throwing reading error and sometimes it is throwing writing error on blob storage, even if SAS token has all required permissions (Read, Write, Delete, Create, List, Add etc..), and even private storage account is also having DNS configuration available. And for other use, I am able to add/update/delete blobs on the same private storage account.
Only problem I am facing is while calling ImportDevicesAsync method of IoTHub's RegistryManager.
My sample code is as below:
To create devices.txt file and upload it on proper container.
for (int i = deviceIndex; i < deviceCount + deviceIndex; i++)
{
var deviceToAdd = new ExportImportDevice()
{
Id = $"{devicePrefix}{i.ToString().PadLeft(6, '0')}",
ImportMode = importMode == "delete" ? ImportMode.Delete : ImportMode.Create,
Status = DeviceStatus.Enabled,
Authentication = new AuthenticationMechanism()
{
SymmetricKey = new SymmetricKey()
{
PrimaryKey = CryptoKeyGenerator.GenerateKey(32),
SecondaryKey = CryptoKeyGenerator.GenerateKey(32)
}
},
Tags = new TwinCollection(initialTags.SerializeObject())
};
serializedDevices.Add(deviceToAdd.SerializeObject());
}
// Write the list to the blob
StringBuilder sb = new();
serializedDevices.ForEach(serializedDevice => sb.AppendLine(serializedDevice));
Uri uri = new(assetsBlockBlobUrl + "?" + assetsBlobContainerSas);
CloudBlockBlob blob = new(uri);
await blob.DeleteIfExistsAsync();
using (CloudBlobStream stream = await blob.OpenWriteAsync())
{
byte[] bytes = Encoding.UTF8.GetBytes(sb.ToString());
for (var i = 0; i < bytes.Length; i += 500)
{
int length = Math.Min(bytes.Length - i, 500);
await stream.WriteAsync(bytes.AsMemory(i, length));
}
}
To import devices from the same container using registryManager.ImportDeviceAsync method:
RegistryManager registryManager = RegistryManager.CreateFromConnectionString(Environment.GetEnvironmentVariable("iotHubConnectionString"));
JobProperties importJob = await registryManager.ImportDevicesAsync(containerSasUri, containerSasUri);
////Wait until job is finished
while (true)
{
importJob = await registryManager.GetJobAsync(importJob.JobId);
_logger.LogInformation("import job " + importJob.Status);
if (importJob.Status == JobStatus.Completed)
{
return Common.Utils.GetObjectResult(importMode == "delete" ? MessageConstants.SuccessDeletedAsset : MessageConstants.SuccessCreatedAsset);
}
else if (importJob.Status == JobStatus.Failed)
{
return Common.Utils.GetObjectResult(importMode == "delete" ? MessageConstants.DeleteDeviceFail : MessageConstants.CreateDeviceFail);
}
else if (importJob.Status == JobStatus.Cancelled)
{
return Common.Utils.GetObjectResult(importMode == "delete" ? MessageConstants.DeviceDeletionCancel : MessageConstants.DeviceCreationCancel);
}
await Task.Delay(TimeSpan.FromSeconds(5));
}
solution/code how to upload large files more than 100 mb of size to blob storage in azure hosted app service (webapi) using .Net Core, But from local machine it's working not from azure app service.
Error showing file is too large to upload
Tried like one Example below -
[RequestFormLimits(MultipartBodyLengthLimit = 6104857600)]
[RequestSizeLimit(6104857600)]
public async Task<IActionResult> Upload(IFormfile filePosted)
{
string fileName = Path.GetFileName(filePosted.FileName);
string localFilePath = Path.Combine(fileName);
var fileStream = new FileStream(localFilePath, FileMode.Create);
MemoryStream ms = new MemoryStream();
filePosted.CopyTo(ms);
ms.WriteTo(fileStream);
BlobServiceClient blobServiceClient = new BlobServiceClient("ConnectionString");
var containerClient = new blobServiceClient.GetBlobContainerClient("Container");
BlobUploadOptions options = new BlobUploadOptions
{
TransferOptions = new StorageTransferOptions
{
MaximumConcurrency = 8,
MaximumTransferSize = 220 * 1024 * 1024
}
}
Blobsclient bc = containerClient.GetBlobClient("Name");
await bc.UploadAsync(fileStream, options);
ms.Dispose();
return Ok()
}
I tried in my environment and got below results:
To upload large files from local storage to Azure blob storage or file storage, you can use Azure data movement library,It provides high-performance for uploading, downloading larger files.
Code:
using Microsoft.Azure.Storage;
using Microsoft.Azure.Storage.Blob;
using Microsoft.Azure.Storage.DataMovement;
class program
{
public static void Main(string[] args)
{
string storageConnectionString = "<Connection string>";
CloudStorageAccount account = CloudStorageAccount.Parse(storageConnectionString);
CloudBlobClient blobClient = account.CreateCloudBlobClient();
CloudBlobContainer blobContainer = blobClient.GetContainerReference("test");
blobContainer.CreateIfNotExists();
string sourceBlob = #"C:\Users\download\sample.docx";
CloudBlockBlob destPath = blobContainer.GetBlockBlobReference("sample.docx");
TransferManager.Configurations.ParallelOperations = 64;
// Setup the transfer context and track the download progress
SingleTransferContext context = new SingleTransferContext
{
ProgressHandler = new Progress<TransferStatus>(progress =>
{
Console.WriteLine("Bytes Upload: {0}", progress.BytesTransferred);
})
};
// download thee blob
var task = TransferManager.UploadAsync(
sourceBlob, destPath, null, context, CancellationToken.None);
task.Wait();
}
}
Console:
Portal:
After I executed the above code and got successfully uploaded large file to azure blob storage.
We have a file system abstraction that allows us to easily switch between local and cloud (Azure) storage.
For reading and writing files we have the following members:
Stream OpenRead();
Stream OpenWrite();
Part of our application "bundles" documents into one file. For our local storage provider OpenWrite returns an appendable stream:
public Stream OpenWrite()
{
return new FileStream(fileInfo.FullName, FileMode.Open, FileAccess.ReadWrite, FileShare.ReadWrite, BufferSize, useAsync: true);
}
For Azure blob storage we do the following:
public Stream OpenWrite()
{
return blob.OpenWrite();
}
Unfortunately this overrides the blob contents each time. Is it possible to return a writable stream that can be appended to?
Based on the documentation for OpenWrite here http://msdn.microsoft.com/en-us/library/microsoft.windowsazure.storage.blob.cloudblockblob.openwrite.aspx, The OpenWrite method will overwrite an existing blob unless explicitly prevented using the accessCondition parameter.
One thing you could do is read the blob data in a stream and return that stream to your calling application and let that application append data to that stream. For example, see the code below:
static void BlobStreamTest()
{
storageAccount = CloudStorageAccount.DevelopmentStorageAccount;
CloudBlobContainer container = storageAccount.CreateCloudBlobClient().GetContainerReference("temp");
container.CreateIfNotExists();
CloudBlockBlob blob = container.GetBlockBlobReference("test.txt");
blob.UploadFromStream(new MemoryStream());//Let's just create an empty blob for the sake of demonstration.
for (int i = 0; i < 10; i++)
{
try
{
using (MemoryStream ms = new MemoryStream())
{
blob.DownloadToStream(ms);//Read blob data in a stream.
byte[] dataToWrite = Encoding.UTF8.GetBytes("This is line # " + (i + 1) + "\r\n");
ms.Write(dataToWrite, 0, dataToWrite.Length);
ms.Position = 0;
blob.UploadFromStream(ms);
}
}
catch (StorageException excep)
{
if (excep.RequestInformation.HttpStatusCode != 404)
{
throw;
}
}
}
}
There is now a CloudAppendBlob class that allows you to add content to an existing blob :
var account = CloudStorageAccount.Parse("storage account connectionstring");
var client = account.CreateCloudBlobClient();
var container = client.GetContainerReference("container name");
var blob = container.GetAppendBlobReference("blob name");
In your case you want to append from a stream:
await blob.AppendFromStreamAsync(new MemoryStream());
But you can append from text, byte array, file. Check the documentation.
I am developing a social network application.
I want to creat ,in azure storage, a container for each user (client) joining a social network, which means user1 has a container named container 1, and inside container 1 there will be user 1 profile in xml format and a profile picture of user 1.
Similarly, for user2 there will be container 2 created in azure blob storage and then user 2 profile is saved in xml format and profile picture of user 2, and it goes like this so for, let say, 10 users there will be 10 containers.
If i want to list all 9 users' information stored in Azure storage in different 9 containers from user client 1, how could i do that?
I am using webservice, but the challenge I am facing is how to collect all the 9 user profile information located in 9 different containers.
The following should do the trick -
CloudStorageAccount account =
CloudStorageAccount.FromConfigurationSetting("DataConnectionString");
// We need to access blobs now, so create a CloudBlobClient
CloudBlobClient blobClient = account.CreateCloudBlobClient();
IEnumerable<CloudBlobContainer> containers = blobClient.ListContainers();
For newer versions of Microsoft.WindowsAzure.Storage, use the right variant of CloudBlobClient.ListContainersSegmentedAsync method. Example usage is as follows:
private static async Task<IEnumerable<CloudBlobContainer>> ListContainersAsync(CloudBlobClient cloudBlobClient)
{
BlobContinuationToken continuationToken = null;
var containers = new List<CloudBlobContainer>();
do
{
ContainerResultSegment response = await cloudBlobClient.ListContainersSegmentedAsync(continuationToken);
continuationToken = response.ContinuationToken;
containers.AddRange(response.Results);
} while (continuationToken != null);
return containers;
}
Usage of the above method will be as follows:
string connectionString = "<your connection string>";
CloudStorageAccount.TryParse(connectionString, out CloudStorageAccount storageAccount);
if (storageAccount == null)
{
Console.WriteLine("Connection string did not work");
}
CloudBlobClient cloudBlobClient = storageAccount.CreateCloudBlobClient();
var containers = await ListContainersAsync(cloudBlobClient);
Alternately, you can create an extension method as follows:
public static class CloudBlobClientExtension
{
public static async Task<IEnumerable<CloudBlobContainer>> ListContainersAsync(this CloudBlobClient cloudBlobClient)
{
BlobContinuationToken continuationToken = null;
var containers = new List<CloudBlobContainer>();
do
{
ContainerResultSegment response = await cloudBlobClient.ListContainersSegmentedAsync(continuationToken);
continuationToken = response.ContinuationToken;
containers.AddRange(response.Results);
} while (continuationToken != null);
return containers;
}
}
Usage of the above extension method will be as follows:
string connectionString = "<your connection string>";
CloudStorageAccount.TryParse(connectionString, out CloudStorageAccount storageAccount);
if (storageAccount == null)
{
Console.WriteLine("Connection string did not work");
}
CloudBlobClient cloudBlobClient = storageAccount.CreateCloudBlobClient();
IEnumerable<CloudBlobContainer> containers = await cloudBlobClient.ListContainersAsync();
For more details, please refer to MSDN and Balkan's Blog
For older Azure Storage versions, use CloudBlobClient.ListContainers method. Example usage is as follows:
string connectionString = "<your connection string>";
CloudStorageAccount.TryParse(connectionString, out CloudStorageAccount storageAccount);
if (storageAccount == null)
{
Console.WriteLine("Connection string did not work");
}
CloudBlobClient cloudBlobClient = storageAccount.CreateCloudBlobClient();
var containers = cloudBlobClient.ListContainers();
Azure Blob Storage client library v12 for .NET
using Azure;
using Azure.Storage.Blobs;
BlobServiceClient blobServiceClient = new BlobServiceClient(azStorageConnString);
var containerList = blobServiceClient.GetBlobContainers();
foreach (var container in containerList)
// doImportantWork
CloudStorageAccount account =
CloudStorageAccount.FromConfigurationSetting("DataConnectionString");
// We need to access blobs now, so create a CloudBlobClient
var blobClient = account.CreateCloudBlobClient();
IEnumerable<CloudBlobContainer> containers = blobClient.ListContainers();
// This will return you list of containers
var containerList = containers.Select(e => e.Name).Tolist();
What is the most efficient way to get the count on the number of blobs in an Azure Storage container?
Right now I can't think of any way other than the code below:
CloudBlobContainer container = GetContainer("mycontainer");
var count = container.ListBlobs().Count();
If you just want to know how many blobs are in a container without writing code you can use the Microsoft Azure Storage Explorer application.
Open the desired BlobContainer
Click the Folder Statistics icon
Observe the count of blobs in the Activities window
I tried counting blobs using ListBlobs() and for a container with about 400,000 items, it took me well over 5 minutes.
If you have complete control over the container (that is, you control when writes occur), you could cache the size information in the container metadata and update it every time an item gets removed or inserted. Here is a piece of code that would return the container blob count:
static int CountBlobs(string storageAccount, string containerId)
{
CloudStorageAccount cloudStorageAccount = CloudStorageAccount.Parse(storageAccount);
CloudBlobClient blobClient = cloudStorageAccount.CreateCloudBlobClient();
CloudBlobContainer cloudBlobContainer = blobClient.GetContainerReference(containerId);
cloudBlobContainer.FetchAttributes();
string count = cloudBlobContainer.Metadata["ItemCount"];
string countUpdateTime = cloudBlobContainer.Metadata["CountUpdateTime"];
bool recountNeeded = false;
if (String.IsNullOrEmpty(count) || String.IsNullOrEmpty(countUpdateTime))
{
recountNeeded = true;
}
else
{
DateTime dateTime = new DateTime(long.Parse(countUpdateTime));
// Are we close to the last modified time?
if (Math.Abs(dateTime.Subtract(cloudBlobContainer.Properties.LastModifiedUtc).TotalSeconds) > 5) {
recountNeeded = true;
}
}
int blobCount;
if (recountNeeded)
{
blobCount = 0;
BlobRequestOptions options = new BlobRequestOptions();
options.BlobListingDetails = BlobListingDetails.Metadata;
foreach (IListBlobItem item in cloudBlobContainer.ListBlobs(options))
{
blobCount++;
}
cloudBlobContainer.Metadata.Set("ItemCount", blobCount.ToString());
cloudBlobContainer.Metadata.Set("CountUpdateTime", DateTime.Now.Ticks.ToString());
cloudBlobContainer.SetMetadata();
}
else
{
blobCount = int.Parse(count);
}
return blobCount;
}
This, of course, assumes that you update ItemCount/CountUpdateTime every time the container is modified. CountUpdateTime is a heuristic safeguard (if the container did get modified without someone updating CountUpdateTime, this will force a re-count) but it's not reliable.
The API doesn't contain a container count method or property, so you'd need to do something like what you posted. However, you'll need to deal with NextMarker if you exceed 5,000 items returned (or if you specify max # to return and the list exceeds that number). Then you'll make add'l calls based on NextMarker and add the counts.
EDIT: Per smarx: the SDK should take care of NextMarker for you. You'll need to deal with NextMarker if you're working at the API level, calling List Blobs through REST.
Alternatively, if you're controlling the blob insertions/deletions (through a wcf service, for example), you can use the blob container's metadata area to store a cached container count that you compute with each insert or delete. You'll just need to deal with write concurrency to the container.
Example using PHP API and getNextMarker.
Counts total number of blobs in an Azure container.
It takes a long time: about 30 seconds for 100000 blobs.
(assumes we have a valid $connectionString and a $container_name)
$blobRestProxy = ServicesBuilder::getInstance()->createBlobService($connectionString);
$opts = new ListBlobsOptions();
$nblobs = 0;
while($cont) {
$blob_list = $blobRestProxy->listBlobs($container_name, $opts);
$nblobs += count($blob_list->getBlobs());
$nextMarker = $blob_list->getNextMarker();
if (!$nextMarker || strlen($nextMarker) == 0) $cont = false;
else $opts->setMarker($nextMarker);
}
echo $nblobs;
If you are not using virtual directories, the following will work as previously answered.
CloudBlobContainer container = GetContainer("mycontainer");
var count = container.ListBlobs().Count();
However, the above code snippet may not have the desired count if you are using virtual directories.
For instance, if your blobs are stored similar to the following: /container/directory/filename.txt where the blob name = directory/filename.txt the container.ListBlobs().Count(); will only count how many "/directory" virtual directories you have. If you want to list blobs contained within virtual directories, you need to set the useFlatBlobListing = true in the ListBlobs() call.
CloudBlobContainer container = GetContainer("mycontainer");
var count = container.ListBlobs(null, true).Count();
Note: the ListBlobs() call with useFlatBlobListing = true is a much more expensive/slow call...
Bearing in mind all the performance concerns from the other answers, here is a version for v12 of the Azure SDK leveraging IAsyncEnumerable. This requires a package reference to System.Linq.Async.
public async Task<int> GetBlobCount()
{
var container = await GetBlobContainerClient();
var blobsPaged = container.GetBlobsAsync();
return await blobsPaged
.AsAsyncEnumerable()
.CountAsync();
}
With Python API of Azure Storage it is like:
from azure.storage import *
blob_service = BlobService(account_name='myaccount', account_key='mykey')
blobs = blob_service.list_blobs('mycontainer')
len(blobs) #returns the number of blob in a container
If you are using Azure.Storage.Blobs library, you can use something like below:
public int GetBlobCount(string containerName)
{
int count = 0;
BlobContainerClient container = new BlobContainerClient(blobConnctionString, containerName);
container.GetBlobs().ToList().ForEach(blob => count++);
return count;
}
Another Python example, works slow but correctly with >5000 files:
from azure.storage.blob import BlobServiceClient
constr="Connection string"
container="Container name"
blob_service_client = BlobServiceClient.from_connection_string(constr)
container_client = blob_service_client.get_container_client(container)
blobs_list = container_client.list_blobs()
num = 0
size = 0
for blob in blobs_list:
num += 1
size += blob.size
print(blob.name,blob.size)
print("Count: ", num)
print("Size: ", size)
I have spend quite period of time to find the below solution - I don't want to some one like me to waste time - so replying here even after 9 years
package com.sai.koushik.gandikota.test.app;
import com.microsoft.azure.storage.CloudStorageAccount;
import com.microsoft.azure.storage.blob.*;
public class AzureBlobStorageUtils {
public static void main(String[] args) throws Exception {
AzureBlobStorageUtils getCount = new AzureBlobStorageUtils();
String storageConn = "<StorageAccountConnection>";
String blobContainerName = "<containerName>";
String subContainer = "<subContainerName>";
Integer fileContainerCount = getCount.getFileCountInSpecificBlobContainersSubContainer(storageConn,blobContainerName, subContainer);
System.out.println(fileContainerCount);
}
public Integer getFileCountInSpecificBlobContainersSubContainer(String storageConn, String blobContainerName, String subContainer) throws Exception {
try {
CloudStorageAccount storageAccount = CloudStorageAccount.parse(storageConn);
CloudBlobClient blobClient = storageAccount.createCloudBlobClient();
CloudBlobContainer blobContainer = blobClient.getContainerReference(blobContainerName);
return ((CloudBlobDirectory) blobContainer.listBlobsSegmented().getResults().stream().filter(listBlobItem -> listBlobItem.getUri().toString().contains(subContainer)).findFirst().get()).listBlobsSegmented().getResults().size();
} catch (Exception e) {
throw new Exception(e.getMessage());
}
}
}
Count all blobs in a classic and new blob storage account. Building on #gandikota-saikoushik, this solution works for blob containers with a very large number of blobs.
//setup set values from Azure Portal
var accountName = "<ACCOUNTNAME>";
var accountKey = "<ACCOUTNKEY>";
var containerName = "<CONTAINTERNAME>";
uristr = $"DefaultEndpointsProtocol=https;AccountName={accountName};AccountKey={accountKey}";
var storageAccount = Microsoft.WindowsAzure.Storage.CloudStorageAccount.Parse(uristr);
var client = storageAccount.CreateCloudBlobClient();
var container = client.GetContainerReference(containerName);
BlobContinuationToken continuationToken = new BlobContinuationToken();
blobcount = CountBlobs(container, continuationToken).ConfigureAwait(false).GetAwaiter().GetResult();
Console.WriteLine($"blobcount:{blobcount}");
public static async Task<int> CountBlobs(CloudBlobContainer container, BlobContinuationToken currentToken)
{
BlobContinuationToken continuationToken = null;
var result = 0;
do
{
var response = await container.ListBlobsSegmentedAsync(continuationToken);
continuationToken = response.ContinuationToken;
result += response.Results.Count();
}
while (continuationToken != null);
return result;
}
List blobs approach is accurate but slow if you have millions of blobs. Another way that works in a few cases but is relatively fast is querying the MetricsHourPrimaryTransactionsBlob table. It is at the account level and metrics get aggregated hourly.
https://learn.microsoft.com/en-us/azure/storage/common/storage-analytics-metrics
You can use this
public static async Task<List<IListBlobItem>> ListBlobsAsync()
{
BlobContinuationToken continuationToken = null;
List<IListBlobItem> results = new List<IListBlobItem>();
do
{
CloudBlobContainer container = GetContainer("containerName");
var response = await container.ListBlobsSegmentedAsync(null,
true, BlobListingDetails.None, 5000, continuationToken, null, null);
continuationToken = response.ContinuationToken;
results.AddRange(response.Results);
} while (continuationToken != null);
return results;
}
and then call
var count = await ListBlobsAsync().Count;
hope it will be useful