In the article How to use the Windows Azure Blob Storage Service in .NET the following code is used to demonstrate how one might upload a file
// Retrieve storage account from connection string.
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(
CloudConfigurationManager.GetSetting("StorageConnectionString"));
// Create the blob client.
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
// Retrieve reference to a previously created container.
CloudBlobContainer container = blobClient.GetContainerReference("mycontainer");
// Retrieve reference to a blob named "myblob".
CloudBlockBlob blockBlob = container.GetBlockBlobReference("myblob");
// Create or overwrite the "myblob" blob with contents from a local file.
using (var fileStream = System.IO.File.OpenRead(#"path\myfile"))
{
blockBlob.UploadFromStream(fileStream);
}
If you had a long running service that was accepting files and storing them in blob storage would you perform all of these steps every time? Or would you maybe have a class that had a reference to blockBlob that was used by multiple requests? How much (if any) of this is it okay to cache and use from multiple requests? (which I guess means threads)
I concur with #knightpfhor, there is nothing to cache. Until you call UploadFromStream, no long-running transactions have been called. Everything is in memory, constructing objects.
This is not like a Sql Connection, where programmers would find clever ways to cache connections because they were expensive to open - this is REST calls, so every data-changing action is an https call and all the preparation prior to it, is simply light-weight object manipulation
Most of those objects have pretty light weight constructors and are not guaranteed to be thread safe (check the MSDN documentation) so I wouldn't be too concerned about caching them. The only one I tend to keep as a static object is the cloud storage account.
Related
I am using libraries Microsoft.Azure.Storage.Blob 11.2.3.0 and Microsoft.Azure.Storage.Common 11.2.3.0 to connect to an Azure BlobStorage from a .NET Core 3.1 application.
Users of my application are supposed to supply connection information to an Azure BlobStorage to/from where the application will deposit/retrieve data.
Initially, I had assumed allowing users to specify a connection string and a custom blob container name (as an optional override of the default) would be sufficient. I could simply stuff that connection string into the CloudStorageAccount.Parse method and get back a storage account instance to call CreateBlobCloudClient on.
Now that I'm trying to use this method to connect using a container-specific SAS (also see my other question about that), it appears that the connection string might not be the most universal way to go.
Instead, it now seems a blob container URL, plus a SAS token or an account key (and possibly an account name, thought that seems to be included in the blob container URL already) are more versatile. However, I am concerned that the next way of pointing to a blob storage that I need to support (whichever that may be) might require yet another kind of information - hence my question:
What set of "fields" do I need to support in the configuration files of my application to make sure my users can point to their BlobStorage whichever way they want, as long as they have a BlobStorage?
(Is there maybe even a standard solution or best practice recommendation by Microsoft?)
Please note that I am exclusively concerned with what to store. An arbitrarily long string? A complex object of sorts? If so, with what fields?
I am not asking how to store that configuration once I know what it must comprise. For example, this is not about securely encrypting credentials etc.
On Workaround To access the Storage account using the SAS Token you need to pass the Account Name along with the SAS Token and Blob Name if you trying to upload and You need give the permission for your SAS Token .
Microsoft recommends using Azure Active Directory (Azure AD) to authorize requests against blob and queue data if possible, instead of Shared Key. Azure AD provides superior security and ease of use over Shared Key. For more information about authorizing access to data with Azure AD, see Authorize access to Azure blobs and queues using Azure Active Directory..
Note: Based on my testes you need to pass the Storage Account Name And SAS Token and the Container Name And Blob name
Example: I tried with uploading file to container using container level SAS Token . able to upload the file successfully.
const string sasToken = "SAS Token";
StorageCredentials storageCredentials = new StorageCredentials(sasToken);
const string accountName = "teststorage65";//Account Name
const string blobContainerName = "test";
const string blobName = "test.txt";
const string myFileLocation = #"Local Path ";
var storageAccount = new CloudStorageAccount(storageCredentials, accountName, null, true);
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
CloudBlobContainer blobContainer = blobClient.GetContainerReference(blobContainerName);
//blobContainer.CreateIfNotExists();
CloudBlockBlob cloudBlob = blobContainer.GetBlockBlobReference(blobName);
cloudBlob.UploadFromFile(myFileLocation);
As you already know You can use the Storage connection string to connect with Storage.
CloudStorageAccount storageAccount = CloudStorageAccount.Parse("Connection string");
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
CloudBlobContainer container = blobClient.GetContainerReference("test");
Your application needs to access the connection string at runtime to authorize requests made to Azure Storage.
You have several options for storing your connection string Or SAS Token
1) You can store your connection string in an environment variable.
2) An application running on the desktop or on a device can store the connection string in an app.config or web.config file. Add the connection string to the AppSettings section in these files.
3) An application running in an Azure cloud service can store the connection string in the Azure service configuration schema (.cscfg) file. Add the connection string to the ConfigurationSettings section of the service configuration file.
Reference: https://learn.microsoft.com/en-us/azure/storage/common/storage-configure-connection-string
I have saved pdf files in azure blob storage blob, I want to show these files on my website but when a file render on html its link should be deactivated means no one can use that link to download the file again. Is this possible in azure blob storage?
You can use the blob policy to make it:
CloudStorageAccount account = CloudStorageAccount.Parse("yourStringConnection");
CloudBlobClient serviceClient = account.CreateCloudBlobClient();
var container = serviceClient.GetContainerReference("yourContainerName");
container
.CreateIfNotExistsAsync()
.Wait();
CloudBlockBlob blob = container.GetBlockBlobReference("test/helloworld.txt");
blob.UploadTextAsync("Hello, World!").Wait();
SharedAccessBlobPolicy policy = new SharedAccessBlobPolicy();
// define the expiration time
policy.SharedAccessExpiryTime = DateTime.UtcNow.AddMinutes(1);
// define the permission
policy.Permissions = SharedAccessBlobPermissions.Read;
// create signature
string signature = blob.GetSharedAccessSignature(policy);
// get full temporary uri
Console.WriteLine(blob.Uri + signature);
If I understand correctly, you're looking for single use links to Azure Blobs. Natively this feature is not available in Azure Storage. You would need to write code to implement something like this where you would keep track of the number of times a link has been used and in case the limit exceeds, you will not process that link.
I would like to put audit data in immutable blob storage. Is there a specific format that needs to be written in?
Also once written how do we query or see this data in case we have to look at the data.
Would log analytic be able to this?
You have to create an immutable policy. You can do it using one of the management sdks or using the portal:
Create a blob container in any General Purpose V2 storage account. Then navigate to its properties and create a policy:
I used a legal policy, there is also a time based retention policy.
Offical documentation: here
You can read and write it using standard methods like (C#):
var storageAccount = CloudStorageAccount.Parse(CloudStorageConnectionString);
var blobClient = storageAccount.CreateCloudBlobClient();
var container = blobClient.GetContainerReference(StorageContainer);
var blob = container.GetBlockBlobReference("test");
blob.UploadText("Content");
Console.WriteLine(blob.DownloadText());
blob.UploadText("Modified Content"); // fails with a (409) conflict error with ErrorCode: BlobImmutableDueToLegalHold
I have a .NET app which uses the WebClient and the SAS token to upload a blob to the container. The default behaviour is that a blob with the same name is replaced/overwritten.
Is there a way to change it on the server, i.e. prevents from replacing the already existing blob?
I've seen the Avoid over-writing blobs AZURE but it is about the client side.
My goal is to secure the server from overwritting blobs.
AFAIK the file is uploaded directly to the container without a chance to intercept the request and check e.g. existence of the blob.
Edited
Let me clarify: My client app receives a SAS token to upload a new blob. However, an evil hacker can intercept the token and upload a blob with an existing name. Because of the default behavior, the new blob will replace the existing one (effectively deleting the good one).
I am aware of different approaches to deal with the replacement on the client. However, I need to do it on the server, somehow even against the client (which could be compromised by the hacker).
You can issue the SAS token with "create" permissions, and without "write" permissions. This will allow the user to upload blobs up to 64 MB in size (the maximum allowed Put Blob) as long as they are creating a new blob and not overwriting an existing blob. See the explanation of SAS permissions for more information.
There is no configuration on server side but then you can implement some code using the storage client sdk.
// retrieve reference to a previously created container.
var container = blobClient.GetContainerReference(containerName);
// retrieve reference to a blob.
var blobreference = container.GetBlockBlobReference(blobName);
// if reference exists do nothing
// else upload the blob.
You could do similar using the REST api
https://learn.microsoft.com/en-us/rest/api/storageservices/fileservices/blob-service-rest-api
GetBlobProperties which will return 404 if blob does not exists.
Is there a way to change it on the server, i.e. prevents from replacing the already existing blob?
Azure Storage Services expose the Blob Service REST API for you to do operations against Blobs. For upload/update a Blob(file), you need invoke Put Blob REST API which states as follows:
The Put Blob operation creates a new block, page, or append blob, or updates the content of an existing block blob. Updating an existing block blob overwrites any existing metadata on the blob. Partial updates are not supported with Put Blob; the content of the existing blob is overwritten with the content of the new blob.
In order to avoid over-writing existing Blobs, you need to explicitly specify the Conditional Headers for your Blob Operations. For a simple way, you could leverage Azure Storage SDK for .NET (which is essentially a wrapper over Azure Storage REST API) to upload your Blob(file) as follows to avoid over-writing Blobs:
try
{
var container = new CloudBlobContainer(new Uri($"https://{storageName}.blob.core.windows.net/{containerName}{containerSasToken}"));
var blob = container.GetBlockBlobReference("{blobName}");
//bool isExist=blob.Exists();
blob.UploadFromFile("{filepath}", accessCondition: AccessCondition.GenerateIfNotExistsCondition());
}
catch (StorageException se)
{
var requestResult = se.RequestInformation;
if(requestResult!=null)
//409,The specified blob already exists.
Console.WriteLine($"HttpStatusCode：{requestResult.HttpStatusCode},HttpStatusMessage：{requestResult.HttpStatusMessage}");
}
Also, you could combine your blob name with the MD5 code of your blob file before uploading to Azure Blob Storage.
As I known, there is no any configurations on Azure Portal or Storage Tools for you to achieve this purpose on server-side. You could try to post your feedback to Azure Storage Team.
I am developing an azure application which needs at some point to upload(download) a large amount of small blobs to a single container (more than 1k blobs, less than 1 Mb each). In order to speed up this process I'd like to use multiple threads for uploading(downloading) blobs.
This is routine I use for uploading single blob:
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(ConnectionString);
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
CloudBlobContainer blobContainer =
blobClient.GetContainerReference(ContainerName);
blobContainer.CreateIfNotExist();
CloudBlob blob = blobContainer.GetBlobReference(Id);
blob.UploadByteArray(Data);
For each type used in the code above MSDN says following:
Any public static (Shared in Visual Basic) members of this type are
thread safe. Any instance members are not guaranteed to be thread
safe.
Does it mean that I need to execute following code in every thread? Or maybe I can execute it only once and share single instance of CloudBlobContainer among different threads?
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(ConnectionString);
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
CloudBlobContainer blobContainer =
blobClient.GetContainerReference(ContainerName);
I would be really happy to use single instance of CloudBlobContainer in different threads otherwise it seriously slows down the whole uploading(downloading) process.
You should be fine sharing a single blob container reference as long as you are not trying to perform an update on the container itself (even then, I think it would still be fine in most scenarios like List). In fact, you don't really even need the container reference if you are sure it exists:
client.GetContainerReference("foo").GetBlobReference("bar");
client.GetBlobReference("foo/bar"); //same
As you can see, the only reason to get a container reference is if you want to perform an operation on the container itself (list, delete, etc.). If you keep the blob references in separate threads, you will be fine.