I am trying to accomplish, that when I create a snapshot with an Azure function on create/modify, that it also sets the snapshot to the cool tier.
The storage account itself, needs to be the Hot tier, while snapshots should not.
How do i accomplish this?
I have tried to look through the documentation, but are only able to see the GUI guide for doing it on single blobs manually.
Can I not accomplish this in the blobtrigger itself?
[FunctionName("CreateSnapshotAndCool")]
public static void Run([BlobTrigger("images/{name}", Connection = "AzureWebJobsStorage")]CloudBlockBlob myBlob, string name, ILogger log)
{
log.LogInformation($"C# Blob trigger function Processed blob\n Name:{name}");
log.LogInformation("take snapshots for blob: " + name);
myBlob.SnapshotAsync().Wait();
// Change snapshot just created to cool tier.
}
Yes, we can accomplish the same with blobtrigger. Coming to the pricing tier, if you have changed a blob or snapshot's tier, then you are billed for the entire object, regardless of whether the blob and snapshot are eventually in the same tier again.
Hope it will helpful. Refer below link for more details
Blob snapshots - Azure Storage | Microsoft Learn
Steps to create blobtrigger:
Create / Add Function App into an App Services
Create an Azure Blob Storage trigger into that.
We can add input / output parameters as required.
Related
I am facing a problem. I have to implement a solution to load multiple images from Azure Blob Storage in Xamarin.Forms app.
I would like to know how to send List of image names and get List of images back from Azure Blob Storage Account.
Thank you !
Azure Blob doesn't have a strong query\search story. If you know the container you could always iterate through the blobs in that container but this is going to be very inefficient and less then ideal. You can also use Azure Search to index blob storage (and it's contents) but I find that overkill for your use case.
My suggestion is to store a reference to the blob in a searchable data source. If you already have a data source in place for your app (such as SQL) I would use that.
One inexpensive way to get this into your data source to query later is to use an Azure Function or Logic App to trigger when a new blob is created and store the data you need to later find it (e.g. the filename) along with the blob reference. Table storage is a very inexpensive method or you can use any data store you like. You can then host an API endpoint in Azure Functions (or your host of choice) where your Xamarin app can pass in the filenames and get the results back.
With Azure Functions the code for a a blob trigger that writes to table storage is pretty minimal and would follow something like the pattern below:
public static void Run(
[BlobTrigger("mycollection/{name}")] BlobProperties blobProperties,
[Table("blobdata")]IAsyncCollector<MyBlobDataPoco> tableData,
string name, TraceWriter log)
{
var myData = new MyBlobDataPoco() { Filename = name, Container= "mycollection" };
tableData.AddAsync(myData);
}
I'm trying to set up a function to take a snapshot of a blob container every time a change is pushed to it. There is some pretty simple functionality in Azure Functions to do this, but it only works for general purpose storage accounts. I'm trying to do this with a blob only storage account. I'm very new to Azure so I may be approaching this all wrong, but I haven't been able to find much helpful information. Is there any way to do this?
As #joy-wang mentioned, the Azure Functions Runtime requires a general purpose storage account.
A general purpose storage account is required to configure the AzureWebJobsStorage and the AzureWebJobsDashboard settings (local.settings.json or Appsettings Blade in the Azure portal):
{
"IsEncrypted": false,
"Values": {
"AzureWebJobsStorage": "my general purpose storage account connection string",
"AzureWebJobsDashboard": "my general purpose storage account connection string",
"MyOtherStorageAccountConnectionstring": "my blob only storage connection string"
}
}
If you want to create a BlobTrigger Function, you can specify another connection string and create a snapshot everytime a blob is created/updated:
[FunctionName("Function1")]
public static async Task Run([BlobTrigger("test-container/{name}",
Connection = "MyOtherStorageAccountConnectionstring")]CloudBlockBlob myBlob,
string name, TraceWriter log)
{
log.Info($"C# Blob trigger function Processed blob\n Name:{name}");
await myBlob.CreateSnapshotAsync();
}
In the Visual Studio:
I have tried to create snapshot for a blob-only storage
named joyblobstorage , but it failed. I supposed you should get the same error in the screenshot.
As the error information says Microsoft.Azure.WebJobs.Host: Storage account 'joyblobstorage' is of unsupported type 'Blob-Only/ZRS'. Supported types are 'General Purpose'.
In the portal:
I try to create a Function App and use the existing Storage, but it could not find my blob-only storage account. Azure Function setup in portal should not allow we to select a blob-only storage account. Please refer to the screenshot.
Conclusion:
It is not possible to create snapshot for a blob-only storage. In the official documentation, you could see the Storage account requirements.
When creating a function app in App Service, you must create or link to a general-purpose Azure Storage account that supports Blob, Queue, and Table storage.
Also, in the App settings reference, you could see
AzureWebJobsStorage
The Azure Functions runtime uses this storage account connection string for all functions except for HTTP triggered functions. The storage account must be a general-purpose one that supports blobs, queues, and tables.
AzureWebJobsDashboard
Optional storage account connection string for storing logs and displaying them in the Monitor tab in the portal. The storage account must be a general-purpose one that supports blobs, queues, and tables.
Here is the Feedback, Azure App Service Team has explained the requirements on storage account, you could refer to it.
I want to create a blob storage trigger that takes any files put into blob storage (a fast process) and transfers them to Data Lake storage (NOT to another Blob Storage location).
Can this be done?
Can it be done using JavaScript, or does it require C#?
Does sample code exist showing how to do this? If so, would you be so kind as to point me to it?
Note: we've created a pipeline that will go from Blob Storage to Data lake storage. That's not what I'm asking about here.
You could potentially use an Azure Function or Azure Logic App to detect new files on Blob Storage and either call your webhook to trigger the pipeline or do the move itself.
Can this be done?
As jamesbascle mentioned that we could use Azure function to do that.
Can it be done using JavaScript, or does it require C#?
It can be done with javascript or C#.
Does sample code exist showing how to do this? If so, would you be so kind as to point me to it?
How to create a Blob storage triggered function, please refer to this document. We also could get the C#/javascript demo code from this document.
JavaScript code
module.exports = function(context) {
context.log('Node.js Blob trigger function processed', context.bindings.myBlob);
context.done();
};
C# code
[FunctionName("BlobTriggerCSharp")]
public static void Run([BlobTrigger("samples-workitems/{name}")] Stream myBlob, string name, TraceWriter log)
{
log.Info($"C# Blob trigger function Processed blob\n Name:{name} \n Size: {myBlob.Length} Bytes");
}
I have to get the file Contents from Azure Blob Storage only when it is updated/Created the same file in the Azure Storage. This has to be done through C#.
I have to get the file Contents from Azure Blob Storage only when it is updated/Created the same file in the Azure Storage. This has to be done through C#.
According to your description, I suggest you could try to use azure webjobs or functions blob trigger to get the file content from the blob storage.
The blobtrigger will trigger a process when an Azure blob is created or updated.
More details, you could refer to this article and below code sample.
public static void WriteLog([BlobTrigger("input/{name}")] string logMessage,
string name,
string blobTrigger,
TextWriter logger)
{
logger.WriteLine("Full blob path: {0}", blobTrigger);
logger.WriteLine("Content:");
logger.WriteLine(logMessage);
}
Notice: The SDK scans log files to watch for new or changed blobs. This process is not real-time; a function might not get triggered until several minutes or longer after the blob is created.
If the speed and reliability limitations of blob triggers are not acceptable for your application, the recommended method is to create a queue message when you create the blob, and use the QueueTrigger attribute instead of the BlobTrigger attribute on the function that processes the blob.
Azure WebJob SDK uses the storage connection string defined in the AzureWebJobsStorage and AzureWebJobsDashboard app settings for its logging and dashboard.
WebJob SDK creates the following blob container in AzureWebJobsStorage:
azure-webjobs-hosts
WebJob SDK creates the following blob containers in AzureWebJobsDashboard
azure-jobs-host-output
azure-webjobs-hosts
Many blobs are created in the above blob containers as the WebJob runs. The containers can be bloated or saturated if there is no clean-up mechanism.
What is the cleanup mechanism for the above blob containers?
Update
The answer below is a workaround. At this point, there is no built-in mechanism to clean up the WebJobs logs. The logs can pile up quite large as the Job runs in a long term. Developers must create the cleanup mechanism on their own. Azure Functions is a good way of implementing such cleanup process. An example is provided in the below answer.
What is the clean up mechanism for the blobs that WebJobs SDK creates in the AzureWebJobsDashboard connection?
I haven’t found a way to do it. There is an open issue on GitHub which related to this topic, but haven’t been closed.
No way to set webjob logging retention policy
In a similar issue on GitHub we found that Azure WebJob SDK have changed a way of saving logs to multi tables of Azure Table Storage. We can easily delete the table per month. For logs writen in Azure Blob Storage haven’t been grouped by month until now.
WebJobs.Logging needs to support log purge / retention policies
To delete the older WebJob log, I suggest you create a time triggered WebJob to delete the logs which you wanted.
Is there any AzureFunction code sample shows how to do the blob cleanup?
Code below is for your reference.
// Parse the connection string and return a reference to the storage account.
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(storageConnectionString);
// Create the table client.
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
// Retrieve a reference to a container.
var container = blobClient.GetContainerReference("azure-webjobs-hosts");
// Query out all the blobs which created after 30 days
var blobs = container.GetDirectoryReference("output-logs").ListBlobs().OfType<CloudBlob>()
.Where(b => b.Properties.LastModified < new DateTimeOffset(DateTime.Now.AddDays(-30)));
// Delete these blobs
foreach (var item in blobs)
{
item.DeleteIfExists();
}