I have been using the Azure Function App consumption plan with Azure Blob Storage V1 for some time, with no problem.
Two months ago, we changed over to Azure blob Storage V2 and all of a sudden we noticed a drastic increase in costs.
After some investigation, we found that there are several 10's of thousands of API calls for
GetBlob,
GetBlobProperties,
GetBlobServiceProperties,
ListBlobs,
PurBlob,
RenewBlobLease.
My functions do not use this particular storage account for any storage, as I am using an external account for tables.
Has anyone experienced such an issue or know why these API calls are being called?
Maybe this storage is also the AzureWebJobsStorage of your function app?
The files of your function app will be stored in the AzureWebJobsStorage.
You can follow this to check the source of the request:
How to view Blob Storage logs in Azure App Insights?
Related
I have some e-mail attachments being saved to Azure Blob.
I am now trying to write a Azure Functions App that would connect to that blob storage, run some scripts and re-save the file.
However, when selecting a storage account for the function, I couldn't select my blob storage account.
I went on the website and it said this:
When creating a function app, you must create or link to a general-purpose Azure Storage account that supports Blob, Queue, and Table storage. Some storage accounts don't support queues and tables. These accounts include blob-only storage accounts and Azure Premium Storage.
I'm wondering, is there any workaround this? and if not, perhaps any other suggestions? I'm becoming a little lost in all the options, and which one to actually choose.
Thanks!
EDIT: Might I add I writing the function Python
I think you are overlooking the fact that you can have multiple storage accounts. In order for an Azure Function to work you need a storage account. That storage account is used to store runtime information of the Azure Function for internal purposes like state management. This storage account is subject to restrictions as you already found out. There is no workaround for that.
However, if the function you are writing needs to access another storage account it is free to do so. You just have to provide details to connect to that specific storage account. In that case you also have a clear seperation between the storage account that is used by the azure function for its internal operations and the storage account your application needs to connect and which you have total control about withouth having to worry that you break things by deleting internal used blobs/tables/queues.
You can have a blob triggered function that gets triggered when changes occur on your specific blob storage. That doesn't need to be the storage account that the azure function internally uses, which is created/selected when creating the azure function.
Here is a sample that shows how to add a blob triggered azure function in Python. MyStorageAccountAppSetting refers to an app setting that holds the connection string to the storage account that you use for storage.
The snippet from the website you are quoting is for storing the function app code itself and any related modules. It does not pertain to what your function can access when the code of your function executes.
When your function executes it will need to use the Azure Blob Storage SDK/modules to connect to your blob storage account and read the email attachments. Here's a quickstart guide for using Azure Storage with Python: Quickstart with Azure Storage Blobs SDK for Python
General-purpose v2 storage accounts support the latest Azure Storage features and incorporate all of the functionality of general-purpose v1 and Blob storage accounts here
There are more integration options with GPv2 accounts including Azure Function Triggers. See: Azure Blob storage bindings for Azure Functions
Further refer: Types of storage accounts
If Blob, based on your need, you can choose an access tier based on the frequency of access for the data (e-mail attachments)Access tiers for Azure Blob Storage - hot, cool, and archive. If General purpose storage account, its standard performance tier.
We receive large feed files(excel/csv) of ~5gb size every night, need to read and have to post one record after another from the file to azure service bus topic. Is it possible using Azure Data Factory, if yes, how? or any other better azure based solution available? please suggest.
Data Factory doesn't support Azure Service Bus topic. So it's impossible to using Azure Data Factory
For more details, please see : Supported data stores and formats.
I think Azure Function may can help you achieve it.
You can reference :
Azure Blob storage bindings for Azure Functions.
This article explains how to work with Azure Blob storage bindings in Azure Functions. Azure Functions supports trigger, input, and output bindings for blobs. The article includes a section for each binding: Blob trigger, Blob input binding,Blob output binding.
Azure Service Bus bindings for Azure Functions. This article explains how to work with Azure Service Bus bindings in Azure Functions. Azure Functions supports trigger and output bindings for Service Bus queues and topics
Here is a blog about Copy data from Azure Service Bus Topic to a Storage Account inside Azure.
We don't know how much time it will take. Azure Functions has two kinds of pricing plans:Consumption plan and App Service plan.
Azure Functions in a Consumption plan are limited to 10 minutes for a single execution. In the Premium plan, the run duration defaults to 30 minutes to prevent runaway executions. However, you can modify the host.json configuration to make this unbounded for Premium plan apps.
For more details, please reference: Azure Functions Premium plan.
Maybe Azure Function is not fit for this long process, but can help you execute this long process.
Hope this helps.
I'm developing a Stateless Service which will be in charge to process the blobs uploaded to a certain azure storage blob container. Is there a way to bind this Stateless Service to the blob the same way it can be done using Azure WebJobs?
As a workaround I was thinking of having a binded WebJob to that Azure Storage Blob Container and calling the Stateless Service everytime new blobs are uploaded.
Any sugestions or examples?
Thanks in advance!
It's possible to use the WebJob SDK in ASF services and use a blob trigger.
You can also use an Azure Function with a blob trigger, that then calls a service hosting a web api.
We are new to Windows azure and are developing a web application. In the beginning of the project , we have deployed complete code to different environments which actually publish complete code and uploaded blob objects to azure storage as we linked sitefinity to hold blob objects in azure storage . But now as we are in the middle of development , we are just required to upload any new blob files created which can be quite less in numbers (1 or 2 or maybe few).Now I would like to know best process to sync these blob files to different azure storage environments which is for each cloud service. So ideally we would like to update staging cloud service and staging storage first and then test there and then once no bugs are found, then will be required to update UAT and production storages as well with the changed or new blob objects.
Please help.
You can use the Azure Storage Explorer to manually upload/download blobs from storage accounts very easily. For one or two blobs, this would be an easy solution, otherwise you will need to write a tool that connects to the blob storage via an API and does the copying for you.
Why do we link an azure storage account to a cloud service? How does it help? What happens if I do not link them?
Two reasons:
Easier management - you have better idea of what is your overall configuration for a particular deployment
Easier management - upon deleting a resource you are being asked whether you want to delete the linked resources also
By the way, you can also link a Windows Azure SQL Database to a Cloud Service.
The whole idea is to help you better manage the services. There is no other reason and nothing will happen if you do not link. But think a bit - if you manage 3 subscriptions, 2 cloud services deployments each, 2 storage accounts per deployment. That is 6 cloud services, 12 storage accounts. Can you easily tell which service is using which account?
The cloud service depends on the storage account. When deploying the cloud service it will create a container called vsdeploy with a block blob that is used for the VMs it creates.
It also stores crash dump files there as well under the container wad-crashdumps. The folder structure is WAD{GUID}{worker role}{instance}. Then it will store all the .dmp files as block blobs.