How would I find the total cost of an Azure function (including costs incurred by the Azure function on the Azure storage account such as cost of LRS List and Create Container Operations when multiple Azure functions are using the same Azure storage account)?
You can See functions costs by going to
Cost Management >> Cost Analysis >> View CostByResource >> Sort By ResourceType
For me, it doesn't give much details, the best option is to have an App Service Plan which is applied to the function, where you can get CPU/Memory consumption cost:
This time Select previews (Sort By ResourceType):
As for the Storage Account cost related to a function, you'll get all details for Storage costs but not specific to functions ( I'd assume you'll use storage account only for functions)
Related
Do you get billed in Azure for each time you publish an Azure Functions from VS?
The short answer is Yes, you get charged for publishing an Azure Function from Visual Studio. But "each time?" well, not really.
So, let's get to understand how that works. Azure functions although they offer you 1,000,000 executions per month (considering the execution time and memory), your code has to live somewhere which is the Function Storage Account.
Storage Accounts pricing could be broken down into two main costs:
Storage:
You pay for storage per month (pay-as-you-go) unless you are on a Premium storage plan. The first 50 TB of data in a Hot access tier of blob data is ~$0.0184 to $0.0424 per GB depends on where your data is hosted and its redundancy.
Now in your case that cost will incure once per 50 TB per month
Transfer:
When you deploy your data via Visual Studio you're effectively making API calls to Write your data which is charged (also depends on your data host location and redundancy) per 10,000 operations that includes every time you or your function does PutBlob, PutBlock, PutBlockList, AppendBlock, SnapshotBlob, CopyBlob, and SetBlobTier on that Storage Account. The operations cost varies from $0.05 to $0.091 for every 10,000 operations.
Others:
Other costs may incure using features such as Blob Index, Blob Changes, and encryption.
Conclusion
Publishing your Function from Visual Studio contributes to the overall cost of the Functions's Storage Accout. However, the cost is very small (sub $1) even if you published your function thousands of times every month.
For more information about Azure Blob Storage pricing visit https://azure.microsoft.com/en-us/pricing/details/storage/blobs/#pricing
As part of a client requirement, I've been asked to set up central log repository for different Azure workloads including Storage accounts & databases. I see a default diagnostic setting in place but all of those are disabled. To enable these, we need to enable certain logs/metrics which will further be ingested into the workspace. Now I want to make a cost-effective & most accurate selection of the logs/metrics for storage accounts. Can someone with more profound knowledge into this domain enlighten me about it?
Similarly for Postgre SQL & Cosmos DB databases too, I have to make such decision. Please help me with this.
Please check the below points and references in detail.
Selection:
You can select the logs for the operations that you want to Get all the details you wish for.selection depends on the requirement.
A good practice is to go through your agents and monitoring settings
and see exactly what you are logging. Capture logs which are
important for your monitoring purpose.
Choose the cheapest region to create and store your log analytics
workspace.
If you have very high volume of the log ingestion then it would be
prudent to opt for azure commitment tier.
In case you need to export the log analytics data, rather than
exporting all the data, you can filter it and send only relevant log
data
Above things can significantly reduce your azure billing cost and help you to save money in using azure monitor effectively. Understand Azure Monitor and Log Analytics Pricing and Cost Optimization (azurelib.com)
Storing:
Log data can accumulate in your account over time which can increase the cost of storage.
If you need log data for only a small period of time, you can reduce
your costs by modifying the log data retention period to less days.
Use lifecycle policy to move data between access tiers.
Data ingested into Log Analytics workspace can be retained at no
additional charge(free) up to the first 31 days.
See
Design considerations and change the data retention if
not needed more than that. See Monitoring Azure Blob Storage
| Microsoft Docs.
Storage Insights is a dashboard on top of Azure Storage metrics and
logs. You can use Storage Insights to examine the transaction volume
and used capacity of all your accounts. That information can help
you decide which accounts you might want to retire.
Analyze:
Analyze the used capacity and monitor the use of the container.
you can consider reducing the total cost by exporting logs to
storage account, and then using a serverless query solution on top
of log data.See blob storage monitoring/optimize cost for infrequent
queries
Organize data into access tiers.Log Analytics has Commitment Tiers,
which can save you as much as 30 percent compared to the
Pay-As-You-Go price.
You should periodically review this information to determine if you
can reduce your charges by moving to another tier
References:
Plan and manage costs for Azure Blob Storage | Microsoft Docs
Azure Monitor Logs pricing details - Azure Monitor | Microsoft Docs
Azure Monitor Log Analytics too Expensive? Part 2 - Save Some Money
| Thomas Stringer (trstringer.com)
Because I have been charge a lot of money by storage, and I have so many storage account. I want to know how can I check how much my total storage usage in my subscription ?
I have check this this similar question, but they only provide how to check each storage account.
The quickest way:
Nav to azure portal -> Monitor -> Storage accounts: Then select your subscription; For storage accounts, select "All" -> At last, click "Capacity". The screenshot as below:
Remember that you should calculate the total space used one by one, there is no such built-in feature. Or you can write a code calculate total space used one by one.
We are making use of Azure Functions (v2) extensively to fulfill a number of business requirements.
We have recently introduced a durable function to handle a more complex business process which includes both fanning out, as well as a chain of functions.
Our problem is related to how much the storage account is being used. I made a fresh deployment on an account we use for dev testing on Friday, and left the function idling over the weekend to monitor what happens. I also set a budget to alert me if the cost start shooting up.
Less than 48 hours later, I received an alert that I was at 80% of my budget, and saw how the storage account was single handedly responsible for the entire bill. The most baffling part is, that it's mostly egress and ingress on file storage, which I'm entirely not using in the application! So it must be something internal by the azure function implementations. I've dug around and found this. In this case the issue seems to have been solved by switching to an App Service plan, but this is not an option in our case and must stick to consumption. I also double checked and made sure that I don't have the AzureWebJobsDashboard setting.
Any ideas what we can try next?
The below are some interesting charts from the storage account. Note how file egress and ingress makes up most of the activity on the entire account.
A ticket for this issue has also been opened on GitHub
The link you provided actually points to AzureWebJobsDashboard as the culprit. AzureWebJobsDashboard is an optional storage account connection string for storing logs and displaying them in the Monitor tab in the portal. The storage account must be a general-purpose one that supports blobs, queues, and tables.
For performance and experience, it is recommended to use
APPINSIGHTS_INSTRUMENTATIONKEY and App Insights for monitoring instead
of AzureWebJobsDashboard
When creating a function app in App Service, you must create or link to a general-purpose Azure Storage account that supports Blob, Queue, and Table storage. Internally, Functions uses Storage for operations such as managing triggers and logging function executions. Some storage accounts do not support queues and tables, such as blob-only storage accounts, Azure Premium Storage, and general-purpose storage accounts with ZRS replication. These accounts are filtered out of from the Storage Account blade when creating a function app.
When using the Consumption hosting plan, your function code and
binding configuration files are stored in Azure File storage in the
main storage account. When you delete the main storage account, this
content is deleted and cannot be recovered.
If you use the legacy "General Purpose V1" storage accounts, you may see your costs drop by up to 95%. I had a similar use case where my storage account costs exploded after the accounts were upgraded to "V2". In my case, we just went back to V1 instead of changing our application.
Altough V1 is now legacy, I don't see Azure dropping it any time soon. You can still create it using the Azure Portal. Could be a medium-term solution.
Some alternatives to save costs:
Try the "premium" performance tier (V2 only). It is cheaper for such workloads.
Try LRS or ZRS as the redundancy setting. Depends on the criticality of this orchestration data.
PS: Our use case were some EventHub processors which used the storage accounts for coordination and checkpointing.
PS2: Regardless of the storage account configuration, there must be a way reduce the traffic towards the storage account. It is just another thing to try to reduce costs.
Having used Azure for some time now, I'm well aware of the default 20,000 IOPS limit of an Azure Storage Account. What I've yet to find however is up to date documentation on how to monitor an account's IOPS in order to determine whether or not it's being throttled. This is important when debugging performance issues for applications, VMs, and ASR replication - to name but three possible uses.
If anyone knows the correct way to keep track of an account's total IOPS and/or whether it's being throttled at any point in time, I'd appreciate it - if there's a simple solution for monitoring this over time, all the better, otherwise if all that exists is an API/PowerShell cmdlet, I guess I'll have to write something to save the data periodically over time.
You can monitor your storage account for throttling using Azure Monitor | Metrics. There are 3 metrics relevant to your question, which are
AnonymousThrottlingError
SASThrottlingError
ThrottlingError
These metrics exist for each of the 4 storage account abstractions (blob, file, table, queue). If you're unsure how your storage account is being used then monitor these metrics for all 4 services. Things like ASR, Backup and VM's are going to be using the blob service.
To configure this, go to the Azure Monitor | Metrics blade in the portal and select the storage account(s) you want to monitor. Then check off the metrics you're interested in. The image blow shows the chart with these 3 metrics configured for the blob service.
You can also configure an alert based on these metrics to alert you when any of these throttling events occur.
As for measuring the IOPS for the storage account, you could monitor the Transactions metric for the storage account. This is not really measuring the IOPS, but it does give you some visibility into the number of transactions (which sort of relates to IOPS) across the storage account. You can configure this from the storage account blade and clicking Metrics in the Monitoring section as shown below.