Is there currently a way to autoscale an Azure App Service plan by a "v2" storage account message queue? When I attempt to set up autoscaling based on a message queue, it is only showing my "classic" storage accounts. I would prefer to not destroy and recreate the storage account as classic just because of this.
Currently that supports classic storage accounts only. I would suggest you to provide feedback here: https://feedback.azure.com/forums/169385-web-apps-formerly-websites
Related
I set up a simple scenario in Azure using a Storage Account, a Function App, and an Event Grid System Trigger. Blob uploads into the Storage Account should cause the Event Grid System Trigger to send a BlobCreated event to trigger the Azure Function.
I can see that the Event Grid System Topic appears to be configured for the correct storage account according to the overview page in the Azure Portal:
I have a subscription created for the Event Grid System Topic, and it subscribes to all of the events the storage account can generate as I can see in the Azure Portal. This shows all 6 event types enabled, so I'm not filtering them out.
Despite this, when I upload blobs into a container I created in my storage account and watch for the events to show up in the metrics on my Event Grid System Topic, or see my Azure Function trigger, no events appear to ever be generated. Some interesting points about my storage account which may be worth mentioning are:
I am using a premium storage account
I am using a private vnet for my storage account
I suspected the network, but to rule that out I changed my storage account back to public and tried again but it didn't change the behavior. From everything I can tell from documentation, this should be working. Any ideas why it isn't?
I work at MS in the SDK team, and I reached to an EventGrid team member directly for opinion:
I looked into our service logs for last two weeks and I could not find
any events for this topic/event-subscription.
Can you please provide specific time and region when you are
uploading/deleting/editing the blobs to help investigating? Also, is
this specific to this storage account? Was this working before or this
scenario working for other storage accounts? Can you please open a
support ticket to handle this properly.
Thanks! In any doubt on the process, feel free to reply to me, we'll monitor this thread
[Edit: more info from Storage team]
We communicated with Azure Storage team and they confirmed that the behavior as described is by design and expected. Here are some additional details from Azure Storage Team:
The issue is that the customer is using a Premium_LRS StorageV2
account. These accounts only support premium page blobs and premium
disks.
If the customer wants to store block blobs in the premium tier, they
need to create a BlockBlobStorage account.
See subscript 5 in this table:
https://learn.microsoft.com/en-us/azure/storage/common/storage-account-overview
We are making use of Azure Functions (v2) extensively to fulfill a number of business requirements.
We have recently introduced a durable function to handle a more complex business process which includes both fanning out, as well as a chain of functions.
Our problem is related to how much the storage account is being used. I made a fresh deployment on an account we use for dev testing on Friday, and left the function idling over the weekend to monitor what happens. I also set a budget to alert me if the cost start shooting up.
Less than 48 hours later, I received an alert that I was at 80% of my budget, and saw how the storage account was single handedly responsible for the entire bill. The most baffling part is, that it's mostly egress and ingress on file storage, which I'm entirely not using in the application! So it must be something internal by the azure function implementations. I've dug around and found this. In this case the issue seems to have been solved by switching to an App Service plan, but this is not an option in our case and must stick to consumption. I also double checked and made sure that I don't have the AzureWebJobsDashboard setting.
Any ideas what we can try next?
The below are some interesting charts from the storage account. Note how file egress and ingress makes up most of the activity on the entire account.
A ticket for this issue has also been opened on GitHub
The link you provided actually points to AzureWebJobsDashboard as the culprit. AzureWebJobsDashboard is an optional storage account connection string for storing logs and displaying them in the Monitor tab in the portal. The storage account must be a general-purpose one that supports blobs, queues, and tables.
For performance and experience, it is recommended to use
APPINSIGHTS_INSTRUMENTATIONKEY and App Insights for monitoring instead
of AzureWebJobsDashboard
When creating a function app in App Service, you must create or link to a general-purpose Azure Storage account that supports Blob, Queue, and Table storage. Internally, Functions uses Storage for operations such as managing triggers and logging function executions. Some storage accounts do not support queues and tables, such as blob-only storage accounts, Azure Premium Storage, and general-purpose storage accounts with ZRS replication. These accounts are filtered out of from the Storage Account blade when creating a function app.
When using the Consumption hosting plan, your function code and
binding configuration files are stored in Azure File storage in the
main storage account. When you delete the main storage account, this
content is deleted and cannot be recovered.
If you use the legacy "General Purpose V1" storage accounts, you may see your costs drop by up to 95%. I had a similar use case where my storage account costs exploded after the accounts were upgraded to "V2". In my case, we just went back to V1 instead of changing our application.
Altough V1 is now legacy, I don't see Azure dropping it any time soon. You can still create it using the Azure Portal. Could be a medium-term solution.
Some alternatives to save costs:
Try the "premium" performance tier (V2 only). It is cheaper for such workloads.
Try LRS or ZRS as the redundancy setting. Depends on the criticality of this orchestration data.
PS: Our use case were some EventHub processors which used the storage accounts for coordination and checkpointing.
PS2: Regardless of the storage account configuration, there must be a way reduce the traffic towards the storage account. It is just another thing to try to reduce costs.
I'm building a queue messaging system in Azure and what I'm trying to do is an outbound message queue container in Azure Storage Queue that allows my desktop Windows Services to get the latest messages from that queue. The problem I'm facing is that I want to have multiple queues per tenant (each Windows service serves one client) in one storage account. As far as I see, there is no way to restrict the connection string access to each queue. On the other hand, it is not practical for me to create one storage account per tenant. What is the best way to restrict client access to one specific queue with the current security methods available in Azure? I was thinking about using Service Bus Queues, but even that doesn't solve the connection string issue I have in the client application.
I think service bus queues is your answer; they allow a multi-subscriber model with "subjects" and various filters etc.
Storage queues are very simplistic and are not the right answer for this particular scenario.
Sorry, on my mobile so haven't got all the relevant docs to hand.
One option is to use AAD identities and Storage's AAD authentication support (which is currently in public preview).
You would need a Service Principal in Azure AD for each tenant for this,
and add the principal to the Storage Queue Data Reader or Storage Queue Data Contributor role on their respective queue.
You can then use the principal's credentials to get an access token that is tenant-specific.
Documentation:
https://azure.microsoft.com/en-us/blog/announcing-the-preview-of-aad-authentication-for-storage/
https://joonasw.net/view/azure-ad-authentication-with-azure-storage-and-managed-service-identity
I would like to build a trigger like thing that can monitor my Azure File Storage for new coming files. If there are new files then create a VM.
What should I look at? Monitor + Alert or Logic Apps Designer?
Any one done anything similar before? Thanks!
What should I look at? Monitor + Alert or Logic Apps Designer?
I recommand that you could use Azure function+ Event Grid.
You could use the Event Grid to listen the change of the storage and implement the creating VM in the Azure function. About how to handle Event Grid events in Azure Function, please refer to this document.
Note: Currently, the following Azure services support sending events to Event Grid:
Azure Subscriptions (management operations)
Container Registry
Custom Topics
Event Hubs
IoT Hub
Media Services
Resource Groups (management operations)
Service Bus
Storage Blob
Storage General-purpose v2 (GPv2)
If Azure blob storage is possible, you could use the Azure function with blob trigger.
In one of my Azure Web Apps I create an Azure Topic when it doesn't exist yet during the warmup(start) of our Azure Web App.
On topic creation time I want to know whether the service bus has a premium pricing tier, when yes I want to disable express, when no (standard pricing tier) I want to enable express, to prevent exceptions.
Is there a defensive way to check if a premium pricing tier is available on the service bus (for example: using the service bus connection string) ?
When there is no defensive way, I can always catch the web exception that know raises, but I want to prevent the exception if I can.
Edit: After consulting our lead-dev we decided to skip the EnableExpress setting completely in our DTAP. So I don't need to implement the SKU check at all. Be aware to not set the EnableExpress property at all otherwise you get the webexception in Premium SKU.
Is there a defensive way to check if a premium pricing tier is
available on the service bus (for example: using the service bus
connection string)?
Unfortunately there's none as of today. Service Bus Client SDK does not expose this information. This feature has been asked from the Service Bus team and there's an open issue on Github for that: https://github.com/Azure/azure-service-bus/issues/42.
The differences between Premium and Standard tiers are highlighted here: https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-premium-messaging. Kind of an anti-pattern but one thing you could do is perform and operation that is only available in Premium tier (say sending a message greater than 256KB in size) and catch the exception (or lack of) to determine if the Service Bus tier is Premium or not.
If we want to check the Azure Service Bus Pricing tier,we could use the following code with Microsoft.Azure.Management.Fluent SDK.
var credentials = SdkContext.AzureCredentialsFactory.FromFile(#"c:\tom\azureCredential.txt");
var azure = Azure
.Configure()
.WithLogLevel(HttpLoggingDelegatingHandler.Level.Basic)
.Authenticate(credentials)
.WithDefaultSubscription();
var serviceBus = azure.ServiceBusNamespaces.GetByResourceGroup("resourcegroup", "servicebusnamespace");
var priceTier = serviceBus.Sku.Tier;
Before code we need to create an azure active directory application and assign the correspondent role. We could create the azure credential file following the document. The following is the credential file format.
subscription=########-####-####-####-############
client=########-####-####-####-############
tenant=########-####-####-####-############
key=XXXXXXXXXXXXXXXX
managementURI=https\://management.core.windows.net/
baseURL=https\://management.azure.com/
authURL=https\://login.windows.net/
graphURL=https\://graph.windows.net/