From last few days I had charging approx $0.16 everyday for "LRS List and Create Container Operations". I don't know from where request is coming, I have checked all logs and metrics.
So here request is around 35k in last few days. But I don't have any idea from where all this request are coming.
Related
So suppose that you have an application that lets user request a job. For example (hypothetical): user uploads a video. There is an entry made in RDBMs with the URL to video on blob and the status is set to "Pending".
There is a recurring time triggered functionapp that is executed every 10 seconds or so which gets 10 pending jobs from RDBMS and performs some compression etc.
The problem here is that as long as the number of requests stay 10-30 videos per 10 seconds we should be fine. But if the number of requests increase all of a sudden .. say 200 requests per 10 seconds this would mean that there will be a lot of job pending and the user would have to wait 10 times longer than usual to see status change. How do you scale out function app automatically in such scenario? Does it have to be manual?
There's an easier way to get fan out and parallel processing through multiple concurrently running Azure Functions.
Add an Azure Service Bus Queue to your solution.
For each video that needs to be processed, enqueue a service bus message with the appropriate data you'll need to retrieve and process the video (like the BlobId).
Have your Azure Function triggered by an ServiceBusTrigger.
Azure will spin up additional instances of your Azure Function as the queue depth increases. It'll also scale in idle instances after there's no more data to process.
We include dates in the events sent to our hub. Whenever I connect a new Azure Function to our Event Hub with a new consumer group, it seems to receive all events ever sent to the hub. This is somewhat expected, however I set the Message Retention on the hub to 1 day, so I expected at most to receive one day worth of events for the new consumer, but it seems to receive all events, even months old events, based on the date within the message, and lots more events than we generate over a day.
Based on this page:
https://blogs.msdn.microsoft.com/servicebus/2015/03/09/data-retention-in-event-hubs/
It seems like maybe this retention period is somewhat irrelevant, or misleading. If the "container" hasn't filled up yet, it could contain messages forever. If, for example, the container has a limit of 1000 messages before the event hub looks at it, but it takes a year to generate 1000 messages, does that mean any new consumer could get year-old messages, even with a 1-day "retention period"?
When the container does hit the limit of 1000 messages, are the messages older than 1 day discarded and the messages newer than 1 day ago (within the retention period) retained? Or is the whole container discarded?
From looking at our test and prod environments it seems like this container fits at least 50000 messages (or equivalent size).
Is a checkpoint the only way to limit this initial influx of messages for a new consumer group?
Retention time is the minimum guaranteed period, not the maximum or exact. 1 day retention means you will have all the messages from last day, but maybe some more messages too.
So you can rely on 1 day of retention, but be prepared to see older messages too.
I created a QnA bot using the Azure Bot service, and now I'm seeing data transfers out of my subscription of over 1 GB a day! I cannot figure out why, but since it's billable, I'd like to know why and how I can stop it.
The bot isn't being used yet, so no one is sending queries to it. I'm confused how this is happening.
Here's a screen shot of the graph for use in the last hour as well as a screen shot of the billing for the last few days showing the sudden jump in use.
Is this normal?
If you add AzureWebJobsDisableHomepage with a value of true, to the App settings, the data out will stop.
The setting itself is documented here: https://github.com/Azure/azure-webjobs-sdk-script/wiki/Configuration-Settings (although it doesn't provide an explanation for how this setting affects a bot specifically)
The reasoning behind what is happening is a little complex. Azure Functions are not normally "in memory" and available all the time. There is a small spinup time that is not ideal within a bot. So, apparently there is a job setup with consumption plan bots to ping it every 10 seconds (and by 'ping', i mean retrieve the root of the site). If you open the Log Stream, you'll see an http get request every 10 seconds. Adding AzureWebJobsDisableHomepage doesn't disable the request, but changes the status of what is returned from "OK" to "NoContent".
This will be added to the Bot Service arm template soon (so future consumption plan bots do not automatically accrue these data usages).
I've got a nice logging system I've set up that writes to Azure Table Storage and it has worked well for a long time. However, there are certain places in my code where I need to now write a lot of messages to the log (50-60 msgs) instead of just a couple. It is also important enough that I can't start a new thread to finish writing to the log and return the MVC action before I know the log is successful because theoretically that thread could die. I have to write to the log before I return data to the web user.
According to the Azure dashboard, Table Storage transactions take ~37ms to commit, end to end (E2E), while queues only take ~6ms E2E to commit.
I'm now considering not logging directly to table storage, and instead log to an Azure Queue, then have a batch job run that reads off the queue and then puts them in their proper place in table storage. That way I can still index them properly via their partition and row keys. I can also write just a single queue message with all of the log entries. So it should only take 6ms instead of (37 * 50) ms.
I know that there are Table Storage batch operations. However, each of the log entries typically goes to different partition, and batch ops need to stay within a single partition.
I know that queue messages only live for 7 days, so I'll make sure I store queue messages in a new mechanism if they're older than a day (if it doesn't work the first 50 times, it just isn't going to work).
My question, then is: what am I not thinking about? How could this completely kick me in the balls in 4 months down the road?
I have am trying to acquire lease on a blog but it gives me a 400 Bad Request. If the time span is more than 1 minute.
Is this expected? I am not able to find any documentation on it.
Originally, blob leases were simply 60 seconds. Starting with the 2012-02-12 version, blob lease durations can be between 15-60 seconds, or infinite. You can read this blog post for more details around all the updates in the 2012-02-12 update.
If you want to see the actual REST API details behind the SDK, it's documented here. In this page, you'll see that an infinite lease duration is specified as -1 (which is slightly different from, say, the .net SDK call, which uses null to signify infinite lease).