I'm using azure search for my app, and lately i'm facing performance issues.
Currently i'm investigating problem and i came across the following article:
https://learn.microsoft.com/en-us/azure/search/search-performance-optimization#scaling-azure-search-for-high-query-rates-and-throttled-requests
It says:
Scaling Azure Search for high query rates and throttled requests
When you are receiving too many throttled requests or exceed your target
latency rates from an increased query load, you can look to decrease
latency rates in one of two ways: Increase Replicas: A replica is like
a copy of your data allowing Azure Search to load balance requests
against the multiple copies. All load balancing and replication of
data across replicas is managed by Azure Search and you can alter the
number of replicas allocated for your service at any time. You can
allocate up to 12 replicas in a Standard search service and 3 replicas
in a Basic search service. Replicas can be adjusted either from the
Azure portal or PowerShell. Increase Search Tier: Azure Search comes
in a number of tiers and each of these tiers offers different levels
of performance. In some cases, you may have so many queries that the
tier you are on cannot provide sufficiently low latency rates, even
when replicas are maxed out. In this case, you may want to consider
leveraging one of the higher search tiers such as the Azure Search S3
tier that is well suited for scenarios with large numbers of documents
and extremely high query workloads.
Now i can't figure out what throttled requests means.
Google didn't help!
Azure Search starts throttling requests when the error rate (requests failing with 207 or 503 status codes) exceeds a certain threshold. The best strategy is to use an exponential retry policy on 207 and 503 responses to control the load and avoid throttling altogether.
Throttled requests have the throttle-reason response header that contains information about why the request was throttled. It appears we haven't documented that; we'll work on fixing that.
Related
I have one user bulk deleting some 50K documents from one container using a stored procedure.
Meanwhile another user is trying to login to the web app (connected to same cosmos db) but the request fails due to rate limit being exceeded.
What should be the best practice in this case in order to avoid service shortages like the one described?
a) Should I provision RUs by collection?
b) Can I set a cap on the RU's consumed by bulk operations from code when making a request?
c) is there any other approach?
More details on my current (naive/newbie) implementation:
Two collections : RawDataCollection and TransformedDataCollection
Partition key values are the customer account number
RU set at the database level (current dev deployment has minimum 400RUs)
Bulk insert/delete actions are needed in both collections
User profile data (for login purposes, etc.) is stored in RawDataCollection
Bulk actions are low priority in terms of service level, meaning it could be put on hold or something if a higher priority task comes in.
Normally when user logs in, retrieves small amounts of information. This is high priority in terms of service level.
It is recommended to not use Stored Procedures for bulk delete operations. Stored procedures only operate on the primary replica meaning they can only leverage 1/4 of total RU/s provisioned. You will get better throughput usage and more efficiency doing bulk operations using SDK client in Bulk Mode.
Whether you provision throughput at the database level or container level depends on a couple of things. If you have a large number of containers that get roughly the same number of requests and storage, database level throughput is fine. If the requests and storage is asymmetric then provision those containers which diverge greatly from the others with their own dedicated throughput. Learn more about the differences.
You cannot throttle requests on a container directly. You will need to implement Queue-based load leveling in your application.
Overall if you've provisioned 400 RU/s and trying to bulk delete 50K records, you are under provisioned and need to increase throughput. In addition, if you're workload is highly variable with long periods of little to no requests with short periods of high volume, you may want to consider using Serverless throughput or Autoscale
Lately I'm facing some performance issues while querying over my Azure search service index. I'm trying to figure out what happens. I came across the following article:
Azure Search performance and optimization considerations
It says:
Uploading of content to Azure Search will impact the overall performance and latency of the Azure Search service. If you expect to send data while users are performing searches, it is important to take this workload into account in your tests.
I want to clarify something. If, for example, I have two indexes on my search service account, let say: index-a, index-b.
If I upload content to index-a, it will impact the overall performance and latency of index-b?
If both indexes are within the same service, then yes, one index will have its performance affected by the other one. How much it's affected will depend on the service tier and the amount of information you are indexing.
How reliable is the insertion mechanism to azure search?
Say, a call on average to upload to azure search. Are there any slas on this? average insertion time for one document, average failure rate for one document.
I'm trying to send data from my database to azure search and I was wondering if it was more reliable to send data directly to azure search, or do a dual write for example to a high available queue like kafka and read from there.
From SLA for Azure Search:
We guarantee at least 99.9% availability for index query requests when
an Azure Search Service Instance is configured with two or more
replicas, and index update requests when an Azure Search Service
Instance is configured with three or more replicas. No SLA is provided
for the Free tier.
Your client code needs to follow the best practices: batch indexing requests, retry on transient failures with an exponential back-off policy, and scale service appropriately based on the size of the documents and indexing load.
Whether or not use an intermediate buffer depends not so much on SLA, but how spiky your indexing load will be, and how decoupled you want your search indexing component to be.
You may also find Capacity planning for Azure Search useful.
I'm a bit confused by Azure price calculator. In particular it doesn't explain the bandwidth pricing.
I'm considering Azure for a restful api that is going to use blobs for most data storage together with a sql server database for a subset that is easier to manage with a relational approach.
In this application a lot of data will enter the system through the ReST api, but a small fraction will be exposed to the clients (mainly as summary reports). Still the total bandwidth required should be in the order of 50 GiB/mo.
In the Azure's pricing page related to data transfer I see the pricing is only related to outgoing data, but I cannot figure how this relates to a ReST api that will be hosted in Azure App Service.
I mean, it could just mean that I'm going to pay for the bandwidth consumed by HTTPS responses (and not by HTTPS requests), but it seems a bit hard to estimate what this pricing is going to be.
Within a given region, there are no transfer costs at all. You mentioned using App Service, blobs, and SQL Database. As long as those services are within a single region, there are zero bandwidth costs as data flows between them and any other service within that region.
Bandwidth is billed specifically for outbound transfer. So, essentially you're metered for all data leaving a given region.
If you look at the page Data Transfers Pricing Details
Data Transfers refer to data moving in and out of Azure data centres other than those explicitly covered by the Content Delivery Network or ExpressRoute pricing.
Inbound data transfers
(i.e. data going into Azure data centres): Free
Outbound data transfer prices are set at a sliding scale depending on location and bandwidth used.
inbound traffic is free so the data coming in can be removed from the equation. Outbound is not free, and you saw the pricing page.
Data transfer is everything that is going out from every operation you execute.
And it is hard to estimate the traffic pricing - i would recommend to register the Azure trial and test it for a month and see how it is going. Because your data is not only what is returned, there is a lot of payloads coming with that.
But if you estimate 10 GB/month of outbound traffic, then it will start from $0.087 per GB starting from fifth GB (because first 5 are free). There are different regions described at the pricing page as well, so you should apply the pricing according to the region where your website is.
I tried to dig on MSDN but could not get concrete statement for which is the best load balancing method.
could someone please share some light on which of the below are best option for given scenario:
Performance
Failover
Round Robin.
Scenario:
x Web Roleshosted on Large VM on single data center.
Requirement:
must be 100% up 24x7.
Thank you.
First: Do you really want to offer a 100% uptime SLA for your customers, when Azure itself doesn't offer 100% in its SLA's?
That said: Traffic Manager only load-balances your compute, not your storage. So if you're trying to increase uptime by having a set of backup compute nodes running in another data center, you need to think about data access speed and cost:
With round robin, you'll now have distributed traffic across multiple data centers, guaranteed, and constantly. And if your data is in a single data center (which is a good idea to have data in a single System of Record, unless you have replication logic all taken care of), some of your users are going to see increased latency as the nodes separated from your data are going to be requesting data across many miles (potentially between continents). Plus, data egress has a $$$ cost to it.
With performance, your users are directed toward the data center which offers them the lowest latency. Again, this now means traffic across multiple data centers, with the same issues as round robin.
With failover, you now have all traffic going to one data center, with another designated as your failover data center (so it's for High Availability). In the event you have an outage in the primary data center, you'd now have a failover data center to rely on. This may help justify the added latency and cost, as you'd only experience this latency+cost when your primary app location becomes unavailable for some reason.
So: If you're going for the high availability route, to help approach the 100% availability mark, I'm guessing you'd be best off with the failover model.
Traffic manager comes into picture only when your application is deployed across multiple cloud services within same data center or in different data centers. If your application is hosted in a single cloud service (with multiple instances of course) , then the instances are load balanced using Round Robin pattern. This is the default load balancing pattern and comes to you without any extra charge.
You can read more about traffic manager here: https://azure.microsoft.com/en-us/documentation/articles/traffic-manager-overview/
As per my guess there can not be comparison which is best load balancing method of Azure Traffic manager. All of them have unique advantages and vary depending on the requirement of application. Most common scenario is to use performance load balancing option with azure traffic manager. But as Gaurav said, you will have to have your cloud service application hosted on more than one cloud services. If you wish to implement performance load balancing then here is the link to get you started - http://sanganakauthority.blogspot.com/2014/06/performance-load-balancing-using-azure.html