I'm trying to figure out why my CreateItem<T> returns a request charge of 7.05 RU for inserting a simple object with 5 properties, less then 1KB including the id and partition key. Its just a default container, with default indexing etc.
The documentation says it should be 5 RU. I've tried to use the ItemRequestOption to disable returning the object, disable indexing for that item etc. but it stays at 7.05 RU. I've also tried changing the consistency level of the account but nothing changes.
A ReadItem on id and partition key returns with a RU charge of 1.0 as expected and a RU of 2.9 if run as a query.
I'm a little annoyed at the 7.05 RU cost for the CreateItem since its a 41% increase in RU cost over the 5 RUs. Enabling TTL jumped the RU cost to 7.65.
What is the best way to diagnose these kind of issues? I've tried looking at the response diagnostics but its not really helpful.
RU consumption on write operations will be heavily impacted by your indexing policy.
By default CosmosDB container will have indexing enabled for the whole document (each property). If you replace it with specific paths which are used heavily in queries, it can significantly improve performance of write operations as well as reduce consumption costs.
Also please note that there is no guarantee in RU consumption per document in query: documentation. It will vary based on index, partition keys, cardinality of partitions, geo-replication, "fan-out" during execution etc.
Related
By default, all the data in Azure Cosmos DB is indexed i.e. every property inside document/item has by default consistent/automatic indexing.
However, the cost of storing this large amount of data is not clearly visible to the end users.
How would you calculate/track the cost you are bearing for storing or using the index related data?
As the billing is only related to RU/s and data storage, it is not clear how the indexing strategy affects billing.
Also i wonder if the RU/s necessary for intensive writes maybe increased because of the indexes.
If so, indexes in CosmosDB should be excluded, and only necessary properties have to be indexed, thus reducing the overall performance.
As the billing is only related to RU/s and data storage, it is not
clear how the indexing strategy affects billing.
Indexing strategy will affect billing because if you index everything, you will consume more storage and that in turn will increase your bill. When an item is written to Cosmos DB, a part of your RU/s will be spent on indexing that item thus you will end up consuming more RU/s which will increase your bill.
You may find these links helpful in optimizing the costs as far as indexing is concerned:
Optimize cost with indexing: https://learn.microsoft.com/en-us/azure/cosmos-db/optimize-cost-storage#optimize-cost-with-indexing.
Optimize by changing indexing policy: https://learn.microsoft.com/en-us/azure/cosmos-db/optimize-cost-throughput#optimize-by-changing-indexing-policy.
Also i wonder if the RU/s necessary for intensive writes maybe
increased because of the indexes.
That is correct.
If so, indexes in CosmosDB should be excluded, and only necessary
properties have to be indexed, thus reducing the overall performance.
For bulk writes, it is recommended that you turn off indexing completely before doing the bulk writes and enable it once the write operations are completed. You can also request for Lazy Indexing as described here.
UPDATE - Adding comment from #MarkBrown (part of CosmosDB team):
I don't recommend using lazy indexing as there is no way to know when
it is caught up and the data can be queried consistently. Also, a
well-tuned indexing policy can be fine in bulk scenarios. Generally
best reserved for massive one-time data loads. Not recommended for
regular, batch type data ingestion.
We are planning in writing 10000 JSON Documents to Azure Cosmos DB (MongoDB), Does the Throughput Units matter, if so, can we increase for the batch load and set it back to low number
Yes you can do that. The lowest the RUs can be is 400. Scale up before you're about to do your insert and then turn it down again. As always, that part can be automated if you know when the documents are going to be inserted.
Check out the DocumentClient documentation and more specifically ReplaceOfferAsync.
You can scale the RU/sec allocation up or down at any time. You'll want to look at your insertion cost (RU cost is returned in a header) for a typical document, to get an idea of how many documents you might be able to write, per second, before getting throttled.
Also keep in mind: if you scale your RU out beyond what an underlying physical partition can provide, Cosmos DB will scale out your collection to have additional physical partitions. This means you might not be able to scale your RU back down to the bare minimum later (though you will be able to scale down).
At first, I have created DocumentDB Collection with 2000 as Throughput. after I have realized that it is too high, I tried to reduce by clicking (-) button. It got reduce till 1000 after that it doesn't work. I want to set as 400.
Unfortunately it is not possible to do so considering your collection is of unlimited size. Minimum throughput for such collection is 1000 RU/s (earlier it was 2500 RU/s).
If you want to have a collection with 400 RU/s, please create a new collection with fixed storage capacity (currently 10GB) and migrate the data from your existing collection to this new collection.
There were some updates to the performance level of Cosmos DB recently. The previous S1, S2, S3 performance levels have been retired. Now there are only two types of containers, fixed and unlimited. As Gaurav mentioned, the RU/s range for fixed is from 400 - 10,000, and for unlimited is from 1,000 - 100,000. And unlimited must have a partition key, while fixed doesn't have to.
More details can be found in this newly updated document: Partition and scale in Azure Cosmos DB. According to this document, my suggestions would be, even if you go and create a fixed container, you should still give it 1,000 RU/s and a partition key so you don't have to do migration in the future in case your data or load grows beyond the capacity of the fixed container.
We migrated our mobile app (still being developed) from Parse to Azure. Everything is running, but the price of DocumentDB is so high that we can't continue with Azure without fix that. Probably we're doing something wrong.
1) Price seams to have a bottleneck in the DocumentDB requests.
Running a process to load the data (about 0.5 million documents), memory and CPU was ok, but the DocumentDB request limit was a bottleneck, and the price charged was very high.
2) Even after the end of this data migration (few days of processing), azure continue to charge us every day.
We can't understand what is going on here. The graphic for use are flat, but the price is still climbing, as you can see in the imagens.
Any ideas?
Thanks!
From your screenshots, you have 15 collections under the Parse database. With Parse: Aside from the system classes, each of your user-defined classes gets stored in its own collection. And given that each (non-partitioned) collection has a starting run-rate of ~$24/month (for an S1 collection), you can see where the baseline cost would be for 15 collections (around $360).
You're paying for reserved storage and RU capacity. Regardless of RU utilization, you pay whatever the cost is for that capacity (e.g. S2 runs around $50/month / collection, even if you don't execute a single query). Similar to spinning up a VM of a certain CPU capacity and then running nothing on it.
The default throughput setting for the parse collections is set to 1000 RUPS. This will cost $60 per collection (at the rate of $6 per 100 RUPS). Once you finish the parse migration, the throughput can be lowered if you believe the workload decreased. This will reduce the charge.
To learn how to do this, take a look at https://azure.microsoft.com/en-us/documentation/articles/documentdb-performance-levels/ (Changing the throughput of a Collection).
The key thing to note is that DocumentDB delivers predictable performance by reserving resources to satisfy your application's throughput needs. Because application load and access patterns change over time, DocumentDB allows you to easily increase or decrease the amount of reserved throughput available to your application.
Azure is a "pay-for-what-you-use" model, especially around resources like DocumentDB and SQL Database where you pay for the level of performance required along with required storage space. So if your requirements are that all queries/transactions have sub-second response times, you may pay more to get that performance guarantee (ignoring optimizations, etc.)
One thing I would seriously look into is the DocumentDB Cost Estimation tool; this allows you to get estimates of throughput costs based upon transaction types based on sample JSON documents you provide:
So in this example, I have an 8KB JSON document, where I expect to store 500K of them (to get an approx. storage cost) and specifying I need throughput to create 100 documents/sec, read 10/sec, and update 100/sec (I used the same document as an example of what the update will look like).
NOTE this needs to be done PER DOCUMENT -- if you're storing documents that do not necessarily conform to a given "schema" or structure in the same collection, then you'll need to repeat this process for EVERY type of document.
Based on this information, I cause use those values as inputs into the pricing calculator. This tells me that I can estimate about $450/mo for DocumentDB services alone (if this was my anticipated usage pattern).
There are additional ways you can optimize the Request Units (RUs -- metric used to measure the cost of the given request/transaction -- and what you're getting billed for): optimizing index strategies, optimizing queries, etc. Review the documentation on Request Units for more details.
I'm looking into moving to the new partitioned collections for DocumentDB and have a few questions that the documentation and pricing calculator seems to be a little unclear on.
PRICING:
In the below scenario my partitioned collection will be charged $30.02/mo at 1GB of data with a constant hourly RU use of 500:
So does this mean if my users only hit the data for an average of 500 RU's for about 12 hours per day which means that HALF the time my collection goes UNUSED, but is still RUNNING and AVAILABLE (not shut down) the price goes down to $15.13/mo as the calculator indicates here:
Or will I be billed the full $30.01/mo since my collection was up and running?
I get confused when I go to the portal and see an estimate for $606/mo with no details behind it when I attempt to spin up the lowest options on a partition collection:
Is the portal just indicating the MAXIMUM that I COULD be billed that month if I use all my allotted 10,100 RU's a second every second of the hour for 744 consecutive hours?
If billing is based on hourly use and the average RU's used goes down to 100 on some of the hours used in the second scenario does the cost go down even further? Does Azure billing for partitioned collections fluctuates based on hourly usage and not total up time like the existing S1/S2/S3 tiers?
If so then how does the system determine what is billed for that hour? If most of the hour the RU's used are 100/sec but for a few seconds that hour it spikes to 1,000 does it average out by the seconds across that entire hour and only charge me for something like 200-300 RU's for that hour or will I be billed for the highest RU's used that hour?
PERFORMANCE:
Will I see a performance hit by moving to this scenario since my data will be on separate partitions and require partition id/key to access? If so what can I expect, or will it be so minimal that it would be undetected by my users?
RETRIES & FAULT HANDLING:
I'm assuming the TransientFaultHandling Nuget package I use in my current scenario will still work on the new scenario, but may not be used as much since my RU capacity is much larger, or do I need to rethink how I handle requests that go over the RU cap?
So they way that pricing works for an Azure documentDB is that you pay to reserve a certain amount of data storage size (in GB's) and/or throughput (in Request units (RU)). These charges are charged per hour that the reserve is in place (does not require usage). Additionally, just having a Document Account active is deemed to be an active S1 subscription, until a documentDB gets created then the pricing of your db takes over. There are two options available:
Option 1 (Original Pricing)
You can a choose between S1, S2 or S3. Each offering the same 10GB of storage but varying in throughput 250RU/1000RU/2500RU.
Option 2 (User-defined performance)
This is the new pricing structure which better decouples size and throughout. This option additionally provides for partitioning. Note that with user defined performance you are charge per GB of data storage used (Pay as you go storage).
With user-defined performance levels, storage is metered based on
consumption, but with pre-defined performance levels, 10 GB of storage
is reserved at the time of collection creation.
Single Partition Collection
The minimum is set at 400RU and 1GB of data storage.
The maximum is set at 10,000RU and 250GB of data storage.
Partitioned Collections
The minimum is set at 10,000RU and 1GB of data storage.
The maximum is set at 250,000RU and 250GB of data storage (EDIT can request greater).
So at a minimum you will be paying the cost per hour related to the option you selected. The only way to not pay for an hour would be to delete the db and the account, unfortunately.
Cost of Varying RU
In terms of varying your RU within the time frame of 1 hours, you are charged for that hour at the cost of the peak reserve RU you requested. So if you were at 400RU and you up it to 1000RU for 1sec you will be charge at the 1000RU rate for that hour. Even if for the other 59minutes 59secounds you set it back to 400RU.
Will I see a performance hit by moving to this scenario since my data will be on separate partitions and require partition id/key to access?
One the topic of perfromance hit there's a few things to think about, but in general no.
If you have a sane partition key with enough values you should not see a performance penalty. This means that you need to partition data so that you have the partition key available when querying and you need to keep the data you want from a query in the same partition by using the same partiton key.
If you do queries without partitionkey, you will see a sever penalty, as the query is parsed and executed per partition.
One thing to keep in mind when selecting a partition key is the limits for each partition, which are 10GB and 10K RU. This means that you want an even distributions over the partitions in order to avoid a "hot" partition which means that even if you scale to more than enough RU in total, you may recieve 429 for a specific partition.