I am getting 'upload quota exceeded" in fusion table.
This is puzzling because I did not exceed 10% of my google drive storage and my KML file contains only 741 rows. What would cause this?
Fusion Tables have separate quotas:
100 MB per table
250 MB total among all your tables
https://developers.google.com/fusiontables/docs/v1/using#quota
Related
According to Microsoft's documentation on physical partitions, each one can hold 50GB and supply 10,000RU of throughput. Provisioning physical partitions involves initialising the container in increments of 10,000RU max scale.
However, looking at their documentation on Autoscale and storage limits it claims that a max scale of 50,000RU can hold 500GB of data, double the amount stated that 5 partitions should be able to hold.
These two statements seem to be in conflict with each other. Does each partition actually hold 100GB, not 50GB?
It is still 50GB. I believe what the 2nd link does not talk about is the number of physical partitions it would create.
Each physical partition has a limit of 10,000 RU and 50GB so if your storage is 500GB (and max throughput is 50,000 RU), there would be 10 physical partitions where each partition has 5,000 RU and 50GB.
From this link:
When you first select the max RU/s, Azure Cosmos DB will provision:
Max RU/s / 10,000 RU/s = # of physical partitions. Each physical
partition can support up to 10,000 RU/s and 50 GB of storage. As
storage size grows, Azure Cosmos DB will automatically split the
partitions to add more physical partitions to handle the storage
increase, or increase the max RU/s if storage exceeds the associated
limit.
The maximum size of a physical partition is 50GB.
What your second article is describing is how the minimum RU is automatically adjusted based on the size of your container or database. The minimum RU is the bigger value between 400RU/s and the size in GB * 10RU/s (and 1/100 * the max you've ever set).
A few examples what that means for different database sizes:
1GB -> 400RU/s
30GB -> 400RU/s
50GB -> 500RU/s
500GB -> 5000RU/s
600GB -> 6000RU/s
(examples assumes you don't participate in a special program)
When using autoscale you can define a lower/upper limit with 10x difference. You however still have to consider the previous rule. Once you go from 500GB to 600GB in size your autoscale rule also automatically adjusts its lower and upper limit following the ruleset above based on the lower limit.
The current specification of Azure SQL Managed Instance is 8vcore with Storage 1280 GB.
The issue is when I am going through the Size it shows me there is 380 GB of unallocated storage out of 1280 GB How can I allocate this storage.
Azure SQL database_size is always larger than the sum of reserved + unallocated space because it includes the size of log files, but reserved and unallocated_space consider only data pages.
FYI that “database size” property is the size of the current database, it includes both data and log files. “Space Available” is space in the database that has not been reserved for database objects. With our situation, if database objects take up all the unallocated space, data and log files will grow by “autogrowth” value (we can see these property under database, properties, Files tab) until database size reaches 10 GB.
For more detail information, please refer to the following link:
sp_spaceused (Transact-SQL):
http://msdn.microsoft.com/en-us/library/ms188776.aspx
Database Properties (Files Page):
http://msdn.microsoft.com/en-us/library/ms180254.aspx
I have read this article, but I am still not sure whether I should store PDFs as page or block blobs in Azure Blob Storage.
The documents are just corporate documents for archiving, i.e. they will never be modified but need to be accessed via web and downloaded. The size of each document varies between 50 kB and 5 MB.
Any insights would be greatly appreciated.
You should use block blobs since you don't need random read or write operations.
If you really only need to archive files, consider using Azure Archive storage, which is the lowest-priced storage offer in Azure.
#Meneghino Using Block blob would be best for objects such as PDFs. Page blobs are suitable for VHDs, basically, by default when you create a VM resource, the VHDs get stored on Page blobs due to its optimization to read and write operations.
Page Blob: are a collection of 512-byte pages optimized for random read and write operations. To create a page blob, you initialize the page blob and specify the maximum size the page blob will grow. To add or update the contents of a page blob, you write a page or pages by specifying an offset and a range that align to 512-byte page boundaries. A write to a page blob can overwrite just one page, some pages, or up to 4 MB of the page blob. Writes to page blobs happen in-place and are immediately committed to the blob. The maximum size for a page blob is 8 TB.
Block blobs: let you upload large blobs efficiently. Block blobs are comprised of blocks, each of which is identified by a block ID. You create or modify a block blob by writing a set of blocks and committing them by their block IDs. Each block can be a different size, up to a maximum of 100 MB (4 MB for requests using REST versions before 2016-05-31), and a block blob can include up to 50,000 blocks. The maximum size of a block blob is therefore slightly more than 4.75 TB (100 MB X 50,000 blocks). For REST versions before 2016-05-31, the maximum size of a block blob is a little more than 195 GB (4 MB X 50,000 blocks). If you are writing a block blob that is no more than 256 MB (64 MB for requests using REST versions before 2016-05-31) in size, you can upload it in its entirety with a single write operation;
More information can be found here: https://learn.microsoft.com/en-us/rest/api/storageservices/understanding-block-blobs--append-blobs--and-page-blobs
I have a hive table that is of 2.7 MB (which is stored in a parquet format). When I use impala-shell to convert this hive table to kudu, I notice that the /tserver/ folder size increases by around 300 MB. Upon exploring further, I see it is the /tserver/wals/ folder that holds the majority of this increase. I am facing serious issues due to this. If a 2.7 MB file generates a 300 MB WAL, then I cannot really work on bigger data. Is there a solution to this?
My kudu version is 1.1.0 and impala is 2.7.0.
I never used KUDU but I'm able to Google on a few keywords, and read some documentation.
From the Kudu configuration reference section "Unsupported flags"...
--log_preallocate_segments Whether the WAL should preallocate the entire segment before writing to it Default true
--log_segment_size_mb The default segment size for log roll-overs, in MB Default 64
--log_min_segments_to_retain The minimum number of past log segments to keep at all times, regardless of what is required for
durability. Must be at least 1. Default 2
--log_max_segments_to_retain The maximum number of past log segments to keep at all times for the purposes of catching up other
peers. Default 10
Looks like you have a minimum disk requirement of (2+1)x64 MB per tablet, for the WAL only. And it can grow up to 10x64 MB if some tablets are straggling and cannot catch up.
Plus some temp disk space for compaction etc. etc.
[Edit] these default values have changed in Kudu 1.4 (released in June 2017); quoting the Release Notes...
The default size for Write Ahead Log (WAL) segments has been reduced
from 64MB to 8MB. Additionally, in the case that all replicas of a
tablet are fully up to date and data has been flushed from memory,
servers will now retain only a single WAL segment rather than two.
These changes are expected to reduce the average consumption of disk
space on the configured WAL disk by 16x
So I am hitting the 10 GB maximum storage capacity for Azure DocumentDb every few months.
I noticed recently Microsoft has added a partitioned mode that raises the maximum storage capacity to 250 GB but the problem is that the minimum throughput (RU/s) is 10,100 which jumps the price to ~$606 a month from around $25 a month.
Is there anyway to increase storage capacity while keeping the throughput around throughput around 400?
Without using partitioned collections, you'd need to create multiple non-partitioned collections, as needed. The SDKs have partitioning support (or you can shard data across partitions as you see fit).
EDIT: Please note that the minimum RU for partitioned collections is no longer 10,100 RU (as mentioned in the original question). It's now 400 RU (as of late 2018).