Azure Storage Tables have been superseeded by Azure Cosmos Table API at a significantly higher price point but also with new features like automatic secondary indexing.
One of the pain points using Azure Storage Tables was, that in order to achieve custom ordering of query, we have to redundantly store the data with different Partition/Row-Keys as the documentation states, that
Query results returned by the Table service are sorted in ascending
order based on PartitionKey and then by RowKey.
However, the next paragraph states, that
Query results returned by the Azure Table API in Azure DB are not
sorted by partition key or row key. For a detailed list of feature
differences, see differences between Table API in Azure Cosmos DB and
Azure Table storage.
Following, the link, i find that
Query results returned by the Table API aren't sorted in partition
key/row key order as they're in Azure Table storage.
So i am a bit confused now, how to achieve ordering when using Cosmos Table API. Is there no ordering at all? Or can i specify ordering with my querys?
For Azure Cosmos Table API, this one is correct: "Query results returned by the Azure Table API in Azure DB are not sorted by partition key or row key".
So the returned results is no sorting as of now.
Somebody has asked this issue before at GitHub here.
And the MS team suggests vote on this user voice. And they may add this basic sort feature in future.
Hope it helps.
Additional information to this topic i found out from the GitHub thread:
The latest preview of CosmosDB Tables SDK (0.11.0-preview) has OrderBy support:
https://github.com/MicrosoftDocs/azure-docs/issues/26228#issuecomment-471095278
Related
I am trying to use Cosmos DB Tables. What I am noticing is that if I query on Timestamp property, no data is returned.
Here's the query I am using:
Timestamp ge datetime'2010-01-01T00:00:00'
I believe my query is correct because the same query runs perfectly fine against a table in my Storage Account.
If I query on any other attribute, the query runs perfectly fine.
I tried running this query in both Cerebrata Cerulean and in Microsoft Storage Explorer and I am getting no results in both places.
However when I run the same query in Azure Portal Data Explorer, data is returned. I opened developer tools in Azure Portal and noticed that the Portal is not making OData query. Instead it is making SQL API query. For example, in the above case the query that's being sent is:
Select * from c where c._ts > [epoch value indicating time]
Similarly if I query on an attribute using the tools above:
AttributeName eq 'Some Attribute Value'
Same query is being sent in Azure Portal as
SELECT * FROM c WHERE c.AttributeName["$v"] = 'Some Attribute Value'
All the documentation states that I should be able to write OData queries and they should work but I am not finding it to be correct.
So what's the correct way of querying Cosmos DB Tables?
UPDATE
Seems this is not a problem with just Timestamp property but all Edm.DateTime kind of properties.
UPDATE #2
So I opened up my Cosmos DB Table account as SQL API account to see how the data is actually stored under the hood.
First thing I observed is that Timestamp property is not getting stored at all. Value of Timestamp (in Storage Table Entity) is actually getting stored as _ts system property and that too as Epoch seconds.
Next thing I noticed is that all Date/Time kind of properties are actually getting converted into a 20 character long strings and are stored something like the following:
"SourceTimestamp": {
"$t": 9,
"$v": "00637219463290953744"
},
I am wondering if that has something to do with not being able to issue ODATA queries directly.
BTW, I forgot to mention that I am using Azure Storage Node SDK to access my Cosmos Table account (as this is what Microsoft is recommending considering there's no Node SDK specifically for Table API).
Thanks for your patience while I looked into this.
The root cause for this behavior is while Storage table stores with time granularity of ticks, Cosmos DB's _ts is at a second level of granularity. This isn't OData related. We actually block queries for timestamp properties because it was confusing customers and overall Timestamp based queries are not recommended for Storage Tables.
The workaround for this is to add your own custom datetime or long data type property and set the value yourself from the client.
We will address this in a future update but this work is not currently scheduled.
Thanks.
I would like to get a unique list of PrimaryKey's used in an existing Azure Table Storage Account with a large amount of data. In this db there will be more than 20 unique Primary Key's and I would like to get this unique list back. Is there any FilterCondition or API that supports this?
Unfortunately no. There's no API available in Azure Table Storage that will give you the list of partition keys back.
You will need to fetch all entities and extract the unique partition keys from the result set.
To optimize this process, you can do a few things:
Run your application in a VM in the same region as that of the storage account. That way you're avoiding the latency + data egress charges.
Use Query Projection to only return PartitionKey in the result as you're only interested in getting that information.
My website is hosted on Azure, i want to implement Azure search indexing, but there are some limitations of showing data to anyone. So when i retrieve data from a table, i will check user id and all details from different userrole table and pick some data on the basis of userid,
Can i get data from different database tables using Azure search indexing? currently i am getting onlye one table data on one index?
i have to implement Azure index search, or please suggest me any other workaround for my problem.
for example
i have a table of "users","userroles", "projects" and "tasks".
i want to show tasks of projects related to user. foreign key will be used. now if i create azure index, it will only run my query on tasks table, it will not check the tasks details from projects,users, etc tables. so my question is how i can create such type of index or query in Azure search, where i use different tables to get relevant and correct data in my search.
You can add those roles on your Azure Search index, specifying which roles have access, and use Odata to filter the results.
https://.search.windows.net/indexes//docs?search=&$filter=Administrator%20eq%20true
You can learn more about filters on this documentation.
When you are using Azure Storage Explorer you can click on the name of each columns to sort the results by that field.
Is there any way to sort query results in PowerShell using az storage entity query?
In another word, I can get the results in Azure CLI as an object and I can sort it using Sort-Object, but I want to sort entries on the Azure Storage Server and get sorted-results. It's not useful to get all of the data from the server and sort it manually.
Please see this page https://learn.microsoft.com/en-us/rest/api/storageservices/Query-Operators-Supported-for-the-Table-Service?redirectedfrom=MSDN.
There's a complete list of supported operators you can use with Azure Storage Table, OrderBy is sadly not among the supported ones.
This means, you will need to retrieve the data first, then do the sorting.
but I want to sort entries on the Azure Storage Server and get
sorted-results. It's not useful to get all of the data from the server
and sort it manually.
It is not possible as Azure Tables does not support server-side sorting. You will need to fetch the desired data on the client and perform the sorting there only.
I'm trying to find the best solution for storing dynamic spatial data. I wonder if any of Microsoft's Azure solutions could work. Azure Table Storage would let me create a lot of custom and dynamic structures stored on fast SSD disks.
Because of data's dynamic nature, common indexing seems useless. I would also like to create a lot of table-like structures so the whole architecture cannot be static. Using Azure Table Storage I would dynamically create a table based on country, city, etc sorted by latitude or longitude.
I would appreciate any clue.
Azure Table Storage has mostly been replaced by Azure Cosmos DB.
At the time of writing the Table Storage page even says:
The content in this article applies to the original basic Azure Table storage. However, there is now a premium offering for Azure Table storage in public preview that offers throughput-optimized tables, global distribution, and automatic secondary indexes. To learn more and try out the new premium experience, please check out Azure Cosmos DB: Table API.
You can use Cosmos DB via the Table API, but you'll probably find the Document DB API to be more powerful.
Documents are "schema-free". You can just throw your documents in to a collection, and then you can query against them.
You can create documents which have geo-spatial properties which are indexed automatically.
Then you can perform geo-spatial queries against those properties.
For example you might give each of your documents a point, and then create a query to select all documents that are inside of a polygon.
Or maybe you want to find out how far away each document is from a given point.