How to store and read millions of records in Azure with secondary indexing - azure

Question

I need to store tens of millions of records in Azure. I need to be able to sort them by insert date in descending order. Finally, I need to issue where clauses on 3 fields.
public class Record
{
public DateTime CreatedOn {get; set;}
public string Filter1 {get; set;}
public string Filter2 {get; set;}
public bool Filter3 {get; set;}
}
I believe Azure Table Storage does not support secondary indexes, so I'm considering CosmosDB. The problem is each logical partition on CosmosDB is limited to 20 GBs. I can't dump everything in a single partition. I'm thinking of slicing data horizontally based on the CreatedOn field (dd/mm/yyyy without the time component). This helps me distribute the data, but I'm stuck with the query part to read it back.
The data is user generated; thus, I have no idea what values CreatedOn will be assigned. If I partition on CreatedOn, how can I make my query smart so that I know what the next partition key is when I reach the end of the previous partition?
Example:
Record1 CreatedOn => 5/28/2020
Record2 CreatedOn => 5/28/2020
Record3 CreatedOn => 5/22/2020
Record4 CreatedOn => 5/10/2020
There could be a large gap between dates, so I don't want to brute force the next partition by scanning the range day by day.
Regarding queries:
Filter1 and Filter2 can have 5 and 12 different values respectively. Filter3 is just a boolean.
A sample query would look like:
select top 20 from record
where Filter1 = 'Value1' and Filter2 = 'Value2' and Filter3 = false
order by CreatedOn desc
Another sample query is the same as above without Filter2.
select top 20 from record
where Filter1 = 'Value1' and Filter3 = false
order by CreatedOn desc

Answer 1

Use Cosmos SQL API. You can use Order by to sort on query.
All the fields are indexed by default. You can customize the indexing policy to include composite indices to make queries more efficient.
Pick a PK which will have a wide range of possible values to spread the request unit (RU) consumption and data storage evenly across all logical partitions evenly. Also, you should know the PK value for majority of your queries. You will use it in you where clause so that the query engine will only query specific partitions where the data of interest resides.
In you case what should be that PK property?
public class Record
{
public DateTime CreatedOn {get; set;}
public string Filter1 {get; set;}
public string Filter2 {get; set;}
public bool Filter3 {get; set;}
}
"CreatedOn" is not a good choice. First, like you mentioned you would not know its value during query time, so you will have to always do cross partition queries. Second, all your writes for a given day will go to the same partition resulting in hot partition which will result in rate-limiting and in inefficient use of the provisioned throughput, and higher costs.
Do any of the other properties help the cause? Maybe you must create another property. Maybe UserID , I am just making one up since I don’t know the context.
public class Record
{
public DateTime CreatedOn {get; set;}
public string Filter1 {get; set;}
public string Filter2 {get; set;}
public bool Filter3 {get; set;}
public string UserID {get; set;}
}
Now you can query
select top 20 from record
where Filter1 = 'Value1' and Filter2 = 'Value2' and Filter3 = false and UserID = 'somevalue' order by CreatedOn desc