Cosmos DB is very slow with the newer library compared to old - azure

Question

I am in the process of upgrading my application azure function from V3 to V4. In so doing I am also upgrading from the older/no longer supported Microsoft.Azure.DocumentDB (V 2.18.0) to the newest Microsoft.Azure.Cosmos (3.32 as per recommendations. The problem is that doing this is now taking almost 3 times as long to make a basic get request and we see every single request is a query vs a read.
An example is below where we are calling the provided ReadItemAsync(id,partition,options,token). The payload returning is about 589 bytes. The resulting diagnostic on this is that it is taking .400 - 900ms ms to return!
This cannot stand.
I am at a loss as to how to fix this issue. If MS is going to take 500 - 1000ms for every get.. and I want to just run through 26 items.. that is going to take almost 25 seconds of time. How can this be the case? This is crazy bad. When I run through my method, I do a get, a save and an upsert. On the old library, this was taking about 300ms to complete one iteration, on the 3.31.2 it is taking > 1500 ms.
I have no idea where or how to solve a request taking 460ms at the azure farm.
The raw data call looks like this:
response = await _database.GetContainer(containerId)
.ReadItemAsync<T>(id, partitionKey, null, cancellationToken);
LastQueryUsage = response.RequestCharge;
return response;
Diagnostics Dump from the above Read request:
{
"Summary": {
"DirectCalls": {
"(200, 0)": 1
},
"GatewayCalls": {
"(200, 0)": 3,
"(304, 0)": 1
}
},
"name": "ReadItemAsync",
"id": "0add6a37-9928-4145-aed1-b29e910e22f3",
"start time": "12:55:11:446",
"duration in milliseconds": 928.666,
//reduced for brevity in light of initial answer.
}
----------EDITS AFTER MARK'S RESPONSE: ----------
I am still seeing bad performance on my test collections. Very bad.
I spin up a brand new Azure V4 Dotnet6 isolated project.
public class CosmosSingleTonConnection
{
private static TestSettings _settings = new TestSettings();
private static readonly List<(string, string)> containers = new()
{
("myDb", "col1"),
("myDb", "col2")
};
private static CosmosClient cosmosClient;
private static Container Raw;
private static Container State;
public Container Container1=> Raw;
public Container Container2=> State;
public CosmosSingleTonConnection(IOptions<TestSettings> settings)
{
_settings = settings.Value;
cosmosClient = InitializeCosmosClient(_settings.Key, _settings.Endpoint);
Raw = cosmosClient.GetDatabase("myDb").GetContainer("col1");
State = cosmosClient.GetDatabase("myDb").GetContainer("col2");
}
private CosmosClient InitializeCosmosClient(string key, string endpoint)
{
return CosmosClient
.CreateAndInitializeAsync(accountEndpoint: endpoint, authKeyOrResourceToken: key, containers: containers, null, CancellationToken.None)
.Result;
}
---Program.cs ---
var host = new HostBuilder()
.ConfigureFunctionsWorkerDefaults(builder =>
{
builder
.AddApplicationInsights(opt => { opt.EnableHeartbeat = true; })
.AddApplicationInsightsLogger();
})
.ConfigureServices(DoConfiguration)
.Build();
void DoConfiguration(IServiceCollection services)
{
services.AddOptions<TestSettings>()
.Configure<IConfiguration>((settings, configuration) => { configuration.Bind(settings); });
services.AddSingleton<CosmosSingleTonConnection>();
services.AddScoped<IDoStuffService, DoStuffService>();
}
host.Run();
---DoStuffService---
private readonly CosmosSingleTonConnection _db;
public DoStuffService(CosmosSingleTonConnection db)
{
_db = db;
}
public FeedIterator<ObjectDTO> QueryLast30(string sensor)
{
string top30 = #"Select * from Col1 r Where r.paritionKey= #partitionKey"; //" Order by r.DateTimeCreatedUtc";
QueryRequestOptions ops = new QueryRequestOptions()
{
PartitionKey = new PartitionKey(sensor)
};
var query = new QueryDefinition(top30).WithParameter("#partitionKey", sensor);
using FeedIterator<ObjectDTO> feed = _db.Container1().GetItemQueryIterator<ObjectDTO>(queryDefinition: query, null, null);
return feed;
}
---The FUNCTION ---
public Function1(ILoggerFactory loggerFactory, IDoStuffService service)
{
_logger = loggerFactory.CreateLogger<Function1>();
Service = service;
}
[Function("Function1")]
public async Task<HttpResponseData> RunAsync([HttpTrigger(AuthorizationLevel.Function, "get", "post")] HttpRequestData req)
{
var response = req.CreateResponse(HttpStatusCode.OK);
List<string> responseTimes = new();
for (int i = 0; i < 10; i++)
{
var feed = Service.QueryLast30("01020001");
while (feed.HasMoreResults)
{
FeedResponse<RawSensorData> fr = await feed.ReadNextAsync();
responseTimes.Add(fr.Diagnostics.GetClientElapsedTime().TotalMilliseconds.ToString());
}
}
response.WriteString(string.Join(" | ", responseTimes));
return response;
}
'----Initial plus subsequent requests---`
Is this as good as it can get? Because this is not good if I have to do 4 atomic operations against cosmos per iteration.
459.3067 | 86.5555 | 421.989 | 81.4663 | 426.62 | 81.7712 | 82.6038 | 78.9875 | 81.0167 | 79.0283
201.5111 | 86.7607 | 79.1739 | 83.5416 | 79.2815 | 80.5983 | 79.8568 | 83.7092 | 79.7441 | 79.3132
81.8724 | 79.7575 | 91.6382 | 80.5015 | 81.7875 | 87.2023 | 79.3385 | 78.3251 | 78.3159 | 79.2731
82.8567 | 81.5768 | 81.6155 | 81.535 | 81.5871 | 79.2668 | 79.6522 | 78.9888 | 79.2734 | 80.0451
81.1635 | 88.578 | 111.7357 | 84.9948 | 80.207 | 81.2129 | 79.9344 | 80.1654 | 79.4129 | 82.7971

Answer 1

This is likely due to metadata requests on first access of your container object or due to establishing the connection to your container on the first call to it.
Best practice is to use a singleton CosmosClient as well as cache the names or database and container objects and also keep references to these objects within scope for the lifetime of your application.
If you are creating and destroying these references on every invocation, your app will suffer performance high latency as the Cosmos Client fetches metadata from the master partition as well as establishes connections to the service. Once these references and connections are in place (after the first call to the service) all subsequent calls to the service will be fast.
Also good idea to do a quick check for our .NET v3 Migration Guideance and these performance tips:
Migrate your application to use the Azure Cosmos DB .NET SDK v3
Best practices for Azure Cosmos DB .NET SDK Checklist
Performance tips for Azure Cosmos DB and .NET
Manage Connections for Azure Functions