Unable to connect to Azure Cosmos Db Account using Microsoft.EntityFrameworkCore.Cosmos - Response status code - azure

Question

The CosmosDb provider is sending this message:
“Response status code does not indicate success: 503 Substatus: 0 Reason: (The request failed because the client was unable to establish connections to 3 endpoints across 1 regions. Please check for client resource starvation issues and verify connectivity between client and server.”
In my tests, it works (.net core 3.1):
Task.Run(async () =>
{
var endpoint = “test”;
var masterKey = “test”;
using (var client = new DocumentClient(new Uri(endpoint), masterKey))
{
//Insert new Document
Console.WriteLine("\r\n>>>>>>>>>>>>>>>> Creating Document <<<<<<<<<<<<<<<<<<<");
dynamic candidato = new
{
Id = 1,
Nome = "Test"
};
var document1 = await client.CreateDocumentAsync(
UriFactory.CreateDocumentCollectionUri("Test", "Test"),
candidato);
Console.ReadKey();
}
}).Wait();
It does not:
Task.Run(async () =>
{
using (var context = new StudentsDbContext())
{
context.Add(new FamilyContainer(2, "Test"));
await context.SaveChangesAsync();
}
}).Wait();
public class FamilyContainer
{
public int Id { get; set; }
public string Nome { get; set; }
public FamilyContainer(int id, string nome)
{
Id = id;
Nome = nome;
}
}
public class StudentsDbContext : DbContext
{
public DbSet<FamilyContainer> FamilyContainer { get; set; }
protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
{
optionsBuilder.UseCosmos(
"test",
"test",
"FamilyDatabase",
options =>
{ }
);
}
protected override void OnModelCreating(ModelBuilder modelBuilder)
{
modelBuilder.Entity<FamilyContainer>(x =>
{
x.ToContainer("FamilyContainer");
});
}
}
Packages
Can anyone help me? Thanks
fail: Microsoft.EntityFrameworkCore.Update[10000]
An exception occurred in the database while saving changes for context type '...'.
Microsoft.EntityFrameworkCore.Storage.RetryLimitExceededException: Maximum number of retries (6) exceeded while executing database operations with 'CosmosExecutionStrategy'. See inner exception for the most recent failure.
---> Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: 503 Substatus: 0 Reason: (Microsoft.Azure.Documents.ServiceUnavailableException: Service is currently unavailable.ActivityId: 07fbf539-0d44-4e5a-89d0-cd46838ee605, {"RequestStartTimeUtc":"2020-02-21T16:34:09.1834993Z","RequestEndTimeUtc":"2020-02-21T16:34:41.3484203Z","RequestLatency":"00:00:32.1649210","IsCpuOverloaded":false,"NumberRegionsAttempted":1,"ResponseStatisticsList":[{"ResponseTime":"2020-02-21T16:34:11.5964152Z","ResourceType":2,"OperationType":0,"StoreResult":"StorePhysicalAddress: rntbd:.../, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: True, StatusCode: 410, SubStatusCode: 0, RequestCharge: 0, ItemLSN: -1, SessionToken: , UsingLocalLSN: False, TransportException: A client transport error occurred: Failed to connect to the remote endpoint. (Time: 2020-02-21T16:34:11.5298608Z, activity ID: 07fbf539-0d44-4e5a-89d0-cd46838ee605, error code: ConnectFailed [0x0005], base error: socket error ConnectionRefused [0x0000274D]...
--- End of inner exception stack trace ---

Answer 1

I was facing same issue.
What worked for me is changing ConnectionMode to ConnectionMode.Gateway while initializing CosmosClient like :
var options = new CosmosClientOptions() { ConnectionMode = ConnectionMode.Gateway };
var client = new CosmosClient(endpoint, key, options);
For more details on refer :
https://learn.microsoft.com/en-us/dotnet/api/microsoft.azure.cosmos.cosmosclientoptions?view=azure-dotnet
https://learn.microsoft.com/en-us/dotnet/api/microsoft.azure.cosmos.connectionmode?view=azure-dotnet

Answer 2

TransportException: A client transport error occurred: Failed to connect to the remote endpoint. (Time: 2020-02-21T16:34:11.5298608Z, activity ID: 07fbf539-0d44-4e5a-89d0-cd46838ee605, error code: ConnectFailed [0x0005], base error: socket error ConnectionRefused
This means that the Connection was refused.
Either your Cosmos DB account has Firewall/VPN enabled and the application is not able to establish a connection due not not being in a whitelisted IP/Network : Try checking your account configuration.
The environment you are executing the code is restricting connections (some corporate Firewall or network might be blocking port ranges): Try running the app in a different network, or use GatewayMode. If that works, then this is related to the network.
The machine might be running low on sockets or high on CPU.

Answer 3

My RCA for this is: Cosmos Partitions where served by individual processes on CosmosDB, each partition serving process has it's own TCP port. When client connects to 443 (Using TCP Direct Mode), CosmosDB Proxy sends partition ports back to client so that client can talk to server-partitions in parallel. Partition ports are random (11000 upwards afaik). Normal company firewall would allow outbound 443 (connection to cosmos works) but blocks the outbound random ports. So at the end, access fails.
Workarounds:
Open firewall
Use Gateway Mode. This uses https/443 only by forwarding internally instead of redirecting to other ports.

Answer 4

It is because Entity framework has a default connection mode of Direct. It worked for me after overriding it to Gateway.
{
optionsBuilder.UseCosmos(
"test",
"test",
"FamilyDatabase",
options =>
{ options.ConnectionMode(ConnectionMode.Gateway); }
);
}

Answer 5

I just want to add this because it wasted a lot of my time. The following code would instantly die with an error message that led me to this S.O. post:
var container = _client.Client.GetContainer(_databaseName, containername);
var result = await container.CreateItemAsync(dataitem, pk);
I disbelieved the error message because everything else has worked, upsert, read, etc. After messing with it for a while, I noticed the documentation shows a template type for CreateItemAsync.
var container = _client.Client.GetContainer(_databaseName, containername);
var result = await container.CreateItemAsync<T>(dataitem, pk);
Changing the code to that fixed it (inside of a templated function).
I wanted to add: if I had been catching exceptions, I would have gotten to the meat of the problem much sooner. The library I am working with is not meant to catch exceptions, they are handled by the layer above it.