The CosmosDb provider is sending this message:
“Response status code does not indicate success: 503 Substatus: 0 Reason: (The request failed because the client was unable to establish connections to 3 endpoints across 1 regions. Please check for client resource starvation issues and verify connectivity between client and server.”
In my tests, it works (.net core 3.1):
Task.Run(async () =>
{
var endpoint = “test”;
var masterKey = “test”;
using (var client = new DocumentClient(new Uri(endpoint), masterKey))
{
//Insert new Document
Console.WriteLine("\r\n>>>>>>>>>>>>>>>> Creating Document <<<<<<<<<<<<<<<<<<<");
dynamic candidato = new
{
Id = 1,
Nome = "Test"
};
var document1 = await client.CreateDocumentAsync(
UriFactory.CreateDocumentCollectionUri("Test", "Test"),
candidato);
Console.ReadKey();
}
}).Wait();
It does not:
Task.Run(async () =>
{
using (var context = new StudentsDbContext())
{
context.Add(new FamilyContainer(2, "Test"));
await context.SaveChangesAsync();
}
}).Wait();
public class FamilyContainer
{
public int Id { get; set; }
public string Nome { get; set; }
public FamilyContainer(int id, string nome)
{
Id = id;
Nome = nome;
}
}
public class StudentsDbContext : DbContext
{
public DbSet<FamilyContainer> FamilyContainer { get; set; }
protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
{
optionsBuilder.UseCosmos(
"test",
"test",
"FamilyDatabase",
options =>
{ }
);
}
protected override void OnModelCreating(ModelBuilder modelBuilder)
{
modelBuilder.Entity<FamilyContainer>(x =>
{
x.ToContainer("FamilyContainer");
});
}
}
Packages
Can anyone help me? Thanks
fail: Microsoft.EntityFrameworkCore.Update[10000]
An exception occurred in the database while saving changes for context type '...'.
Microsoft.EntityFrameworkCore.Storage.RetryLimitExceededException: Maximum number of retries (6) exceeded while executing database operations with 'CosmosExecutionStrategy'. See inner exception for the most recent failure.
---> Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: 503 Substatus: 0 Reason: (Microsoft.Azure.Documents.ServiceUnavailableException: Service is currently unavailable.ActivityId: 07fbf539-0d44-4e5a-89d0-cd46838ee605, {"RequestStartTimeUtc":"2020-02-21T16:34:09.1834993Z","RequestEndTimeUtc":"2020-02-21T16:34:41.3484203Z","RequestLatency":"00:00:32.1649210","IsCpuOverloaded":false,"NumberRegionsAttempted":1,"ResponseStatisticsList":[{"ResponseTime":"2020-02-21T16:34:11.5964152Z","ResourceType":2,"OperationType":0,"StoreResult":"StorePhysicalAddress: rntbd:.../, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: True, StatusCode: 410, SubStatusCode: 0, RequestCharge: 0, ItemLSN: -1, SessionToken: , UsingLocalLSN: False, TransportException: A client transport error occurred: Failed to connect to the remote endpoint. (Time: 2020-02-21T16:34:11.5298608Z, activity ID: 07fbf539-0d44-4e5a-89d0-cd46838ee605, error code: ConnectFailed [0x0005], base error: socket error ConnectionRefused [0x0000274D]...
--- End of inner exception stack trace ---
I was facing same issue.
What worked for me is changing ConnectionMode to ConnectionMode.Gateway while initializing CosmosClient like :
var options = new CosmosClientOptions() { ConnectionMode = ConnectionMode.Gateway };
var client = new CosmosClient(endpoint, key, options);
For more details on refer :
https://learn.microsoft.com/en-us/dotnet/api/microsoft.azure.cosmos.cosmosclientoptions?view=azure-dotnet
https://learn.microsoft.com/en-us/dotnet/api/microsoft.azure.cosmos.connectionmode?view=azure-dotnet
TransportException: A client transport error occurred: Failed to connect to the remote endpoint. (Time: 2020-02-21T16:34:11.5298608Z, activity ID: 07fbf539-0d44-4e5a-89d0-cd46838ee605, error code: ConnectFailed [0x0005], base error: socket error ConnectionRefused
This means that the Connection was refused.
Either your Cosmos DB account has Firewall/VPN enabled and the application is not able to establish a connection due not not being in a whitelisted IP/Network : Try checking your account configuration.
The environment you are executing the code is restricting connections (some corporate Firewall or network might be blocking port ranges): Try running the app in a different network, or use GatewayMode. If that works, then this is related to the network.
The machine might be running low on sockets or high on CPU.
My RCA for this is: Cosmos Partitions where served by individual processes on CosmosDB, each partition serving process has it's own TCP port. When client connects to 443 (Using TCP Direct Mode), CosmosDB Proxy sends partition ports back to client so that client can talk to server-partitions in parallel. Partition ports are random (11000 upwards afaik). Normal company firewall would allow outbound 443 (connection to cosmos works) but blocks the outbound random ports. So at the end, access fails.
Workarounds:
Open firewall
Use Gateway Mode. This uses https/443 only by forwarding internally instead of redirecting to other ports.
It is because Entity framework has a default connection mode of Direct. It worked for me after overriding it to Gateway.
{
optionsBuilder.UseCosmos(
"test",
"test",
"FamilyDatabase",
options =>
{ options.ConnectionMode(ConnectionMode.Gateway); }
);
}
I just want to add this because it wasted a lot of my time. The following code would instantly die with an error message that led me to this S.O. post:
var container = _client.Client.GetContainer(_databaseName, containername);
var result = await container.CreateItemAsync(dataitem, pk);
I disbelieved the error message because everything else has worked, upsert, read, etc. After messing with it for a while, I noticed the documentation shows a template type for CreateItemAsync.
var container = _client.Client.GetContainer(_databaseName, containername);
var result = await container.CreateItemAsync<T>(dataitem, pk);
Changing the code to that fixed it (inside of a templated function).
I wanted to add: if I had been catching exceptions, I would have gotten to the meat of the problem much sooner. The library I am working with is not meant to catch exceptions, they are handled by the layer above it.
I am getting the below error when I try to connect to Azure cosmosdb mongo api from my asp.net core :
Error: TimeoutException: A timeout occured after 30000ms selecting a server using CompositeServerSelector
Code used:
readonly string _cn = #"mongodb://testdb12321:f2iORCXklWzAHJLKbZ6lwesWU32SygfUN2Kud1pu1OtMBoe9lBPqHMWdbc0fF7zgl7FryTSCGv7EGCa6X8gZKw==#testdb12321.documents.azure.com:10255/?ssl=true&replicaSet=globaldb";
var client = new MongoClient(_cn);
var db = client.GetDatabase("researchdb");
var cns = db.ListCollectionNames().First();
I am trying to connect from an App Service to a database, using an MSI token, and am getting
Login failed for user ''.
WHen I deployed the App Service using ARM I produced an Output that ensured that it created a Service Principal:
{
"principalId":"98f2c1f2-0a86-4ff1-92db-d43ec0edxxxx","
tenantId":"e6d2d4cc-b762-486e-8894-4f5f440dxxxx",
"type":"SystemAssigned"
}
I've also checked in Kudu the environment variables and have:
MSI_ENDPOINT = http://127.0.0.1:41239/MSI/token/
MSI_SECRET = 7C1B16Fxxxxxxxxxxxxx
I have provided in the Azure Portal a connection string as follows:
Data Source=nzmoebase0001bt.database.windows.net;Initial Catalog=nzmoebase0001bt;Connect Timeout=300;MultipleActiveResultSets=False;Encrypt=True;TrustServerCertificate=False;Connection Timeout=30;
As an offside, ain't that cool that MSI (theoretically) allows one to cut/paste from source code to a public forum without fear of leaving a secret?! Just sayin'.... Now...if only I could get it to work!!!
I've added the principal to the database as an Owner...
But I'm still getting:
Login failed for user ''.
What else should I try?
Thank you.
#juunas: the token is being added using:
static async Task AttachAccessTokenToDbConnection(IDbConnection dbConnection)
{
SqlConnection sqlConnection = dbConnection as SqlConnection;
if (sqlConnection == null)
{
return;
}
string msiEndpoint = Environment.GetEnvironmentVariable("MSI_ENDPOINT");
if (string.IsNullOrEmpty(msiEndpoint))
{
return;
}
var msiSecret = Environment.GetEnvironmentVariable("MSI_SECRET");
if (string.IsNullOrEmpty(msiSecret))
{
return;
}
// To get around:
// "Cannot set the AccessToken property if 'UserID', 'UID', 'Password', or 'PWD' has been specified in connection string."
var terms = new[] {"UserID","Password","PWD=","UID=" };
string connectionString = dbConnection.ConnectionString;
foreach (var term in terms)
{
if (connectionString.Contains(term, StringComparison.InvariantCultureIgnoreCase))
{
return;
}
}
string accessToken = await AppCoreDbContextMSITokenFactory.GetAzureSqlResourceTokenAsync();
sqlConnection.AccessToken = accessToken;
}
Update:
Some progress (not entirely sure what's changed), getting from Entity Framework
This operation requires a connection to the 'master' database. Unable to create a connection to the 'master' database because the original database connection has been opened and credentials have been removed from the connection string. Supply an unopened connection.
I Added Persist Security Info = True as per several recommendations on the net, but didn't change much.
Data Source=nzmoebase0001bt.database.windows.net;Initial Catalog=nzmoebase0001bt;MultipleActiveResultSets=False;Persist Security Info = True;Encrypt=True;TrustServerCertificate=False;Connection Timeout=30;
As per #evilSnobu, I did finally get tracing on and captured the token:
.eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI....
WHich decoded using jwt.io gave:
{
"typ": "JWT",
"alg": "RS256",
"x5t": "FSimuFrFNoC0sJXGmv13nNZceDc",
"kid": "FSimuFrFNoC0sJXGmv13nNZceDc"
}.{
"aud": "https://database.windows.net/",
"iss": "https://sts.windows.net/e6d2d4cc-b762-486e-8894-4f5f440dxxxx/",
"iat": 1522783025,
"nbf": 1522783025,
"exp": 1522786925,
"aio": "Y2NgYPjNdyJd9zrzpLavJSEzNIuPAAA=",
"appid": "d1057cea-461b-4946-89a9-d76439c2xxxx",
"appidacr": "2",
"e_exp": 262800,
"idp": "https://sts.windows.net/e6d2d4cc-b762-486e-8894-4f5f440dxxxx/",
"oid": "98f2c1f2-0a86-4ff1-92db-d43ec0edxxxx",
"sub": "98f2c1f2-0a86-4ff1-92db-d43ec0edxxxx",
"tid": "e6d2d4cc-b762-486e-8894-4f5f440dxxxx",
"uti": "59bqKWiSL0Gf0bTCI0AAAA",
"ver": "1.0"
}.[Signature]
Which to my untrained eye looks great (?). But I still get:
System.Data.SqlClient.SqlException: Login failed for user ''.
Which appears to be from a deeper message tie to EntityFramework's startup scheam checking process:
[InvalidOperationException: This operation requires a connection to the 'master' database. Unable to create a connection to the 'master' database because the original database connection has been opened and credentials have been removed from the connection string. Supply an unopened connection.]
Any and all advice greatly appreciated. Thanks!
I'm chasing the "This operation requires a connection to the 'master' database" error now, following this document:
https://learn.microsoft.com/en-us/azure/app-service/app-service-web-tutorial-connect-msi
I think it's key to include the following line after you assign your token:
Database.SetInitializer<MyDatabaseContext>(null);
In my case we had some code explicitly running Entity Framework Migrations that resulted in the error. When we disabled it, the app started working as expected.
I have tried as suggested here.
I am able to access the connection object, but it says
IllegalStateException - state should be: open
Below is the snippet which I have tried.
//set connection
testSuite.metaClass.db = db
//get connection (i am trying to access it from the same testSuite for testing purpose)
def connection = testSuite.db.getConnection()
for(def userObj :connection.User.find()){
log.info userObj
}
i am developing a multitenant app in node that switches db based on the request.
i am using mongoose to create the connection to the mongo deployment where i have 3 dbs.
all my code is written in coffeescript.
here is how i create the initial connection:
conn = mongoose.createConnection('mongodb://<user>:<pwd>#<host>:<port>,<host>:<port>/main?replicaSet=set-xxxxxxxxxx');
here the code in the request
db = conn.useDb('demo')
myModel = db.model('mymodel')
for obj in objects
o = new settingModel(obj)
o.save (err, obj) ->
console.log 'err is', err if err
i can switch db and query the db but when i try to write to it i get a:
errmsg: 'not authorized on demo to execute command { insert: "settings", writeConcern: { w: 1 }...
how can i solve this issue? the databases are all hosted on compose.io