CosmosDB Container not getting created - azure

CreateContainerIfNotExistsAsync is throwing an exception with status code "Bad Request" if the container does not exist in the db. If the container exists in the db, then no exception is thrown. Can anyone help me why this is happening.
(Hid the url and key for online posting)
using Microsoft.Azure.Cosmos;
using Microsoft.Azure.Cosmos.Linq;
using System.Threading.Tasks;
namespace CosmosDB // Note: actual namespace depends on the project name.
{
class Program
{
public static async Task Main(string[] args)
{
var cosmosUrl = "###########################";
var cosmoskey = "###########################";
var databaseName = "TestDB";
// var containerId = "ToDo";
CosmosClient client = new CosmosClient(cosmosUrl, cosmoskey);
Database database = await client.CreateDatabaseIfNotExistsAsync(databaseName);
Container container = await database.CreateContainerIfNotExistsAsync(
id: "ToDoList",
partitionKeyPath: "/category",
throughput: 100
);
}
}
}

The command fails because your input is invalid. The throughput must be a value between 400 and 10,000 RU/s (for a normal database or container) and since you are using 100 it will throw the exception.
The error won't occur if your container already exists, because it will not check (server-side) or perform an update on the throughput.
Edit:
Link to Microsoft documentation regarding service limits.
Link to Microsoft REST API (used by the SDK).

Related

Azure Cosmos DB client throws "HttpRequestException: attempt was made to access a socket in a way forbidden by its access permissions" underneath

I use CosmosClient from SDK Microsoft.Azure.Cosmos 3.28.0 in ASP.NET Core 3.1 in Azure Durable Function. This client is getting and sending data from/to my cosmos instance (Core (SQL)) and it works fine but I see that it constantly throws exception in following http request for metadata
GET 169.254.169.254/metadata/instance
System.Net.Http.HttpRequestException: An attempt was made to access a socket in a way forbidden by its access permissions.
I use following configuration:
private static void RegisterCosmosDbClient(ContainerBuilder builder)
{
builder.Register(c => new SocketsHttpHandler()
{
PooledConnectionLifetime = TimeSpan.FromMinutes(10), // Customize this value based on desired DNS refresh timer
MaxConnectionsPerServer = 20, // Customize the maximum number of allowed connections
}).As<SocketsHttpHandler>().SingleInstance();
builder.Register(
x =>
{
var cosmosDbOptions = x.Resolve<CosmosDbOptions>();
var socketsHttpHandler = x.Resolve<SocketsHttpHandler>();
return new CosmosClient(cosmosDbOptions.ConnectionString, new CosmosClientOptions()
{
ConnectionMode = ConnectionMode.Direct,
PortReuseMode = PortReuseMode.PrivatePortPool,
IdleTcpConnectionTimeout = new TimeSpan(0, 23, 59, 59),
SerializerOptions = new CosmosSerializationOptions()
{
PropertyNamingPolicy = CosmosPropertyNamingPolicy.CamelCase
},
HttpClientFactory = () => new HttpClient(socketsHttpHandler, disposeHandler: false)
});
})
.AsSelf()
.SingleInstance();
}
I also tried approach with passing IHttpClientFactory from this blog but it didn't help.
It looks like there are no new sockets available in your environment therefore you are getting the socket forbidden error. Please review how to manage connection for Azure Cosmos DB clients and you should use a singleton Azure Cosmos DB client for the lifetime of your application to resolve the issue. In case if you still facing the issue leveraging the singleton object then please let me know so we can further review it.
That particular IP and path is for https://learn.microsoft.com/azure/virtual-machines/windows/instance-metadata-service?tabs=windows
The SDK is attempting to detect the Azure information. It could mean for Durable Functions, this information and endpoint is not available.
This does not affect SDK operations and should not block you from performing other actions on the CosmosClient instance.

Unable to Connect from Azure Function to Azure Cosmos DB. Getting "Microsoft.Azure.Cosmos.Direct: Object reference not set to an instance of an object

Got a .Net code from Udemy course and ran in my local. Wrote an Azure Function which connects to Azure Cosmos DB and creates an item. But not getting connected to Azure Cosmos DB. See below the code and error. Appreciate any help. In the debug, found out some issue with the line
_container.CreateItemAsync(_blobdetails, newPartitionKey(_message.VideoName)).GetAwaiter().GetResult();
Code :
using System;
using System.Text;
using System.Text.Json;
using System.Threading.Tasks;
using Azure.Messaging.ServiceBus;
using Azure.Storage.Blobs;
using Azure.Storage.Blobs.Models;
using Microsoft.Azure.Cosmos;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Host;
using Microsoft.Extensions.Logging;
namespace ReceiveMessages
{
public static class Process
{
private static string blob_connection_string = "<blob connection string>";
private static string source_container_name = "unprocessed";
private static string destination_container_name = "processed";
private static readonly string _connection_string = "<cosmos connection string>";
private static readonly string _database_name = "appdb";
private static readonly string _container_name = "video";
[FunctionName("ProcessVideos")]
public static async Task Run([ServiceBusTrigger("videoqueue", Connection = "connection")]ServiceBusReceivedMessage myQueueItem, ILogger log)
{
try
{
ReceivedMessage _message = JsonSerializer.Deserialize<ReceivedMessage>(Encoding.UTF8.GetString(myQueueItem.Body));
BlobServiceClient _client = new BlobServiceClient(blob_connection_string);
BlobContainerClient _source_container_client = _client.GetBlobContainerClient(source_container_name);
BlobClient _source_blob_client = _source_container_client.GetBlobClient(_message.VideoName);
BlobContainerClient _destination_container_client = _client.GetBlobContainerClient(destination_container_name);
BlobClient _destination_blob_client = _destination_container_client.GetBlobClient(_message.VideoName);
CosmosClient _cosmosclient = new CosmosClient(_connection_string, new CosmosClientOptions());
Container _container = _cosmosclient.GetContainer(_database_name, _container_name);
BlobDownloadInfo _info = _source_blob_client.Download();
// Copy the blob to the destination container
await _destination_blob_client.StartCopyFromUriAsync(_source_blob_client.Uri);
log.LogInformation(_info.Details.LastModified.ToString());
log.LogInformation(_info.ContentLength.ToString());
BlobDetails _blobdetails = new BlobDetails();
_blobdetails.BlobName = _message.VideoName;
_blobdetails.BlobLocation = "https://videostorage100.blob.core.windows.net/processed/" + _message.VideoName;
_blobdetails.ContentLength = _info.ContentLength.ToString();
_blobdetails.LastModified = _info.Details.LastModified.ToString();
_blobdetails.id = Guid.NewGuid().ToString();
_container.CreateItemAsync(_blobdetails, new PartitionKey(_message.VideoName)).GetAwaiter().GetResult();
Console.WriteLine("Item created");
// Delete the blob from the unprocessed container
_source_blob_client.Delete();
// Add the details of the blob to an Azure Cosmos DB account
}
catch (Exception ex)
{
string s = ex.Message;
}
}
}
}
* Executed 'ProcessVideos' (Failed, Id=53b3d0b2-d46a-4ba9-bf26-d8de76af0bce, Duration=41001ms)
[2022-03-19T23:07:25.845Z] Executed 'ProcessVideos' (Failed, Id=48b50a3d-f69f-436f-accf-5140c3d7f8a0, Duration=41001ms)
[2022-03-19T23:07:25.854Z] System.Private.CoreLib: Exception while executing function: ProcessVideos. Microsoft.Azure.Cosmos.Direct: Object reference not set to an instance of an object.{"name":"CreateItemAsync","id":"c16e23cd-badc-4f0b-a940-3fac7f52c4f7","caller info":{"member":"OperationHelperWithRootTraceAsync","file":"ClientContextCore.cs","line":219},"start time":"11:06:46:894","duration in milliseconds":36808.7271,"data":{"Client Configuration":{"Client Created Time Utc":"2022-03-19T23:06:45.5706176Z","NumberOfClientsCreated":3,"User Agent":"cosmos-netstandard-sdk/3.19.0|3.19.1|08|X64|Microsoft Windows 10.0.19043|.NET Core 3.1.20|N|","ConnectionConfig":{"gw":"(cps:50, urto:10, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: False, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[])"}},"children":[{"name":"ItemSerialize","id":"647e3fbb-bd9f-4b12-9367-3ed1f6c4d436","caller info":{"member":"ExtractPartitionKeyAndProcessItemStreamAsync","file":"ContainerCore.Items.cs","line":931},"start time":"11:06:46:921","duration in milliseconds":20.2124},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","id":"23650859-deb4-44e1-a696-daeeab7564c8","start time":"11:06:47:893","duration in milliseconds":35799.6699,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","id":"f18265a9-2a93-46c8-aa05-78b69ed30086","start time":"11:06:47:927","duration in milliseconds":35763.5861,"data":{"CPU Load History":{"CPU History":"(2022-03-19T23:06:47.9376170Z 38.049)"}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","id":"876ad64c-0dc3-42a2-9a47-4054ec301b57","start time":"11:06:47:945","duration in milliseconds":35744.6099,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","id":"01ca2b19-39c0-477d-ad85-f930397e682a","start time":"11:06:47:954","duration in milliseconds":35729.8987,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","id":"e372af16-b68c-438d-9ae1-2fdaad2d5f23","start time":"11:06:47:955","duration in milliseconds":35720.3525,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","id":"9995478e-5933-4b21-8c08-cf03501ebe03","caller info":{"member":"ProcessMessageAsync","file":"TransportHandler.cs","line":109},"start time":"11:06:47:963","duration in milliseconds":35704.0196,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-westus1-fd76.documents.azure.com:14059/apps/0152c08e-edca-4977-bca0-40bb4325ee70/services/117845df-eb50-4f9a-8f97-0a5981cfeaae/partitions/1e48d158-7844-4a7c-89a0-aa99c17adcb8/replicas/132920901445077053s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-westus1-fd76.documents.azure.com:14352/apps/0152c08e-edca-4977-bca0-40bb4325ee70/services/117845df-eb50-4f9a-8f97-0a5981cfeaae/partitions/1e48d158-7844-4a7c-89a0-aa99c17adcb8/replicas/132920901532421814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-westus1-fd76.documents.azure.com:14095/apps/0152c08e-edca-4977-bca0-40bb4325ee70/services/117845df-eb50-4f9a-8f97-0a5981cfeaae/partitions/1e48d158-7844-4a7c-89a0-aa99c17adcb8/replicas/132920901532421816s/"}],"RegionsContacted":["https://videodbupdate-westus.documents.azure.com/"],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2022-03-19T23:06:48.3431877Z","EndTimeUTC":"2022-03-19T23:06:48.4598190Z","TargetEndpoint":"https://videodbupdate-westus.documents.azure.com//addresses/?$resolveFor=dbs%2fHdYjAA%3d%3d%2fcolls%2fHdYjAIRIK9s%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=0"},{"StartTimeUTC":"2022-03-19T23:06:54.7678280Z","EndTimeUTC":"2022-03-19T23:06:54.8820135Z","TargetEndpoint":"https://videodbupdate-westus.documents.azure.com//addresses/?$resolveFor=dbs%2fHdYjAA%3d%3d%2fcolls%2fHdYjAIRIK9s%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=0"},{"StartTimeUTC":"2022-03-19T23:07:01.6288211Z","EndTimeUTC":"2022-03-19T23:07:01.7399788Z","TargetEndpoint":"https://videodbupdate-westus.documents.azure.com//addresses/?$resolveFor=dbs%2fHdYjAA%3d%3d%2fcolls%2fHdYjAIRIK9s%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=0"},{"StartTimeUTC":"2022-03-19T23:07:09.6372169Z","EndTimeUTC":"2022-03-19T23:07:09.7484346Z","TargetEndpoint":"https://videodbupdate-westus.documents.azure.com//addresses/?$resolveFor=dbs%2fHdYjAA%3d%3d%2fcolls%2fHdYjAIRIK9s%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=0"},{"StartTimeUTC":"2022-03-19T23:07:17.9939496Z","EndTimeUTC":"2022-03-19T23:07:18.1025134Z","TargetEndpoint":"https://videodbupdate-westus.documents.azure.com//addresses/?$resolveFor=dbs%2fHdYjAA%3d%3d%2fcolls%2fHdYjAIRIK9s%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=0"}],"*
*[2022-03-19T23:07:25.848Z] System.Private.CoreLib: Exception while executing function: ProcessVideos. Microsoft.Azure.Cosmos.Client: Response status code does not indicate success: ServiceUnavailable (503); Substatus: 0; ActivityId: 349d6ef1-4696-4ec8-88c9-5913129164ec; Reason: (Service is currently unavailable. More info: https://aka.ms/cosmosdb-tsg-service-unavailable
[2022-03-19T23:07:26.012Z] System.Private.CoreLib: Exception while executing function: ProcessVideos. Microsoft.Azure.Cosmos.Client: Response status code does not indicate success: ServiceUnavailable (503); Substatus: 0; ActivityId: 40f51724-2bf3-46b2-ac99-c9030aed41c6; Reason: (Service is currently unavailable. More info: https://aka.ms/cosmosdb-tsg-service-unavailable
[2022-03-19T23:07:26.040Z] ActivityId: 349d6ef1-4696-4ec8-88c9-5913129164ec, Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, Windows/10.0.19043 cosmos-netstandard-sdk/3.19.1);. Microsoft.Azure.Cosmos.Direct: Message: The requested resource is no longer available at the server.*
This was fixed in SDK 3.20: https://github.com/Azure/azure-cosmos-dotnet-v3/blob/master/changelog.md#-3200---2021-06-21
Please upgrade to the recommended version to get this and other fixes.
More fixes
You are creating a client (both Blob and Cosmos) per Function execution, that goes against the recommendations and will bring problems as the number of queue messages increase, please follow https://learn.microsoft.com/azure/azure-functions/manage-connections?tabs=csharp#static-clients and use Singleton/static instances. We have a complete example on how to use DI and Functions with the CosmosClient at https://github.com/Azure/azure-cosmos-dotnet-v3/tree/master/Microsoft.Azure.Cosmos.Samples/Usage/AzureFunctions.
Also, do not block threads, since your Function is already async, do await _container.CreateItemAsync(_blobdetails, new PartitionKey(_message.VideoName)) instead.
These 2 points is what will generate these Service Unavailable errors in most cases.
ALSO VERY IMPORTANT, YOUR POST CONTAINED SERVICE KEYS AND CONNECTIONSTRINGS (I EDITED TO REMOVE THEM BUT THEY WERE ALREADY EXPOSED), ROTATE THEM IMMEDIATELY

Unable to connect to Azure Cosmos Db Account using Microsoft.EntityFrameworkCore.Cosmos - Response status code

The CosmosDb provider is sending this message:
“Response status code does not indicate success: 503 Substatus: 0 Reason: (The request failed because the client was unable to establish connections to 3 endpoints across 1 regions. Please check for client resource starvation issues and verify connectivity between client and server.”
In my tests, it works (.net core 3.1):
Task.Run(async () =>
{
var endpoint = “test”;
var masterKey = “test”;
using (var client = new DocumentClient(new Uri(endpoint), masterKey))
{
//Insert new Document
Console.WriteLine("\r\n>>>>>>>>>>>>>>>> Creating Document <<<<<<<<<<<<<<<<<<<");
dynamic candidato = new
{
Id = 1,
Nome = "Test"
};
var document1 = await client.CreateDocumentAsync(
UriFactory.CreateDocumentCollectionUri("Test", "Test"),
candidato);
Console.ReadKey();
}
}).Wait();
It does not:
Task.Run(async () =>
{
using (var context = new StudentsDbContext())
{
context.Add(new FamilyContainer(2, "Test"));
await context.SaveChangesAsync();
}
}).Wait();
public class FamilyContainer
{
public int Id { get; set; }
public string Nome { get; set; }
public FamilyContainer(int id, string nome)
{
Id = id;
Nome = nome;
}
}
public class StudentsDbContext : DbContext
{
public DbSet<FamilyContainer> FamilyContainer { get; set; }
protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
{
optionsBuilder.UseCosmos(
"test",
"test",
"FamilyDatabase",
options =>
{ }
);
}
protected override void OnModelCreating(ModelBuilder modelBuilder)
{
modelBuilder.Entity<FamilyContainer>(x =>
{
x.ToContainer("FamilyContainer");
});
}
}
Packages
Can anyone help me? Thanks
fail: Microsoft.EntityFrameworkCore.Update[10000]
An exception occurred in the database while saving changes for context type '...'.
Microsoft.EntityFrameworkCore.Storage.RetryLimitExceededException: Maximum number of retries (6) exceeded while executing database operations with 'CosmosExecutionStrategy'. See inner exception for the most recent failure.
---> Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: 503 Substatus: 0 Reason: (Microsoft.Azure.Documents.ServiceUnavailableException: Service is currently unavailable.ActivityId: 07fbf539-0d44-4e5a-89d0-cd46838ee605, {"RequestStartTimeUtc":"2020-02-21T16:34:09.1834993Z","RequestEndTimeUtc":"2020-02-21T16:34:41.3484203Z","RequestLatency":"00:00:32.1649210","IsCpuOverloaded":false,"NumberRegionsAttempted":1,"ResponseStatisticsList":[{"ResponseTime":"2020-02-21T16:34:11.5964152Z","ResourceType":2,"OperationType":0,"StoreResult":"StorePhysicalAddress: rntbd:.../, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: True, StatusCode: 410, SubStatusCode: 0, RequestCharge: 0, ItemLSN: -1, SessionToken: , UsingLocalLSN: False, TransportException: A client transport error occurred: Failed to connect to the remote endpoint. (Time: 2020-02-21T16:34:11.5298608Z, activity ID: 07fbf539-0d44-4e5a-89d0-cd46838ee605, error code: ConnectFailed [0x0005], base error: socket error ConnectionRefused [0x0000274D]...
--- End of inner exception stack trace ---
I was facing same issue.
What worked for me is changing ConnectionMode to ConnectionMode.Gateway while initializing CosmosClient like :
var options = new CosmosClientOptions() { ConnectionMode = ConnectionMode.Gateway };
var client = new CosmosClient(endpoint, key, options);
For more details on refer :
https://learn.microsoft.com/en-us/dotnet/api/microsoft.azure.cosmos.cosmosclientoptions?view=azure-dotnet
https://learn.microsoft.com/en-us/dotnet/api/microsoft.azure.cosmos.connectionmode?view=azure-dotnet
TransportException: A client transport error occurred: Failed to connect to the remote endpoint. (Time: 2020-02-21T16:34:11.5298608Z, activity ID: 07fbf539-0d44-4e5a-89d0-cd46838ee605, error code: ConnectFailed [0x0005], base error: socket error ConnectionRefused
This means that the Connection was refused.
Either your Cosmos DB account has Firewall/VPN enabled and the application is not able to establish a connection due not not being in a whitelisted IP/Network : Try checking your account configuration.
The environment you are executing the code is restricting connections (some corporate Firewall or network might be blocking port ranges): Try running the app in a different network, or use GatewayMode. If that works, then this is related to the network.
The machine might be running low on sockets or high on CPU.
My RCA for this is: Cosmos Partitions where served by individual processes on CosmosDB, each partition serving process has it's own TCP port. When client connects to 443 (Using TCP Direct Mode), CosmosDB Proxy sends partition ports back to client so that client can talk to server-partitions in parallel. Partition ports are random (11000 upwards afaik). Normal company firewall would allow outbound 443 (connection to cosmos works) but blocks the outbound random ports. So at the end, access fails.
Workarounds:
Open firewall
Use Gateway Mode. This uses https/443 only by forwarding internally instead of redirecting to other ports.
It is because Entity framework has a default connection mode of Direct. It worked for me after overriding it to Gateway.
{
optionsBuilder.UseCosmos(
"test",
"test",
"FamilyDatabase",
options =>
{ options.ConnectionMode(ConnectionMode.Gateway); }
);
}
I just want to add this because it wasted a lot of my time. The following code would instantly die with an error message that led me to this S.O. post:
var container = _client.Client.GetContainer(_databaseName, containername);
var result = await container.CreateItemAsync(dataitem, pk);
I disbelieved the error message because everything else has worked, upsert, read, etc. After messing with it for a while, I noticed the documentation shows a template type for CreateItemAsync.
var container = _client.Client.GetContainer(_databaseName, containername);
var result = await container.CreateItemAsync<T>(dataitem, pk);
Changing the code to that fixed it (inside of a templated function).
I wanted to add: if I had been catching exceptions, I would have gotten to the meat of the problem much sooner. The library I am working with is not meant to catch exceptions, they are handled by the layer above it.

Cosmos DB 408 response in Azure Function

I have an Azure Function (v2) that accesses Cosmos DB, but not through a binding (we need to use custom serialization settings). I've followed the example here for setting up an object that should then be available to all instances of the activity function. Mine is a little different because our custom CosmosDb object requires an await for setup.
public static class AnalyzeActivityTrigger
{
private static readonly Lazy<Task<CosmosDb>> LazyCosmosDb = new Lazy<Task<CosmosDb>>(InitializeDocumentClient);
private static Task<CosmosDb> CosmosDb => LazyCosmosDb.Value;
private static Task<CosmosDb> InitializeDocumentClient()
{
return StorageFramework.CosmosDb.GetCosmosDb(DesignUtilities.Storage.CosmosDbContainerDefinitions, DesignUtilities.Storage.CosmosDbMigrations);
}
[FunctionName(nameof(AnalyzeActivityTrigger))]
public static async Task<Guid> Run(
[ActivityTrigger]DurableActivityContext context,
ILogger log)
{
var analyzeActivityRequestString = context.GetInput<string>();
var analyzeActivityRequest = StorageFramework.Storage.Deserialize<AnalyzeActivityRequest>(analyzeActivityRequestString);
var componentDesign = StorageFramework.Storage.Deserialize<ComponentDesign>(analyzeActivityRequest.ComponentDesignString);
var (analysisSet, _, _) = await AnalysisUtilities.AnalyzeComponentDesignAndUploadArtifacts(componentDesign,
LogVariables.Off, new AnalysisLog(), Stopwatch.StartNew(), analyzeActivityRequest.CommitName, await CosmosDb);
return analysisSet.AnalysisReport.Guid;
}
}
We fan out, calling this activity function in parallel. Our documents are fairly large, so updating them is expensive, and that happens as part of this code.
I sometimes get this error when container.ReplaceItemAsync is called:
Response status code does not indicate success: 408 Substatus: 0 Reason: (Message: Request timed out. ...
The obvious thing to do seems to be to increase the timeout, but could this be indicative of some other problem? Increasing the timeout seems like addressing the symptom rather than the problem. We have code that scales up our RUs before all this happens, too. I'm wondering if it has to do with Azure Functions fanning out and that putting too much load on it. So I've also played around with adjusting the host.json settings for durableTask like maxConcurrentActivityFunctions and maxConcurrentOrchestratorFunctions, but to no avail so far.
How should I approach this 408 error? What steps can I consider to mitigate it other than increasing the request timeout?
Update 1: I increased the default request timeout to 5 minutes and now I'm getting 503 responses.
Update 2: Pointing to a clone published to an Azure Function on the Premium plan seems to work after multiple tests.
Update 3: We weren't testing it hard enough. The problem is exhibited on the Premium plan as well. GitHub Issue forthcoming.
Update 4: We seem to have solved this by a combination of using Gateway mode in connecting to Cosmos and increasing RUs.
A timeout can indeed signal issues regarding instance resources. Reference: https://learn.microsoft.com/azure/cosmos-db/troubleshoot-dot-net-sdk#request-timeouts
If you are running on Functions, take a look at the Connections. Also verify CPU usage in the instances. If CPU is high, it can affect requests latency and end up getting timeouts.
For Functions, you can certainly use DI to avoid the whole Lazy declaration: https://github.com/Azure/azure-cosmos-dotnet-v3/tree/master/Microsoft.Azure.Cosmos.Samples/Usage/AzureFunctions
Create a Startup.cs file with:
using System;
using Microsoft.Azure.Cosmos;
using Microsoft.Azure.Functions.Extensions.DependencyInjection;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
[assembly: FunctionsStartup(typeof(YourNameSpace.Startup))]
namespace YourNameSpace
{
public class Startup : FunctionsStartup
{
public override void Configure(IFunctionsHostBuilder builder)
{
builder.Services.AddSingleton((s) => {
CosmosClient cosmosClient = new CosmosClient("connection string");
return cosmosClient;
});
}
}
}
And then you can make your Functions not static and inject it:
public class AnalyzeActivityTrigger
{
private readonly CosmosClient cosmosClient;
public AnalyzeActivityTrigger(CosmosClient cosmosClient)
{
this.cosmosClient = cosmosClient;
}
[FunctionName(nameof(AnalyzeActivityTrigger))]
public async Task<Guid> Run(
[ActivityTrigger]DurableActivityContext context,
ILogger log)
{
var analyzeActivityRequestString = context.GetInput<string>();
var analyzeActivityRequest = StorageFramework.Storage.Deserialize<AnalyzeActivityRequest>(analyzeActivityRequestString);
var componentDesign = StorageFramework.Storage.Deserialize<ComponentDesign>(analyzeActivityRequest.ComponentDesignString);
var (analysisSet, _, _) = await AnalysisUtilities.AnalyzeComponentDesignAndUploadArtifacts(componentDesign,
LogVariables.Off, new AnalysisLog(), Stopwatch.StartNew(), analyzeActivityRequest.CommitName, this.cosmosClient);
return analysisSet.AnalysisReport.Guid;
}
}

Error "Exception while executing function" from Azure Service Bus Listener

We use an Azure Service Bus to post all of our requests from our Xamarin mobile app. The Azure Service Bus is bound to an Azure Function which is triggered each time a requests hits the Azure Service Bus.
We have found that we are getting errors from this Azure Function when we send data above a certain size. We can send up to 800 records without a problem but when we send >=850 records we get the following error:
[Error] Exception while executing function:
Functions.ServiceBusQueueTrigger. mscorlib: Exception has been thrown
by the target of an invocation. mscorlib: One or more errors occurred.
A task was canceled.
The service that is being invoked is an ASP.NET Web API RESTful service that saves the data records into a database. This doesn't generate any errors at all.
Here is my Azure Function code.
#r "JWT.dll"
#r "Common.dll"
using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using System.Net.Http;
using System.Net.Http.Headers;
using System.Text;
using Microsoft.ServiceBus.Messaging;
public static void Run(BrokeredMessage message, TraceWriter log)
{
log.Info($"C# ServiceBus queue trigger function processed message: {message.MessageId}");
if (message != null)
{
Common.Entities.MessageObjectEntity messageObject = message?.GetBody<Common.Entities.MessageObjectEntity>();
string msgType = messageObject?.MessageType;
var msgContent = messageObject?.MessageContent;
log.Info($"Message type: {msgType}");
double timestamp = (DateTime.UtcNow - new DateTime(1970, 1, 1)).TotalSeconds;
string subscriber = "MYSUBSCRIBER";
string privatekey = "MYPRIVATEKEY";
Dictionary<string, object> payload = new Dictionary<string, object>()
{
{"iat", timestamp},
{"subscriber", subscriber}
};
string token = JWT.JsonWebToken.Encode(payload, privatekey, JWT.JwtHashAlgorithm.HS256);
using (var client = new HttpClient())
{
string url = $"http://myexamplewebservices.azurewebsites.net/api/routingtasks?formname={msgType}";
client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue(subscriber, token);
HttpContent content = new StringContent((string)msgContent, Encoding.UTF8, "application/json");
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
var response = client.PostAsync(new Uri(url), content);
if (response == null)
{
log.Info("Null response returned from request.");
}
else
{
if (response.Result.IsSuccessStatusCode && response.Result.StatusCode == System.Net.HttpStatusCode.OK)
{
log.Info("Successful response returned from request.");
}
else
{
log.Info($"Unsuccessful response returned from request: {response.Result.StatusCode}.");
}
}
}
log.Info("Completing message.");
}
}
This code has been working for several years and works across all our other apps / web sites.
Any ideas why we're getting errors wehen we post large amounts of data to our Azure Service Bus / Azure Function?
It may caused by "new httpclient", there is a limit to how quickly system can open new sockets so if you exhaust the connection pool, you may get some errors. You can refer to this link: https://aspnetmonsters.com/2016/08/2016-08-27-httpclientwrong/
And could you please share some more error message ?
I can see that you are creating httpclient connection on each request which possibly be causing this issue. Httpclient creates a socket connection underneath it and has hard limit on it. Even when you dispose it it remains there for couple of mins that can't be used. A good practice is to create single static httpclient connection and reuse it. I am attaching some documents for you to go through.
AzFunction Static HttpClient , Http Client Working , Improper instantiation

Resources