I'm pushing bunch of data in Cosmos DB using MongoDB NodeJS drivers. I'm running into the following error
{"Errors":["Request rate is large"]}
I realize this error occurs when the RU/s have exceeded the set threshold. I have partitioned my collection and set 2000 RU/s. I also have set an exponential backoff delay if that error occurs. It was working fine earlier i.e. it retried after a while if it ran into that error but succeeded after waiting. But currently it's consistently throwing the 'Request rate is large' even though I have a mechanism to try again after some delay. Any idea what might be going on and how can I resolve the issue?
Along with that is there a way to extract the raw response headers sent from cosmos DB using MongoDB NodeJS driver as I would like to extract x-ms-retry-after-ms and x-ms-request-charge headers.
Related
On occasion I'm getting a rate limit error without being over my rate limit. I'm using the text completions endpoint on the paid api which has a rate limit of 3,000 requests per minute. I am using at most 3-4 requests per minute.
Sometimes I will get the following error from the api:
Status Code: 429 (Too Many Requests)
Open Ai error type: server_error
Open Ai error message: That model is currently overloaded with other requests. You can retry your request, or contact us through our help center at help.openai.com if the error persists.
Open ai documentation states that a 429 error indicates that you have exceeded your rate limit which clearly I have not.
https://help.openai.com/en/articles/6891829-error-code-429-rate-limit-reached-for-requests
The weird thing is the open ai error message is not stating that. It is giving the response I usually get from a 503 error (service unavailable).
I'd love to hear some thoughts on this, any theories, or if anyone else has been experiencing this.
I have seen a few message on the OpenAI community forum with similar messages. I suggest checking out the error code guide we have with suggestion to mitigate these errors, in general though, it's possible the model itself was down and has nothing to do with your rate limit: https://platform.openai.com/docs/guides/error-codes
This error indicates the OpenAI servers have too many requests from all users and their servers have reached their capacity to service your request. It's pretty common at the moment.
Hopefully they will upgrade their servers soon. Not really sure why it is a big problem since they run on Azure and should be able to scale based on ramped up demand. Maybe they are just trying to minimise costs.
I'm using boto3 to write a lot of objects to S3. I keep hitting an error like:
botocore.exceptions.ClientError: An error occurred (SlowDown) when calling the PutObject operation (reached max retries: 0): Please reduce your request rate.
This is clearly coming from the server and asking my code to slow down the requests it's making. My question is whether there's a way to get aiobotocore to handle this automatically with its retry logic?
It should be theoretically possible for the response to be handled automatically, including a wait. The issue with doing this in my own code is that there are many tasks all hitting the same bucket, so negotiating rate limiting between them will be very complex indeed.
So I am getting an error in Azure Data Factory that I haven't been able to find any information about. I am running a data flow and eventually (after an hour or so) get this error
{"StatusCode":"DFExecutorUserError","Message":"Job failed due to
reason: The service has encountered an error processing your request.
Please try again. Error code 1204.","Details":"The service has
encountered an error processing your request. Please try again. Error
code 1204."}
Troubleshooting I have already done :
I have successfully ran the data flow using the sample option. Did this with 1 million rows.
I am processing 3 years of data and I have successfully processed all the data by filter the data by year and running the data flow once for each year.
So I think I have shown the data isn't the problem, because I have processed all of it by breaking it down into 3 runs.
I haven't found a pattern in the time the pipeline runs for before the error occurs that would indicate I am hitting any timeout value.
The source and sink for this data flow are both an Azure SQL Server database.
Does anyone have any thoughts? Any suggestions for getting a more verbose error out of data factory (I already have the pipeline set with verbose logging).
We are glad to hear that you has found the cause:
"I opened a Microsoft support ticket and they are saying it is a
database transient caused failure."
I think the error will be resolved automatically. I post this as answer and this can be beneficial to other community members. Thank you.
Update:
The most important thing is that you have resolved it by increase the vCorces in the end.
"The only thing they gave me was their BS article on handling
transient errors. Maybe I’m just old but a database that cannot
maintain connections to it is not very useful. What I’ve done to
workaround this is increase my vCores. This sql database was a
serverless one. While performance didn’t look bad my guess is the
database must be doing some sort of resize in the background to
handle the hour long data builds I need it to do. I had already tried
setting the min/max vCores to be the same. The connection errors
disappeared when I increased the vCores count to 6."
We are getting "Operation was cancelled" exception while Azure Indexer is running for larger records (around 2M+). Here are the log details -
"The operation was canceled. Unable to read data from the transport connection: The I/O operation has been aborted because of either a thread exit or an application request. The I/O operation has been aborted because of either a thread exit or an application request "
We are running the indexer under thread. It is working for smaller records but for larger records (1M+), it is throwing Socket Exception.
Does anyone saw this error while running Azure Indexer for larger records (running for long time)?
(we have already increase httpclient timeout to maximum value for serviceClient object.)
This could happen because of happen because of excess http connections. Try to make your **HttpClient** static and see if anything improves. **HttpClient** timeout to maximum value is required to execute with maximum records.
You may also want to consider working to reduce your sql query time for best indexer performance. Also please share you code if possible.
Hope it helps.
Try set SearchServiceClient.HttpClient.Timeout to Timeout.InfiniteTimeSpan. You have to set the timeout before you send any request to Azure Cognitive Search.
client.HttpClient.Timeout = Timeout.InfiniteTimeSpan;
I developed an application that backed up with nodeJs/mongodb and frontend is in Angularjs 1.6.5. Application is used to provide realtime analytics and data is being increased in size every single minute with new data.
As the database size is increasing, queries are taking much longer time to execute and even sometime fails and giving 404 error. (Nginx server) But the strange thing is that whenever i ran the same query in mongo shell in server with .explain('executionStats') method its giving immediate responses with execution time 0 millisecond.
So as per above screenshots, with limit of 10 data mongo shell executing it in no time but in browser when i am hitting the db through node and mongoose it took 1 min but couldn't possible return the result (maybe nginx returning 404 after specif time)
but i try in mongo shell after that without setting up any limit that also returning data with 6331522 records in less then 4 seconds.
I have no clue what is the issue exactly. Any help would be appriciated