Im using a Cloud Run to run my deployment test suite. It takes about 3 minutes to run the suite and my instance timeou is set to 5 minutes.
I've set up a Cloud Run project that will accept an http request (from my CI provider) triggering the tests to run, and then report back pass fail.
Even though the containers are set to only handle 1 concurrent request they are accepting a second request after the first test run completes. As the first run took up 3 of the available 5 minutes, the Second request times-out at 2 minutes.
So, does anyone know of a way to either self terminate a given instance (preferably from within) or to set the total number of requests an instance will accept before closing itself?
Thank you very much for reading my question. Any help would be greatly appreciated!
You don't have instance timeout in Cloud Run. The timeout is on the request processing. You set the maximum duration time to process a request (up to 3600 seconds). So, in your case, you haven't this timeout issue, or I didn't understand your configuration and current issue.
The other part of your question is "how to stop an instance". Simply exit it! According to your language the method are different. In python for example exit(0).
Related
I’m learning to work with Cloud Functions (and Cloud Run) and would like to know it’s behaviour when (HTTP triggered) function already running at Max instances capacity and more HTTP requests come-in
1)
Here’s my function basic code (simple function with approx. 1000ms execution time per invocation):
ctr = 0
def hello_world(request):
global ctr
print("hello_world(): "+str(ctr))
ctr=ctr+1
time.sleep(10)
response = flask.Response("success::", 200)
return response
2)
deployed this function with flag --max-instances=1 (just to ensure no new VM instances come up to handle concurrent requests)
3)
and then send 5 concurrent results
From what I observe, only one of the requests get processed. Other 4 requests just dropped (client received HTTP status code 500, and no trace for these dropped requests in Stackdriver Logging either)
In the link here https://cloud.google.com/functions/docs/max-instances it says:
In that case, incoming requests queue for up to 60 seconds. During this 60 second window, if an instance finishes processing a request, it becomes available to process queued requests. If no instances become available during the 60 second window, the request fails.
Based on which I expected, that while one request is being handled, others would be queued up for max 60 seconds. Therefore, all 5 requests should have been processed (or at-least >1 requests if not all 5!). However actual behaviour that I'm seeing is different from this.
Can someone explain pls.
UPDATE-1: It seems the fix has been released and documentations updated
1) It still continued to return status code 500 during initial colds start (when no instances running) for some of the concurrent requests.. EXPECTED I suppose
2) Also, temporarily exceeded max-instances=1 during very initial bursts of 10 requests, launching upto 4 instances AND successfully server all 4 requests.
3) thereafter, # instanced did come down to respect --max-instances=1settings and all but one requests returned withstatus code 429`
The Cloud Function engineering team is now aware of this and they will proceed to change the documentation to reflect the changes made to the feature still in active development. Stay tuned for the documentation, the update will come shortly =). Thank you for noticing this with your tests!
Aside from this, and as a suggestion from the team, if you are interested on queuing jobs, it is recommended to use Cloud Tasks
My App Engine Flexible cron sometimes takes more than 120 seconds. So, whenever it exceeds 120 seconds, app engine throws 502 error. It doesn't terminate my nodejs task, it only terminates the http request started by App Engine Cron job.
There is one value 240 seconds, I didn't understand where its coming from. I guess this is a retry request. It would be helpful if anyone can highlight this as well.
As per App Engine documentation, a cron can run for an hour. Is this true for http requests started by cron job as well?
To be clear, I want to run my cron for more than 120 seconds and http request to be active for 1 hour.
Even though you have switched to Kubernetes Engine, I would like to take the chance and clarify the purpose of cron jobs here.
As it is stated in the official documentation, cron jobs are used to perform jobs at regular time intervals. They involve invoking a URL through an HTTP request and run for up to 60 minutes while respecting the request's own limitations.
Some good uses for cron jobs: sending report emails on a daily basis, update cached data at regular time intervals or update summary information every hour. When a task involves obtaining external information, especially when there is a large number of operations involved that may exceed the time an HTTP connection remains open, or when there are different types of data that are coming from the external application, I would not consider it a good use of cron jobs.
If you are using Kubernetes now, and consider it to be more useful for the tasks you need to perform, go ahead and continue with it.
I'm currently building web API using AWS Lambda with Serverless Framework.
In my lambda functions, each of them connects to Redis (elasticache) and RDB (Aurora, RDS) or DynamoDB to retrieve data or write new data.
And all my lambda functions are running in my VPC.
Everything works fine except that when a lambda function is first executed or executed a while after last execution, it takes quite a long time (1-3 seconds) to execute the lambda function, or sometimes it even respond with a gateway timeout error (around 30 seconds), even though my lambda functions are configured to 60 seconds timeout.
As stated in here, I assume 1-3 seconds is for initializing a new container. However, I wonder if there is a way to reduce this time, because 1-3 seconds or gateway timeout is not really an ideal for production use.
You've go two issues:
The 1-3 second delay. This is expected and well-documented when using Lambda. As #Nick mentioned in the comments, the only way to prevent your container from going to sleep is using it. You can use Lambda Scheduled Events to execute your function as often as every minute using a rate expression rate(1 minute). If you add some parameters to your function to help you distinguish between a real request and one of these ping requests you can immediately return on the ping requests and then you've worked around your problem. It will cost you more, but we're probably talking pennies per month if anything. Lambda has a generous free tier.
The 30 second delay is unusual. I would definitely check your CloudWatch logs. If you see logs from when your function is working normally but no logs from when you see the 30 second timeout then I would assume the problem is with API Gateway and not with Lambda. If you do see logs then maybe they can help you troubleshoot. Another place to check is the AWS Status Page. I've seen sometimes where Lambda functions timeout and respond intermittently and I pull my hair out only to realize that there's a problem on Amazon's end and they're working on it.
Here's a blog post with additional information on Lambda Container Reuse that, while a little old, still has some good information.
I have a ajax that calls a function.
This function spend 5 minutes to complete.
When I run in my machine, it's everything ok.
But when I run in my deployed web site in azure, the request return with error 500 when past 3.5 minutes. But it's continue running and complete the work, I see in the database.
The response is blank.
Any help?
Thanks!
You can change approach and use web sockets.
5 minutes is a long time to hold a connection, a lot can happen in 5 minutes,
Different approach would be to return a guid before you start the process and make a lull request from the client every 10 sec or so until the process state is changed to finished and you can return the result.
Good luck.
Is there an option or a setting somewhere to control the timeout for an aws ec2 wait command?
Or the number of attempts or waiting period between attempts?
I want to be able to aws ec2 wait instance-terminated for some instances I'm quickly spinning up to perform a few task then terminating. It times out on some longer running tasks with "Waiter InstanceTerminated failed: Max attempts exceeded".
I can't seem to find any info anywhere. I've grepped the cli source code, but my knowledge of Python is too limited for me to understand what's going on. I see there might be something in this test using maxAttempts and delay, but can't figure out how to leverage that from the cli.
So far my suboptimal solution is to sleep first, then start the wait.
There is not a timeout option in the AWS CLI, but you can just use the native timeout command from coreutils to do what you want.
timeout 10 aws ec2 wait instance-terminated
will abort if the command does not return within 10 seconds. A timeout will automatically return error code 124, otherwise it returns the error code of the command.
There's an open Github issue about adding configurable parameters https://github.com/aws/aws-cli/issues/1295
You can also find some environment variables you can define here
https://docs.aws.amazon.com/cli/latest/topic/config-vars.html
One of them being AWS_MAX_ATTEMPTS Number of total requests
But for my use case (restore dynamo table from snapshot) does not seem to be working