Hi I am using google app engine to host a single instance nodejs application. The application works fine and my scripts are showing no errors in the logs. The application is currently just in testing and is not getting used over night, however often I come to work the next day and the server is just returning internal server errors. No errors are shown in my application log other then the 502 errors which i get when trying to access the next day. I see like 100s of calls for /_ah/_background/ overnight some appear to have timed out. At this point I must restart my instance for the app to continue to function.
I am completely stumped.. Because my app using web-sockets I must use manual scaling and a single instance. Would appreciate any help / suggestions.
I would venture a gues that you have a deferred task stuck running. Tasks that run in the taskqueue api are set by default to continuously retry. You can visit the taskqueue api TaskQueue API
To get the tasks to stop running right now visit the Google Cloud Console
select your project. Then select App Engine. Then select Task queues. Click on the task that is running (probably default). There should be a option to Pause the queue. This should prevent the 500 errors from occurring but will not fix the reason the task is failing.
Related
I have a very basic NodeJS application hosted on Google App Engine that executes an async function on 15 second intervals. The deployment is successful and the app starts and runs fine, but stops after about 30 minutes with the following error logs. This runs fine locally, though.
Quitting on terminated signal
Start program failed: user application failed with exit code -1 (refer to stdout/stderr logs for more detail): signal: terminated
I have used App Engine before with no issues, so I'm not sure why this is happening. I used https://github.com/GoogleCloudPlatform/nodejs-docs-samples/tree/main/appengine/typescript as a reference and am still not able to resolve this issue. Any ideas?
Quitting on terminated signal
You may receive this error if your App Engine instances is down scaling or shutting down due to some reasons and possibly due to:
Your application runs out of Instance Hours quota.
Your instance is moved to a different machine, either because the current machine that is running the instance is restarted, or App Engine moved your instance to improve load distribution.
There are good strategies to avoid the downtime of your instance and here are additional:
You can try to have a minimum number of idle instances
Use manual scaling which you can specify the number of instances
will continuously run regardless of the load level.
Increase the maximum instance.
Asynchronous background work is not recommended in App Engine. It can result in higher billing and users may also experience increased latency because of high pushback or request queuing. Google recommend to use Cloud Tasks. With Cloud Tasks, HTTP requests are long-lived and return a response only after any asynchronous work ends.
I recently deployed a Node.js Backend Service to Azure and have the following problem. The service becomes unresponsive after a certain amount of time, and only comes back to life if a external request is sent. The problem is, that it takes about 3 minutes for the Container to start back up and actually return the request. I'm running Node 14 LTS. I also added a health check yesterday, but azure simply doesn't bother actually keeping the app alive, here is the metric off azure
I verified azure is actually trying to reach the correct endpoint, and it does. I also have "Always On" enabled. I also verified that the app itself, is not crashing. I log every request and all of a sudden requests are no longer received, which means the health endpoint doesn't respond either, but it does not result in a container restart. It just waits for an external request to appear and then decides to start everything back up, which takes too long.
I feel like it's some kind of configuration issue, because the app itself is not very complex and I never experienced crashes when doing local development.
The official document tells us that the Free pricing tier you are currently using, Always on does not take effect.
How do I decrease the response time for the first request after idle time?
i'm having hard time figuring out a blocking issue on a client.
the situation is:
A WebSite on Azure WebApp with Vue and Js and some api deployed on a vm on azure.
one (and only one) of the apis is having serious issues lately. Basically we upload images on a DAM using web api 2.0 as a wrapper.
When we upload multiple images together, the client application iterate through an array and perform X calls to the server(form-Data with images around 1mb). those calls arrives pretty much simultaneously on the server.
on dev/qa, everything works fine. i can see from the logs that different thread are created to handle X requests.
on production, this doesn't work anymore. different thread are created but basically the first request freeze everything for a couple of minutes (iis timeout), not even the log file is updated, and return a "task was canceled" error.
From the log I can see that the LogControllerHandler (that overrides the send async method) is hit but the controller is not; this seems not to happen with small images (150 kb).
the iis on prod and QA should be exactly the same, but for some reason, on prod I got this behavior I cannot figure out.
Any pointer would be apreciated.
Thanks
I was following the chnage notification example in
https://github.com/microsoftgraph/msgraph-training-changenotifications.
I had succesfully replicated it and deployed it to Azure.
But the timer function in the Get method of the Notification Controller stops getting triggered sometimes. Sometimes it happens just like that and sometimes after a fresh deployment of code to Azure.
Is there any work around or any alternative for the Timer class which is good for production app?
I have an Azure Functions application which once in a while "freezes" and stops processing messages and timed events.
When this happens I do not see anything in the logs (AppInsight), neither exceptions nor any kind of unfamiliar traces.
The application has following functions:
One processing messages from a Service Bus topic subscription (belonging to another application)
One processing from an internal storage queue
One timer based function triggered every half hour
Four HTTP endpoints
Our production app runs fine. This is due to an internal dashboard (on big screen in the office), which polls one of the HTTP endpoints every 5 minutes, there by keeping it alive.
Our test, stage and preproduction apps stop after a while, stopping to process messages and timer events.
This question is more or less the same as my previous question, but the without error message that was in focus then. Much fewer error messages now, as our deployment has been fixed.
A more detailed analysis can be found in the GitHub issue.
On a consumption plan, all triggers are registered in the host, so that these can be handled, leading to my functions being called at the right time. This part of the host also handles scalability.
I had two bugs:
Wrong deployment. Do zip based deployment as described in the Docs.
Malformed host.json. Comments in JSON are not right, although it does work in most circumstances in Azure Functions. But not all.
The sites now works as expected, both concerning availability and scalability.
Thanks to the people in the Azure Functions team (Ling Toh, Fabio Cavalcante, David Ebbo) for helping me out with this.