I am using pm2 to keep a script running and making api calls in an interval. Sometimes the api calls fail, the script is running and not restarted but the calls just dont go out. I set up a health monitor in node.js to inform me whenever these api calls cease.
Now whenever I see downtime I go into the pm2 dashboard and restart my app. I would like to skip this step and have my health monitor just make a call to the pm2 instance that it should restart exactly like the dashboard is doing.
Is this possible? If so how can it be done? I see some pm2.restart options in the docs but Im not sure how this is supposed to occur from a different machine.
Related
I am running a nodejs server locally with Firebase emulators (using firebase emulators:start). I have another version running on google cloud functions - but I also need something running locally.
When errors happen, my whole node process dies, and the emulator dies etc... and I have to go back and restart the process manually. Is there a way to automatically restart it when an error happens, such as using forever? I tried with something like forever start firebase emulators:start but that didn't work.
Due to how Firebase is built and operates, it does not support any such functionality of staying awake as a process virtually non-stop. You would most likely need to refactor your code in such a way that the process you need to keep alive indefinitely is separated from your code hosted by the firebase emulator.
We have multiple Google Cloud Run services running for an API. There is one parent service and multiple child services. When the parent service starts it loads a schema from all the children.
Currently there isn't a way to tell the parent process to reload the schema so when a new child is deployed the parent service needs to be restarted to reload the schema.
We understand there there are 1 or more instances of Google Cloud Run running and have ideas on dealing with this, but are wondering if there is a way to restart the parent process at all. Without a way to achieve it, one or more is irrelevant for now. The only way found it by deploying the parent which seems like overkill.
The containers running in google cloud are Alpine Linux with Nodejs, running an express application/middleware. I can stop the node application running but not restart it. If I stop the service Google Cloud Run may still continue to serve traffic to that instances causing errors.
Perhaps I can stop the express service so Google Cloud run will replace that instance? Is this a possibility? Is there a graceful way to do it so it tries to complete and current requests first (not simply kill express)?
Looking for any approaches to force Google Cloud Run to restart or start new instances. Thoughts?
Your design seems, at high level, be a cache system: The parent service get the data from the child service and cache the data.
Therefore, you have all the difficulties of cache management, especially cache invalidation. There is no easy solution for that, but my recommendation will be to use memorystore where all child service publish the latest version number of their schema (at container startup for example). Then, the parent service checks (at each requests, for example) the status in memory store (single digit ms latency) if a new version is available of not. If a new, request the child service, and update the parent service schema cache.
If applicable, you can also set a TTL on your cache and reload it every minute for example.
EDIT 1
If I focus only on Cloud Run, you can in only one condition, restart your container without deploying a new version: set the max-instance param to 1, and implement an exit endpoint (simply do os.exit() or similar in your code)
Ok, you loose all the scale up capacity, but it's the only case where, with a special exit endpoint, you can exit the container and force Cloud Run to reload it at the next request.
If you have more than 1 instance, you won't be able to restart all the running instances but only this one which handle the "exit" request.
Therefore, the only one solution is to deploy a new revision (simply deploy, without code/config change)
I have a Google VM, and i can start a web server. The command i issue is: python server.py.
My plan is to have the application running.
Since i will eventually close my pc (and thus the terminal), will the application continue running normally?
Or, do i have to start the server and then use disown, to make the app run in the background?
NOTE: If the second option is the case, does this mean that when i re-log in, and i want to shut down the server, the only way to do it is with pkill?
I'd like to check the status of an app registered with pm2 remotely such that other web-based monitoring services can give us a notification when something breaks.
Are there any options available to remotely check the status of a process in pm2 remotely? One possibility is to have a web script remotely eval() the pm2 status command and look for certain keywords, and make that script accessible on the web for the notification tool. This doesn't seem ideal, though, as we're using an eval command and maybe a regex of that output just to see what is going on.
Any advice?
I wrote a simple web interface for PM2.
You can simply start a websocket connection to /logs and get your application(s) stats updates such as status, uptime, cpu usage, memory usage, restarts in realtime.Feel free to use and contribute. Cheers!
https://github.com/doorbash/pm2-web
The best option is to use keymetrics. It's free to monitor upto 4 processes(great for development and side projects), easy to link an instance/server but quickly turns out to be very expensive when you scale up.
You could always try switching to other alternatives like upstart or pm2-gui.
I've written two simple Node.js apps, each of which run a server on a port (1337 and 1338), and using PM2 (and Keymetrics) to keep them alive, but every few days all three Node.js apps (including PM2) crash simultaneously. I'm fairly new to Node.js and am not sure how to investigate the source of the problem.
DETAILS:
The two servers respond to incoming GET requests. One is to automatically do a GIT fetch/pull in response to a Bitbucket trigger, and the other converts a URL to PDF and sends the PDF back to the browser (using wkhtmltopdf). They both work fine while they're running.
I'm running Node.js v0.12.4.
When the three apps go down, my first indication is from my Uptime (https://github.com/fzaninotto/uptime) instance on OpenShift which emails me to say one of my Node.js servers is down, returning "connect ECONNREFUSED".
A couple of minutes later Keymetrics emails me saying "Agent is offline: Keymetrics Agent seems to be offline".
In other words, ALL my node.js stuff seems to go down at the same time; neither of the two apps nor even PM2 are running. I thought Node spawned a process for each app, so how could they all go down at once?
WHAT I'VE TRIED:
I've tried digging through the ~/.pm2/pm2.log but haven't been able to find anything useful in there. Maybe I just don't know what to look for.
$ pm2 resurrect always brings everything back to life just fine.
Please upgrade to latest PM2 version:
$ npm install pm2 -g
$ pm2 update