I am running my node application on AWS EC2 server.
For continuous integration I installed Jenkins, on EC2, jenkins polls a code commit continuously and when a commit occurs executes some commands written in a script.
The last command is
pm2 start server.js
Everything works fine and the build is shown successful but later when I access the URL the site doesn't show.
I have a nginx server in front of node server, it gives
502 Bad Gateway
On checking I realised the node application is not running, so I cehcked pm2 logs and found this happening -
2016-08-12 07:53:28: [[[[ PM2/God daemon launched ]]]]
2016-08-12 07:53:28: BUS system [READY] on port /var/lib/jenkins/.pm2/pub.sock
2016-08-12 07:53:28: RPC interface [READY] on port /var/lib/jenkins/.pm2/rpc.sock
2016-08-12 07:53:28: Starting execution sequence in -fork mode- for app name:server id:0
2016-08-12 07:53:28: App name:server id:0 online
2016-08-12 07:53:28: pm2 has been killed by signal, dumping process list before exit...
2016-08-12 07:53:28: Deleting process 0
2016-08-12 07:53:28: Stopping app:server id:0
2016-08-12 07:53:28: App [server] with id [0] and pid [25822], exited with code [0] via signal [SIGTERM]
2016-08-12 07:53:28: Proc is not defined anymore or is being killed
2016-08-12 07:53:28: [PM2] Exited peacefully
pm2 is getting killed as soon as starting, don't know why, tried reinstalling pm2 from npm didn't work.
Ubuntu 14.04 LTS
Node v4.4.7 LTS
npm v2.15.8
pm2 v1.1.3
Work is halted, need help immediately, please.
The answer is in the comments given by Dusan Bajic, if anyone is doing continous integration with jenkins on EC2 using a node application and similar thing happens with pm2 just add this line in script before starting pm2.
export BUILD_ID=dontKillMePlease
and instead of
pm2 start server.js
use
pm2 restart server.js
server.js being your application or else build will fail if in jenkins if you commit and it runs script again cause pm2 is already running server.js and won't stop.
From ProcessTreeKiller
To reliably kill processes spawned by a job during a build, Jenkins
contains a bit of native code to list up such processes and kill them
...
How it works
The ProcessTreeKiller takes advantage of the fact that by
default a new process gets a copy of the environment variables of its
spawning/creating process.
It sets a specific environment variable in the process executing the
build job. Later, when the user requests to stop the build job's
process it gets a list of all processes running on the computer and
their environment variables, and looks for the environment variable
that it initially set for the build job's process.
Every job with that environment variable in its environment is then
terminated.
If your build wants to leave a daemon running behind...
A convenient way to achieve that is to change the environment variable
BUILD_ID which Jenkins's ProcessTreeKiller is looking for. This will
cause Jenkins to assume that your daemon is not spawned by the Jenkins
build. For example:
BUILD_ID=dontKillMe /usr/apache/bin/httpd
Related
My NodeJs App was successfully deployed inside a Docker Container. I put this command in Dockerfile:
CMD ["pm2-runtime", "app.js"]
After restarting the app from the PM2 key metrics panel, the container disappeared (docker ps shows nothing) but strangely the app kept running. After a couple of minutes I figured out that pm2 was running it globally in the server. pm2 ls shows the app process.
Is this behaviour considered normal? Can this be prevented?
This is not the expected behvour, there might be the case the same application is running on host with same keys, so by issue restart command can restart both process, but pm2 dashboard able to recognise the process even if it running against same keys.
What I assume, That after the restart from the dashboard there were some errors that kill the container and you were not able to see the container when you run docker ps, as pm2-runtime retires 3 times if there is any error occurs.
One way to double-check is to verify, run docker ps -a and grab the container id and check docker logs stopeed_container_id, you will probely see somethign like
2019-1-02T20:53:03: PM2 log: 0 application online, retry = 3
2019-1-02T20:53:05: PM2 log: 0 application online, retry = 2
2019-1-02T20:53:07: PM2 log: 0 application online, retry = 1
2019-1-02T20:53:09: PM2 log: 0 application online, retry = 0
2019-1-02T20:53:09: PM2 log: Stopping app:www id:0
2019-1-02T20:53:09: PM2 error: app=www id=0 does not have a pid
2019-1-02T20:53:09: PM2 log: PM2 successfully stopped
So overide this behour to do not stopped container even if there is error, I will not recomend this approch but You can run with --no-auto-exit flag.
--no-auto-exit
do not exit if all processes are errored/stopped or 0 apps launched
pm2-runtime option
CMD pm2-runtime --no-auto-exit app.js
I have tried so many ways to keeping node server alive on Linux environment but nothing has worked. Sometime the server runs only 4-5 hours and sometime it runs 10-12 hours and after that server goes shut down automatically.
I have tried forever start, pm2, nodemon but nothing has worked.
I have also tried shell script with forever start for running it but that also not worked.
Applications that are running under PM2 will be restarted automatically if the application crashes or is killed, but an additional step needs to be taken to get the application to launch on system startup (boot or reboot). Luckily, PM2 provides an easy way to do this, the startup subcommand.
The startup subcommand generates and configures a startup script to launch PM2 and its managed processes on server boots:
$ pm2 startup systemd
Run the command that was generated (similar to the highlighted output above, but with your username instead of sammy) to set PM2 up to start on boot (use the command from your own output):
sudo env PATH=$PATH:/usr/bin /usr/lib/node_modules/pm2/bin/pm2 startup systemd -u sammy --hp /home/sammy
check here for details https://www.digitalocean.com/community/tutorials/how-to-set-up-a-node-js-application-for-production-on-ubuntu-16-04
You can increase the size of memory restar- check this: pm2 process crashed on server. it gives an error
Try using process manager for making the application run all the time. Here is the link for Pm2 . It will restart you application once it crashes also automatically
Use a NPM package called nodemon
npm install -g nodemon
nodemon index.js
If the application fails or crashes for any reason it will restart
Read more at
https://www.npmjs.com/package/nodemon
CI/CD services will often use SUCCEDED or FAILED to display if a deployment succeded and if an error occurred. However, with PM2 this is not so easy.
If you do:
pm2 start ecosystem.config.js in one of your deployment steps, PM2 will start the process but won't report an error if one of the processes fails. To check it, you would need to execute pm2 status, and then judging by the number of restarts and by executing pm2 logs --err you would be able to see what happens. I would like PM2 to return a non-zero code or be verbose about a process failing on first launch, rather than what it does now (failing silently).
Is there a native PM2 command that makes it integrate with CI/CD release status reporting easily?
What I want to achieve:
pm2 start ecosystem.config.js
# one of the processes in ecosystem.config.js fails, because eg. the database engine is not running and the driver can't connect to the db
# pm2 prints to stderr and a failure shows up on my CI/CD service. pm2 closes and returns non-zero if it failed to launch one of the processes
What's happening currently:
pm2 start ecosystem.config.js
# one of the processes fails
# no error is printed, everything is ok according to pm2. pm2 keeps running and the problematic process fails silently
I want to rebuild my node+vue.js app to update the changes I made. The app was started with pm2, but the problem is that I can't see anything when I do pm2 ls. The list is empty. On the other hand, the node app is running. I tried to kill it with pid, but it is restarting immediately.
Any suggestions?
I have a node chat application that needs to keep running on my server (ubuntu with nginx). The problem is that the application stops after a few hours or days.
When I check on the server I see that my pm2 list is empty.
The code I use to start my app:
pm2 start notification_server/index.js
It somehow looks as if pm2 is reset after a while. I also tried using forever, but then I run into the same problem. Is there some way to prevent the pm2 list from getting empty?
This is most likely an indication that your server is rebooting. When your server reboots, PM2 shuts down and deletes all Node instances from its "status" list.
You can perform the following steps to make PM2 relaunch your Node programs start back up on reboot:
Run pm2 startup and follow the directions (you will have to perform a sudo command; PM will tell you exactly what to do).
Through pm2 start, get your Node processes up and running just like you like them.
Run pm2 save to register the current state of things as what you want to see on system startup.
Source: http://pm2.keymetrics.io/docs/usage/startup/
Did you try checking logs $ pm2 logs for you application?
Most likely it will tell you why your application was terminated or maybe it just exited as it supposed to. You could find something like that there:
PM2 | App [app] with id [0] and pid [11982], exited with code [1] via signal [SIGINT]
This can tell you what happened. Without more details, it's hard to give you a better answer.