Recently I made a simple API server using node.js+express. And the script below is a part of my package.json file I use to run with npm commands.
"scripts": {
...
"release": "cross-env NODE_ENV=production MODE=release node server/app.js",
}
After I starts the server with npm run release, I can see multiple processes such as below are running on my Linux server.
/bin/sh /api/node_modules/.bin/cross-env NODE_ENV=development MODE=test node server/app.js
node /api/node_modules/.bin/../cross-env/bin/cross-env.js NODE_ENV=development MODE=test node server/app.js
node server/app.js
I read a related documentation here, but I don't understand what actually happens in background.
What is the order of creating processes? npm => /bin/sh => node /api/.. => node server/app.js ?
What does each process do? All three processes are necessary to run my server?
If I wants to kill the server with pid, which process id should I use?
What is the order of creating processes? npm => /bin/sh => node /api/.. => node server/app.js ?
What does each process do? All three processes are necessary to run my server?
Well, the flow is like this:
NPM is spawned (you run it) inside your shell, npm itself runs with NPX in order to set the local path.
Your npm script spawns a process from a package called cross-env for cross-OS environment variable setting.
That process in turn spawns Node.js (after setting the environment variables)
That's why you see 3 processes. After your server itself run - only the actual server process is needed to run the server.
If I wants to kill the server with pid, which process id should I use?
This one: node server/app.js - since that's your actual server, the others are just "utility processes" (one for the npm script you ran and the other for the environment variables").
It's worth mentioning that in general - servers are run inside containers or other orchestrators/managers that have built-in logic for restarting/killing the process. Typically the orchestrator sends SIGTERM to the process.
Related
I read some Docker and Node.js Best Practices articles, e.g. https://github.com/nodejs/docker-node/blob/main/docs/BestPractices.md or 10 best practices to containerize Node.js web applications with Docker or Dockerfile good practices for Node and NPM. All these article were written or updated at least in 2021, I don't list the articles written before 2021 but there quite some.
They are all against CMD ["npm", "run", "start"]. The main reason is npm will swallow the exit signals such as SIGTERM and SIGINT, so the graceful shutdown code in my node app won't run.
I guess it was the case for the old npm (although I didn't test it), but I have tested node14+npm6 and node16+npm8 and I can verify that npm6/8 do NOT swallow those events and my graceful shutdown code is run. Not sure if that was because npm fixed it.
So the only problem remains is there is 1 more process, npm, to run, i.e. NPM run as PID 1. Some articles said the problem with that is "PID 1 will not respond to SIGINT" but as I have verified that is not the case.
Many articles (e.g. this nodejs doc) suggest just CMD [ "node", "server.js" ] but also in https://github.com/nodejs/docker-node/blob/main/docs/BestPractices.md#handling-kernel-signals said "Node.js process running as PID 1 will not respond to SIGINT (CTRL-C) and similar signals.", i.e. nodejs own documents contradict themselves (but I do see nodejs as PID 1 responds to SIGINT)
So I am confused with the problem with CMD ["npm", "run", "start"] or CMD [ "node", "server.js" ]
For my app there is 1 more consideration, my npm scripts has pre hook to make the app run correctly, I have prestart npm script to make npm start work. So currently I just use CMD ["npm", "run", "start"] but I am confused with the "best practice" of how to start my node app in docker.
--- update ---
I found this closed issue for npm lifecycle: propagate SIGTERM to child
So they did fix it but the latest comment in that issue was in 2017, which said "Yes, this isn’t working, at least with bash; npm runs its lifecycle processes in a shell, and bash doesn’t forward SIGTERM to its children."
I realize I only tested that on my mac and on our CentOS server, and the alpine based docker. It may also because I use exec form, not shell form in CMD so I got the exit signal.
Graceful shutdown with Node.js and Kubernetes said their alpine image didn't get SIGTERM using npm start, while I test on alpine3.15 and I can get.
To work around the swallow of SIGINT and SIGTERM, I do this:
CMD [ "bash", "-c", "npm run db:migrate && node ./dist/server/index.js" ]
where npm run db:migrate && node ./dist/server/index.js was the content of my npm start
I attempted to use the command Heroku pg:psql to connect to my database addon in heroku but got a response below
--> Connecting to postgresql-regular-61345
unrecognized win32 error code: 123could not find a "psql" to execute
unrecognized win32 error code: 123could not find a "psql" to execute
psql: fatal: could not find own program executable
! psql exited with code 1
After using the heroku logs --tail command i got the following errors
sh: 1: nodemon: not found
Process exited with status 127
State changed from starting to crashed
I can also see all processes stopping with SIGTERM and the process exiting with status 143
Resolution steps I have taken
Verified that the environment variables have the path for installed postgress14 on my PC
Added a procfile to the root file in my backend code and spcified "web: node matthewfaceappback/server.js in the file"
Changed my set port to a variable port using process.env.PORT || 3000
Set all environment variable including my database url(set by default) on config variable in heroku
Verified there is a start up script
Updated all my packages using "npm update". after doing this i started expereincing the issue of processes stopping with SIGTERM and the process exiting with status 143
I moved nodemon from devDependencies to dependencies. nodemon version is 2.0.15
In package.json i inputed an engines parameter using the version of node in my case
{"engines": {
"node": "14.17.4"
}}
I restarted heroku using "heroku restart"
Below are links to the screenshots of the error
https://www.dropbox.com/s/5bdbyi9e99lbxhu/pic1.PNG?dl=0
https://www.dropbox.com/s/41euniaes5q68c9/pic2.PNG?dl=0
https://www.dropbox.com/s/50oqzbwmwrqogax/pic3.PNG?dl=0
Put nodemon back in the devDependencies and add it as a second node script in package.json:
"scripts": {
"start": "node matthewfaceappback/server.js",
"dev": "nodemon matthewfaceappback/server.js"
},
These two errors are completely unrelated.
The database connection error
The first issue, which I believe is the one you actually care about at the moment based on the title of the question, indicates that the Heroku CLI can't find a PostgreSQL client on your local machine.
The documentation makes the following recommendation
Set up Postgres on Windows
Install Postgres on Windows by using the Windows installer.
Remember to update your PATH environment variable to add the bin directory of your Postgres installation. The directory is similar to: C:\Program Files\PostgreSQL\<VERSION>\bin. Commands like heroku pg:psql depend on the PATH and do not work if the PATH is incorrect.
If you haven't already installed Postgres locally, do so. (This is a good idea anyway as you should be developing locally and you'll probably need a database.)
Then make sure to add its bin/ directory to your PATH environment variable.
The Nodemon error
The second issue is because you are trying to use nodemon in production. Heroku strips development dependencies out of Node.js applications after building them, which normally makes sense. Nodemon is a development tool, not something that should be used for production hosting.
Depending on the contents of your package.json, this might be as simple as changing your start script from nodemon some-script.js to node some-script.js. Alternatively, you can add a Procfile with the command you actually want to run on Heroku:
web: node some-script.js
See also Need help deploying a RESTful API created with MongoDB Atlas and Express
I have a script that sequentially executes 2 instructions as following:
node server.js
node tests.js
Server.js initializes a local Node.js + Express server
Tests.js executes some unit tests on that server
The problem is that the first instruction keeps listening for requests, so the second is never executed.
Is it possible to solve this behaviour?
I advise you to use the npm package start-server-and-test to be sure that your api server is ready to be tested
start-server-and-test 'node server.js' http://localhost:{API_PORT}/ 'node tests.js'
I have npm task that loads concurrently, node-inspector, and node-supervisor.
{
"start": "concurrently --kill-others \"node-inspector\" \"set NODE_PATH=.&&supervisor -n error -- ./bin/www --debug\"",
"prestart": "start chrome http://localhost:3000 http://localhost:8080/debug?port=5858"
}
Result:
[1] set NODE_PATH=.&&supervisor -n error -- ./bin/www --debug exited with code 0
--> Sending SIGTERM to other processes..
[0] Node Inspector v0.12.7
[0] Visit http://127.0.0.1:8080/?port=5858 to start debugging.
But when I'm trying localhost:3000 I get ERR_CONNECTION_REFUSED.
QUESTION 1: How I can pass NODE_PATH=. (or any other ENV variable) to supervisor - so my node runs correctly?
QUESTION 2: Or maybe you know, how to organize two watch tasks within npm scripts? for example: node-supervisor and node-inspector
Note1: If I run set NODE_PATH=.&&supervisor -n error -- ./bin/www --debug directly from command line - works well.
Note2: If I run npm start without NODE_PATH=.&& it works well but node error - cause it can't find modules for example require('helpers/log') or require('models/user') cause i don't want to place everything in node_modules.
Stats:
Windows 8.1
Node 4.4.0
npm 2.4.0
For your first question, check out the handy helper module cross-env
It's designed for exactly this purpose, and is easy to use. Just use cross-env where you'd typically use UNIX-style env, and away you go. Mine, for example, looks like:
"scripts": {
"start": "cross-env DEBUG=express:router webpack-dev-server ...."
}
A web app I am writing in JavaScript using node.js. I use Foreman, but I don't want to manually restart the server every time I change my code. Can I tell Foreman to reload the entire web app before handling an HTTP request (i.e. restart the node process)?
Here's an adjusted version of Pendlepants solution. Foreman looks for an .env file to read environment variables. Rather than adding a wrapper, you can just have Foreman switch what command it uses to start things up:
In .env:
WEB=node app.js
In dev.env:
WEB=supervisor app.js
In your Procfile:
web: $WEB
By default, Foreman will read from .env (in Production), but in DEV just run this:
foreman start -e dev.env
You can use rerun for this purpose
You might implement just 2 commands for this:
gem install rerun
rerun foreman start
Then rerun will automatically restart process after any change in your files.
If you use nodemon
, you can do
nodemon --exec "foreman start"
The problem isn't with Foreman so much as it's with how node doesn't reload code on new requests. The solution is to use an npm package like supervisor along with an environment wrapper for Foreman.
First, install supervisor:
npm install -g supervisor
Then, write a wrapper shell script that Foreman can call:
if [ "$NODE_ENV" == "production" ]; then
node /path/to/app.js
else
supervisor /path/to/app.js
fi
Set the wrapper script's permissions to executable by running chmod a+x /path/to/wrapper_script.sh
Lastly, update foreman to use the wrapper script. So in your Procfile:
web: /path/to/wrapper_script.sh
Now when you run Foreman and your node app isn't running in production, it should reload on every request.
I feel like Peter Ehrlich's comment on the original question deserves to be an answer on its own. I think a different Procfile for local/dev is definitely the best solution: https://stackoverflow.com/a/10790514/133720
You don't even need to install anything new if you use node-dev.
Your .env file loaded from Procfile:
NODECMD=node-dev
Your Procfile:
web: $NODECMD app/server.js
Your foreman command
foreman start -e dev.env -p 9786
And in your production env (heroku) set an environment variable:
NODECMD=node