If not using pm2, my cluster.fork will assign different env for works, like following
if (cluster.isMaster) {
x = [1, 2, 3, 4];
for (let i = 0; i < 4; i++) {
cluster.fork({x:x[i]}); //set different env for each worker
}} else {
console.log(`Worker ${process.pid} started`);
console.log(process.env.x);
//using its own process.env.x
}
But with pm2 I haven't figure out how to cluster.fork(env) for each worker or does pm2 support this ?
---- update ---
I briefly check pm2 cluster code https://github.com/Unitech/pm2/blob/master/lib/God/ClusterMode.js#L48
clu = cluster.fork({pm2_env: JSON.stringify(env_copy), windowsHide: true});
So I would assume pm2 doesn't do this currently and I opened an issue against it
Related
I have a very simple program developed with ES6 and transpiled with Babel.
import kue from 'kue';
import cluster from 'cluster';
const queue = kue.createQueue();
const clusterWorkerSize = require('os').cpus().length;
if (cluster.isMaster) {
kue.app.listen(3000);
for (var i = 0; i < clusterWorkerSize; i++) {
cluster.fork();
}
} else {
queue.process('email', 10, function(job, done){
...
});
}
The problem comes when I run the program with
$ babel-node --presets es2015 program.js
The master process run without problem but the children crash with:
import kue from 'kue';
SyntaxError: Unexpected reserved word
Any idea of how run the children with Babel?
NOTE: one option is to generate a dist/ folder with all the code transpiled to ES5, but I leave that for the last.
The problem here is that child processes are run under the node, not babel-node.
Try to use babel require hook instead of CLI.
Until recently I started my node server like such
node server/server.js
where server.js was a simple expresss server
var app = express();
app.listen(PORT...
Now I am using cluster.js to start my application
var cluster = require('cluster');
if (cluster.isMaster) {
// Count the machine's CPUs
var cpuCount = require('os').cpus().length;
// Create a worker for each CPU
for (var i = 0; i < cpuCount; i += 1) {
cluster.fork();
}
// Listen for dying workers
cluster.on('exit', function () {
cluster.fork();
});
} else {
require('./server');
}
When i start my server using
node server/cluster.js
It works fine. Cluster.js starts server.js which then starts my express server.. so far so good.
However when I attempt o use forever with cluster.js .. something goes amiss. it does start the cluster.js file..however the below command does not work
forever stop server/cluster.js
I do need the above command to work because I am using flightplan for deployment and it needs to stop and start forever before each deployment.
Any suggestions on how I can make this happen ?
Here is a simplified version of my cluster Express app:
/index.js
module.exports = process.env.CODE_COV
? require('./lib-cov/app')
: require('./lib/app');
/lib/app.js
var cluster = require('cluster'),
express = require('express'),
app = module.exports = express.createServer();
if (cluster.isMaster) {
// Considering I have 4 cores.
for (var i = 0; i < 4; ++i) {
cluster.fork();
}
} else {
// do app configurations, then...
// Don't listen to this port if the app is required from a test script.
if (!module.parent.parent) {
app.listen(8080);
}
}
/test/test1.js
var app = require('../');
app.listen(7777);
// send requests to app, then assert the response.
Questions:
var app = require('../'); will not work in this cluster environment. Which of the worker apps should it return? Should it return the cluster object instead of an Express app?
Now, obviously setting the port in the test script will not work. How would you set a port within a test script to a cluster of apps?
How would you send requests to this cluster of apps?
The only solution I can think of is to conditionally turn off the clustering feature and run only one app if the app is requested from a test script (if (module.parent.parent) ...).
Any other way to test a clustered Express app with Mocha?
It's been quite a long time since I have posted this question. Since no one has answered, I will answer to this question myself.
I kept the /index.js as it is:
module.exports = process.env.CODE_COV
? require('./lib-cov/app')
: require('./lib/app');
In /lib/app.js which starts the cluster, I have the following code. In brief, I start the cluster only in non-test environment. In test environment the cluster is not started but only one app/worker itself is started as defined in the cluster.isMaster && !module.parent.parent condition.
var cluster = require('cluster'),
express = require('express'),
app = module.exports = express.createServer();
if (cluster.isMaster && !module.parent.parent) {
// Considering I have 4 cores.
for (var i = 0; i < 4; ++i) {
cluster.fork();
}
} else {
// do app configurations, then...
// Don't listen to this port if the app is required from a test script.
if (!module.parent.parent) {
app.listen(8080);
}
}
In the above case !module.parent.parent will be evaluated as a truthful object only if the application was not started by a test script.
module is the current /lib/app.js script.
module.parent is its parent /index.js script.
module.parent.parent is undefined if the application was started directly via node index.js.
module.parent.parent is the test script if the application was started via one of the scripts.
Thus, I can safely start the script where I can set a custom port.
/test/test1.js
var app = require('../');
app.listen(7777);
// send requests to app, then assert the response.
At the same time if I need to run the application in real, i.e. not for testing, then I run node index.js and it will start up the cluster of applications.
I have a much simpler way of doing this
if (process.env.NODE_ENV !== 'test') {
if (cluster.isMaster) {
var numCPUs = require('os').cpus().length;
console.log('total cpu cores on this host: ', numCPUs);
for (var i = 0; i < numCPUs; i++) {
console.log('forking worker...');
cluster.fork();
}
cluster.on('online', function(worker) {
console.log('Worker ' + worker.process.pid + ' is online.');
});
cluster.on('exit', function(worker, code, signal) {
console.log('worker ' + worker.process.pid + ' died.');
});
} else {
console.log('Im a worker');
// application code
setupServer()
}
} else {
// when running tests
setupServer();
}
Just make sure to set the env to test when running the tests
ex: NODE_ENV=test grunt test
I kind of liked your solution because of it's simplicity, however, in an environment like an MVC framework for node, you may end up chaining module.parent up to 11 times (seriously).
I think a better approach would be to simply check which script node started processing with. The node's command-line arguments are available at process.argv.
The first item in this array would be 'node', the executable and the second argument would be the path to the file that node start executing. This would be index.js in your case.
So instead of checking
module.parent.parent
^ ^
(app.js) |
(index.js)
You could do something like this
var starter = process.argv[1].split(path.sep).pop();
Where starter would be index or index.js depending on what you started your server with.
node index.js vs node index
The check would then look like:
if (cluster.isMaster && starter === 'index.js') {
cluster.fork();
}
Worked in my environments—I hope this helps!
I'm trying to debug the child Node.JS process created using:
var child = require('child_process');
child .fork(__dirname + '/task.js');
The problem is that when running in IntelliJ/WebStorm both parent and child process start on the same port.
debugger listening on port 40893
debugger listening on port 40893
So it only debugs the parent process.
Is there any way to set IntelliJ to debug the child process or force it to start on a different port so I can connect it in Remote debug?
Yes. You have to spawn your process in a new port. There is a workaround to debug with clusters, in the same way you can do:
Start your app with the --debug command and then:
var child = require('child_process');
var debug = typeof v8debug === 'object';
if (debug) {
//Set an unused port number.
process.execArgv.push('--debug=' + (40894));
}
child.fork(__dirname + '/task.js');
debugger listening on port 40894
It is a known bug in node.js that has been recently fixed (although not backported to v0.10).
See this issue for more details: https://github.com/joyent/node/issues/5318
There is a workaround where you alter the command-line for each worker process, although the API was not meant to be used this way (the workaround might stop working in the future). Here is the source code from the github issue:
var cluster = require('cluster');
var http = require('http');
if (cluster.isMaster) {
var debug = process.execArgv.indexOf('--debug') !== -1;
cluster.setupMaster({
execArgv: process.execArgv.filter(function(s) { return s !== '--debug' })
});
for (var i = 0; i < 2; ++i) {
if (debug) cluster.settings.execArgv.push('--debug=' + (5859 + i));
cluster.fork();
if (debug) cluster.settings.execArgv.pop();
}
}
else {
var server = http.createServer(function(req, res) {
res.end('OK');
});
server.listen(8000);
}
Quick simple fix ( where using chrome://inspect/#devices )
var child = require('child_process');
child.fork(__dirname + '/task.js',[],{execArgv:['--inspect-brk']});
Then run your app without any --inspect-brk and the main process won't debug but the forked process will and no conflicts.
To stop a fork conflicting when debugging the main process ;
child.fork(__dirname + '/task.js',[],{execArgv:['--inspect=xxxx']});
where xxxx is some port not being used for debugging the main process. Though I haven't managed to easily connect to both at the same time in the debugger even though it reports as listening.
I find that setting the 'execArgv' attribute in the fork func will work:
const child = fork('start.js', [], {
cwd: startPath,
silent: true,
execArgv: ['--inspect=10245'] });
if "process.execArgv" doenst work you have to try:
if (debug) {
process.argv.push('--debug=' + (40894));
}
this worked for me..
There are one more modern way to debug child (or any) process with Chrome DevTools.
Start your app with arg
--inspect
like below:
node --debug=9200 --inspect app/main.js
You will see the message with URL for each child process:
Debugger listening on port 9200.
Warning: This is an experimental feature and could change at any time.
To start debugging, open the following URL in Chrome:
chrome-devtools://devtools/bundled/inspector.html?experiments=true&v8only=true&ws=127.0.0.1:9200/207f2ab6-5700-4fc5-b6d3-c49a4b34a311
Debugger listening on port 9201.
Warning: This is an experimental feature and could change at any time.
To start debugging, open the following URL in Chrome:
chrome-devtools://devtools/bundled/inspector.html?experiments=true&v8only=true&ws=127.0.0.1:9201/97be3351-2ea1-4541-b744-e720188bacfa
Debugger listening on port 9202.
Warning: This is an experimental feature and could change at any time.
To start debugging, open the following URL in Chrome:
chrome-devtools://devtools/bundled/inspector.html?experiments=true&v8only=true&ws=127.0.0.1:9202/8eb8384a-7167-40e9-911a-5a8b902bb8c9
If you want to debug the remote processes, just change the address 127.0.0.1 to your own.
I uses the following code to take advantage of cluster npm for my node app.
form = require("connect-form");
express = require("express");
app = express.createServer(form({ keepExtensions: true }));
cluster = require("cluster");
// App configuration
// ....
// Run on cluster
cluster( app )
.use(cluster.logger('logs'))
.use(cluster.stats())
.use(cluster.pidfiles('pids'))
.use(cluster.cli())
.listen(port);
This was working fine on node 0.4.4 but I end up with the following error on node 0.6.5
luc#localhost:~/server$ node app.js
node.js:201
throw e; // process.nextTick error, or 'error' event on first tick
^
TypeError: Property 'cluster' of object #<Object> is not a function
at Object.<anonymous> (/home/luc/server/app.js:15:1)
at Module._compile (module.js:432:26)
at Object..js (module.js:450:10)
at Module.load (module.js:351:31)
at Function._load (module.js:310:12)
at Array.0 (module.js:470:10)
at EventEmitter._tickCallback (node.js:192:40)
I know 'cluster' has been tested on 0.4.x node version but the error seems strange though.
Any idea ?
#mengxy nicely pointed to the documentation, but I was running into this same problem. I thought it would be good to point out a few things:
The example given in the LearnBoost cluster git repo (http://github.com/LearnBoost/cluster/blob/master/examples/express.js) does not work with Node 0.6 as of this writing.
As a recent adopter of Node, I'm at the mercy of many examples; I suspect others are as well. I haven't yet seen a good example of Node 0.6 + cluster + express (probably because Node 0.6 is still fairly recent).
So, derived from the nonworking example and the Node 0.6 docs #mengxy pointed to, here's an example of cluster serving an express app:
#!/usr/bin/env node
var cluster = require('cluster');
var express = require('express');
var os = require('os');
var app = express.createServer();
app.get('/', function(req, res){
res.send('Hello World from worker ' + process.env.NODE_WORKER_ID);
});
if (cluster.isMaster) {
// Fork workers. Two per core sounds reasonable to me.
for (var i = 0; i < os.cpus().length * 2; i++) {
var worker = cluster.fork();
}
} else {
// Worker processes have a http server.
app.listen(3000);
}
Cluster npm module has been integrated into node core in node 0.6, and there are some API changes.
You can find the API documents at http://nodejs.org/docs/v0.6.5/api/cluster.html#cluster
From Node.js on multi-core machines:
Node.JS v0.6.X includes the "cluster" module straight out of the box, which makes it easy to set up multiple node workers that can listen on a single port.
This is NOT the same as the learnboost "cluster" module.
http://nodejs.org/docs/latest/api/cluster.html
if (cluster.isMaster) {
// Fork workers.
for (var i = 0; i < numCPUs; i++) {
cluster.fork();
}
} else {
http.Server(function(req, res) { ... }).listen(8000);
}