Until recently I started my node server like such
node server/server.js
where server.js was a simple expresss server
var app = express();
app.listen(PORT...
Now I am using cluster.js to start my application
var cluster = require('cluster');
if (cluster.isMaster) {
// Count the machine's CPUs
var cpuCount = require('os').cpus().length;
// Create a worker for each CPU
for (var i = 0; i < cpuCount; i += 1) {
cluster.fork();
}
// Listen for dying workers
cluster.on('exit', function () {
cluster.fork();
});
} else {
require('./server');
}
When i start my server using
node server/cluster.js
It works fine. Cluster.js starts server.js which then starts my express server.. so far so good.
However when I attempt o use forever with cluster.js .. something goes amiss. it does start the cluster.js file..however the below command does not work
forever stop server/cluster.js
I do need the above command to work because I am using flightplan for deployment and it needs to stop and start forever before each deployment.
Any suggestions on how I can make this happen ?
Related
If not using pm2, my cluster.fork will assign different env for works, like following
if (cluster.isMaster) {
x = [1, 2, 3, 4];
for (let i = 0; i < 4; i++) {
cluster.fork({x:x[i]}); //set different env for each worker
}} else {
console.log(`Worker ${process.pid} started`);
console.log(process.env.x);
//using its own process.env.x
}
But with pm2 I haven't figure out how to cluster.fork(env) for each worker or does pm2 support this ?
---- update ---
I briefly check pm2 cluster code https://github.com/Unitech/pm2/blob/master/lib/God/ClusterMode.js#L48
clu = cluster.fork({pm2_env: JSON.stringify(env_copy), windowsHide: true});
So I would assume pm2 doesn't do this currently and I opened an issue against it
Here is a simplified version of my cluster Express app:
/index.js
module.exports = process.env.CODE_COV
? require('./lib-cov/app')
: require('./lib/app');
/lib/app.js
var cluster = require('cluster'),
express = require('express'),
app = module.exports = express.createServer();
if (cluster.isMaster) {
// Considering I have 4 cores.
for (var i = 0; i < 4; ++i) {
cluster.fork();
}
} else {
// do app configurations, then...
// Don't listen to this port if the app is required from a test script.
if (!module.parent.parent) {
app.listen(8080);
}
}
/test/test1.js
var app = require('../');
app.listen(7777);
// send requests to app, then assert the response.
Questions:
var app = require('../'); will not work in this cluster environment. Which of the worker apps should it return? Should it return the cluster object instead of an Express app?
Now, obviously setting the port in the test script will not work. How would you set a port within a test script to a cluster of apps?
How would you send requests to this cluster of apps?
The only solution I can think of is to conditionally turn off the clustering feature and run only one app if the app is requested from a test script (if (module.parent.parent) ...).
Any other way to test a clustered Express app with Mocha?
It's been quite a long time since I have posted this question. Since no one has answered, I will answer to this question myself.
I kept the /index.js as it is:
module.exports = process.env.CODE_COV
? require('./lib-cov/app')
: require('./lib/app');
In /lib/app.js which starts the cluster, I have the following code. In brief, I start the cluster only in non-test environment. In test environment the cluster is not started but only one app/worker itself is started as defined in the cluster.isMaster && !module.parent.parent condition.
var cluster = require('cluster'),
express = require('express'),
app = module.exports = express.createServer();
if (cluster.isMaster && !module.parent.parent) {
// Considering I have 4 cores.
for (var i = 0; i < 4; ++i) {
cluster.fork();
}
} else {
// do app configurations, then...
// Don't listen to this port if the app is required from a test script.
if (!module.parent.parent) {
app.listen(8080);
}
}
In the above case !module.parent.parent will be evaluated as a truthful object only if the application was not started by a test script.
module is the current /lib/app.js script.
module.parent is its parent /index.js script.
module.parent.parent is undefined if the application was started directly via node index.js.
module.parent.parent is the test script if the application was started via one of the scripts.
Thus, I can safely start the script where I can set a custom port.
/test/test1.js
var app = require('../');
app.listen(7777);
// send requests to app, then assert the response.
At the same time if I need to run the application in real, i.e. not for testing, then I run node index.js and it will start up the cluster of applications.
I have a much simpler way of doing this
if (process.env.NODE_ENV !== 'test') {
if (cluster.isMaster) {
var numCPUs = require('os').cpus().length;
console.log('total cpu cores on this host: ', numCPUs);
for (var i = 0; i < numCPUs; i++) {
console.log('forking worker...');
cluster.fork();
}
cluster.on('online', function(worker) {
console.log('Worker ' + worker.process.pid + ' is online.');
});
cluster.on('exit', function(worker, code, signal) {
console.log('worker ' + worker.process.pid + ' died.');
});
} else {
console.log('Im a worker');
// application code
setupServer()
}
} else {
// when running tests
setupServer();
}
Just make sure to set the env to test when running the tests
ex: NODE_ENV=test grunt test
I kind of liked your solution because of it's simplicity, however, in an environment like an MVC framework for node, you may end up chaining module.parent up to 11 times (seriously).
I think a better approach would be to simply check which script node started processing with. The node's command-line arguments are available at process.argv.
The first item in this array would be 'node', the executable and the second argument would be the path to the file that node start executing. This would be index.js in your case.
So instead of checking
module.parent.parent
^ ^
(app.js) |
(index.js)
You could do something like this
var starter = process.argv[1].split(path.sep).pop();
Where starter would be index or index.js depending on what you started your server with.
node index.js vs node index
The check would then look like:
if (cluster.isMaster && starter === 'index.js') {
cluster.fork();
}
Worked in my environments—I hope this helps!
I'm trying to debug the child Node.JS process created using:
var child = require('child_process');
child .fork(__dirname + '/task.js');
The problem is that when running in IntelliJ/WebStorm both parent and child process start on the same port.
debugger listening on port 40893
debugger listening on port 40893
So it only debugs the parent process.
Is there any way to set IntelliJ to debug the child process or force it to start on a different port so I can connect it in Remote debug?
Yes. You have to spawn your process in a new port. There is a workaround to debug with clusters, in the same way you can do:
Start your app with the --debug command and then:
var child = require('child_process');
var debug = typeof v8debug === 'object';
if (debug) {
//Set an unused port number.
process.execArgv.push('--debug=' + (40894));
}
child.fork(__dirname + '/task.js');
debugger listening on port 40894
It is a known bug in node.js that has been recently fixed (although not backported to v0.10).
See this issue for more details: https://github.com/joyent/node/issues/5318
There is a workaround where you alter the command-line for each worker process, although the API was not meant to be used this way (the workaround might stop working in the future). Here is the source code from the github issue:
var cluster = require('cluster');
var http = require('http');
if (cluster.isMaster) {
var debug = process.execArgv.indexOf('--debug') !== -1;
cluster.setupMaster({
execArgv: process.execArgv.filter(function(s) { return s !== '--debug' })
});
for (var i = 0; i < 2; ++i) {
if (debug) cluster.settings.execArgv.push('--debug=' + (5859 + i));
cluster.fork();
if (debug) cluster.settings.execArgv.pop();
}
}
else {
var server = http.createServer(function(req, res) {
res.end('OK');
});
server.listen(8000);
}
Quick simple fix ( where using chrome://inspect/#devices )
var child = require('child_process');
child.fork(__dirname + '/task.js',[],{execArgv:['--inspect-brk']});
Then run your app without any --inspect-brk and the main process won't debug but the forked process will and no conflicts.
To stop a fork conflicting when debugging the main process ;
child.fork(__dirname + '/task.js',[],{execArgv:['--inspect=xxxx']});
where xxxx is some port not being used for debugging the main process. Though I haven't managed to easily connect to both at the same time in the debugger even though it reports as listening.
I find that setting the 'execArgv' attribute in the fork func will work:
const child = fork('start.js', [], {
cwd: startPath,
silent: true,
execArgv: ['--inspect=10245'] });
if "process.execArgv" doenst work you have to try:
if (debug) {
process.argv.push('--debug=' + (40894));
}
this worked for me..
There are one more modern way to debug child (or any) process with Chrome DevTools.
Start your app with arg
--inspect
like below:
node --debug=9200 --inspect app/main.js
You will see the message with URL for each child process:
Debugger listening on port 9200.
Warning: This is an experimental feature and could change at any time.
To start debugging, open the following URL in Chrome:
chrome-devtools://devtools/bundled/inspector.html?experiments=true&v8only=true&ws=127.0.0.1:9200/207f2ab6-5700-4fc5-b6d3-c49a4b34a311
Debugger listening on port 9201.
Warning: This is an experimental feature and could change at any time.
To start debugging, open the following URL in Chrome:
chrome-devtools://devtools/bundled/inspector.html?experiments=true&v8only=true&ws=127.0.0.1:9201/97be3351-2ea1-4541-b744-e720188bacfa
Debugger listening on port 9202.
Warning: This is an experimental feature and could change at any time.
To start debugging, open the following URL in Chrome:
chrome-devtools://devtools/bundled/inspector.html?experiments=true&v8only=true&ws=127.0.0.1:9202/8eb8384a-7167-40e9-911a-5a8b902bb8c9
If you want to debug the remote processes, just change the address 127.0.0.1 to your own.
I created a simple Webapp using express.js and want to test it with jasmine-node. Works fine so far but my problem is that I have to start the server manually every time before I can run my tests.
Could you help me on how to write a spec-helper that runs the server (with another port then my development one) just for the tests and then kills it afterwards?
This is what I do:
I have a server.js file inside the root of my node project that sets up the node application server (with express) and exports 2 methods:
exports.start = function( config, readyCallback ) {
if(!this.server) {
this.server = app.listen( config.port, function() {
console.log('Server running on port %d in %s mode', config.port, app.settings.env);
// callback to call when the server is ready
if(readyCallback) {
readyCallback();
}
});
}
};
exports.close = function() {
this.server.close();
};
The app.js file will be simple at this point:
var server = require('./server');
server.start( { port: 8000 } );
So the files/folder basic structure would be the following:
src
app.js
server.js
Having this separation will allow you to run the server normally:
node src/app.js
..and/or require it from a custom node script, which could be a node script (or a jake/grunt/whatever task) that executes your tests like this:
/** my-test-task.js */
// util that spawns a child process
var spawn = require('child_process').spawn;
// reference to our node application server
var server = require('./path/to/server.js');
// starts the server
server.start( { port: 8000 }, function() {
// on server ready launch the jasmine-node process with your test file
var jasmineNode = spawn('jasmine-node', [ '.path/to/test/file.js' ]);
// logs process stdout/stderr to the console
function logToConsole(data) {
console.log(String(data));
}
jasmineNode.stdout.on('data', logToConsole);
jasmineNode.stderr.on('data', logToConsole);
jasmineNode.on('exit', function(exitCode) {
// when jasmine-node is done, shuts down the application server
server.close();
}
});
I use Mocha - which is damn similar - but the same principle should apply: you could try requireing your app.js file in a 'beforeEach' hook inside the main describe. That should fire it up for you.
Assuming you use some code that invokes app.listen() in server.js, don't require the file on each run but only once and then have two functions like
startServer = -> app.listen(3000)
stopServer = -> app.close()
Then you can use these in beforeEach and afterEach
If you want then to go one step further in automating your testing while you develop, you can go to your terminal line and execute
jasmine-node . --autotest
Jasmine then will stay listening to every file inside your project and whenever you make changes to one it will tell if that piece of your code breaks any of your tests ;)
I uses the following code to take advantage of cluster npm for my node app.
form = require("connect-form");
express = require("express");
app = express.createServer(form({ keepExtensions: true }));
cluster = require("cluster");
// App configuration
// ....
// Run on cluster
cluster( app )
.use(cluster.logger('logs'))
.use(cluster.stats())
.use(cluster.pidfiles('pids'))
.use(cluster.cli())
.listen(port);
This was working fine on node 0.4.4 but I end up with the following error on node 0.6.5
luc#localhost:~/server$ node app.js
node.js:201
throw e; // process.nextTick error, or 'error' event on first tick
^
TypeError: Property 'cluster' of object #<Object> is not a function
at Object.<anonymous> (/home/luc/server/app.js:15:1)
at Module._compile (module.js:432:26)
at Object..js (module.js:450:10)
at Module.load (module.js:351:31)
at Function._load (module.js:310:12)
at Array.0 (module.js:470:10)
at EventEmitter._tickCallback (node.js:192:40)
I know 'cluster' has been tested on 0.4.x node version but the error seems strange though.
Any idea ?
#mengxy nicely pointed to the documentation, but I was running into this same problem. I thought it would be good to point out a few things:
The example given in the LearnBoost cluster git repo (http://github.com/LearnBoost/cluster/blob/master/examples/express.js) does not work with Node 0.6 as of this writing.
As a recent adopter of Node, I'm at the mercy of many examples; I suspect others are as well. I haven't yet seen a good example of Node 0.6 + cluster + express (probably because Node 0.6 is still fairly recent).
So, derived from the nonworking example and the Node 0.6 docs #mengxy pointed to, here's an example of cluster serving an express app:
#!/usr/bin/env node
var cluster = require('cluster');
var express = require('express');
var os = require('os');
var app = express.createServer();
app.get('/', function(req, res){
res.send('Hello World from worker ' + process.env.NODE_WORKER_ID);
});
if (cluster.isMaster) {
// Fork workers. Two per core sounds reasonable to me.
for (var i = 0; i < os.cpus().length * 2; i++) {
var worker = cluster.fork();
}
} else {
// Worker processes have a http server.
app.listen(3000);
}
Cluster npm module has been integrated into node core in node 0.6, and there are some API changes.
You can find the API documents at http://nodejs.org/docs/v0.6.5/api/cluster.html#cluster
From Node.js on multi-core machines:
Node.JS v0.6.X includes the "cluster" module straight out of the box, which makes it easy to set up multiple node workers that can listen on a single port.
This is NOT the same as the learnboost "cluster" module.
http://nodejs.org/docs/latest/api/cluster.html
if (cluster.isMaster) {
// Fork workers.
for (var i = 0; i < numCPUs; i++) {
cluster.fork();
}
} else {
http.Server(function(req, res) { ... }).listen(8000);
}