Let's say I'd want to control (start/stop) several other NodeJS scripts from within one "main" NodeJS app. However, not necessarly NodeJS Scripts exclusively, but also simple bash scripts.
I'm looking into the following solution, using execa
// simplified
const managedProcesses = [];
async function Start (pkg) {
const runningProcess = await execa(pkg.run_command, {
cwd : pkg.path
});
return runningProcess;
}
async function Stop (pkg) {
// somehow stop the child process
return
}
const someProcess = await Start({
run_command : 'node app.js',
path : './path/to/my/script/'
});
// Keep Reference of process
managedProcesses.push(someProcess);
I first thought pm2 would be a viable solution, but I guess this would only fit for NodeJS-only scripts.
What Problems could I run into using the approach above ?
Should I consider moving forward with this idea ?
For node.js subprocesses there is the cluster module and I strongly recommend using this. For general subprocesses (e.g. bash scripts as you mentioned) you have to use child_process (-> execa). Communication between processes may then be accomplished via grpc. Your approach is fine, so you can consider moving forward with it.
I decided to go full with pm2 for the time being, as they have an excellent programmatic API - also (which I only just learned about) you can specify different interpreters to run your script. So not only node apps are possible but also bash, python, php and so on - which is exactly what I am looking for.
Related
I am writing some tests for a Node/MongoDB project that runs various modules via command line entries. My question is, for the tests, is there a way I can simulate a command line entry? For instance, if what I write in the command line is:
TASK=run-comparison node server
... is there a way I can effectively simulate that within my tests?
The common practice here as far as I know, is to wrap as much of your app as you can within a function/class where you pass the arguments, so you can easily test it with unit tests:
function myApp(args, env){
// My app code with given args and env variables
}
In your test file:
// Run app with given env variable
myApp("", { TASK: "run-comparison"});
In your particular case, if all your tasks are set through env variables, through editing of process.env, mocks, or .env files you may be able to test that without modifications on your code.
If that is not enough for your case (i.e. you really need to exactly simulate command line execution) I wrote a small library to solve this exact issue some time ago: https://github.com/angrykoala/yerbamate (I'm not sure if there are other alternatives available now).
With the example you provided, A test case could be something like this:
const yerbamate = require('yerbamate');
// Gets the package.json information
const pkg = yerbamate.loadPackage(module);
//Test the given command in the package.json root dir
yerbamate.run("TASK=run-comparison node server", pkg.dir, {}, function(code, out, errs) {
// This callback will be called once the script finished
if (!yerbamate.successCode(code)) console.log("Process exited with error code!");
if (errs.length > 0) console.log("Errors in process:" + errs.length);
console.log("Output: " + out[0]); // Stdoutput
});
In the end, this is a fairly simple wrapper of native child_process which you could also use to solve your problem by directly executing subprocesses.
I want to decrypt several config items based on environment variables before anything else starts running in a Node.js app.
I'm starting my app using the standard node ./app.js. Then I call a simple method from the top of my app.js file:
function setConfig() {
var pass = process.env.pass;
var conf = Encrypt.decrypt(encryptedConfig, pass);
var configObj = JSON.parse(conf);
// do stuff with the configObj
}
This works fine, but since everything is async other processes, which need the config variables, are already running and throwing errors.
What I want is to run my setConfig() before anything else. Is this doable?
Apart from accepted answer, what might be useful in some situations (where you can't/don't want to modify the executed file) is NODE_OPTIONS environmental variable + --require (-r) param of node executable
NODE_OPTIONS='--require "./first.js"' node second.js
That way, first.js executes before second.js.
Docs:
https://nodejs.org/api/cli.html#cli_node_options_options
https://nodejs.org/api/cli.html#cli_r_require_module
If a routine is synchronous, it can be executed before routines that depend on it. Executing it before anything else at the top of main module guarantees that there will be no race conditions:
setConfig();
require('module-that-depends-on-config');
If a routine is asynchronous, it should be treated as such in order to avoid race conditions. It's preferable for all asynchronous routines to return promises, so they could be chained with async function in main module:
(async () => {
await setConfigAsync();
require('module-that-depends-on-config');
...
})().catch(console.error);
How might I set up node.js as a shell replacement for bash? For example I should be able to run vi('file') to open a file and cd('location') to change between directories.
Is this even possible?
Sure you can! It will become much less straightforward to use your computer, though.
First off, you will need to know how to set this up. While you could likely set your user shell in Linux to usr/bin/node, this will leave you with only a Node.js REPL with no additional programs set up. What you're going to want to do is write a setup script that can do all of the below setup/convenience steps for you, essentially something that ends with repl.start() to produce a REPL after setting everything up. Of course, since UNIX shell settings can't specify arguments, you will need to write a small C program that executes your shell with those arguments (essentially, exec("/usr/bin/node", "path/to/setup/script.js");) and set that as your UNIX shell.
The main idea here is that any commands that you use beyond the very basics must be require()d into your shell - e.g. to do anything with your filesystem, execute
var fs = require("fs")
and do all of your filesystem calls from the fs object. This is analogous to adding things to your PATH. You can get basic shell commands by using shelljs or similar, and to get at actual executable programs, use Node's built-in child_process.spawnSync for a foreground task or child_process.spawn for a background task.
Since part of your requirement is that you want to call each of your programs like a function, you will need to produce these functions yourself, getting something like:
function ls(path) {
child_process.spawnSync('/bin/ls', [path], { stdio: 'inherit' });
}
for everything you want to run. You can probably do this programmatically by iterating through all the entries in your PATH and using something involving eval() or new Function() to generate execute functions for each, assigning them to the global object so that you don't have to enter any prefixes.
Again, it will become much less straightforward to use your computer, despite having these named functions. Lots of programs that cheat and use bash commands in the background will likely no longer work. But I can certainly see the appeal of being able to leverage JavaScript in your command-line environment.
ADDENDUM: For writing this setup script, the REPLServer object returned by repl.start() has a context object that is the same as the global object accessible to the REPL session it creates. When you write your setup script, you will need to assign everything to the context object:
const context = repl.start('> ').context;
context.ls = function ls(path) { /* . . . */ }
context.cd = function cd(path) { /* . . . */ }
I think it would be an intersting proposition. Create a test account and tell it to use node as it's shell. See 'man useradd' for all options
$ useradd -s /usr/bin/node test
$ su - test
This works on mac and linux.
require('child_process').spawnSync('vi', ['file.txt'], { stdio: 'inherit' })
You could bootstrap a repl session with your own commands, then run the script
#!/bin/bash
node --experimental-repl-await -i -e "$(< scripts/noderc.js)"`
This allows for things like:
> ls()
> run('vi','file.txt')
> await myAsyncFunc()
I think you're looking for something like this https://youtu.be/rcwcigtOwQ0 !
If so.... YES you can!
If you like I can share my code. But I need to fix some bugs first!
tell me if you like.
my .sh function:
const hey = Object.create(null),
sh = Object.create(null);;
hey.shell = Object.create(null);
hey.shell.run = require('child_process').exec;
sh.help = 'Execute an OS command';
sh.action = (...args) => {
// repl_ is the replServer
// the runningExternalProgram property is one way to know if I should
// render the prompt and is not needed. I will create a better
// way to do this (action without if/decision)!
repl_.runningExternalProgram = true;
hey.shell.run(args.join(' '),
(...args) => {
['error', 'log'].forEach((command, idx) => {
if (args[idx]) {
console[command](args[idx]);
}
});
repl_.runningExternalProgram = false;
});
};
PS: to 'cd' into some directory you just need to change the process.cwd (current working directory)
PS2: to avoid need to write .sh for every OS program/command you can use Proxy on the global object.
I'm about to start coding a chat bot. However, I plan on running more than one, using a wrapper to communicate and restart them. I have done this in the past with child_process.fork(), but it was incredibly inefficient. I've looked into spawn and cluster as well, but they all seem to focus on running the same thing, not unique bots. As for plugins, I've looked into fleet, forkfriend, and workerfarm, but none seem to fit my needs.
Is there any plugin or way I'm not seeing to help me do this? Or am I just going to have o wing it again?
You can have as many chat bots as you wish in a single process. The rule of thumb in Node.js is using one process per processor core since Node has slightly different multithreading model you might got used to.
Assuming you still need some multithreading on top of this, here is a couple of node modules you might find fitting your needs:
node-webworker-threads, dnode.
UPDATE:
Now I see what you need. There is a nice example in Node.js docs, which I saw recently. I just copy & paste it here:
var normal = require('child_process').fork('child.js', ['normal']);
var special = require('child_process').fork('child.js', ['special']);
// Open up the server and send sockets to child
var server = require('net').createServer();
server.on('connection', function (socket) {
// if this is a VIP
if (socket.remoteAddress === '74.125.127.100') {
special.send('socket', socket);
return;
}
// just the usual dudes
normal.send('socket', socket);
});
server.listen(1337);
child.js looks like this:
process.on('message', function(m, socket) {
if (m === 'socket') {
socket.end('You were handled as a ' + process.argv[2] + ' person');
}
});
I believe it's pretty much what you need. Launch several processes with different configs (if number of configs is relatively low) and pass socket to a particular one from master process.
One of the pleasures of frameworks like Rails is being able to interact with models on the command line. Being very new to node.js, I often find myself pasting chunks of app code into the REPL to play with objects. It's dirty.
Is there a magic bullet that more experienced node developers use to get access to their app specific stuff from within the node prompt? Would a solution be to package up the whole app, or parts of the app, into modules to be require()d? I'm still living in one-big-ol'-file land, so pulling everything out is, while inevitable, a little daunting.
Thanks in advance for any helpful hints you can offer!
One-big-ol'-file land is actually a good place to be in for what you want to do. Nodejs can also require it's REPL in the code itself, which will save you copy and pasting.
Here is a simple example from one of my projects. Near the top of your file do something similar to this:
function _cb() {
console.log(arguments)
}
var repl = require("repl");
var context = repl.start("$ ").context;
context.cb = _cb;
Now just add to the context throughout your code. The _cb is a dummy callback to play with function calls that require one (and see what they'll return).
Seems like the REPL API has changed quite a bit, this code works for me:
var replServer = repl.start({
prompt: "node > ",
input: process.stdin,
output: process.stdout,
useGlobal: true
});
replServer.on('exit', function() {
console.log("REPL DONE");
});
You can also take a look at this answer https://stackoverflow.com/a/27536499/1936097. This code will automatically load a REPL if the file is run directly from node AND add all your declared methods and variables to the context automatically.