I'm trying to generate specific values in a Node.js script and then pass them as environment variables to a shell command, but I can't seem to get it working. What's the right way to execute a string as a shell command from Node.js while passing in my own environment variables?
The following doesn't seem to be working as I would expect:
const shell = require("shelljs");
const { findPort } = require("./find-port");
async function main() {
// imagine that `findPort(value)` starts at the provided `value` and
// increments it until it finds an available port
const PORT = await findPort(8000); // 8000, 8001, etc
const DB_PORT = await findPort(3306); // 3306, 3307, etc
shell.exec(`yarn run dev`, {
env: {
PORT,
DB_PORT,
},
async: true,
});
}
main();
When I try to run this, I get the following error:
env: node: No such file or directory
Important: I don't want any values to leak out of this specific script, which is why I'm trying to avoid the export FOO=bar syntax, but I may be misunderstanding how that works.
I'd prefer a solution that uses shelljs, but I'm open to other solutions that use child_process.exec, execa, etc.
The script does exactly what you ask for: it runs yarn run dev with your environment variables. Unfortunately, that means that it does not run with the system's environment variables that yarn depends on, like PATH.
You can instead run it with both your variables and the system's variables:
shell.exec(`yarn run dev`, {
env: {
...process.env,
PORT,
DB_PORT,
}
});
Related
Imagine this contrived scenario:
./main.sh
source ./config.sh
SOME_CONFIG="${SOME_CONFIG}bar"
./output.sh
./config.sh
export SOME_CONFIG='foo'
./output.sh
echo "Config is: ${SOME_CONFIG}"
I am trying to replace ./main.sh with a Node.js powered ./main.js WITHOUT replacing the other shell files. The exported ./config.sh functions/variables must also be fully available to ./output.sh
Here is a NON working ./main.js. I have written this for the sole purpose to explain what I want the final code to look like:
const terminal = require('child_process').spawn('bash')
terminal.stdin.write('source ./config.sh\n')
process.env.SOME_CONFIG = `${process.env.SOME_CONFIG}bar` // this must be done in JS
terminal.stdin.write('./output.sh\n') // this must be able to access all exported functions/variables in config.sh, including the JS modified SOME_CONFIG
How can I achieve this? Ideally if there's a library that can do this I'd prefer that.
While this doesn't fully answer my question, it solves the contrived problem I had at hand and could help others if need be.
In general, if bash scripts communicate with each other via environment variables (eg. using export/source), this will allow you to start moving bash code to Node.js.
./main.js
const child_process = require("child_process");
const os = require("os");
// Source config.sh and print the environment variables including SOME_CONFIG
const sourcedConfig = child_process
.execSync(". ./config.sh > /dev/null 2>&1 && env")
.toString();
// Convert ALL sourced environment variables into an object
const sourcedEnvVars = sourcedConfig
.split(os.EOL)
.map((line) => ({
env: `${line.substr(0, line.indexOf("="))}`,
val: `${line.substr(line.indexOf("=") + 1)}`,
}))
.reduce((envVarObject, envVarEntry) => {
envVarObject[envVarEntry.env] = envVarEntry.val;
return envVarObject;
}, {});
// Make changes
sourcedEnvVars["SOME_CONFIG"] = `${sourcedEnvVars["SOME_CONFIG"]}bar`;
// Run output.sh and pass in the environment variables we got from the previous command
child_process.execSync("./output.sh", {
env: sourcedEnvVars,
stdio: "inherit",
});
I am trying to get command line argument values using NPM package "yargs" instead of retrieving the values from config.ts file.
Example:
My config.ts file looks like
export const config: Config = {
username: process.argv[3],
password: process.argv[4]
}
And in my package.json file I already defined my scripts for "test": "./node_modules/.bin/protractor typeScript/config/config.js",
In my testcase i am retrieving values as "config.username" & "config.password"
So when i execute npm test --username1 --password1 , my testcase which is executing is picking values from config.ts file and not the values which i passed "username1" and "password1" from command line.
Can someone please help me in picking the values from command line and not from the config.ts file.
Thanks in advance.. !!!
1.
For protractor side check out this page
Per its content, having this in your conf.js:
module.exports = {
params: {
login: {
email: 'default',
password: 'default'
}
},
// * other config options *
}
you can pass any parameter to it in CMD as follows:
protractor --baseUrl='http://some.server.com' conf.js --parameters.login.email=example#gmail.com
--parameters.login.password=foobar
so you end up having this in your specs:
describe('describe some test', function() {
it('describe some step', function() {
browser.get(browser.baseUrl);
$('.email').sendKeys(browser.params.login.email);
$('.password').sendKeys(browser.params.login.password);
});
});
BUT! These variables are not available in your config and only in specs.
2.
If you need variables to be reused in config (conditional suites etc) The way I'm doing this is following
Lets assume you have a command to start your tests
protractor config.js --baseUrl="www.example.com"
you can declare environment variables BEFORE that command like so
INFO=something PARAM=parameter protractor config.js --baseUrl="www.example.com"
And then in your protractor config you can refer to those variables by typing
console.log(process.env.INFO);
console.log(process.env.PARAM);
Note that you don't even need additional packages for this approach
I want to update my env variable in node js, but i am not able to update its env variable, i tried with console.log(process.env.DB_HOST) but still i am getting old value, here i have added my whole code, can anyone please look in to it, and help me to resolve this issue,
function exec_api() {
return new Promise(async function (resolve) {
const execSync = require('child_process').exec;
//let child_process_obj = execSync('DB_HOST='+process.env.UNITTEST_DB_HOST+' DB_DATABASE='+process.env.UNITTEST_DB_DATABASE+' DB_USERNAME='+process.env.UNITTEST_DB_USERNAME+' DB_PASSWORD='+process.env.UNITTEST_DB_PASSWORD+' PORT='+process.env.UNITTEST_SERVICE_PORT+' ./node_modules/.bin/nodemon main.js');
await execSync('export DB_HOST=' + process.env.UNITTEST_DB_HOST);
await execSync('export DB_DATABASE=' + process.env.UNITTEST_DB_DATABASE);
await execSync('export DB_USERNAME=' + process.env.UNITTEST_DB_USERNAME);
await execSync('export DB_PASSWORD=' + process.env.UNITTEST_DB_PASSWORD);
await execSync('export PORT=' + process.env.UNITTEST_API_BACKEND_PORT);
let child_process_obj = await execSync('node main.js');
unittest_api_backend_process_id = child_process_obj.pid;
resolve(true);
});
}
TLDR: Just change process.env
To change, add or delete environment variables, use process.env. The following is test code showing how this works:
In main.js:
console.log(process.env.DB_DATABASE);
In exec.js:
const execSync = require ('child_process').execSync;
process.env.DB_DATABASE = 'foo'; // this is ALL you need to do
console.log(execSync('node main.js').toString('utf8'));
With the two files above, if you run node exec.js you will see foo printed out in the console. This is printed from main.js which inherits the environment from exec.js.
So all you need to do in your code is:
I want to update my env variable in node js, but i am not able to update its env variable, i tried with console.log(process.env.DB_HOST) but still i am getting old value, here i have added my whole code, can anyone please look in to it, and help me to resolve this issue,
function exec_api() {
return new Promise(function (resolve) {
const exec = require('child_process').exec;
// The following is node.js equivalent of bash "export":
process.env.DB_HOST = process.env.UNITTEST_DB_HOST;
process.env.DB_DATABASE = process.env.UNITTEST_DB_DATABASE;
process.env.DB_USERNAME = process.env.UNITTEST_DB_USERNAME;
process.env.DB_PASSWORD = process.env.UNITTEST_DB_PASSWORD;
process.env.PORT = process.env.UNITTEST_SERVICE_PORT;
let child_process_obj = exec('node main.js', {
stdio: ['inherit', 'inherit', 'inherit']
});
unittest_api_backend_process_id = child_process_obj.pid;
resolve(true);
});
}
Note that if you want the promise to return when the main.js ends you need to do:
function exec_api() {
return new Promise(function (resolve) {
const exec = require('child_process').exec;
// The following is node.js equivalent of bash "export":
process.env.DB_HOST = process.env.UNITTEST_DB_HOST;
process.env.DB_DATABASE = process.env.UNITTEST_DB_DATABASE;
process.env.DB_USERNAME = process.env.UNITTEST_DB_USERNAME;
process.env.DB_PASSWORD = process.env.UNITTEST_DB_PASSWORD;
process.env.PORT = process.env.UNITTEST_SERVICE_PORT;
let child_process_obj = exec('node main.js', {
stdio: ['inherit', 'inherit', 'inherit']
});
unittest_api_backend_process_id = child_process_obj.pid;
child_process_obj.on('exit', () => resolve(true));
// ^^^ Cannot use `await` as the API is not promise based
// but event based instead.
});
}
Long story: The full explanation of why export doesn't work
On unixen, environment variables, and indeed, the entire environment including current working directory, root directory (which can be changed via chroot) etc. are not features of shells. They are features of processes.
We may be familiar with the export syntax of some shells to set environment variables for child processes but that is the shell's syntax. It has nothing to do with environment variables themselves. C/C++ for example don't use export instead uses the setenv() function do set environment variables (indeed, internally that's what bash/sh/ksh etc do when implementing export).
In node.js, the mechanism for reading and setting environment variables is via process.env.
Why asking a shell to do it don't work
This is not merely a node.js issue. It also won't work in bash:
In exporter.sh:
#! /bin/bash
export DB_DATABASE=$1
In exec.sh:
#! /bin/bash
./exporter.sh foo
echo $DB_DATABASE ;# does not print "foo"
This is a core security feature of unixen: other users should not be allowed to mess with your process. The way this policy is enforced in the case of the environment is that only a parent process can set the environment of the child process. A child process is not allowed to set the environment of the parent process. The assumption is that the child process belongs to the parent process so you should be allowed to do what you want to a program - but since the parent process (you) don't belong to the child process the child is not allowed to mess with the parent's environment.
That's why your attempt to use export doesn't work. It actually works (the variables are indeed created in the subshell) but is not allowed to change the environment of it's parent (the node.js process)
When you use export in a terminal, it instructs the shell to set environment variables.
When you call exec from your code, you are not running such a shell, with the reason being that it would become a challenge to extract the output of every command.
This makes export an ignored command.
You can solve this by passing an option object to execSync:
execSync('node main.js', {
env: {
DB_HOST: 'localhost',
// More envs...
}
}
I would like to set the NODE_ENV variable at the beginning of a Grunt task, to development or production, but it looks it's not as simple as I thought.
The reason, why I would like this is that I use grunt-webpack, which expects NODE_ENV to be set correctly to "development" or "production". But I also would like to initialize my tasks exclusively from grunt, if possible.
I created the following test Gruntfile, using the grunt-shell and cross-env modules:
function log(err, stdout, stderr, cb, e) {
if (err) {
cb(err);
return;
}
console.log(process.env.NODE_ENV);
console.log(stdout);
cb();
}
module.exports = function(grunt) {
grunt.initConfig({
shell: {
dev: {
command : 'cross-env NODE_ENV="development"',
options: {
callback: log
}
},
dist: {
command : 'cross-env NODE_ENV="production"',
options: {
callback: log
}
}
}
});
grunt.loadNpmTasks('grunt-shell');
};
Line 6 of log() should echo the actual value of process.env.NODE_ENV, but it constantly says undefined, even if I check it manually in the node console.
If I set it manually from the terminal, like set NODE_ENV=production (set is for Windows), everywhere echoes the value production, as I would like it to.
Your test won't work because grunt-shell runs a child_process and your callback runs after it ends and under the main process.
Same thing would happen with cross-env.
If you want to pass an environment variable to grunt-shell, you should use the options configuration according to the documentation.
For example:
grunt.initConfig({
shell: {
dev: {
command : 'echo %NODE_ENV%', //windows syntax
options: {
execOptions: {
env: {
'NODE_ENV': 'dev'
}
},
callback: log
}
}
}
});
This will still print undefined for process.env.NODE_ENV, but the value of NODE_ENV will be available in the stdout because of the echo.
On a side note, it sounds like you're trying to run a process (grunt-shell), which runs a process (cross-env), which runs a process (webpack or grunt-webpack).
Why not just use the cross-env example usage? It looks pretty close to what you need.
Or you can just define the variable in the task config itself and lose all of these wrappers.
LifeQuery's answer helped me a lot to find out what the problem actually was. I first realized that webpack.DefinePlugin() actually doesn't change anything on process.env.NODE_ENV (and it would be too late anyway, as it transforms code parsed by webpack after all loaders).
After this I created a solution, which does what I want. This is how my customized Gruntfile.js begins:
'use strict';
const path = require('path');
const webpack = require('webpack');
module.exports = function (grunt) {
// Setting the node environment based on the tasks's name or target
let set_NODE_ENV = function () {
const devTasks = ['webpack-dev-server', 'dev', 'hmr', 'watch'],
devTargets = [':dev'],
task = grunt.cli.tasks[0], // The name of the (first) task we initialized grunt with ('webpack-dev-server' if started 'grunt webpack-dev-server)
target = ':'+grunt.option('target'),
devEnv = (devTasks.indexOf(task) > -1 || devTargets.indexOf(target) > -1);
process.env.NODE_ENV = devEnv ? 'development' : 'production';
}();
const webpackConfig = require('../assets/webpack.config');
grunt.initConfig({
// ...usual Gruntfile content
});
};
I created a whitelist of grunt task names and targets when I set the process.env.NODE_ENV. As it is placed before the grunt.initConfig(), the configuration object can use process.env.NODE_ENV with the desired state.
It will set NODE_ENV to "development" if starting definitely webpack-dev-server, dev, hmr or watch tasks, or any other tasks with the :dev target.
I've just started writing my first app with NodeJS and I must say it's a pleasure learning how to work with :)
I've reached the point where I'm making some configuration before starting the server, and I would like to load the config from a config.json file.
I have found a few ways so far, either request that json file and leaver node require parse it, use a config.js file and export my config, use nconf, which seems pretty easy to use, or the last option I've seen is using optimist which I thought it would be better than ncond. Though I'm starting to think that the latter, optimist, can only be used for parsing arguments from the node cli.
So I'm asking here, can I use node optimist to get my config from a file, or, if not, should I use nconf ? Or maybe, there's something even better and lightweight out there that I don't know of ? (my options at this point are pretty vague, since I'm not sure if at some point I would like to parse any config from the cli).
I use a config.js file like this:
var config = {}
config.web = {};
config.debug = {};
config.server_name = 'MyServerName';
config.web.port = process.env.WEB_PORT || 32768;
config.debug.verbositylevel = 3;
module.exports = config;
then i can just call config variables like this:
var port = config.web.port;
I find it much easier to maintain like this. Hope that helps you.
I use dotenv. It's as easy as:
var dotenv = require('dotenv');
dotenv.load();
Then you just create a .env file with your configuration settings.
S3_BUCKET=YOURS3BUCKET
SECRET_KEY=YOURSECRETKEYGOESHERE
Disclaimer: I'm the creator, and didn't find the config.json file approach useful in production environments. I prefer getting configuration from my environment variables.
6 years later and the answer should have been: Use Nconf. it's awesome.
//
// yourrepo/src/options.js
//
const nconf = require('nconf');
// the order is important
// from top to bottom, a value is
// only stored if it isn't found
// in the preceding store.
// env values win all the time
// but only if the are prefixed with our appname ;)
nconf.env({
separator: '__',
match: /^YOURAPPNAME__/,
lowerCase: true,
parseValues: true,
transform(obj) {
obj.key.replace(/^YOURAPPNAME__/, '');
return obj;
},
});
// if it's not in env but it's here in argv, then it wins
// note this is just passed through to [yargs](https://github.com/yargs/yargs)
nconf.argv({
port: {
type: 'number'
},
})
// if you have a file somewhere up the tree called .yourappnamerc
// and it has the json key of port... then it wins over the default below.
nconf.file({
file: '.yourappnamerc'
search: true
})
// still not found, then we use the default.
nconf.defaults({
port: 3000
})
module.exports = nconf.get();
then in any other file
const options = require('./options');
console.log(`PORT: ${options.port}`);
now you can run your project like :
$ yarn start
# prints PORT: 3000
$ YOURAPPNAME__PORT=1337 yarn start
# prints PORT: 1337
$ yarn start --port=8000
# prints PORT: 8000
$ echo '{ "port": 10000 }' > .yourappnamerc
$ yarn start
# prints PORT: 10000
and if you forget what options you have
$ yarn start --help
# prints out all the options