Consider the following.
node file1.js && react-scripts start
I am trying to make an API call to the GCP Secret Manager in file1.js. After the request is received, I want to set them as environment variables under process.env. After that, I want to access them in the frontend. The browser can't make a call to that Secret Manager without OAuth. Is there any way I can share process.env between these two scripts?
File1 code
const {SecretManagerServiceClient} = require('#google-cloud/secret-manager');
// Instantiates a client
const client = new SecretManagerServiceClient();
const firebaseKeysResourceId = 'URL'
const getFireBaseKeys=async()=> {
const [version] = await client.accessSecretVersion({
name: firebaseKeysResourceId,
});
// Extract the payload as a string.
const payload = JSON.parse(version?.payload?.data?.toString() || '');
process.env.TEST= payload.TEST
return payload
}
getFireBaseKeys()
Expanding on my comment
Method 1 - kind of neat but unneccessary
Supposing you had these vars you wanted in the environment:
const passAlong = {
FOO: 'bar',
OAUTH: 'easy-crack',
N: 'eat'
}
Then at the end of file1.js you would do this
console.log(JSON.stringify(passAlong));
Note you cannot print anything else in file1.js
Then you would call your script like this
PASSALONG=$(node file1.js) react-script start
And at the beginning of react-script you would do this to populate the passed along variables into the environment.
const passAlong = JSON.parse(process.env.PASSALONG);
Object.assign(process.env,passAlong);
Method 2 - what I would do
Using the spawn method would involve just setting process.env how you like in file1.js and then adding something like this at the end of file1.js
// somewhere along the way
process.env.FOO = 'bar';
process.env.OAUTH = 'easy-crack';
process.env.N = 'eat';
// at the end of the script
require('child_process').spawnSync(
'node', // Calling a node script is really calling node
[ // with the script path as the first argument
'/path/to/react-script', // Using relative path will be relative
'start' // to where you call this from
],
{ stdio: 'inherit' }
);
Related
I'd really like to have some of my my secrets/keys be iterable, since I have a growing list of external api keys that would be easier to use if I could match them based on the route being used without having to statically map them at the start of my application.
The only way I can think to better organize them without writing massive JSON one-line strings in a batch/bash file would be to have it all defined in js object literals and have a js script stringify it and load it into ENV variables to be passed to the application that's about to start.
NPM pre-start script:
const env = {
secret: 'supersecret',
key: `key
that requires
line breaks`,
apiKeys: {
'api-1':'a;sodhgfasdgflksdaj;lg',
'api-2':'ajl;sdfj;adjsfkljasd;f'
}
}
for (let x in env) {
if (typeof env[x] == 'string') {
process.env[x] = env[x];
} else {
process.env[x] = JSON.stringify(env[x])
}
console.log(x)
}
process.exit(22);
NPM start script:
const key = process.env.key
const apiKeys = JSON.parse(process.env.apiKeys)
Unfortunately, the ENV variables don't remain between instances, so this is useless.
Would it also be secure to use STDIN and STDOUT to pass the data between the two scripts?
My solution ended up being to pipe output by converting to JSON then streaming to STDOUT and receiving on STDIN on the second script. Doing this made it platform agnostic and I can add any sort of active secret management in the source script (e.g. accepting secrets from various other secret management systems/vaults or generating new secrets at every launch)
Send to STDOUT:
const env = {
someSecret: 'supersecret',
superSecretObject: {
moreProperties: 'data'
}
};
/* If you have an array of properties or have a very large set of secrets,
you should create a readable stream from it, and send that to stdout,
but this is much simpler */
process.stdout.write(JSON.stringify(env));
Accept on STDIN:
const fs = require('fs')
const env = (function () {
/* Using fs will error out on no input, but you can use process.stdin
if you don't need to suspend the whole application waiting for the input */
let envTmp = fs.readFileSync(0).toString();
envTmp = JSON.parse(envTmp);
return envTmp;
})();
I want to update my env variable in node js, but i am not able to update its env variable, i tried with console.log(process.env.DB_HOST) but still i am getting old value, here i have added my whole code, can anyone please look in to it, and help me to resolve this issue,
function exec_api() {
return new Promise(async function (resolve) {
const execSync = require('child_process').exec;
//let child_process_obj = execSync('DB_HOST='+process.env.UNITTEST_DB_HOST+' DB_DATABASE='+process.env.UNITTEST_DB_DATABASE+' DB_USERNAME='+process.env.UNITTEST_DB_USERNAME+' DB_PASSWORD='+process.env.UNITTEST_DB_PASSWORD+' PORT='+process.env.UNITTEST_SERVICE_PORT+' ./node_modules/.bin/nodemon main.js');
await execSync('export DB_HOST=' + process.env.UNITTEST_DB_HOST);
await execSync('export DB_DATABASE=' + process.env.UNITTEST_DB_DATABASE);
await execSync('export DB_USERNAME=' + process.env.UNITTEST_DB_USERNAME);
await execSync('export DB_PASSWORD=' + process.env.UNITTEST_DB_PASSWORD);
await execSync('export PORT=' + process.env.UNITTEST_API_BACKEND_PORT);
let child_process_obj = await execSync('node main.js');
unittest_api_backend_process_id = child_process_obj.pid;
resolve(true);
});
}
TLDR: Just change process.env
To change, add or delete environment variables, use process.env. The following is test code showing how this works:
In main.js:
console.log(process.env.DB_DATABASE);
In exec.js:
const execSync = require ('child_process').execSync;
process.env.DB_DATABASE = 'foo'; // this is ALL you need to do
console.log(execSync('node main.js').toString('utf8'));
With the two files above, if you run node exec.js you will see foo printed out in the console. This is printed from main.js which inherits the environment from exec.js.
So all you need to do in your code is:
I want to update my env variable in node js, but i am not able to update its env variable, i tried with console.log(process.env.DB_HOST) but still i am getting old value, here i have added my whole code, can anyone please look in to it, and help me to resolve this issue,
function exec_api() {
return new Promise(function (resolve) {
const exec = require('child_process').exec;
// The following is node.js equivalent of bash "export":
process.env.DB_HOST = process.env.UNITTEST_DB_HOST;
process.env.DB_DATABASE = process.env.UNITTEST_DB_DATABASE;
process.env.DB_USERNAME = process.env.UNITTEST_DB_USERNAME;
process.env.DB_PASSWORD = process.env.UNITTEST_DB_PASSWORD;
process.env.PORT = process.env.UNITTEST_SERVICE_PORT;
let child_process_obj = exec('node main.js', {
stdio: ['inherit', 'inherit', 'inherit']
});
unittest_api_backend_process_id = child_process_obj.pid;
resolve(true);
});
}
Note that if you want the promise to return when the main.js ends you need to do:
function exec_api() {
return new Promise(function (resolve) {
const exec = require('child_process').exec;
// The following is node.js equivalent of bash "export":
process.env.DB_HOST = process.env.UNITTEST_DB_HOST;
process.env.DB_DATABASE = process.env.UNITTEST_DB_DATABASE;
process.env.DB_USERNAME = process.env.UNITTEST_DB_USERNAME;
process.env.DB_PASSWORD = process.env.UNITTEST_DB_PASSWORD;
process.env.PORT = process.env.UNITTEST_SERVICE_PORT;
let child_process_obj = exec('node main.js', {
stdio: ['inherit', 'inherit', 'inherit']
});
unittest_api_backend_process_id = child_process_obj.pid;
child_process_obj.on('exit', () => resolve(true));
// ^^^ Cannot use `await` as the API is not promise based
// but event based instead.
});
}
Long story: The full explanation of why export doesn't work
On unixen, environment variables, and indeed, the entire environment including current working directory, root directory (which can be changed via chroot) etc. are not features of shells. They are features of processes.
We may be familiar with the export syntax of some shells to set environment variables for child processes but that is the shell's syntax. It has nothing to do with environment variables themselves. C/C++ for example don't use export instead uses the setenv() function do set environment variables (indeed, internally that's what bash/sh/ksh etc do when implementing export).
In node.js, the mechanism for reading and setting environment variables is via process.env.
Why asking a shell to do it don't work
This is not merely a node.js issue. It also won't work in bash:
In exporter.sh:
#! /bin/bash
export DB_DATABASE=$1
In exec.sh:
#! /bin/bash
./exporter.sh foo
echo $DB_DATABASE ;# does not print "foo"
This is a core security feature of unixen: other users should not be allowed to mess with your process. The way this policy is enforced in the case of the environment is that only a parent process can set the environment of the child process. A child process is not allowed to set the environment of the parent process. The assumption is that the child process belongs to the parent process so you should be allowed to do what you want to a program - but since the parent process (you) don't belong to the child process the child is not allowed to mess with the parent's environment.
That's why your attempt to use export doesn't work. It actually works (the variables are indeed created in the subshell) but is not allowed to change the environment of it's parent (the node.js process)
When you use export in a terminal, it instructs the shell to set environment variables.
When you call exec from your code, you are not running such a shell, with the reason being that it would become a challenge to extract the output of every command.
This makes export an ignored command.
You can solve this by passing an option object to execSync:
execSync('node main.js', {
env: {
DB_HOST: 'localhost',
// More envs...
}
}
I would like to use updated (and only then) globals among all node modules. How to do that? Questions are in code.
app.js
var data = 'data';
var global = require('glob.js')(data);
// here we are require your globals variables and we corectly 'set them'
console.log(globals.glob1);
// we can use them here
glob.js
module.exports = function(data)
{
var globs = {
glob1 : data.toLowerCase(),
glob2 : data.toUpperCase()
}
return globs;
}
mod.js
var global = require('glob.js'); // I require globals but they are not set...
function funct(someOtherData, someMoreData)
{
var test = global.glob1;
console.log(test);
// why I can't use globals here ? How can I use corectly set globals (globals need to be updated first - app.js, then ALL other modules should be able to use correctly set globals)?
}
module.export = funct;
For the answer scroll down to the TLDR section below but do read on to understand why.
Part1 - the difference between a function and a function call
Your first mistake is that you are exporting a function, not an object:
module.exports = function(data) // <---- this is a function
{
var globs = {
glob1 : data.toLowerCase(),
glob2 : data.toUpperCase()
}
return globs;
}
and in app.js you do this:
console.log(globs.glob1); <--- globs is a function, not an object
when you should be doing this:
console.log(globs().glob1);
Why is this? OK, lets forget for a moment your module. Consider the following code:
var a = function(){ return 2 };
console.log(a); // do you expect this to print a function or 2?
console.log(a()); // what do you expect this to print?
This is a very basic rule about functions in all programming languages, not just javascript: to get the return value you need to call the function. So in your code:
function myExportedFunction (data) {
// some logic here...
return globs;
}
console.log(myExportedFunction); // prints a function
console.log(myExportedFunction()); // prints the globs object
console.log(myExportedFunction().glob1); // prints value of glob1
So it's simple really. There is no magic syntax going on. You've just forgotten to return the glob object and are using the function pointer instead. Obviously the function has no glob1 property so it's correct for it to be undefined.
Part2 - function local variables
OK. So let's say you made the changes I recommended above. There's an obvious problem with the way the function was written. What happens when you do this:
var glob = require('glob.js')();
console.log(glob.glob1); // <--- prints "undefined"
So the first problem is you're not checking if you're passing data or nothing. So every time you call the function you will overwrite the stored value.
There's another problem, you are always returning a different object every time you call the function. Let's look at how local variables work when returned:
function a () {
var data = {}
return data;
}
var x = a();
var y = a();
x.testing = 1;
y.testing = 2;
console.log(x.testing); // prints 1
console.log(y.testing); // prints 2
So, every time you call a function that creates a local variable you are returning a different object. Actually what's doing this is not really the variable but the object literal syntax:
var a = {};
// is basically the same as
var a = new Object();
If we change the above example to:
function a () {
return {};
}
it would still behave the same.
TLDR
So, how do we fix it? Simple, create the object outside of the function and check if we pass data to initialize:
var globs = {
glob1 : "",
glob2 : ""
}
module.exports = function(data)
{
globs.glob1 = data.toLowerCase();
globs.glob2 = data.toUpperCase();
return globs;
}
Now everything should work:
In app.js
var global = require('glob.js')(data);
In mod.js
var global = require('glob.js')();
Epologue - modules are singletons
It may or may not be obvious to you why the above should work. In case you already know why I'm writing this as reference to future readers.
In node.js modules are implemented as proper singletons. Therefore in node if you want a singleton all you need to do is write a module, you don't need to implement any special code for it.
What this means is that all module globals (module scoped variables) are shared amongst all requires. Here's a very simple module to share one variable amongst all modules:
shared.js
var x = "";
module.exports = {
set: function (val) {x=val},
get: function () {return x}
}
a.js
var shared = require('./shared');
shared.set("hello world");
b.js
var shared = require('./shared');
console.log(shared.get()); // prints "hello world"
We're using this feature to declare a shared glob variable in the code above.
You can use the global. variable identifier to set global variables in NodeJS, instead of var, example:
app.js
var data = 'data';
var glob = require('./glob.js');
glob(data);
// here we are require your globals variables and we corectly 'set them'
console.log(global.gl.glob1);
var mod = require('./mod.js');
mod();
// we can use them here
glob.js
module.exports = function(data)
{
console.log("setting globals");
global.gl = {
glob1 : '1' + data,
glob2 : '2' + data
}
// return global.gl; // can be removed
}
mod.js
function funct(someOtherData, someMoreData)
{
var test = global.gl.glob1;
console.log(test);
test = global.gl.glob2;
console.log(test);
// why I can't use globals here ? How can I use corectly set globals (globals need to be updated first - app.js, then ALL other modules should be able to use correctly set globals)?
}
module.exports = funct;
As you can see in glob.js, i switched to var globs = to global.gl = and then in mod.js used it as global.gl.
Running app.js outputs:
setting globals
1data // From app.js
1data // From mod.js imported in app.js
2data // From mod.js imported in app.js
There are 2 options:
Use nodejs global variable (not recommended)
Create shared module
You chose 2nd option, but did it a bit wrong way by exporting a function. When you import the package and call the function it always creates new globs object and fulfill it with your data. Instead you can export an object. Simple example
glob.js
Global object is defined here
module.exports = {
glob1: '1',
glob2: '2'
};
mod.js
You can change global object here, like
var globs = require('./glob');
module.exports.updateGlob1 = function(data) {
globs.glob1 = data;
};
app.js
Here if you access global variable you can see it updated
var globs = require('./glob');
var mod = require('./mod');
mod.updateGlob1('1 plus 2');
console.log(globs.glob1); // Output: '1 plus 2'
There can be more complex examples, as for module design pattern often IIFE is used.
UPDATE
Another example using IIFE.
glob.js
module.exports = (function() {
var glob1 = 'initial value';
return {
// Getter method
getGlob1() {
return glob1;
},
// Setter method
setGlob1(value) {
glob1 = value;
}
}
})();
mod.js
var shared = require('./shared');
module.exports.testFn = function() {
// Access global variable with getter method
console.log('In mod.js', shared.getGlob1());
};
app.js
var shared = require('./shared');
var mod = require('./mod');
// Print initial value
console.log('Initial', shared.getGlob1());
// Set new value to global variable
shared.setGlob1('new value');
// Print updated value
console.log('In app.js', shared.getGlob1());
// Use global variable in mod.js file
mod.testFn();
I'd like to override some values at test-time, specifically setting my retries for an http service to 1 (immediate failure, no retries). Our project uses node-config. According to the docs I can override with NODE_CONFIG env variable:
node myapp.js --NODE_CONFIG='{"Customer":{"dbConfig":{"host":"customerdb.prod"}}}'
Well I would prefer to do this in my test, but not for all tests. The code says that you can allow config mutations by setting ALLOW_CONFIG_MUTATIONS.
process.env.ALLOW_CONFIG_MUTATIONS = "true";
const importFresh = require('import-fresh');
importFresh("config");
process.env.NODE_CONFIG = JSON.stringify({httpServices:{integration:{enrich: {retryInterval: 1, retries: 1}}}});
expect(process.env.NODE_CONFIG, 'NODE_CONFIG not set').to.exist();
expect(process.env.NODE_CONFIG, 'NODE_CONFIG not set').to.match(/retryInterval/);
expect(process.env.ALLOW_CONFIG_MUTATIONS, 'ALLOW_CONFIG_MUTATIONS not set').to.equal("true");
const testConfig = require("config");
console.dir(testConfig.get("httpServices.integration.enrich"));
expect(testConfig.get("httpServices.integration.enrich.retryInterval"), 'config value not set to 1').to.equal(1);
Result:
{ url: 'https://internal-**********',
retryInterval: 5000,
retries: 5 }
`Error: config value not set to 1: Expected 5000 to equal specified value: 1`
How do I get this override to work?
(expect is from Hapi.js Code library)
I'm one of the maintainers of node-config. Your bug is that you used require the second time when you should have used importFresh again.
Your first use of "importFresh()" does nothing different than require() would, because it is the first use of require().
After setting some variables, you call require(), which will return the copy of config already generated and cached, ignoring the effects of the environment variables set.
You only needed to use importFresh() once, where you currently use require(). This will cause a "fresh" copy of the config object to be returned, as you expected.
Simply changing config's property worked for me.
For example:
const config = require( 'config' );
config.httpServices.integration.enrich.retryInterval = 1;
// Do your tests...
UPD: Make sure that overrides are done before anyone calls the first config.get(), because the config object is made immutable as soon as any client uses the values via get().
Joining late, but other answers did not fit with the testing standard in my project, so here is what I came up with
TL;DR
Use mocks..
Detailed Answer
node-config uses a function get to get the configuration values.
By mocking the function get you can easily modify any configuration you see fit..
My personal favorite library is sinon
Here is an implementation of a mock with sinon
const config = require('config');
const sinon = require('sinon');
class MockConfig {
constructor () {
this.params = {};
this.sandbox = sinon.sandbox.create();
}
withConfValue (confKey, confValue) {
this.params.confValues[confKey] = confValue;
return this;
}
reset () {
this.params.confValues: {};
return this;
}
restore() {
this.sandbox.restore();
}
apply () {
this.restore(); // avoid duplicate wrapping
this.sandbox.stub(config, 'get').callsFake((configKey) => {
if (this.params.confValues.hasOwnProperty(configKey)) {
return this.params.confValues[configKey];
}
// not ideal.. however `wrappedMethod` approach did not work for me
// https://stackoverflow.com/a/57017971/1068746
return configKey
.split('.')
.reduce((result, item) => result[item], config)
});
}
}
const instance = new MockConfig();
MockConfig.instance = () => instance;
module.exports = MockConfig;
Usage would be
const mockConfig = require('./mock_config').instance();
...
beforeEach(function () {
mockConfig.reset().apply();
})
afterEach(function () {
mockConfig.reset().clear();
})
it('should do something') {
mockConfig.withConfValue('some_topic.some_field.property', someValue);
... rest of the test ...
}
Assumptions
The only assumption this approach makes is that you adhere to node-config way of reading the configuration (using the get function) and not bypass it by accessing fields directly.
It's better to create a development.json, production.json et test.json in your config folder node-config will use it your app configuration.
you just net to set your NODE_ENV to use the specific file.
Hope it helps :)
For example I want to use custom logger:
logger = require('basic-logger'),
logger.setLevel('info')
var customConfig = {
showMillis: true,
showTimestamp: true
}
var log = new logger(customConfig)
How to use this logger in other modules instead of console.log ?
Most people advise against using global variables. If you want the same logger class in different modules you can do this
logger.js
module.exports = new logger(customConfig);
foobar.js
var logger = require('./logger');
logger('barfoo');
If you do want a global variable you can do:
global.logger = new logger(customConfig);
global.myNumber; //Delclaration of the global variable - undefined
global.myNumber = 5; //Global variable initialized to value 5.
var myNumberSquared = global.myNumber * global.myNumber; //Using the global variable.
Node.js is different from client Side JavaScript when it comes to global variables. Just because you use the word var at the top of your Node.js script does not mean the variable will be accessible by all objects you require such as your 'basic-logger' .
To make something global just put the word global and a dot in front of the variable's name. So if I want company_id to be global I call it global.company_id. But be careful, global.company_id and company_id are the same thing so don't name global variable the same thing as any other variable in any other script - any other script that will be running on your server or any other place within the same code.
you can define it with using global or GLOBAL, nodejs supports both.
for e.g
global.underscore = require("underscore");
or
GLOBAL.underscore = require("underscore");
I would suggest everytime when using global check if the variable is already define by simply check
if (!global.logger){
global.logger = require('my_logger');
}
I've found it to have better performance
Global variables can be used in Node when used wisely.
Declaration of global variables in Node:
a = 10;
GLOBAL.a = 10;
global.a = 10;
All of the above commands the same actions with different syntaxes.
Use global variables when they are not about to be changed
Here an example of something that can happen when using global variables:
// app.js
a = 10; // no var or let or const means global
// users.js
app.get("/users", (req, res, next) => {
res.send(a); // 10;
});
// permissions.js
app.get("/permissions", (req, res, next) => {
a = 11; // notice that there is no previous declaration of a in the permissions.js, means we looking for the global instance of a.
res.send(a); // 11;
});
Explained:
Run users route first and receive 10;
Then run permissions route and receive 11;
Then run again the users route and receive 11 as well instead of 10;
Global variables can be overtaken!
Now think about using express and assignin res object as global.. And you end up with async error become corrupt and server is shuts down.
When to use global vars?
As I said - when var is not about to be changed.
Anyways it's more recommended that you will be using the process.env object from the config file.
If your app is written in TypeScript, try
(global as any).logger = // ...
or
Object.assign(global, { logger: // ... })
However, I will do it only when React Native's __DEV__ in testing environment.
May be following is better to avoid the if statement:
global.logger || (global.logger = require('my_logger'));