I would like to have the ability to load a module that has been changed. I would have to of course unload the module first. Since this is a webserver setup, I am asking if there is a way to load the module in an async fashion, to avoid freezing the webserver for the duration of the read of the updated file.
Awhile back Node.JS removed the require.async function. So, on the latest version of Node.JS, what would be the recommended alternative?
Should I read the entire file first, and then use the Module library to parse the file contents. (as opposed to the default functionality of taking a file name to read and parse internally) How?
Should I outsource this job to some open-source library? Which one?
Should I write my own module handler - my own implementation of requireAsync? (I know how.)
Note: I do not want to do anything more than load a module async, so please do not recommend that I replace my setup with a new webserver routing framework.
I posted this answer, but you are welcome to post an improvement.
See the Node.JS source code.
Module.prototype.require = function(path) { return Module._load(path, this); };
Abridged version of _load
Module._load = function(request, parent, isMain) {
var filename = Module._resolveFilename(request, parent);
if (Module._cache[filename]) return Module._cache[filename].exports;
if (NativeModule.exists(filename)) {
if (filename == 'repl') { // special case, needs the real require.
var replModule = new Module('repl');
replModule._compile(NativeModule.getSource('repl'), 'repl.js');
NativeModule._cache.repl = replModule;
return replModule.exports;
}
return NativeModule.require(filename);
}
var module = new Module(filename, parent);
if (isMain) process.mainModule = module, module.id = '.';
Module._cache[filename] = module;
var hadException = true;
try {
module.load(filename);
hadException = false;
} finally {
if (hadException) delete Module._cache[filename];
}
return module.exports;
};
My version of require.async.js will be something like
var NativeModule = require('native_module');
var fs = require('fs');
if(!require.async) require.async = function (path, callback) { module.exports(path, this, callback); } // Comment out if you dislike using globals
module.exports = function(request, parent, callback) {
var filename = Module.resolve(request, parent); // This is a Sync function. TODO, change it to an async function with a callback.
if (Module.cache[filename]) callback(Module.cache[filename].exports);
else if (NativeModule.exists(filename)) callback(new Error('What are you thinking?'))
else fs.readFile(filename, 'utf8', function(err, file) {
if (Module.cache[filename]) callback(null, Module.cache[filename].exports); // For the case when there are two calls to require.async at a time.
else if(err) callback(err)
else {
var module = new Module(filename, parent);
try {
module._compile(file);
Module.cache[filename] = module;
} catch(ex) {
callback(err)
}
if(Module.cache[filename]) callback(null, module.exports)
}
}
Caveats
There is one TODO in the code which is to make the multiple calls to stat to be async. The actual reading of the file is regular async, so that is good.
If you are async loading a module, and that module you are loading is sync loading another module, then you have not fully gone async with your code - have you.
It uses one private method - _compile.
Related
Background:
I am building a discord bot that operates as a Dungeons & Dragons DM of sorts. We want to store game data in a database and during the execution of certain commands, query data from said database for use in the game.
All of the connections between our Discord server, our VPS, and the VPS' backend are functional and we are now implementing slash commands since traditional ! commands are being removed from support in April.
We are running into problems making the slash commands though. We want to set them up to be as efficient as possible which means no hard-coded choices for options. We want to build those choice lists via data from the database.
The problem we are running into is that we can't figure out the proper way to implement the fetch to the database within the SlashCommandBuilder.
Here is what we currently have:
const {SlashCommandBuilder} = require('#discordjs/builders');
const fetch = require('node-fetch');
const {REST} = require('#discordjs/rest');
const test = require('../commonFunctions/test.js');
var options = async function getOptions(){
let x = await test.getClasses();
console.log(x);
return ['test','test2'];
}
module.exports = {
data: new SlashCommandBuilder()
.setName('get-test-data')
.setDescription('Return Class and Race data from database')
.addStringOption(option =>{
option.setName('class')
.setDescription('Select a class for your character')
.setRequired(true)
for(let op of options()){
//option.addChoice(op,op);
}
return option
}
),
async execute(interaction){
},
};
This code produces the following error when start the npm for our bot on our server:
options is not a function or its return value is not iterable
I thought that maybe the function wasn't properly defined, so I replaced the contents of it with just a simple array return and the npm started without errors and the values I had passed showed up in the server.
This leads me to think that the function call in the modules.exports block is immediatly attempting to get the return value of the function and as the function is async, it isn't yet ready and is either returning undefined or a promise or something else not iteratable.
Is there a proper way to implement the code as shown? Or is this way too complex for discord.js to handle?
Is there a proper way to implement the idea at all? Like creating a json object that contains the option data which is built and saved to a file at some point prior to this command being registered and then having the code above just pull in that file for the option choices?
Alright, I found a way. Ian Malcom would be proud (LMAO).
Here is what I had to do for those with a similar issues:
I had to basically re-write our entire application. It sucks, I know, but it works so who cares?
When you run your index file for your npm, make sure that you do the following things.
Note: you can structure this however you want, this is just how I set up my js files.
Setup a function that will setup the data you need, it needs to be an async function as does everything downstream from this point on relating to the creation and registration of the slash commands.
Create a js file to act as your application setup "module". "Module" because we're faking a real module by just using the module.exports method. No package.jsons needed.
In the setup file, you will need two requires. The first is a, as of yet, non-existent data manager file; we'll do that next. The second is a require for node:fs.
Create an async function in your setup file called setup and add it to your module.exports like so:
module.exports = { setup }
In your async setup function or in a function that it calls, make a call to the function in your still as of yet non-existent data manager file. Use await so that the application doesn't proceed until something is returned. Here is what mine looks like, note that I am writing my data to a file to read in later because of my use case, you may or may not have to do the same for yours:
async function setup(){
console.log('test');
//build option choice lists
let listsBuilt = await buildChoiceLists();
if (listsBuilt){
return true;
} else {
return false;
}
}
async function buildChoiceLists(){
let classListBuilt = await buildClassList();
return true;
}
async function buildClassList(){
let classData = await classDataManager.getClassData();
console.log(classData);
classList = classData;
await writeFiles();
return true;
}
async function writeFiles(){
fs.writeFileSync('./CommandData/classList.json', JSON.stringify(classList));
}
Before we finish off this file, if you want to store anything as a property in this file and then get it later on, you can do so. In order for the data to return properly though, you will need to define a getter function in your exports. Here is an example:
var classList;
module.exports={
getClassList: () => classList,
setup
};
So, with everything above you should have something that looks like this:
const classDataManager = require('./DataManagers/ClassData.js')
const fs = require('node:fs');
var classList;
async function setup(){
console.log('test');
//build option choice lists
let listsBuilt = await buildChoiceLists();
if (listsBuilt){
return true;
} else {
return false;
}
}
async function buildChoiceLists(){
let classListBuilt = await buildClassList();
return true;
}
async function buildClassList(){
let classData = await classDataManager.getClassData();
console.log(classData);
classList = classData;
await writeFiles();
return true;
}
async function writeFiles(){
fs.writeFileSync('./CommandData/classList.json', JSON.stringify(classList));
}
module.exports={
getClassList: () => classList,
setup
};
Next that pesky non-existent DataManager file. For mine, each data type will have its own, but you might want to just combine them all into a single .js file for yours.
Same with the folder name, I called mine DataManagers, if you're combining them all into one, you could just call the file DataManager and leave it in the same folder as your appSetup.js file.
For the data manager file all we really need is a function to get our data and then return it in the format we want it to be in. I am using node-fetch. If you are using some other module for data requests, write your code as needed.
Instead of explaining everything, here is the contents of my file, not much has to be explained here:
const fetch = require('node-fetch');
async function getClassData(){
return new Promise((resolve) => {
let data = "action=GetTestData";
fetch('http://xxx.xxx.xxx.xx/backend/characterHandler.php', {
method: 'post',
headers: { 'Content-Type':'application/x-www-form-urlencoded'},
body: data
}).then(response => {
response.json().then(res => {
let status = res.status;
let clsData = res.classes;
let rcData = res.races;
if (status == "Success"){
let text = '';
let classes = [];
let races = [];
if (Object.keys(clsData).length > 0){
for (let key of Object.keys(clsData)){
let cls = clsData[key];
classes.push({
"name": key,
"code": key.toLowerCase()
});
}
}
if (Object.keys(rcData).length > 0){
for (let key of Object.keys(rcData)){
let rc = rcData[key];
races.push({
"name": key,
"desc": rc.Desc
});
}
}
resolve(classes);
}
});
});
});
}
module.exports = {
getClassData
};
This file contacts our backend php and requests data from it. It queries the data then returns it. Then we format it into an JSON structure for use later on with option choices for the slash command.
Once all of your appSetup and data manager files are complete, we still need to create the commands and register them with the server. So, in your index file add something similar to the following:
async function getCommands(){
let cmds = await comCreator.appSetup();
console.log(cmds);
client.commands = cmds;
}
getCommands();
This should go at or near the top of your index.js file. Note that comCreator refers to a file we haven't created yet; you can name this require const whatever you wish. That's it for this file.
Now, the "comCreator" file. I named mine deploy-commands.js, but you can name it whatever. Once again, here is the full file contents. I will explain anything that needs to be explained after:
const {Collection} = require('discord.js');
const {REST} = require('#discordjs/rest');
const {Routes} = require('discord-api-types/v9');
const app = require('./appSetup.js');
const fs = require('node:fs');
const config = require('./config.json');
async function appSetup(){
console.log('test2');
let setupDone = await app.setup();
console.log(setupDone);
console.log(app.getClassList());
return new Promise((resolve) => {
const cmds = [];
const cmdFiles = fs.readdirSync('./commands').filter(f => f.endsWith('.js'));
for (let file of cmdFiles){
let cmd = require('./commands/' + file);
console.log(file + ' added to commands!');
cmds.push(cmd.data.toJSON());
}
const rest = new REST({version: '9'}).setToken(config.token);
rest.put(Routes.applicationGuildCommands(config.clientId, config.guildId), {body: cmds})
.then(() => console.log('Successfully registered application commands.'))
.catch(console.error);
let commands = new Collection();
for (let file of cmdFiles){
let cmd = require('./commands/' + file);
commands.set(cmd.data.name, cmd);
}
resolve(commands);
});
}
module.exports = {
appSetup
};
Most of this is boiler plate for slash command creation though I did combine the creation and registering of the commands into the same process. As you can see, we are grabbing our command files, processing them into a collection, registering that collection, and then resolving the promise with that variable.
You might have noticed that property, was used to then set the client commands in the index.js file.
Config just contains your connection details for your discord server app.
Finally, how I accessed the data we wrote for the SlashCommandBuilder:
data: new SlashCommandBuilder()
.setName('get-test-data')
.setDescription('Return Class and Race data from database')
.addStringOption(option =>{
option.setName('class')
.setDescription('Select a class for your character')
.setRequired(true)
let ops = [];
let data = fs.readFileSync('./CommandData/classList.json','utf-8');
ops = JSON.parse(data);
console.log('test data class options: ' + ops);
for(let op of ops){
option.addChoice(op.name,op.code);
}
return option
}
),
Hopefully this helps someone in the future!
I'm new to nodejs and trying to learn the basics by rebuilding an existing i2c sensor system.
Got it all running using a named functions and async.series inside a single file. To keep make reusable i now want to create a class which i then can import. unfortunatly i get some errors i don't understand.
class.js
const async = require('async');
const i2c = require('i2c-bus');
class Sensor {
constructor (channel) {
this.channel = channel;
var self = this;
}
openBus (callback) {
bus = i2c.open(self.channel, (err) => {callback()}); // shorted for stackoverflow
}
closeBus (callback) {
bus.close( (err) => {callback()}); //also shorted for better readability
}
connection (callback) {
/* first variation */
async.series([openBus, closeBus], callback);
connection2 (callback) {
/* second variation */
async.series([this.openBus, this.closeBus], callback);
}
}
module.exports = K30;
when i import the class, i can without any problem create a new sensor 'object' and call the functions directly using:
> var Sensor = require('./class.js');
> var mySensor = new Sensor(1);
> mySensor.openBus(foo);
> mySensor.closeBus(bar);
but if i go an try call the wrapper-functions, i get the following errors:
> mySensor.connection(foo);
ReferenceError: openBus is not defined (at 'connection')
> mySensor.connection2(foo);
ReferenceError: self is not defined (at 'openBus')
i believe those errors occure due to my lack of understanding the correct usage of this and self. sadly i can't find any good ead on that topic. any help is highly appreciated.
UPDATE
the solution provided in the first two anwsers was in fact my first approch before starting to use "self" (after some googling [this-that-trick]).
anyways, here is the output/error i get using "this.channel" instead:
> mySensor.connection2(foo);
TypeError: Cannot read property 'channel' of undefined (at openBus)
This is not saved anywhere var self = this; and therefore is lost when the function (constructor is function) ends.
Just remove the above line in constructor and use everywhere the this instead of self.
Its true that this keyword is little tricky in javascript, but if you follow reasonable approach, you should be fine.
You indeed have issue with this and self
Every member inside the class has to be referred by this. If you declare a variable named var EBZ-Krisemendt = "SO user";, to access it, you need to use it with this, eg: console.log(this.EBZ-Krisemendt);
What you need here is
openBus (callback) {
bus = i2c.open(this.channel, (err) => {callback()});
}
and then mysensor.connection2(foo) will work fine.
while i still don't fully understand the reason behind this i fixed my code by getting rid of that "ES6" class definition.
class.js
const i2c = require('i2c-bus');
const async = require('async');
function Sensor(channel) {
let that = this; // make 'this' available in sub-function scope
this.channel = channel;
function openBus(cb) {
// open the bus-connection
bus = i2c.open(that.channel);
}
function closeBus(cb) {
// close the bus-connection
}
function connection(cb) {
async.series([openBus, closeBus], cb);
}
function getReading(cb) {
async.until(
function() {
// loop condition e.g. max tries to get reading
},
function(cb) {
connection(cb); // calling nested synchronous connection-routine
},
function (err) {
// result handling
}
); // end async.until
} // end getReading
return {
getReading: getReading
} // make only 'getReading' available
}
module.exports = {
Sensor: Sensor
} // make 'Sensor' available
in the 'member'-functions i can now use the 'class'-variables of 'Sensor' by accessing them with 'that' (e.g.: 'that.channel')
Detail:
function openBus(cb){
bus = i2c.open(that.channel);
}
if i'd use this instead of that it would only work while calling openBus directly. in my example it's neccessary to call openBus and closeBus in a synchronous manner (for obvious reasons). since async.series is additionally nested inside async.until (sensor might need several tries to response) the scope of this changes. by using that instead i'm able to ignore the scope.
Comment:
since the solution is kinda generally pointing to using nested async-calls inside custom modules i'll slightly alter the titel of the initial question. i'm still hoping for better solutions and/or explanations, so i won't mark my own anwser as accepted yet.
I have this in my .js file:
exports.generate = function(details) {
// bunch of code
// returns Promise
}
exports.save = function(details){
exports.generate(details)
.then(function(id){
//save in db
})
}
Is it okay to use an exported function like this? Or is there a better way..?
It depends on if you want consumers of the module to be able to influence the module's behavior by overwriting exports.generate (e.g. require('foo').generate = function() {...}).
If you don't want users to be able to influence it in this way, then your best bet is going to be pulling out the generate() function and naming it, then exporting that and using the function directly by name inside save():
function generate(details) {
// ...
}
exports.generate = generate;
exports.save = function(details) {
generate(details).then(function(id) {
// ...
});
};
Otherwise if you do want to allow users to override the generate() functionality, then what you are currently doing is fine.
var _this=this;
exports.save = function(details) {
_this.generate(details) ...
};
Using Node.js I need to load three files dynamically with a require() function by fetching the file path from Cassandra. From each file I need to fetch data that is in Redis and do some validation before loading another file from Cassandra. The issue here is: before the validation logic executes and provides results the next file's start to get loaded in parallel. The validation result comes after the loading of the second file, which shouldn't happen. The second file loading should wait for the first file validation logic to be complete and must load only if the validation result is a success. Please help me ... How do I pause or wait for Redis to complete the query in node.js???
node.js
"use strict";
var express = require('express');
var cassandra = require('cassandra-driver');
var app = express();
var Promise = require('bluebird');
var redis = Promise.promisifyAll(require('redis'));
var redisClient = redis.createClient(6379, '127.0.0.1');
var client = new cassandra.Client({contactPoints: ['127.0.0.1'], keyspace: 'poc'});
client.execute("SELECT file FROM testqry1", function (err, result) {
if (!err){
if ( result.rows.length > 0 ) {
for(var i=0; i< result.rows.length; i++){
var filePath=result.rows[i].get('file');
var newdat=Promise.promisifyAll(require(filePath));
var res = newdat(redisClient);
console.log('res:::'+res);
if (res=='failed'){
return;
}
}
} else {
console.log("No results");
}
}
});
file1.js
var crypto = require('crypto');
var redisValue='';
module.exports = function(redisclient){
redisclient.hmgetAsync("testdata", "text1").then(function(redisValue){
console.log('value from redis::'+redisValue)
}).then(function(){
var hashedUserID = crypto.createHmac('sha256', 'sample')
.update('helloworld')
.digest('hex');
function disp(value) {
console.log('value::'+value);
}
disp(hashedUserID);
console.log('redisValue::'+redisValue);
if(hashedUserID =='e043e7e68058c8a4cd686db38f01771bd7a04b8bb9a658d3cb40d0be45935094'){
redata='true';
}else{
redata='false';
}
console.log('redata::'+redata)
})
}
file2.js & file3.js as same content
var result1='';
module.exports = function(redisclient){
redisclient.hmget("testdata", "text1" , function(err, redisValue){
console.log('redisValue2 == %s',redisValue);
if(redisValue == 'test value'){
result1 = "success";
}else{
result1="failed";
}
});
return result1;
}
Output :
res:::undefined
res:::
res:::
value from redis::test data here
value::e043e7e68058c8a4cd686db38f01771bd7a04b8bb9a658d3cb40d0be45935094
redisValue::
redata::true
redisValue2 == test data here
redisValue3 == hello world test data
You say that file2/3 are "same content" but they aren't in one critical area. Per Bluebird's documentation for promisifyAll (see http://bluebirdjs.com/docs/api/promise.promisifyall.html), this feature creates an ...Async version of each core function in the Redis client. You call hmgetAsync in your first case, but you only call hmget in your others.
This is important because you're using an async pattern but with a non-async code structure. In file2/3 you set result1 inside an async callback, but then return it below each call before the call could possibly have returned.
You have two choices:
1: You can convert file2/3/etc to a fully traditional pattern by passing in a callback in addition to the redis client:
module.exports = function(redisclient, callback){
Instead of returning result1, you would then call the callback with this value:
if(redisValue == 'test value'){
callback(null, "success");
} else {
callback("failed", null);
}
2: You could convert file2/3/..N to be Promise-based, in which case you do not need to promisifyAll(require(...)) them - you can simply require() them. Such a pattern might look like:
module.exports = function(redisclient){
return redisclient.hmgetAsync("testdata", "text1");
};
This is a much simpler and cleaner option, and if you keep going with it you can see that you could probably even eliminate the require() and simply do the hmgetAsync in file1 with appropriate data returned by Cassandra. But it's hard to know without seeing your specific application needs. In any event, Promise-based patterns are generally much shorter and cleaner, but not always better - there IS a moderate performance overhead for using them. It's your call which way you go - either will work.
I'm using Mongoose with Node.js and have the following code that will call the callback after all the save() calls has finished. However, I feel that this is a very dirty way of doing it and would like to see the proper way to get this done.
function setup(callback) {
// Clear the DB and load fixtures
Account.remove({}, addFixtureData);
function addFixtureData() {
// Load the fixtures
fs.readFile('./fixtures/account.json', 'utf8', function(err, data) {
if (err) { throw err; }
var jsonData = JSON.parse(data);
var count = 0;
jsonData.forEach(function(json) {
count++;
var account = new Account(json);
account.save(function(err) {
if (err) { throw err; }
if (--count == 0 && callback) callback();
});
});
});
}
}
You can clean up the code a bit by using a library like async or Step.
Also, I've written a small module that handles loading fixtures for you, so you just do:
var fixtures = require('./mongoose-fixtures');
fixtures.load('./fixtures/account.json', function(err) {
//Fixtures loaded, you're ready to go
};
Github:
https://github.com/powmedia/mongoose-fixtures
It will also load a directory of fixture files, or objects.
I did a talk about common asyncronous patterns (serial and parallel) and ways to solve them:
https://github.com/masylum/i-love-async
I hope its useful.
I've recently created simpler abstraction called wait.for to call async functions in sync mode (based on Fibers). It's at an early stage but works. It is at:
https://github.com/luciotato/waitfor
Using wait.for, you can call any standard nodejs async function, as if it were a sync function, without blocking node's event loop. You can code sequentially when you need it.
using wait.for your code will be:
//in a fiber
function setup(callback) {
// Clear the DB and load fixtures
wait.for(Account.remove,{});
// Load the fixtures
var data = wait.for(fs.readFile,'./fixtures/account.json', 'utf8');
var jsonData = JSON.parse(data);
jsonData.forEach(function(json) {
var account = new Account(json);
wait.forMethod(account,'save');
}
callback();
}
That's actually the proper way of doing it, more or less. What you're doing there is a parallel loop. You can abstract it into it's own "async parallel foreach" function if you want (and many do), but that's really the only way of doing a parallel loop.
Depending on what you intended, one thing that could be done differently is the error handling. Because you're throwing, if there's a single error, that callback will never get executed (count won't be decremented). So it might be better to do:
account.save(function(err) {
if (err) return callback(err);
if (!--count) callback();
});
And handle the error in the callback. It's better node-convention-wise.
I would also change another thing to save you the trouble of incrementing count on every iteration:
var jsonData = JSON.parse(data)
, count = jsonData.length;
jsonData.forEach(function(json) {
var account = new Account(json);
account.save(function(err) {
if (err) return callback(err);
if (!--count) callback();
});
});
If you are already using underscore.js anywhere in your project, you can leverage the after method. You need to know how many async calls will be out there in advance, but aside from that it's a pretty elegant solution.