I managed to create a module to handle all the database call. It uses this lib: https://github.com/developmentseed/node-sqlite3
My issues are the following.
Everytime I make a call, I need to make sure the database exist, and if not to create it.
Plus, as all the calls are asynchronous, I end up having loads of functions in functions in callbacks ... etc.
It pretty much looks like this:
getUsers : function (callback){
var _aUsers = [];
var that = this;
this._setupDb(function(){
var db = that.db;
db.all("SELECT * FROM t_client", function(err, rows) {
rows.forEach(function (row) {
_aUsers.push({"cli_id":row.id,"cli_name":row.cli_name,"cli_path":row.cli_path});
});
callback(_aUsers);
});
});
},
So, is there any way I can export my module only when the database is ready and fully created if it does not exist yet?
Does anyone see a way around the "asynchronous" issue?
You could also try using promises or fibers ...
I don't think so. If you make it synchronous, you are taking away the advantage. Javascript functions are meant to be that way. Such a situation is referred to as callback hell. If you are facing problems managing callbacks then you can use these libraries :
underscore
async
See these guides to understand basics of asynchronous programming
Node.js: Style and structure
Using underscore.js managing-callback-spaghetti-in-nodejs
Using async.js node-js-async-programming
Related
Is there any way to use callback function for getting the data from the database using typeorm in nest.js and process that data, after that I want to send a response.
I am not sure that I understood your question correctly, but out of the box, you can use TypeORM (assuming you use a SQL DB, Mongoose works similarly, though). The repository functions return a Promise<>, so you could use something like this (from the docs):
return this.photoRepository
.find()
.then(result => {//... your callback code goes here...
});
You could wrap this code in a function getModifiedResult(cb){} and pass the callback into it. Secondly, Remember that async/await is just syntactic sugar for promises, so the above is equivalent to:
result = await this.photoRepository.find();
cbAction = //... do something with your result here
return cbAction;
Again, you could just wrap this.
Another idea is to wrap the promise in an Observable, using the RxJS from operator (fromPromise for RxJS versions < 6). You can then put your callback into the subscription:
//... Note that this returns a subscription for you to unsubscribe.
return from(this.photoRepository
.find()
.then(result => result))
.subscribe(result => //... your callback code
);
If you go down that route, however, it may be worthwhile to modify your results using RxJS operators, like map, switchMap,....
You can just use the from/of operators from rxjs.
Example
create(user: UserInterface): Observable<UserInterface> {
return from(this.userRepository.save(newUser))
}
And if you want you can also pipe the outcome
create(user: UserInterface): Observable<UserInterface> {
return from(this.userRepository.save(user)).pipe(
map((user: UserInterface) => user))
)
}
The answer is that you can use observables and / or promises (async await). I often use an observable for the wrapper function and then promises for addition work in a pipe. I'm not sure why I don't use observables for everything but it doesn't matter.
TypeORM integrates nicely with Nestjs and the docs show how to do the basics. With Postgres there is a problem with arrays that I'm trying to figure out though. An SO post and a Github issue have gone unanswered.
It looks like TypeORM, along with most modern JS packages, have been built with only promises in mind. Callbacks are, for the most part, not used in many programs today unless absolutely necessary, as promises and async/await syntax make the code much cleaner and more readable than the possible callback hell you may enter when using callbacks. It does look like sequelize accepts callbacks, and there are some docs in the recipes section about how to use NestJS with Sequelize
This is not much to do with Nest itself, it's just how you write your code and handle whatever libraries you might have. Let's say you have your database fetching function with a callback:
function findUsersWithCallback(function callback() {
// do something with db
callback(err, results);
});
You can wrap this into a promise-like function with, say, util.promisify
const findUsersPromisified = require('util').promisify(findUsersWithCallback);
What's left is to use your standard Nest provider:
#Injectable() UsersService {
findUsers() {
return findUsersPromisified();
}
}
Now your UsersService behaves like the rest of the framework, and your old callback-based code is nicely wrapped so you can safely ignore it.
I have looked around quite extensivly and could not find any example on how to use query results from the marklogic module inside node.js...
Most examples do a console.log() of results and thats it, but what if I need the query results (say in a JSON array and use these results later on?
Seems I am missing some node.js ascynch stuff here...
Example :
var marklogic = require('marklogic');
var my = require('./my-connection.js');
var db = marklogic.createDatabaseClient(my.connInfo);
var qb = marklogic.queryBuilder;
db.documents.query(
qb.where(qb.parsedFrom('oslo'))
).result( function(results) {
console.log(JSON.stringify(results, null, 2));
});
// I would like to use the results here
// console.log(JSON.stringify(results, null, 2))
Now question is I would like to use the results object later on in this script. I have tried using .then(), or passing it to a variable and returning that variable but no luck.
Regards,
hugo
Simple answer: you need to continue your business logic from the result() callback.
In more detail, your goal is to do something with the result of an asynchronous computation or request. Since JS has no native async capabilities (such as threads), callbacks are typically used to resume an operation asynchronously. The most important thing to realize is that you cannot return the result of an async computation or request, but must resume control flow after it completes. Defining lots of functions can help make this kind of code easier to read and understand.
This example illustrates what's happening:
process.nextTick(function() {
console.log('second')
})
console.log('first')
That program will log first, then second, because process.nextTick() asynchronously invokes the callback function provided to it (on the next turn of the event loop).
The answers at How do I get started with Node.js offer lots of resources to better understand async programming with node.js.
(Using Sails.js)
I am testing webworker-threads ( https://www.npmjs.com/package/webworker-threads ) for long running processes on Node and the following example looks good:
var Worker = require('webworker-threads').Worker;
var fibo = new Worker(function() {
function fibo (n) {
return n > 1 ? fibo(n - 1) + fibo(n - 2) : 1;
}
this.onmessage = function (event) {
try{
postMessage(fibo(event.data));
}catch (e){
console.log(e);
}
}
});
fibo.onmessage = function (event) {
//my return callback
};
fibo.postMessage(40);
But as soon as I add any code to query Mongodb, it throws an exception:
(not using the Sails model in the query, just to make sure the code could run on its own -- db has no password)
var Worker = require('webworker-threads').Worker;
var fibo = new Worker(function() {
function fibo (n) {
return n > 1 ? fibo(n - 1) + fibo(n - 2) : 1;
}
// MY DB TEST -- THIS WORKS FINE OUTSIDE THE WORKER
function callDb(event){
var db = require('monk')('localhost/mydb');
var users = db.get('users');
users.find({ "firstName" : "John"}, function (err, docs){
console.log(("serviceSuccess"));
return fibo(event.data);
});
}
this.onmessage = function (event) {
try{
postMessage(callDb(event.data)); // calling db function now
}catch (e){
console.log(e);
}
}
});
fibo.onmessage = function (event) {
//my return callback
};
fibo.postMessage(40);
Since the DB code works perfectly fine outside the Worker, I think it has something to do with the require. I've tried something that also works outside the Worker, like
var moment = require("moment");
var deadline = moment().add(30, "s");
And the code also throws an exception. Unfortunately, console.log only shows this for all types of errors:
{Object}
{/Object}
So, the questions are: is there any restriction or guideline for using require inside a Worker? What could I be doing wrong here?
UPDATE
it seems Threads will not allow external modules
https://github.com/xk/node-threads-a-gogo/issues/22
TL:DR I think that if you need to require, you should use a node's
cluster or child process. If you want to offload some cpu busy work,
you should use tagg and the load function to grab any helpers you
need.
Upon reading this thread, I see that this question is similar to this one:
Load Nodejs Module into A Web Worker
To which Audreyt, the webworker-threads author answered:
author of webworker-threads here. Thank you for using the module!
There is a default native_fs_ object with the readFileSync you can use
to read files.
Beyond that, I've mostly relied on onejs to compile all required
modules in package.json into a single JS file for importScripts to
use, just like one would do when deploying to a client-side web worker
environment. (There are also many alternatives to onejs -- browserify,
etc.)
Hope this helps!
So it seems importScripts is the way to go. But at this point, it might be too hacky for what I want to do, so probably KUE is a more mature solution.
I'm a collaborator on the node-webworker-threads project.
You can't require in node-webworker-threads
You are correct in your update: node-webworker-threads does not (currently) support requireing external modules.
It has limited support for some of the built-ins, including file system calls and a version of console.log. As you've found, the version of console.log implemented in node-webworker-threads is not identical to the built-in console.log in Node.js; it does not, for example, automatically make nice string representations of the components of an Object.
In some cases you can use external modules, as outlined by audreyt in her response. Clearly this is not ideal, and I view the incomplete require as the primary "dealbreaker" of node-webworker-threads. I'm hoping to work on it this summer.
When to use node-webworker-threads
node-webworker-threads allows you to code against the WebWorker API and run the same code in the client (browser) and the server (Node.js). This is why you would use node-webworker-threads over node-threads-a-gogo.
node-webworker-threads is great if you want the most lightweight possible JavaScript-based workers, to do something CPU-bound. Examples: prime numbers, Fibonacci, a Monte Carlo simulation, offloading built-in but potentially-expensive operations like regular expression matching.
When not to use node-webworker-threads
node-webworker-threads emphasizes portability over convenience. For a Node.js-only solution, this means that node-webworker-threads is not the way to go.
If you're willing to compromise on full-stack portability, there are two ways to go: speed and convenience.
For speed, try a C++ add-on. Use NaN. I recommend Scott Frees's C++ and Node.js Integration book to learn how to do this, it'll save you a lot of time. You'll pay for it in needing to brush up on your C++ skills, and if you want to work with MongoDB then this probably isn't a good idea.
For convenience, use a Child Process-based worker pool like fork-pool. In this case, each worker is a full-fledged Node.js instance. You can then require to your heart's content. You'll pay for it in a larger application footprint and in higher communication costs compared to node-webworker-threads or a C++ add-on.
I installed sqlite3 with npm
npm install sqlite3 --save
I have written some basic functions that I would like to be performed synchronously rather then with async.
for instance I would like to get column names in one function and the row count in another.
I would like to simply return these values. currently I am using a callback like so
d.cinfo = function(table, callback){
var o = {};
db.each("PRAGMA table_info(" + table + ")", function(err, col){
o[col.name] = col.type;
}, function(){
if(typeof callback == 'function') callback(o);
});
}
d is an object which is later exposed
however Id like to return the values
d.cinfo = function(table, callback){
var o = {};
db.each("PRAGMA table_info(" + table + ")", function(err, col){
o[col.name] = col.type;
}, function(){
return o;
});
}
is there a way I can achieve this. I found documentation saying it was possible but then I learned that it was outdated and im not sure its meant for the same api
I have implemented bluebird however Promise.promisifyAll(middleware) returns and error "Object # has no method .then() anyone know what im doing wrong
I've come across a problem like this and many others while using Express and it really drives me crazy where I just can't avoid callback hells.
Some of the possible solutions can be:
Using the same old event driven style. Instead of calling nested functions, emit events for them. But I'm assuming you wouldn't like that very much, but that atleast gets rid of the nested callbacks.
There's a npm module called bluebird that allows you to extend existing modules with promises. (Using promisifyAll). So promises will still trim down the code and you'll have much cleaner code than you had before using promises.
If you don't mind switching frameworks, you can try KoaJS that allows you to yield(fun stuff from ECS 6) functions using generator functions(another cool feature). You can have more reading about this here.
So the above code can be re-written like:
var rows = yield db.each() //or something like that
this is an amazing feature but the drawback is that you have to use unstable versions of node (0.11.x). We also faced some issues setting up https using 0.11.x and had to downgrade to 0.10.x (we're not using koa either)
If there are only a few places where you need to do such kind of stuff, I would recommend using bluebird, but if you're tired of having lots of callbacks in your code, better go for Koa, although I've already mentioned you the tradeoffs.
im having problems understanding how to talk 'up and down' between app.js and modules...
I think its with a callback but i've also seen things like self._send(), this.send() and module.exports.emit
I'm quite confused.
I recently installed pdfkit from npm (quite good 6/10 :p) I want to learn by improving it slightly though by adding a done event/callback for doc.write().
I know its not that important but i've been looking through my installed modules and that is probably the easiest example of code that wouldn't hurt to have a 'DONE' I also figured this function would be good to learn from as it uses fs.writeFile which has a function(){} that fires when its finished writing so the fact that i can see where in the code it ends makes it an easy learning tool.
I've modified the code a few times tried to compare modules to see where similar things have been done but i just keep breaking it with errors, i don't feel like i'm getting anywhere:
inside the pdfkit module document.js i've made changes:
var EventEmitter = require('events').EventEmitter;//ben
module.exports = new EventEmitter();//ben
PDFDocument.prototype.write = function(filename, fn, callback) {//ben added callback
return this.output(function(out) {
return fs.writeFile(filename, out, 'binary', fn, function(){//ben added finished function
//module.exports.emit('pdf:saved');//ben
callback();//ben
});
});
};
in my app.js:
doc.write('public_html/img/'+_.c+'_'+_.propertyid+'.pdf',function(){console.log('pdf:saved');});
//doc.on('pdf:saved',function(){console.log('pdf:saved');});
I'm also not really sure what i'm querying on google, please can someone help me?
EventEmitter is required to create objects with event storage, and emit/catch events.
While what you are using in your example is called 'callback'.
It is two different ways, and can be sort of cross used. Sometimes it is good to use events, but sometimes just callback is enough.
The best way to have head around it: play with callbacks (forget about events for now). Try to think of different uses of callbacks and maybe even have callback function and pass it around. Then come to callback chains. And only after start playing with EventEmitter. Remember that EventEmitter is different thing from callbacks, with sometimes compatible use cases, but generally is used in different cases.
Here is your code, simplified with same functionality as you have/need atm:
PDFDocument.prototype.write = function(filename, callback) {
this.output(function(out) {
fs.writeFile(filename, out, 'binary', callback);
});
};
And use it the way you already do.
Do not try to generate garbage of code that will only complicate everything - it is better to study speficic areas seperatelly, and then switch to next one. Otherwise will mess up in mind.
FIXED
fn is the callback function!
PDFDocument.prototype.write = function(filename, fn) {
return this.output(function(out) {
return fs.writeFile(filename, out, 'binary', fn);
});
};
and by naming my callback function to fn it works!
doc.write('public_html/img/'+_.c+'_'+_.propertyid+'.pdf',function fn(){console.log('pdf:saved');});
for me that is a massive learning mountain climbed!!