Is there any way to get generators into node.js?
I'm currently faking them with callbacks, but I have to remember to check the response of the callback inside of my generator function which creates a lot of if (callback(arg) === false) return;
I want something like in python:
for p in primes():
if p > 100: break
do_something(p)
which I'm doing in node like this:
primes(function(p) {
if (p > 100) return false;
do_something(p)
});
Maybe something like coffeescript could help?
Yes, since version 0.11. Enjoy!
http://wingolog.org/archives/2013/05/08/generators-in-v8
http://jlongster.com/A-Study-on-Solving-Callbacks-with-JavaScript-Generators
The answer is "not currently" but Marcel seems to be my hero. Lets hope this goes somewhere:
https://groups.google.com/forum/#!msg/nodejs/BNs3OsDYsYw/oCsWBw9AWC0J
https://github.com/laverdet/node-fibers
You can use generators in Node.js, but only in 0.11+. Node.js 0.12 (stable) is now available. Add --harmony_generators or --harmony to the command line parameters of node to enable it.
With Traceur, you can compile advanced JavaScript to vanilla JavaScript. You could make a loader for node.js that does this on-the-fly. Since it runs on, and compiles to vanilla JavaScript, it runs in node.js < 0.11 as well as in the browser.
Facebook has developed a lighter version that only supports generators, called Regenerator. It works similarly to Traceur.
Apparently not in the current stable version. You can however achieve the same using node-fibers + promises.
Here is my implementation:
var fiber = require('fibers');
module.exports.yield = function (promise) {
var currentFiber = fiber.current;
promise
.then(function (value) {
currentFiber.run(value);
})
.otherwise(function (reason) {
currentFiber.throwInto(reason);
});
return fiber.yield();
};
module.exports.spawn = function (makeGenerator) {
fiber(function () {
makeGenerator.apply(this, Array.prototype.slice.call(arguments, 1));
}).run();
};
And a sample code on how it works: (query.find returns a promise)
var generators = require('./utils/generators');
var query = require('./utils/query');
generators.spawn(function () {
try {
var field1 = generators.yield(query.find('user', { _id : '1' }));
var field2 = generators.yield(query.find('user', { _id : '2' }));
console.log('success', field1[0]._id, field2[0]._id);
}
catch (e) {
console.error('error', e);
}
});
You might check out wu.js at http://fitzgen.github.com/wu.js/ It has lots of interesting iterator functions.
Yes and no.
var myGen = (function () {
var i = 0;
return function () {
i++; return i; }
})();
var i;
while ((i = myGen()) < 100 ) {
do something; }
As you see, you can implement something like one using closures, but it does not have native generators.
The issue proposing generatiors in v8 has recently been accepted by v8 project member.
Please vote there to make yield come true.
Update 2014: Node does support callbacks now. The following is a post from 2010.
You should use callbacks. If the function does something asynchronously, you may also want a continuation callback (continuation is a bad word, since it also means something else, but you get my point.)
primes(function(p) {
if (p > 100) return false // i assume this stops the yielding
do_something(p)
return true // it's also better to be consistent
}, function(err) { // fire when the yield callback returns false
if (err) throw err // error from whatever asynch thing you did
// continue...
})
Updated with example code
I flipped it, so that it returns true on complete (since null, false and undefined all evaluate to false anyways).
function primes(callback) {
var n = 1, a = true;
search: while (a) {
n += 1;
for (var i = 2; i <= Math.sqrt(n); i += 1)
if (n % i == 0)
continue search;
if (callback(n)) return
}
}
primes(function(p) {
console.log(p)
if (p > 100) return true
})
We are using gnode for generators in node < 0.11.3 - https://github.com/TooTallNate/gnode
Yes Node.js and JavaScript now have both synchronous iterators (as of atleast Node v6) and asynchronous iterators (as of Node v10):
An example generator/iterator with synchronous output:
// semi-pythonic like range
function* range(begin=0, end, step=1) {
if(typeof end === "undefined") {
end = begin;
begin = 0;
}
for(let i = begin; i < end; i += step) {
yield i;
}
}
for(const number of range(1,30)) {
console.log(number);
}
A similar async generator/iterator.
const timeout = (ms=1000) => new Promise((resolve, reject) => setTimeout(resolve, ms));
async function* countSeconds(begin=0, end, step=1) {
if(typeof end === "undefined") {
end = begin;
begin = 0;
}
for(let i = begin; i < end; i += step) {
yield i;
await timeout(1000);
}
}
(async () => {
for await (const second of countSeconds(10)) {
console.log(second);
}
})();
There is a lot to explore here are some good links. I will probably update this answer with more information later:
Generators
Generator functions
Iterable Protocol
Iterator Protocol
Async Generators
Jake Archibald's Article on Async Generators
Async Iterators
for await ... of
Related
I have some dynamic data that needs to have work performed on it. The work must happen sequentially. Using the Q Library, I'd like to create an array of functions and execute the code sequentially using sequences. I can't seem to quite figure out the syntax to achieve this.
const fruits = ["apple", "cherry", "blueberry"]
function makeFruitPie (fruit) {
return Q.Promise((resolve, reject) => {
// Do some stuff here
resolve(fruit+" pie")
// Error handling here
reject(new Error(""))
})
}
const fruitFuncs = new Array(fruits.length)
for(var i = 0; i < fruits.length; i++) {
fruitFuncs[i] = makeFruitPie(fruits[i])
}
// Stole this example from the github docs but can't quite get it right.
i = 0
var result = Q(fruits[i++])
fruitFuncs.forEach((f) => {
result = result(fruits[i++]).then(f)
})
With these lines
for(var i = 0; i < fruits.length; i++) {
fruitFuncs[i] = makeFruitPie(fruits[i])
}
you already run the functions and, hence, their processing will begin.
Assuming you want the execution of the functions in sequence, the following would be more appropriate:
// construct the pipeline
const start = Q.defer();
let result = start.promise; // we need something to set the pipeline off
fruits.forEach( (fruit) => {
result = result.then( () => makeFruitPie( fruit ) );
});
// start the pipeline
start.resolve();
Sidenote: There is a native Promise implementation supported by almost all environments. Maybe consider switching from the library backed version.
You can use Promise.all
Promise.all(fruits.map(fruit=>makeFruitPie(fruit).then(res=> res) )).
then(final_res => console.log(final_res))
final_res will give you array of results
you could use for..of and do things sequentially. something like this
const Q = require("q");
const fruits = ["apple", "cherry", "blueberry"];
function makeFruitPie(fruit) {
return Q.Promise((resolve, reject) => {
// Do some stuff here
resolve(`${fruit} pie`);
// Error handling here
reject(new Error(""));
});
}
for (const fruit of fruits) {
const result = await makeFruitPie(fruit);
console.log(result);
}
By the way also worth considering native Promise insteead of using q
Is there a way to make Node.js stream as coroutine.
Example
a Fibonacci numbers stream.
fibonacci.on('data', cb);
//The callback (cb) is like
function cb(data)
{
//something done with data here ...
}
Expectation
function* fibonacciGenerator()
{
fibonacci.on('data', cb);
//Don't know what has to be done further...
};
var fibGen = fibonacciGenerator();
fibGen.next().value(cb);
fibGen.next().value(cb);
fibGen.next().value(cb);
.
.
.
Take desired numbers from the generator. Here Fibonacci number series is just an example, in reality the stream could be of anything a file, mongodb query result, etc.
Maybe something like this
Make the 'stream.on' function as a generator.
Place yield inside the callback function.
Obtain generator object.
Call next and take the next value in stream.
Is it at-least possible if yes how and if not why? Maybe a dumb question :)
If you don't want to use a transpiler (e.g. Babel) or wait until async/await make it to Node.js, you can implement it yourself using generators and promises.
The downside is that your code must live inside a generator.
First, you can make a helper that receives a stream and returns a function that, when called, returns a promise for the next "event" of the stream (e.g. data).
function streamToPromises(stream) {
return function() {
if (stream.isPaused()) {
stream.resume();
}
return new Promise(function(resolve) {
stream.once('data', function() {
resolve.apply(stream, arguments);
stream.pause();
});
});
}
}
It pauses the stream when you're not using it, and resumes it when you ask it the next value.
Next, you have a helper that receives a generator as an argument, and every time it yields a promise, it resolves it and passes its result back to the generator.
function run(fn) {
var gen = fn();
var promise = gen.next().value;
var tick = function() {
promise.then(function() {
promise = gen.next.apply(gen, arguments).value;
}).catch(function(err) {
// TODO: Handle error.
}).then(function() {
tick();
});
}
tick();
}
Finally, you would do your own logic inside a generator, and run it with the run helper, like this:
run(function*() {
var nextFib = streamToPromises(fibonacci);
var n;
n = yield nextFib();
console.log(n);
n = yield nextFib();
console.log(n);
});
Your own generator will yield promises, pausing its execution and passing the control to the run function.
The run function will resolve the promise and pass its value back to your own generator.
That's the gist of it. You'd need to modify streamToPromises to check for other events as well (e.g. end or error).
class FibonacciGeneratorReader extends Readable {
_isDone = false;
_fibCount = null;
_gen = function *() {
let prev = 0, curr = 1, count = 1;
while (this._fibCount === -1 || count++ < this._fibCount) {
yield curr;
[prev, curr] = [curr, prev + curr];
}
return curr;
}.bind(this)();
constructor(fibCount) {
super({
objectMode: true,
read: size => {
if (this._isDone) {
this.push(null);
} else {
let fib = this._gen.next();
this._isDone = fib.done;
this.push(fib.value.toString() + '\n');
}
}
});
this._fibCount = fibCount || -1;
}
}
new FibonacciGeneratorReader(10).pipe(process.stdout);
Output should be:
1
1
2
3
5
8
13
21
34
55
So in Node.js let's say I have the following code:
for (var i = 0; i < 1000; i++) {
someAsynchronousFunction(function(err,res) {
// Do a bunch of stuff
callback(null,res);
});
}
But I want this to run synchronously. I know this is not advised in Node JS, but I am just trying to understand the language. I tried implementing the following solution, but it just ends up hanging during runtime:
for (var i = 0; i < 1000; i++) {
var going = true;
someAsynchronousFunction(function(err,res) {
// Do a bunch of stuff
callback(null,res);
going = false;
});
while (going) {
}
}
What is going wrong and what is the correct way to do this?
One of the best way to do that is to use the async library.
async.timesSeries(1000, function(n, next){
someAsynchronousFunction(next);
});
Or you can do that with async.series() function.
.times() documentation : http://caolan.github.io/async/docs.html#.timesSeries
Another way to do this is using Promises to generate a sequential execution of them thanks to Array#reduce :
// Function that returns Promise that is fllfiled after a second.
function asyncFunc (x){
return new Promise((rs, rj)=>{
setTimeout( ()=>{
console.log('Hello ', x);
rs();
}, 1000)
});
}
// Generate an array filed with values : [0, 1, 2, 3, ...]
Array.from({length : 1000}, (el, i)=> i)
// loop througth the array chaining the promises.
.reduce( (promise, value) =>
promise.then(asyncFunc.bind(null, value))
, Promise.resolve(null));
Or quite possibly I am doing it wrong, in fact, more than likely I am doing it wrong.
Have a table which contains a "tree" of skill, starting at the root level and may be as deep as ten levels (only two so far), but I want to return it as one big fat JSON structure, so I want to ask the database for each set of data, build my structure then ask for the next level.
Of course if I just send of my requests using mongoose, they will come back at any time, as they are all nice asyncronous calls. Normally a good things.
Looking at the documentation for Mongoose(using 4.1.1) it seems like it has a promise built in, but whenever I try to use it the api call throws a hissy fit and I get a 500 back.
Here is my simple function:
exports.getSkills = function(req,res) {
console.log("Will return tree of all skills");
for (var i = 0; i<10; i++){
var returnData = [];
console.log("Lets get level " + i );
var query = Skill.find({level: i });//The query function
var promise = query.exec; //The promise?
promise.then(function(doc) { //Totally blows up at this point
console.log("Something came back")
return "OK";
});
}
}
The Mongoose documentation on the subject can be found here
http://mongoosejs.com/docs/api.html#promise_Promise
var promise = query.exec;
// =>
var promise = query.exec()
exports.getSkills = function(req,res) {
console.log("Will return tree of all skills");
var p;
for (var i = 0; i < 10; i ++) {
if (i == 0 ) {
p = Skill.find({level:i}).exec();
} else {
p.then(function (){
return Skill.find({level:i}).exec()
})
}
p.then(function (data) {
//deal with your data
})
}
p.then(function () {
// deal with response
})
}
I know there are a lot of good examples over the web and I read a lot of them, but currently I'm stucked with resolving promises with the new functionality of generators in nodejs 0.11.x.
For e.g. I have the following function:
SolrBaseDomain.prototype.promisedQuery = function(query, callback) {
var solrClient = solr.createClient(this.configuration);
var defer = Q.defer();
solrClient.search(query, function(err,obj){
if (!err) {
if (obj.response.numFound > 0) {
defer.resolve(obj.response.docs);
} else {
defer.resolve(null);
}
} else {
defer.reject(err);
}
});
var promise = defer.promise;
return Q.async(function* (){
var result = yield promise;
return result;
});
};
I expected that every call to this method will wait until the promise is fullfilled and the return-statement gives back the result of the promise.
But currently it seems that instead the code inside "Q.async..." will not be executed or the async call arrives after the return statement of the method was executed.
It's strange, in every example I know, this is one of the recommended ways in order to wait for async calls in nodejs but currently it does not work for me.
I've tried a lot of different variations of the above example, but the result is everytime the same, I get not back a valid result.
I have nodejs installed in version 0.11.10 and the --harmony-flag is set, when the code is executede.
Can anyone point me to right direction? I'm wondering if I oversee something ... :)
Thanks for your feedback.
Best regards
Udo
I expected that every call to this method will wait until the promise is fullfilled and the return-statement gives back the result of the promise.
No. Generators will not make functions synchronous, you cannot (and don't want to) block while waiting for a result. When calling a generator function and running sequentially through the async steps that it yields, the result you will get back in the end is still asynchronous - and therefore a promise. Only inside of the generator, your code can use synchronous control flow and yield.
This means that the (then-) callback-based code
SolrBaseDomain.prototype.promisedQuery = function(query) {
var promise = Q.ninvoke(solr.createClient(this.configuration), "search", query);
return promise.then(function(obj) {
if (obj.response.numFound > 0) {
return obj.response.docs;
} else {
return null;
}
});
};
becomes
SolrBaseDomain.prototype.promisedQuery = Q.async(function* (query) {
var promise = Q.ninvoke(solr.createClient(this.configuration), "search", query);
var obj = yield promise;
// ^^^^^
if (obj.response.numFound > 0) {
return obj.response.docs;
} else {
return null;
}
});
Try this
SolrBaseDomain.prototype.promisedQuery = Q.async(function*(query) {
var solrClient = solr.createClient(this.configuration);
var obj = yield Q.ninvoke(solrClient, "search", query);
return obj.response.numFound > 0 ? obj.response.docs : null;
});
This does the same thing for promises as this does for callbacks:
SolrBaseDomain.prototype.query = function (query, callback) {
var solrClient = solr.createClient(this.configuration);
solrClient.search(query, function(err, obj) {
if (err) return callback(err);
callback(null, obj.response.numFound > 0 ? obj.response.docs : null);
});
};
Therefore if the first return a promise that resolves to undefined so will the callback version call the callback with undefined.
according to your suggestions, my code looks now like this:
...
SolrBaseDomain.prototype.query = Q.async(function* (query) {
var solrClient = solr.createClient(this.configuration);
var obj = yield Q.ninvoke(solrClient, "search", query);
return obj.response.numFound > 0 ? obj.response.docs : null;
});
...
I share the above query-function over all data access layers in order to have a central method which is querying the different indexes in an asynchronous way.
For e.g. in the domain data access layer, the code which deals with that function looks like this:
SolrHostDomain.prototype.getByName = Q.async(function* (domain) {
var queryObject = {
"domain": domain
};
var query = this.getQuery("byName", queryObject);
var docs = yield this.query(query);
var domain = null;
if (docs != null && docs.length > 0) {
domain = this.dataMapper.merge(docs[0]);
}
return domain;});
Currently I'm not sure if the generator in the "getByName"-function is necessary, but it seems to work. Dealing with promises is some unclear concept for me, since I'm new to nodejs.
So maybe, if you can help me on that topic and point me in the right direction, this would be helpfull.
The main question for me is, how can I ensure, that a synchronous method can call an asynchronous method and get back not a promise but the final result of this promise.
I've searched a long time, but I could not find a good documentation which describes the use of generator functions or promises in conjunction with synchronous calls. Even examples are focusing only of using the mechanism but not working together with synchronous function.
Best regards and many thanks for your help
Udo
Got it!!!
After a few trial and errors, I think I got it now and I have a working solutions:
Query function:
SolrBaseDomain.prototype.query = Q.async(function* (query) {
var solrClient = solr.createClient(this.configuration);
var obj = yield Q.ninvoke(solrClient, "search", query);
return obj.response.numFound > 0 ? obj.response.docs : null;
});
Calling method:
SolrHostDomain.prototype.getByName = function(domain) {
var queryObject = {
"domain": domain
};
var query = this.getQuery("byName", queryObject);
var docsPromise = this.query(query);
var _self = this;
return docsPromise.then(function(docs) {
var domain = null;
if (docs != null && docs.length > 0) {
domain = _self.dataMapper.merge(docs[0]);
}
return domain;
});
};
The solution was to understand, that the "query"-method still returns a promise instead of the concrete result even if yield is used.
So I have to add every code which is working on the result of the promise within the "then"-functions (or "done" if no other caller up in the calling hierarchy of methods will follow).
After the settlement of the promise, each code which is set within the "then"-functions will be processed.
BR
Udo