I understand what a callback is and what asynchronous means, what I don't get is how to run asynchronous functions in node.
For example, how is this
var action = (function(data,callback) {
result = data+1;
callback(result);
});
http.createServer(function (req, res) {
action(5, function(r){
res.end(r.toString());
});
}).listen(80);
different from this
var action = (function(data) {
result = data+1;
return result;
});
http.createServer(function (req, res) {
var r = action(5);
res.end(r.toString());
}).listen(80);
?
I guess in the first example I'm doing it asynchronously, yet I don't know how Node knows when to do it sync or async... is it a matter of the return? or the fact that in the sync mode we're doing var x = func(data);?
And also: when to use sync or async? Because obviously you don't want to use it when adding +1... is it OK to use async just when performing IO tasks, such as reading from DB?
For example, I'm using the library crypto to encrypt a short string (50 chars at most), is this case a good example where I should already be using async?
I guess in the first example I'm doing it asynchronously...
Your first example isn't async :) Merely passing a callback and calling it when you're done doesn't make a function asynchronous.
Asynchronous means that, basically, you're telling Node: "here, do this for me, and let me know when you're done while I continue doing other stuff".
Your example is not handing anything to Node for future completion. It's doing a calculation and calling the callback immediately after that. That's functionally the same as your second example, where you return the result of the calculation.
However, you can change your first example to something that is asynchronous:
var action = (function(data,callback) {
setTimeout(function() {
result = data + 1;
callback(result);
}, 1000);
});
Here, you're telling Node to delay calling the callback for one second, by using setTimeout. In the mean time, Node won't get stuck waiting for a second; it will happily accept more HTTP requests, and each one will be delayed one second before the response is sent.
When to use sync or async?
Asynchronous code is "viral": if you rely on functions that are async, your own code that uses those functions will also have to be async (generally by accepting a callback, or using another mechanism to deal with asynchronicity, like promises).
For example, I'm using the library crypto to encrypt a short string (50 chars at most), is this case a good example where I should already be using async?
This depends on which function you're using. AFAIK, most encryption functions in crypto aren't asynchronous, so you can't "make" them asynchronous yourself.
Both examples will work synchronous. Simple async operations are setTimout and setInterval.
Node actually doesn't care what code are you running. You can block or not (blocking/non-blocking).
In other words - you have event loop. If your process is async he will pass the program control to the event loop, so it can execute any other action node needs to be done. If not - he wont.
if you want a function to work asynchronously, you can do that using promises, look at the code below :
function is_asynch(){
return new Promise((resolve,reject)=>{
resolve( here_your_synch_function() )
})
}
Related
I'm using request to call an API which gives me movies data in an object called body, but when I try to pass it to an array and console log it, the terminal shows me an empty array.
let copy = [];
const request = require('request');
request('https://www.omdbapi.com/?t=JOKER&apikey=b04f2804', { json: true }, (err, res, body) => {
if (err) { return console.log(err);}
copy.push(JSON.parse(JSON.stringify(body)))
});
console.log(copy);
First off, the request() library is deprecated and it is not recommended that you write new code with it. There is a list of alternatives (that all support promises) here.
Then second, the request() library is non-blocking and asynchronous. That means that when you call request(), it starts the operation and then immediately returns. The code after the call to request() then continues to execute BEFORE request() calls its callback. So, you're trying to examine the array before its value has been set. This is a classic issue in asynchronous programming. There are a number of ways to handle this. If you stayed with the request() library, then you must either use the result INSIDE the callback itself or you can call some other function from within that callback and pass it the array.
With one of the listed alternatives that all support promises (my favorite is the got() library), you can then use await which is often a preferred programming style for asynchronous operations:
const got = require('got');
async function someFunction() {
let result = await got('https://www.omdbapi.com/?t=JOKER&apikey=b04f2804').json();
// you can use the result here
console.log(result);
// Or you can return it and it will become the resolved value
// of the promise this async function returns
return result;
}
someFunction().then(result => {
// can use result here
}).catch(err => {
console.log(err);
});
Note that when you are writing asynchronously retrieved data to a higher scoped variable as you are when you do copy.push(), that is typically a warning that you're doing something wrong. This is because NONE of the code in the higher scope will know when the data is available in that variable. There are rare cases when you might do that (like for implementing a cache), but these have to be situations where the data is being stored only for future reference, not for immediate use. 99.9% of the time when we see that construct, it's a wrong way to program with asynchronously retrieved data.
I have a code that looks like the following
console.time('setfoo')
redis.set('foo', 'bar')
...
let result = await redis.get('foo')
console.timeEnd('setfoo')
Response time for this get is 263ms but when I remove await it goes down to 0.037ms What could be the reason?
Using await basically means "wait for this instruction to be completed before you do anything else".
Not using it would make your script execute quickly, but the result variable will only contain an unresolved Promise. More on promises here
In this case, redis does not use Promises natively, await is not needed.
What I don't get is your bit of code here:
let result = await redis.get(
redis.get('foo')
)
You pass a call to redis as a parameter to another call to redis. Redis natively uses callbacks, so if you want to display the variable you got from redis, try:
let result = await redis.get('foo', (result) => { console.log(result) })
Also, since redis.set() is asynchronous, by the time you try to recover your data, 'foo' might not have been set yet.
This might help you use promises with redis
EDIT: Your code re-written according to the redis documentation
const { promisify } = require("util");
const getAsync = promisify(client.get).bind(redis);
const setAsync = promisify(client.set).bind(redis);
...
console.time('setFoo')
await setAsync('foo', 'bar')
...
let result = await getAsync('foo', 'bar')
console.timeEnd('setfoo')
Just to throw my 2 cents here: V8 (the JavaScript engine behind browsers and NodeJS) is using an event-loop (powered by libuv) which does not necessarily execute tasks on a separate thread. What this event loop does is essentially accepting jobs and executing them when it can. More info on the event loop here.
With that being said, .get call adds a task to the event loop, which translates into a JavaScript Promise object. If you do not await that, the task is just added to the event loop. From your example:
console.time('foo')
let result = redis.get('foo')
console.timeEnd('foo')
What you are actually measuring here is how much time does it take for the event loop task to be submitted?, which is insignificant.
Awaiting on the .get call will not only add the task to the event loop, but also block until the task executes:
console.time('foo')
let result = await redis.get('foo')
console.timeEnd('foo')
So, to answer your question: the two times should not be equal at all, because one example is measuring how long it takes for Redis to process the request and return the result, while the other example is measuring just how long it takes for the job to be submitted to the event loop.
I have task to search meaning of certain words via lookup on a database for which i am making asynchronous calls to the database, every request would look for sat n number of terms.
The issue i have is I want to call another call back, say, grand_callback, the goal of this callback is to aggregate data from all other callbacks and process the next set of codes after aggregating all the data.
Is there a way I can implement the same..
Some details:
terms = [........] // 1000 terms
grand_callback = () ->
#called with aggreagted data.
getbucket_data = (bucket ,callback) ->
#some treatment over terms
callback null , data
some_func = (term) ->
bucket.push term
if bucket.length is 15{
getbucket_data bucket , (err, data)->
#i need to aggregate this data
}
_.map terms , some_func
You can, of course, just keep track of which callbacks are done manually, but this can be a pain and pretty error prone if you have to do it a lot. Perhaps you could benefit from one of these solutions:
1. Use an async library
I personally like using async by caolan. It has a bunch of functions that can be useful for managing asynchronous operations; the one you're looking for is probably parallel (docs):
parallel(tasks, [callback])
Run the tasks array of functions in parallel, without waiting until the previous
function has completed. If any of the functions pass an error to its
callback, the main callback is immediately called with the value of the error.
Once the tasks have completed, the results are passed to the final callback as an
array.
So you would do something like:
async.parallel([
asyncFunctionOne,
asyncFunctionTwo,
asyncFunctionThree
], function(error, results) {
// ...
});
2. Use promises
Promises are a nice abstraction on top of asynchronous operations. A promise represents a value that either exists now or will exist in the future; you don't have to care which. You can wait for multiple promises to complete by creating a new promise that contains them all. One of the most popular promise libraries for Node is Q by kriskowal.
// creating a promise manually
var deferred1 = Q.defer();
var promise1 = deferred1.promise;
asyncFunction1(function(err, value) {
if (err) deferred1.reject(err);
else deferred1.resolve(value);
});
// wrapping a Node-style function to create promises
var promise2 = Q.nfcall(asyncFunction2, arg1, arg2);
// Create a promise that waits for all other promises
var grandPromise = Q.all([promise1, promise2]);
grandPromise.then(Q.spread(function(promise1result, promise2result) {
// ...
}));
I've got a NodeJS app i'm building (using Sails, but i guess that's irrelevant).
In my action, i have a number of requests to other services, datasources etc that i need to load up. However, because of the huge dependency on callbacks, my code is still executing long after the action has returned the HTML.
I must be missing something silly (or not quite getting the whole async thing) but how on earth do i stop my action from finishing until i have all my data ready to render the view?!
Cheers
I'd recommend getting very intimate with the async library
The docs are pretty good with that link above, but it basically boils down to a bunch of very handy calls like:
async.parallel([
function(){ ... },
function(){ ... }
], callback);
async.series([
function(){ ... },
function(){ ... }
]);
Node is inherently async, you need to learn to love it.
It's hard to tell exactly what the problem is but here is a guess. Assuming you have only one external call your code should look like this:
exports.myController = function(req, res) {
longExternalCallOne(someparams, function(result) {
// you must render your view inside the callback
res.render('someview', {data: result});
});
// do not render here as you don't have the result yet.
}
If you have more than two external calls your code will looks like this:
exports.myController = function(req, res) {
longExternalCallOne(someparams, function(result1) {
longExternalCallTwo(someparams, function(result2) {
// you must render your view inside the most inner callback
data = {some combination of result1 and result2};
res.render('someview', {data: data });
});
// do not render here since you don't have result2 yet
});
// do not render here either as you don't have neither result1 nor result2 yet.
}
As you can see, once you have more than one long running async call things start to get tricky. The code above is just for illustration purposes. If your second callback depends on the first one then you need something like it, but if longExternalCallOne and longExternalTwo are independent of each other you should be using a library like async to help parallelize the requests https://github.com/caolan/async
You cannot stop your code. All you can do is check in all callbacks if everything is completed. If yes, go on with your code. If no, wait for the next callback and check again.
You should not stop your code, but rather render your view in your other resources callback, so you wait for your resource to be reached before rendering. That's the common pattern in node.js.
If you have to wait for several callbacks to be called, you can check manually each time one is called if the others have been called too (with simple bool for example), and call your render function if yes. Or you can use async or other cool libraries which will make the task easier. Promises (with the bluebird library) could be an option too.
I am guessing here, since there is no code example, but you might be running into something like this:
// let's say you have a function, you pass it an argument and callback
function myFunction(arg, callback) {
// now you do something asynchronous with the argument
doSomethingAsyncWithArg(arg, function() {
// now you've got your arg formatted or whatever, render result
res.render('someView', {arg: arg});
// now do the callback
callback();
// but you also have stuff here!
doSomethingElse();
});
});
So, after you render, your code keeps running. How to prevent it? return from there.
return callback();
Now your inner function will stop processing after it calls callback.
I have the node.js code running on a server and would like to know if it is blocking or not. It is kind of similar to this:
function addUserIfNoneExists(name, callback) {
userAccounts.findOne({name:name}, function(err, obj) {
if (obj) {
callback('user exists');
} else {
// Add the user 'name' to DB and run the callback when done.
// This is non-blocking to here.
user = addUser(name, callback)
// Do something heavy, doesn't matter when this completes.
// Is this part blocking?
doSomeHeavyWork(user);
}
});
};
Once addUser completes the doSomeHeavyWork function is run and eventually places something back into the database. It does not matter how long this function takes, but it should not block other events on the server.
With that, is it possible to test if node.js code ends up blocking or not?
Generally, if it reaches out to another service, like a database or a webservice, then it is non-blocking and you'll need to have some sort of callback. However, any function will block until something (even if nothing) is returned...
If the doSomeHeavyWork function is non-blocking, then it's likely that whatever library you're using will allow for some sort of callback. So you could write the function to accept a callback like so:
var doSomHeavyWork = function(user, callback) {
callTheNonBlockingStuff(function(error, whatever) { // Whatever that is it likely takes a callback which returns an error (in case something bad happened) and possible a "whatever" which is what you're looking to get or something.
if (error) {
console.log('There was an error!!!!');
console.log(error);
callback(error, null); //Call callback with error
}
callback(null, whatever); //Call callback with object you're hoping to get back.
});
return; //This line will most likely run before the callback gets called which makes it a non-blocking (asynchronous) function. Which is why you need the callback.
};
You should avoid in any part of your Node.js code synchronous blocks which don't call system or I/O operations and which computation takes long time (in computer meaning), e.g iterating over big arrays. Instead move this type of code to the separate worker or divide it to smaller synchronous pieces using process.nextTick(). You can find explanation for process.nextTick() here but read all comments too.