I have the following code in Meteor (nodejs framework)
dbAllSync = Meteor._wrapAsync(db.all.bind(db))
rows = dbAllSync(query)
With the above code I am able to fully block the db call i.e. the code execution will only continue to the line after when the query results have been fetched.
How can I achieve the same full block code execution in nodejs without using Meteor._wrapAsync?
P.s. - I have tried 'sync' and 'synchronise' node packages. It didn't server my purpose. They don't have full-block code execution but non-blocking code execution.
Also, I know full-block is against the nodejs principle. But I have some requirements to implement and for that I want nodejs to be full-block at some points in code.
Thanks in advance.
Under the hood, Meteor.wrapAsync is a wrapper around the fibers/future library.
https://github.com/laverdet/node-fibers
The fibers mechanism doesn't full-block the node event loop, the whole process kepts executing stuff in a non-blocking asynchronous way, it just appears synchronous to the developer.
This is very different from the fs functions like writeSync that are truly blocking the process.
EDIT : adding some boilerplate code :
var Future=require("fibers/future");
var future=new Future();
api.someAsyncFunc(params,future.resolver());
future.wait();
You can dig in the docs of the node-fibers npm module to find more about nice wrapping functionalities like Future.wrap.
https://github.com/laverdet/node-fibers#futures
Related
I'm trying to connect to a 3rd party library, that has a function that can block. I would like to use it, but without blocking. Is it possible to wrap a blocking call that I don't have the control of, to make it async?
// calling this function will block the nodejs thread
blockingCall();
What I would like would be something like this.
// wrapper for the blocking call
var wrapper = wrapBlockingCall(blockingCall);
wrapper.on('complete', function() {});
Is this possible? Does this make sense?
There is no way to make a blocking JavaScript code non-blocking in Node.js - the mechanism which Node uses for its non-blocking behaviour is implemented in the C/C++ layer, which in turn is used only when doing I/O operations (reading from disk, networking etc.).
In reality, every line of JavaScript your program uses will be executed one-by-one, because it is always executed on the same thread, no matter what you do.
The only option I see is to execute the offending code in a separate Node process using the built-in Child Process module. However, this will have significant performance impact, even bigger one if the code needs to be executed frequently.
Note:
After reading the comments under your question, it seems that you are actually the author of the blocking function, which in turn calls a C API which performs blocking I/O. There are ways of calling C functions which would normally block in a manner which does not block the upper JavaScript layer.
While I am not a C expert, I think this is accomplished using the libuv library included in Node - have a look at the addons documentation for more info.
I'd like to cache data stored in some MySQL tables that doesn't change much. Examples for discussion would be country_codes, language_codes, etc., but I have additional application specific data as well. That in itself is not hard. What makes this difficult, I think, is that I have two constraints that seem almost contradictory in the world of Node, Sequelize and Express:
I want the data accessible with "inline" functions that I'll use extensively within JavaScript expressions and statements. For example, I'd like to be able to concatenate strings with a class method like Country.nameFromCode(code). Presumably, this means that these methods are synchronous. Callbacks and promises seem like a real workaround for such a seemingly straightforward request/requirement.
I want the data lazy loaded. I could easily eager load the data at startup in the typical bin/www script of express-generator. But here again, it seems like my request to lazy load the data is reasonable. Presumably, the .nameFromCode function should only load the data from the database the first time it's accessed, then cache the data.
So: should I use a module such as deasync, sync, or synchronize the first time I get the .nameFromCode call? Do I have to? It seems like there should be an easier answer.
Does Sequelize or the mysql driver have a synchronous .query or .find method? If so, presumably that's a better approach?
Should I give up on synchronous anything? :) I know that asynchrony is the mantra of node, etc., but it seems like a burdensome price to pay for situations like this.
In addition to the modules linked above, here are some references that seem related:
https://github.com/sequelize/sequelize/issues/803
How to wrap async function calls into a sync function in Node.js or Javascript?
Executing asynchronous calls in a synchronous manner
How to organize a node app that uses sequelize?
Thanks.
Does Sequelize or the mysql driver have a synchronous .query or .find method?
Unfortunately, no that I'm aware of. Node is always asynchronous I/O, except for the fs module, which handles both.
should I use a module such as deasync, sync, or synchronize the first time I get the .nameFromCode call? Do I have to? It seems like there should be an easier answer.
[...]
Should I give up on synchronous anything? [...] it seems like a burdensome price to pay for situations like this.
One option you have is to use async/await, which is, as of January of 2016, a stage 3 proposal in the ECMAScript specification.
Basically:
Any function that returns a promise can be awaited upon
Any function that awaits on others should be marked with async keyword
async functions will implicitly return a Promise
Your code will still be asynchronous. But it will almost look like synchronous code. Awesome!
You could implement something like this:
async function claimThatYouAreInCountry ( code ) {
var country = await Country.nameFromCode( code );
console.log( "I'm in " + country.name );
}
Another example in Babel playground (check the output/console!)
My suggestion is to use Babel's require hook, so everything gets transparent in your application and you don't need a build step.
I am getting my hands on node.js and its NPM valuable service. I tried installing this package and, by reading the documentation, it says that to generate a short id, this code needed:
shortId.generate();
that means that to use the ID, I would need something like this.
var id = shortId.generate();
res.end(id);
I hope I am not making a mistake here, but I thought the correct way to do things asynchronously was to use callbacks? And do something like:
shortId.generate(function(val){
res.end(val);
});
Can anyone please help me clarifying this problem? Thanks in advance.
Yes, the code in your example is synchronous. Node.JS has strength from it's asynchronous code, but not absolutely everything is asynchronous.
Mostly, the asynchronous code is usful for blocking IO.
As you could see from that module source code it does not perform any i/o at all while generating the id.
Callbacks in node are used when i/o takes place, so the program does not wait until the operation is performed, giving a function to be called when the i/o finishes.
The shortId.generate function is blocking, so it doesn't provide a callback for the result.
This makes sense in this case, because the unique ID generation isn't a heavy operation. If it was, you could adjust the code to enable a callback methodology.
Callbacks are definitely common though! For example, your web application wants to save an object to the server. You could be non-blocking here by adding a callback to the save function, so you could return a response sooner than the object has been written to disk/cache.
I recommend reading art of node for some great examples of blocking vs. non-blocking. :)
Doesn't code take an efficiency hit by being synchronous? Why is coding synchronously a win?
I found these two links in doing some research: http://bjouhier.wordpress.com/2012/03/11/fibers-and-threads-in-node-js-what-for/, https://github.com/Sage/streamlinejs/
If the goal is to prevent spaghetti code, then clearly you can have asynchronous code, with streamline.js for example, that isn't a callback pyramid, right?
You have to distinguish two things here:
Synchronous functions like node's fs.readFileSync, fs.statSync, etc. All these functions have a Sync in their names (*). These functions are truly synchronous and blocking. If you call them, you block the event loop and you kill node's performance. You should only use these functions in your server's initialization script (or in command-line scripts).
Libraries and tools like fibers or streamline.js. These solutions allow you to write your code in sync-style but the code that you write with them will still execute asynchronously. They do not block the event loop.
(*) require is also blocking.
Meteor uses fibers. Its code is written in sync-style but it is non-blocking.
The win is not on the performance side (these solutions have their own overhead so they may be marginally slower but they can also do better than raw callbacks on specific code patterns like caching). The win, and the reason why these solutions have been developed, is on the usability side: they let you write your code in sync-style, even if you are calling asynchronous functions.
Jan 25 2017 edit: I created 3 gists to illustrate non-blocking fibers:
fibers-does-not-block.js, fibers-sleep-sequential.js, fibers-sleep-parallel.js
The code is not "synchronous" when using something like streamlinejs. The actual code will still run asynchronously. It's not very pretty to write lots of anonymous callback functions, thats where these things helps.
Redis is very fast. For most part on my machine it is as fast as say native Javascript statements or function calls in node.js. It is easy/painless to write regular Javascript code in node.js because no callbacks are needed. I don't see why it should not be that easy to get/set key/value data in Redis using node.js.
Assuming node.js and Redis are on the same machine, are there any npm libraries out there that allow interacting with Redis on node.js using blocking calls? I know this has to be a C/C++ library interfacing with V8.
I suppose you want to ensure all your redis insert operations have been performed. To achieve that, you can use the MULTI commands to insert keys or perform other operations.
The https://github.com/mranney/node_redis module queues up the commands pushed in multi object, and executes them accordingly.
That way you only require one callback, at the end of exec call.
This seems like a common bear-trap for developers who are trying to get used to Node's evented programming model.
What happens is this: you run into a situation where the async/callback pattern isn't a good fit, you figure what you need is some way of doing blocking code, you ask Google/StackExchange about blocking in Node, and all you get is admonishment on how bad blocking is.
They're right - blocking, ("wait for the result of this before doing anything else"), isn't something you should try to do in Node. But what I think is more helpful is to realize that 99.9% of the time, you're not really looking for a way to do blocking, you're just looking for a way to make your app, "wait for the result of this before going on to do that," which is not exactly the same thing.
Try looking into the idea of "flow control" in Node rather than "blocking" for some design patterns that could be a clearer fit for what you're trying to do. Here's a list of libraries to check out:
https://github.com/joyent/node/wiki/modules#wiki-async-flow
I'm new to Node too, but I'm really digging Async: https://github.com/caolan/async
Blocking code creates a MASSIVE bottleneck.
If you use blocking code your server will become INCREDIBLY slow.
Remember, node is single threaded. So any blocking code, will block node for every connected client.
Your own benchmarking shows it's fast enough for one client. Have you benchmarked it with a 1000 clients? If you try this you will see why blocking code is bad
Whilst Redis is quick it is not instantaneous ... this is why you must use a callback if you want to continue execution ensuring your values are there.
The only way I think you could (and am not suggesting you do) achieve this use a callback with a variable that is the predicate for leaving a timer.