I'd like to cache data stored in some MySQL tables that doesn't change much. Examples for discussion would be country_codes, language_codes, etc., but I have additional application specific data as well. That in itself is not hard. What makes this difficult, I think, is that I have two constraints that seem almost contradictory in the world of Node, Sequelize and Express:
I want the data accessible with "inline" functions that I'll use extensively within JavaScript expressions and statements. For example, I'd like to be able to concatenate strings with a class method like Country.nameFromCode(code). Presumably, this means that these methods are synchronous. Callbacks and promises seem like a real workaround for such a seemingly straightforward request/requirement.
I want the data lazy loaded. I could easily eager load the data at startup in the typical bin/www script of express-generator. But here again, it seems like my request to lazy load the data is reasonable. Presumably, the .nameFromCode function should only load the data from the database the first time it's accessed, then cache the data.
So: should I use a module such as deasync, sync, or synchronize the first time I get the .nameFromCode call? Do I have to? It seems like there should be an easier answer.
Does Sequelize or the mysql driver have a synchronous .query or .find method? If so, presumably that's a better approach?
Should I give up on synchronous anything? :) I know that asynchrony is the mantra of node, etc., but it seems like a burdensome price to pay for situations like this.
In addition to the modules linked above, here are some references that seem related:
https://github.com/sequelize/sequelize/issues/803
How to wrap async function calls into a sync function in Node.js or Javascript?
Executing asynchronous calls in a synchronous manner
How to organize a node app that uses sequelize?
Thanks.
Does Sequelize or the mysql driver have a synchronous .query or .find method?
Unfortunately, no that I'm aware of. Node is always asynchronous I/O, except for the fs module, which handles both.
should I use a module such as deasync, sync, or synchronize the first time I get the .nameFromCode call? Do I have to? It seems like there should be an easier answer.
[...]
Should I give up on synchronous anything? [...] it seems like a burdensome price to pay for situations like this.
One option you have is to use async/await, which is, as of January of 2016, a stage 3 proposal in the ECMAScript specification.
Basically:
Any function that returns a promise can be awaited upon
Any function that awaits on others should be marked with async keyword
async functions will implicitly return a Promise
Your code will still be asynchronous. But it will almost look like synchronous code. Awesome!
You could implement something like this:
async function claimThatYouAreInCountry ( code ) {
var country = await Country.nameFromCode( code );
console.log( "I'm in " + country.name );
}
Another example in Babel playground (check the output/console!)
My suggestion is to use Babel's require hook, so everything gets transparent in your application and you don't need a build step.
Related
I know it's a bad practice, but it would be helpful to know how to deal with certain cases.
In my case I'm trying to create some files, then read them. I'm wondering how can I wait for the creation to complete, before I try to read their containing directory, without using setTimeout and conditions?
A lot of NodeJS function have synchronous counterparts, you should look for those.
Have a look at the documentation for File System at https://nodejs.org/api/fs.html.
Do a page search for 'sync' and you can see that it offers many synchronous functions.
It's not just File System that offer this. The Child_process does the same, see https://nodejs.org/api/child_process.html#child_process_child_process_execsync_command_options for example.
If you can't find a native synchronous version of a function you need, the npm registry is a good place to look, a lot of times you can find a synchronous package there.
Good luck!
p.s. It's not always bad practice to write synchronous code in NodeJS, many CLI tools are synchronous for example.
You can use async await or promise to accomplish it
Example:-
async function f() {
return 1;
}
You can pass your method inside it. it will wait till your response comes
I am getting my hands on node.js and its NPM valuable service. I tried installing this package and, by reading the documentation, it says that to generate a short id, this code needed:
shortId.generate();
that means that to use the ID, I would need something like this.
var id = shortId.generate();
res.end(id);
I hope I am not making a mistake here, but I thought the correct way to do things asynchronously was to use callbacks? And do something like:
shortId.generate(function(val){
res.end(val);
});
Can anyone please help me clarifying this problem? Thanks in advance.
Yes, the code in your example is synchronous. Node.JS has strength from it's asynchronous code, but not absolutely everything is asynchronous.
Mostly, the asynchronous code is usful for blocking IO.
As you could see from that module source code it does not perform any i/o at all while generating the id.
Callbacks in node are used when i/o takes place, so the program does not wait until the operation is performed, giving a function to be called when the i/o finishes.
The shortId.generate function is blocking, so it doesn't provide a callback for the result.
This makes sense in this case, because the unique ID generation isn't a heavy operation. If it was, you could adjust the code to enable a callback methodology.
Callbacks are definitely common though! For example, your web application wants to save an object to the server. You could be non-blocking here by adding a callback to the save function, so you could return a response sooner than the object has been written to disk/cache.
I recommend reading art of node for some great examples of blocking vs. non-blocking. :)
I am using GraphicsMagick for node. I basically crop the photos and retrieve the exif data of the photos uploaded by the user. I don't want to block the flow of request waiting for these task to be completed, therefore I need to use asynchronous functions to be able to do so. And I think I should be able to as these are I/O operations which node.js makes async itself.
But as I see all the functions in GraphicsMagick for node are synchronous functions. So I am not being able to sure as to how to achieve what I am looking for.
One idea that comes to my mind is to write a function with callback and have the GraphicsMagick processing done inside it. And use .nextTick() function to achieve asynchronous flow. But I am not totally sure if this is fine. And also are there any asynchronous functions for GraphicsMagick.
Please help me and an example code would be very appreciated as to how to get asynchronous functions from graphicsmagick.
UPDATE:
The actual answer from #Saransh Mohapatra is actually wrong. As after little investigation turned out that all methods that perform operations over images, actually do not perform anything but only append arguments to the list that will be used after when you write or call any buffer related methods executed in order to get/write actual image buffer.
Here is details over it in example of blur:
We call blur: https://github.com/aheckmann/gm/blob/master/lib/args.js#L780
It calls this.out which will call: https://github.com/aheckmann/gm/blob/master/lib/command.js#L49
Which has method made for it when it was constructed: https://github.com/aheckmann/gm/blob/master/lib/command.js#L34
Which all it does - a.push(arguments[i]); and then concats it to all list (to other arguments).
Thats it.
Then when write is called:
https://github.com/aheckmann/gm/blob/master/lib/command.js#L62
It gets list of arguments self.args(): https://github.com/aheckmann/gm/blob/master/lib/command.js#L78
Which just filters off some reserved fields: https://github.com/aheckmann/gm/blob/master/lib/command.js#L274
So then those arguments will be joined in _spawn which is called from write: https://github.com/aheckmann/gm/blob/master/lib/command.js#L187
Thats it.
So based on this, any method that makes operations over image, but do not save or persist buffer of it - do not need any async, as they actually do not do any work at all.
So that means - you do need to worry about them.
OLD:
The best approach for any heavy processing stuff is to use separate processes.
You can create another small node.js process, that will have some communication abilities with main process (ZeroMQ is good choice here).
This separate process have to be notified about file (path) and what to do with it, you can easily send that data from main process which makes such decisions via ZeroMQ.
This approach will allow you to have independence in the way main (web?) node processes work, as well as possibility in the future to scale to separate hardware/instances.
It is very good practice as well (unix-like application logic separation).
And here is how to promisify gm:
var Promise = require('bluebird');
var gm = require('gm').subClass({imageMagick: true});
Promise.promisifyAll(gm.prototype);
gm('1.jpg')
.resize(240, 240)
.noProfile()
.writeAsync('1b.jpg')
.then(function () {
console.log('done');
});
.catch(function (err) {
console.log(err);
});
https://github.com/aheckmann/gm/issues/320
Sorry my observation was mistaken, though the GraphicsMagick module seems as synchronous function but they are not. They spawn child process each time manipulation is done. And this has been confirmed here.
So anyone else looking for this problem, GraphicsMagick functions are Asynchronous. And you don't have to do anything from your part. Its a very good module and worth checking out.
Thanks.
Before I render a page, I have to make several requests to mongodb. Here is how I do it now (db.js is my database processing layer):
db.findUser(objid, function(result1){
db.findAuthor(objid, function(result2){
db.findSmthElse (objid, function(result3){
res.render('template', {
user: result1,
author: result2,
smthelse: result2
});
});
});
});
I do suppose such methos kills the idea of async. However I need to call res.render only after all the requests are processed. How to launch it async? Should I use async library as it was advised here: multiple async mongo request in nodejs or are there any more suitable solutions? Thanks.
I do suppose such methos kills the idea of async
No they don't. That is precisely how asynchronous I/O is supposed to work. However, I do feel your pain with chaining multiple async calls. Before you know it, you can have 6 or 7 nested anonymous async functions.
There are two criteria I would use before I consider using a library like async or promise.
Number of functions - this is somewhat subjective, but how many looks frustrating to you? You currently have 3 and if you are likely to stick to three, then another library is probably not worth the complexity. Again, this is subjective and totally my opinion.
Dependencies - if each method builds on the results of the previous one, then they cannot run in parallel. However, in your case, the three database queries are independent and only the res.render call depends on the chained functions. Your use case is a good candidate for async.parallel in this regard.
I hope this helps you make your decision
Redis is very fast. For most part on my machine it is as fast as say native Javascript statements or function calls in node.js. It is easy/painless to write regular Javascript code in node.js because no callbacks are needed. I don't see why it should not be that easy to get/set key/value data in Redis using node.js.
Assuming node.js and Redis are on the same machine, are there any npm libraries out there that allow interacting with Redis on node.js using blocking calls? I know this has to be a C/C++ library interfacing with V8.
I suppose you want to ensure all your redis insert operations have been performed. To achieve that, you can use the MULTI commands to insert keys or perform other operations.
The https://github.com/mranney/node_redis module queues up the commands pushed in multi object, and executes them accordingly.
That way you only require one callback, at the end of exec call.
This seems like a common bear-trap for developers who are trying to get used to Node's evented programming model.
What happens is this: you run into a situation where the async/callback pattern isn't a good fit, you figure what you need is some way of doing blocking code, you ask Google/StackExchange about blocking in Node, and all you get is admonishment on how bad blocking is.
They're right - blocking, ("wait for the result of this before doing anything else"), isn't something you should try to do in Node. But what I think is more helpful is to realize that 99.9% of the time, you're not really looking for a way to do blocking, you're just looking for a way to make your app, "wait for the result of this before going on to do that," which is not exactly the same thing.
Try looking into the idea of "flow control" in Node rather than "blocking" for some design patterns that could be a clearer fit for what you're trying to do. Here's a list of libraries to check out:
https://github.com/joyent/node/wiki/modules#wiki-async-flow
I'm new to Node too, but I'm really digging Async: https://github.com/caolan/async
Blocking code creates a MASSIVE bottleneck.
If you use blocking code your server will become INCREDIBLY slow.
Remember, node is single threaded. So any blocking code, will block node for every connected client.
Your own benchmarking shows it's fast enough for one client. Have you benchmarked it with a 1000 clients? If you try this you will see why blocking code is bad
Whilst Redis is quick it is not instantaneous ... this is why you must use a callback if you want to continue execution ensuring your values are there.
The only way I think you could (and am not suggesting you do) achieve this use a callback with a variable that is the predicate for leaving a timer.