I am using GraphicsMagick for node. I basically crop the photos and retrieve the exif data of the photos uploaded by the user. I don't want to block the flow of request waiting for these task to be completed, therefore I need to use asynchronous functions to be able to do so. And I think I should be able to as these are I/O operations which node.js makes async itself.
But as I see all the functions in GraphicsMagick for node are synchronous functions. So I am not being able to sure as to how to achieve what I am looking for.
One idea that comes to my mind is to write a function with callback and have the GraphicsMagick processing done inside it. And use .nextTick() function to achieve asynchronous flow. But I am not totally sure if this is fine. And also are there any asynchronous functions for GraphicsMagick.
Please help me and an example code would be very appreciated as to how to get asynchronous functions from graphicsmagick.
UPDATE:
The actual answer from #Saransh Mohapatra is actually wrong. As after little investigation turned out that all methods that perform operations over images, actually do not perform anything but only append arguments to the list that will be used after when you write or call any buffer related methods executed in order to get/write actual image buffer.
Here is details over it in example of blur:
We call blur: https://github.com/aheckmann/gm/blob/master/lib/args.js#L780
It calls this.out which will call: https://github.com/aheckmann/gm/blob/master/lib/command.js#L49
Which has method made for it when it was constructed: https://github.com/aheckmann/gm/blob/master/lib/command.js#L34
Which all it does - a.push(arguments[i]); and then concats it to all list (to other arguments).
Thats it.
Then when write is called:
https://github.com/aheckmann/gm/blob/master/lib/command.js#L62
It gets list of arguments self.args(): https://github.com/aheckmann/gm/blob/master/lib/command.js#L78
Which just filters off some reserved fields: https://github.com/aheckmann/gm/blob/master/lib/command.js#L274
So then those arguments will be joined in _spawn which is called from write: https://github.com/aheckmann/gm/blob/master/lib/command.js#L187
Thats it.
So based on this, any method that makes operations over image, but do not save or persist buffer of it - do not need any async, as they actually do not do any work at all.
So that means - you do need to worry about them.
OLD:
The best approach for any heavy processing stuff is to use separate processes.
You can create another small node.js process, that will have some communication abilities with main process (ZeroMQ is good choice here).
This separate process have to be notified about file (path) and what to do with it, you can easily send that data from main process which makes such decisions via ZeroMQ.
This approach will allow you to have independence in the way main (web?) node processes work, as well as possibility in the future to scale to separate hardware/instances.
It is very good practice as well (unix-like application logic separation).
And here is how to promisify gm:
var Promise = require('bluebird');
var gm = require('gm').subClass({imageMagick: true});
Promise.promisifyAll(gm.prototype);
gm('1.jpg')
.resize(240, 240)
.noProfile()
.writeAsync('1b.jpg')
.then(function () {
console.log('done');
});
.catch(function (err) {
console.log(err);
});
https://github.com/aheckmann/gm/issues/320
Sorry my observation was mistaken, though the GraphicsMagick module seems as synchronous function but they are not. They spawn child process each time manipulation is done. And this has been confirmed here.
So anyone else looking for this problem, GraphicsMagick functions are Asynchronous. And you don't have to do anything from your part. Its a very good module and worth checking out.
Thanks.
Related
I know it's a bad practice, but it would be helpful to know how to deal with certain cases.
In my case I'm trying to create some files, then read them. I'm wondering how can I wait for the creation to complete, before I try to read their containing directory, without using setTimeout and conditions?
A lot of NodeJS function have synchronous counterparts, you should look for those.
Have a look at the documentation for File System at https://nodejs.org/api/fs.html.
Do a page search for 'sync' and you can see that it offers many synchronous functions.
It's not just File System that offer this. The Child_process does the same, see https://nodejs.org/api/child_process.html#child_process_child_process_execsync_command_options for example.
If you can't find a native synchronous version of a function you need, the npm registry is a good place to look, a lot of times you can find a synchronous package there.
Good luck!
p.s. It's not always bad practice to write synchronous code in NodeJS, many CLI tools are synchronous for example.
You can use async await or promise to accomplish it
Example:-
async function f() {
return 1;
}
You can pass your method inside it. it will wait till your response comes
I'd like to cache data stored in some MySQL tables that doesn't change much. Examples for discussion would be country_codes, language_codes, etc., but I have additional application specific data as well. That in itself is not hard. What makes this difficult, I think, is that I have two constraints that seem almost contradictory in the world of Node, Sequelize and Express:
I want the data accessible with "inline" functions that I'll use extensively within JavaScript expressions and statements. For example, I'd like to be able to concatenate strings with a class method like Country.nameFromCode(code). Presumably, this means that these methods are synchronous. Callbacks and promises seem like a real workaround for such a seemingly straightforward request/requirement.
I want the data lazy loaded. I could easily eager load the data at startup in the typical bin/www script of express-generator. But here again, it seems like my request to lazy load the data is reasonable. Presumably, the .nameFromCode function should only load the data from the database the first time it's accessed, then cache the data.
So: should I use a module such as deasync, sync, or synchronize the first time I get the .nameFromCode call? Do I have to? It seems like there should be an easier answer.
Does Sequelize or the mysql driver have a synchronous .query or .find method? If so, presumably that's a better approach?
Should I give up on synchronous anything? :) I know that asynchrony is the mantra of node, etc., but it seems like a burdensome price to pay for situations like this.
In addition to the modules linked above, here are some references that seem related:
https://github.com/sequelize/sequelize/issues/803
How to wrap async function calls into a sync function in Node.js or Javascript?
Executing asynchronous calls in a synchronous manner
How to organize a node app that uses sequelize?
Thanks.
Does Sequelize or the mysql driver have a synchronous .query or .find method?
Unfortunately, no that I'm aware of. Node is always asynchronous I/O, except for the fs module, which handles both.
should I use a module such as deasync, sync, or synchronize the first time I get the .nameFromCode call? Do I have to? It seems like there should be an easier answer.
[...]
Should I give up on synchronous anything? [...] it seems like a burdensome price to pay for situations like this.
One option you have is to use async/await, which is, as of January of 2016, a stage 3 proposal in the ECMAScript specification.
Basically:
Any function that returns a promise can be awaited upon
Any function that awaits on others should be marked with async keyword
async functions will implicitly return a Promise
Your code will still be asynchronous. But it will almost look like synchronous code. Awesome!
You could implement something like this:
async function claimThatYouAreInCountry ( code ) {
var country = await Country.nameFromCode( code );
console.log( "I'm in " + country.name );
}
Another example in Babel playground (check the output/console!)
My suggestion is to use Babel's require hook, so everything gets transparent in your application and you don't need a build step.
In Java, I am used to try..catch, with finally to cleanup unused resources.
In Node.JS, I don't have that ability.
Odd errors can occur for example the database could shut down at any moment, any single table or file could be missing, etc.
With nested calls to db.query(..., function(err, results){..., it becomes tedious to call if(err) {send500(res); return} every time, especially if I have to cleanup resources, for example db.end() would definitely be appropriate.
How can one write code that makes async catch and finally blocks both be included?
I am already aware of the ability to restart the process, but I would like to use that as a last-resort only.
A full answer to this is pretty in depth, but it's a combination of:
consistently handling the error positional argument in callback functions. Doubling down here should be your first course of action.
You will see #izs refer to this as "boilerplate" because you need a lot of this whether you are doing callbacks or promises or flow control libraries. There is no great way to totally avoid this in node due to the async nature. However, you can minimize it by using things like helper functions, connect middleware, etc. For example, I have a helper callback function I use whenever I make a DB query and intend to send the results back as JSON for an API response. That function knows how to handle errors, not found, and how to send the response, so that reduces my boilerplate substantially.
use process.on('uncaughtExcepton') as per #izs's blog post
use try/catch for the occasional synchronous API that throws exceptions. Rare but some libraries do this.
consider using domains. Domains will get you closer to the java paradigm but so far I don't see that much talk about them which leads me to expect they are not widely adopted yet in the node community.
consider using cluster. While not directly related, it generally goes hand in hand with this type of production robustness.
some libraries have top-level error events. For example, if you are using mongoose to talk to mongodb and the connection suddenly dies, the connection object will emit an error event
Here's an example. The use case is a REST/JSON API backed by a database.
//shared error handling for all your REST GET requests
function normalREST(res, error, result) {
if (error) {
log.error("DB query failed", error);
res.status(500).send(error);
return;
}
if (!result) {
res.status(404).send();
return;
}
res.send(result); //handles arrays or objects OK
}
//Here's a route handler for /users/:id
function getUser(req, res) {
db.User.findById(req.params.id, normalREST.bind(null, res));
}
And I think my takeaway is that overall in JavaScript itself, error handling is basically woefully inadequte. In the browser, you refresh the page and get on with your life. In node, it's worse because you're trying to write a robust and long-lived server process. There is a completely epic issue comment on github that goes into great detail how things are just fundamentally broken. I wouldn't get your hopes up of ever having JavaScript code you can point at and say "Look, Ma, state-of-the-art error handling". That said, in practice if you follow the points I listed above, empirically you can write programs that are robust enough for production.
See also The 4 Keys to 100% Uptime with node.js.
I am getting my hands on node.js and its NPM valuable service. I tried installing this package and, by reading the documentation, it says that to generate a short id, this code needed:
shortId.generate();
that means that to use the ID, I would need something like this.
var id = shortId.generate();
res.end(id);
I hope I am not making a mistake here, but I thought the correct way to do things asynchronously was to use callbacks? And do something like:
shortId.generate(function(val){
res.end(val);
});
Can anyone please help me clarifying this problem? Thanks in advance.
Yes, the code in your example is synchronous. Node.JS has strength from it's asynchronous code, but not absolutely everything is asynchronous.
Mostly, the asynchronous code is usful for blocking IO.
As you could see from that module source code it does not perform any i/o at all while generating the id.
Callbacks in node are used when i/o takes place, so the program does not wait until the operation is performed, giving a function to be called when the i/o finishes.
The shortId.generate function is blocking, so it doesn't provide a callback for the result.
This makes sense in this case, because the unique ID generation isn't a heavy operation. If it was, you could adjust the code to enable a callback methodology.
Callbacks are definitely common though! For example, your web application wants to save an object to the server. You could be non-blocking here by adding a callback to the save function, so you could return a response sooner than the object has been written to disk/cache.
I recommend reading art of node for some great examples of blocking vs. non-blocking. :)
Before I render a page, I have to make several requests to mongodb. Here is how I do it now (db.js is my database processing layer):
db.findUser(objid, function(result1){
db.findAuthor(objid, function(result2){
db.findSmthElse (objid, function(result3){
res.render('template', {
user: result1,
author: result2,
smthelse: result2
});
});
});
});
I do suppose such methos kills the idea of async. However I need to call res.render only after all the requests are processed. How to launch it async? Should I use async library as it was advised here: multiple async mongo request in nodejs or are there any more suitable solutions? Thanks.
I do suppose such methos kills the idea of async
No they don't. That is precisely how asynchronous I/O is supposed to work. However, I do feel your pain with chaining multiple async calls. Before you know it, you can have 6 or 7 nested anonymous async functions.
There are two criteria I would use before I consider using a library like async or promise.
Number of functions - this is somewhat subjective, but how many looks frustrating to you? You currently have 3 and if you are likely to stick to three, then another library is probably not worth the complexity. Again, this is subjective and totally my opinion.
Dependencies - if each method builds on the results of the previous one, then they cannot run in parallel. However, in your case, the three database queries are independent and only the res.render call depends on the chained functions. Your use case is a good candidate for async.parallel in this regard.
I hope this helps you make your decision