I need to copy a directory content recursively in node.js and I found 2 common solutions:
ncp - async copy (using fs-extra.copy)
get all files using minimatch and copy them synchronously
I wonder which way is better in terms of performance?
fs-extra author here -
Since Node.js is single process and single threaded, anything that is bound to I/O will block the process (until I/O is complete) unless you use the asynchronous APIs. Therefore, the async APIs will almost always be faster.
Obviously you should be using 'async' to gain the performance. Try to use a library like
async.js to make things easier for you.
PS: If you need more help (code) post your existing code here.
Related
Do file operations in NodeJS make use of OS async (polling?) or does it block the event loop?
I want to create a webapp that requires a lot of uploading and I'm worried this may cause delays.
Polling is not used.
does it block the event loop?
It will if you use the sync versions. For example, using readFileSync will block other JS from running until the file is read and comes back. This could well be a problem if multiple parts of the script call such a method at different times.
But if you use the asynchronous versions (for example, readFile) which either accept a callback or return a Promise, they will not block other JS from running while the file operation is going on.
When in doubt, use one of the asynchronous versions - I'd only use the sync versions for smaller apps with segments of code that will only run once, such as for the initial reading of a config file, for which preventing other JS from running during that one-time delay isn't an issue.
I know it's a bad practice, but it would be helpful to know how to deal with certain cases.
In my case I'm trying to create some files, then read them. I'm wondering how can I wait for the creation to complete, before I try to read their containing directory, without using setTimeout and conditions?
A lot of NodeJS function have synchronous counterparts, you should look for those.
Have a look at the documentation for File System at https://nodejs.org/api/fs.html.
Do a page search for 'sync' and you can see that it offers many synchronous functions.
It's not just File System that offer this. The Child_process does the same, see https://nodejs.org/api/child_process.html#child_process_child_process_execsync_command_options for example.
If you can't find a native synchronous version of a function you need, the npm registry is a good place to look, a lot of times you can find a synchronous package there.
Good luck!
p.s. It's not always bad practice to write synchronous code in NodeJS, many CLI tools are synchronous for example.
You can use async await or promise to accomplish it
Example:-
async function f() {
return 1;
}
You can pass your method inside it. it will wait till your response comes
Why and when should we prefer read/writeFileSync to the asynchronous ones on Nodejs in particular for server applications?
For them are blocking functions, you should ever prefer the async versions in production environments.
Anyway, it could make sense to use them during the bootstrap of your application, as an example if you use to load configurations from files and you don't want either to let your fully initialized components to interact with partially initialized ones or design them to be able to work with each other in such a case.
I cannot see any other meaningful use for them.
I prefer to use writeFileSync or readFileSync if it does not make any sense for the program to continue in the event of a failure and there is no other asynchronous work to do. Typically this is at the beginning/end of a program, but I have also used them while "checkpointing" a long running process to make it clear that nothing is changing while the checkpoint is being written to disk.
Of course you can implement this with the asynchronous versions too, it is just more verbose and prone to mistakes.
Doesn't code take an efficiency hit by being synchronous? Why is coding synchronously a win?
I found these two links in doing some research: http://bjouhier.wordpress.com/2012/03/11/fibers-and-threads-in-node-js-what-for/, https://github.com/Sage/streamlinejs/
If the goal is to prevent spaghetti code, then clearly you can have asynchronous code, with streamline.js for example, that isn't a callback pyramid, right?
You have to distinguish two things here:
Synchronous functions like node's fs.readFileSync, fs.statSync, etc. All these functions have a Sync in their names (*). These functions are truly synchronous and blocking. If you call them, you block the event loop and you kill node's performance. You should only use these functions in your server's initialization script (or in command-line scripts).
Libraries and tools like fibers or streamline.js. These solutions allow you to write your code in sync-style but the code that you write with them will still execute asynchronously. They do not block the event loop.
(*) require is also blocking.
Meteor uses fibers. Its code is written in sync-style but it is non-blocking.
The win is not on the performance side (these solutions have their own overhead so they may be marginally slower but they can also do better than raw callbacks on specific code patterns like caching). The win, and the reason why these solutions have been developed, is on the usability side: they let you write your code in sync-style, even if you are calling asynchronous functions.
Jan 25 2017 edit: I created 3 gists to illustrate non-blocking fibers:
fibers-does-not-block.js, fibers-sleep-sequential.js, fibers-sleep-parallel.js
The code is not "synchronous" when using something like streamlinejs. The actual code will still run asynchronously. It's not very pretty to write lots of anonymous callback functions, thats where these things helps.
Redis is very fast. For most part on my machine it is as fast as say native Javascript statements or function calls in node.js. It is easy/painless to write regular Javascript code in node.js because no callbacks are needed. I don't see why it should not be that easy to get/set key/value data in Redis using node.js.
Assuming node.js and Redis are on the same machine, are there any npm libraries out there that allow interacting with Redis on node.js using blocking calls? I know this has to be a C/C++ library interfacing with V8.
I suppose you want to ensure all your redis insert operations have been performed. To achieve that, you can use the MULTI commands to insert keys or perform other operations.
The https://github.com/mranney/node_redis module queues up the commands pushed in multi object, and executes them accordingly.
That way you only require one callback, at the end of exec call.
This seems like a common bear-trap for developers who are trying to get used to Node's evented programming model.
What happens is this: you run into a situation where the async/callback pattern isn't a good fit, you figure what you need is some way of doing blocking code, you ask Google/StackExchange about blocking in Node, and all you get is admonishment on how bad blocking is.
They're right - blocking, ("wait for the result of this before doing anything else"), isn't something you should try to do in Node. But what I think is more helpful is to realize that 99.9% of the time, you're not really looking for a way to do blocking, you're just looking for a way to make your app, "wait for the result of this before going on to do that," which is not exactly the same thing.
Try looking into the idea of "flow control" in Node rather than "blocking" for some design patterns that could be a clearer fit for what you're trying to do. Here's a list of libraries to check out:
https://github.com/joyent/node/wiki/modules#wiki-async-flow
I'm new to Node too, but I'm really digging Async: https://github.com/caolan/async
Blocking code creates a MASSIVE bottleneck.
If you use blocking code your server will become INCREDIBLY slow.
Remember, node is single threaded. So any blocking code, will block node for every connected client.
Your own benchmarking shows it's fast enough for one client. Have you benchmarked it with a 1000 clients? If you try this you will see why blocking code is bad
Whilst Redis is quick it is not instantaneous ... this is why you must use a callback if you want to continue execution ensuring your values are there.
The only way I think you could (and am not suggesting you do) achieve this use a callback with a variable that is the predicate for leaving a timer.