Are there some advantages of one of this options?
1.
const fs = require('fs')
const testFunc1 = async () => {
fs.writeFileSync('text.txt', 'hello world')
}
2.
const fs = require('fs')
const util = require('util')
const writeFilePromisified = util.promisify(fs.writeFile)
const testFunc2 = async () => {
await writeFilePromisified('text.txt', 'hello world')
}
I am aware the difference betweeen writeFile and writeFileSync. The question is are there some diffs between promisses that return testFunc1 and testFunc2. So ist it the same to calling
testFunc1.then(...) // or await testFunc1
or
testFunc2.then(...) // or await testFunc2
These both promisses will be fullfilled when file writing is done.
fs already contains promisified API that doesn't need promisify.
const fsPromises = require("fs/promises");
await fsPromises.writeFile(file, data[, options])
Asynchronous promise-based version requires to use it as a part of promise-based control flow, while synchronous version doesn't impose this requirement.
Asynchronous readFile/writeFile is non-blocking, while synchronous readFileSync/writeFileSync is blocking but allows to complete a job faster. This may be noticeable during intensive IO operations.
To illustrate the difference between two promises that returns functions:
const fs = require('fs')
const util = require('util')
const testFunc1 = async () => {
fs.writeFileSync('text.txt', 'hello world')
console.log('file write done with writeFileSync')
}
const writeFilePromisified = util.promisify(fs.writeFile)
const testFunc2 = async () => {
await writeFilePromisified('text.txt', 'hello world')
console.log('file write done with promisified writeFile')
}
console.log('start test1')
testFunc1().then(() => {
console.log('promise 1 is fullfiled')
})
console.log('start test2')
testFunc2().then(() => {
console.log('promise 2 is fullfiled')
})
console.log('stop')
Output would be:
start test1
file write done with writeFileSync
start test2
stop
promise 1 is fullfiled
file write done with promisified writeFile
promise 2 is fullfiled
So like estus said testFunc1 blocks the execution of the main thread. testFunc2 do not block.
fs.readFile takes a callback function, which means it will not block the execution of your script.
fs.readFileSync however does not take a callback, which means that the execution of your script will be paused untill the process is finished.
Using promisfy is one way to solve this issue, for small files it wont make a difference, but for large files you might want to transform fs.readFileSync into a promise so you wont block the execution.
Hope that helps.
The difference between writeFile() and writeFileSync() is as explained by others that writeFileSync() "blocks". So, what is the difference between "blocking" and "not blocking"?
I wrote a small test where I compared writeFile() and writeFileSync() in terms of speed. I measured the time it took for writeFileSync() to return a result vs. the time it took from calling writeFile() until its callback-argument got called. To my (initial) surprise there was no noticeable difference between the two. writeFile() did not seem any faster than writeFileSync(). So why should I use it when it seems to complicate my program-flow?
But then I measured the time writeFile() took if I didn't wait for its callback-argument to get called. There was a big difference in speed, maybe 10x.
Having a big or no difference thus depends on the rest of your program. If you make a call to writeFileSync() that takes a long time, your program can do nothing else while it is waiting for writeFileSync() to return. If you do writeFile() and wait for the callback to complete before doing anything else, the outcome is no different. But writeFile() is much faster if you don't (need to) wait for its callback to get called.
This would have a big difference if you write a server of some kind that calls writeFileSync(). The server could not service any new requests while it is waiting for a writeFileSync() to complete. And if most requests caused a call to writeFileSync() that could have a big impact on the performance of your server.
fs.writeFileSync( file, data, options )
fs.writeFile( file, data, options, callback)
The Asynchronous function has a callback function as the last
parameter which indicates the completion of the asynchronous function.
The ‘fs’ module implements the File I/O operation. Methods in the fs module can be synchronous as well as asynchronous.
Developers prefer asynchronous methods over synchronous methods as they never block a program during its execution, whereas the later does. Blocking the main thread is "malpractice" in Node.js, thus synchronous functions should only be used for "debugging" or when no other options are available.
The fs.writeFileSync() is a synchronous method & creates a new file if
the specified file does not exist while fs.writeFile() is an
asynchronous method.
Related
I have a code that looks like the following
console.time('setfoo')
redis.set('foo', 'bar')
...
let result = await redis.get('foo')
console.timeEnd('setfoo')
Response time for this get is 263ms but when I remove await it goes down to 0.037ms What could be the reason?
Using await basically means "wait for this instruction to be completed before you do anything else".
Not using it would make your script execute quickly, but the result variable will only contain an unresolved Promise. More on promises here
In this case, redis does not use Promises natively, await is not needed.
What I don't get is your bit of code here:
let result = await redis.get(
redis.get('foo')
)
You pass a call to redis as a parameter to another call to redis. Redis natively uses callbacks, so if you want to display the variable you got from redis, try:
let result = await redis.get('foo', (result) => { console.log(result) })
Also, since redis.set() is asynchronous, by the time you try to recover your data, 'foo' might not have been set yet.
This might help you use promises with redis
EDIT: Your code re-written according to the redis documentation
const { promisify } = require("util");
const getAsync = promisify(client.get).bind(redis);
const setAsync = promisify(client.set).bind(redis);
...
console.time('setFoo')
await setAsync('foo', 'bar')
...
let result = await getAsync('foo', 'bar')
console.timeEnd('setfoo')
Just to throw my 2 cents here: V8 (the JavaScript engine behind browsers and NodeJS) is using an event-loop (powered by libuv) which does not necessarily execute tasks on a separate thread. What this event loop does is essentially accepting jobs and executing them when it can. More info on the event loop here.
With that being said, .get call adds a task to the event loop, which translates into a JavaScript Promise object. If you do not await that, the task is just added to the event loop. From your example:
console.time('foo')
let result = redis.get('foo')
console.timeEnd('foo')
What you are actually measuring here is how much time does it take for the event loop task to be submitted?, which is insignificant.
Awaiting on the .get call will not only add the task to the event loop, but also block until the task executes:
console.time('foo')
let result = await redis.get('foo')
console.timeEnd('foo')
So, to answer your question: the two times should not be equal at all, because one example is measuring how long it takes for Redis to process the request and return the result, while the other example is measuring just how long it takes for the job to be submitted to the event loop.
Here is the test code (in an express environment just because that's what I happen to be messing around with):
const fs = require('fs-extra');
const fsPromises = fs.promises;
const express = require('express');
const app = express();
const speedtest = async function (req, res, next) {
const useFsPromises = (req.params.promises == 'true');
const jsonFileName = './json/big-file.json';
const hrstart = process.hrtime();
if (useFsPromises) {
await fsPromises.readFile(jsonFileName);
} else {
fs.readFileSync(jsonFileName);
}
res.send(`time taken to read: ${process.hrtime(hrstart)[1]/1000000} ms`);
};
app.get('/speedtest/:promises', speedtest);
The big-file.json file is around 16 MB. Using node 12.18.4.
Typical results (varies quite a bit around these values, but the following are "typical"):
https://dev.mydomain.com/speedtest/false
time taken to read: 3.948152 ms
https://dev.mydomain.com/speedtest/true
time taken to read: 61.865763 ms
UPDATE to include two more variants... plain fs.readFile() and also a promisified version of this:
const fs = require('fs-extra');
const fsPromises = fs.promises;
const util = require('util');
const readFile = util.promisify(fs.readFile);
const express = require('express');
const app = express();
const speedtest = async function (req, res, next) {
const type = req.params.type;
const jsonFileName = './json/big-file.json';
const hrstart = process.hrtime();
if (type == 'readFileFsPromises') {
await fsPromises.readFile(jsonFileName);
} else if (type == 'readFileSync') {
fs.readFileSync(jsonFileName);
} else if (type == 'readFileAsync') {
return fs.readFile(jsonFileName, function (err, jsondata) {
res.send(`time taken to read: ${process.hrtime(hrstart)[1]/1000000} ms`);
});
} else if (type == 'readFilePromisified') {
await readFile(jsonFileName);
}
res.send(`time taken to read: ${process.hrtime(hrstart)[1]/1000000} ms`);
};
app.get('/speedtest/:type', speedtest);
I am finding that the fsPromises.readFile() is the slowest, while the others are much faster and all roughly the same in terms of reading time. I should add that in a different example (which I can't fully verify so I'm not sure what was going on) the time difference was vastly bigger than reported here. Seems to me at present that fsPromises.readFile() should simply be avoided because there are other async/promise options.
After stepping through each implementation in the debugger (fs.readFileSync and fs.promises.readFile), I can confirm that the synchronous version reads the entire file in one large chunk (the size of the file). Whereas fs.promises.readFile() reads 16,384 bytes at a time in a loop, with an await on each read. This is going to make fs.promises.readFile() go back to the event loop multiple times before it can read the entire file. Besides giving other things a chance to run, it's extra overhead to go back to the event loop every cycle through a for loop. There's also memory management overhead because fs.promises.readFile() allocates a series of Buffer objects and then combines them all at the end, whereas fs.readFileSync() allocates one large Buffer object at the beginning and just reads the entire file into that one Buffer.
So, the synchronous version, which is allowed to hog the entire CPU, is just faster from a pure time to completion point of view (it's significantly less efficient from a CPU cycles used point of view in a multi-user server because it blocks the event loop from doing anything else during the read). The asynchronous version is reading in smaller chunks, probably to avoid blocking the event loop too much so other things can effectively interleave and run while fs.promises.readFile() is doing its thing.
For a project I worked on awhile ago, I wrote my own simple asynchronous version of readFile() that reads the entire file at once and it was significantly faster than the built-in implementation. I was not concerned about event loop blockage in that particular project so I did not investigate if that's an issue.
In addition, fs.readFile() reads the file in 524,288 byte chunks (much larger chunks that fs.promises.readFile()) and does not use await, using just plain callbacks. It is apparently just coded more optimally than the promise implementation. I don't know why they rewrote the function in a slower way for the fs.promises.readFile() implementation. For now, it appears that wrapping fs.readFile() with a promise would be faster.
I need to read and return an JSON file in a AWS lambda function (using Node 8+) which is done as below.
Just wanted to know if the readFileSync method in this async lambda function will have any impact on the server by blocking the nodejs thread?
I know that the readFileSync will actually wait for the file to be loaded before proceeding to the next line, but doesn't the async which is the wrapper method itself keep clear of any thread blocking scenarios? Not sure how this works. Please advise.
exports.handler= async (event) => {
var response = {
"statusCode": 200,
"body": ""
};
try {
var data = fs.readFileSync('test.json');
} catch (err) {
return response;
}
response.body = data;
return response;
};
No, in this case that's actually probably the most appropriate approach. AWS will spin up new containers when you receive multiple concurrent requests -- that's kinda crux of the serverless concept; you write functions are oblivious to concurrency management, so you're free to focus on what the function should actually do. In return the AWS lambda framework will make sure to scale up and down the actual number of containers executing your function based on current demand.
In fact, if you were to change the readFileSync to the async variant, you'd have to call the appropriate event callbacks to ensure AWS wouldn't kill your lambda while it was waiting for the file async result.
i'm trying to understand asynchronous callbacks in NodeJS by following this guide and i have a question about the position of callback() in the code snippet below.
var fs = require('fs')
var myNumber = undefined
function addOne(callback) {
fs.readFile('number.txt', function doneReading(err, fileContents) {
myNumber = parseInt(fileContents)
myNumber++
callback()
})
}
function logMyNumber() {
console.log(myNumber)
}
addOne(logMyNumber)
Here, my file 'number.txt' contains the number 1 and the output of this entire code snippet is 2. This appears to be invoking callback() after the file is read, and the output is expected. However, moving callback() outside of fs.readFile() but inside addOne() as shown below has confused me as the output is now undefined.
var fs = require('fs')
var myNumber = undefined
function addOne(callback) {
fs.readFile('number.txt', function doneReading(err, fileContents) {
myNumber = parseInt(fileContents)
myNumber++
})
callback()
}
function logMyNumber() {
console.log(myNumber)
}
addOne(logMyNumber)
Does this mean that in the second example, callback() is invoked before fs.readFile() has completed?
Your thinking is right. Node.js executes fs.readFile but DOES NOT wait for it's completion.
So the execution moves to next statement, which invokes the callback and the result is undefined because the previous command has not yet finished.
Asynchronous programming is very interesting, but easy thing.
Asynchronous executions means that program won't run your code synchronously line by line waiting for completing each line of code WHEN YOU ARE EXECUTING BLOCKING CODE, but instead it will do concurrent execution (concurrency).
Blocking code is when you are doing for example http requests, reading files as in your example or DOM events in browser. Basically blocking code is code which is not depended on your code. Basically blocking code is when your program should wait unknown duration till their completion to continue.
On basic javascript operations like arithmetic and array operations, loops e.g.
Code runs synchronously because they are not depended on external resources.
That is why we have callbacks in JavaScript to not wait for their completion.
Callbacks helps us to run code after asynchronous code executed.
Asynchronous means that our code is NON-BLOCKING.
So in your code when you run
fs.readFile('number.txt', function doneReading(err, fileContents) {
myNumber = parseInt(fileContents)
myNumber++
})
callback()
It will start running fs.readFile and immediately will start execution of callback function.
But when when you run
fs.readFile('number.txt', function doneReading(err, fileContents) {
myNumber = parseInt(fileContents)
myNumber++
callback()
})
It will start executing fs.readFile and within his callback doneReading function will do operations with myNumber and will execute callback
Most of JavaScript asynchronous functions have callback functions.
You can read about blocking and non-blocking codes here, also here about callbacks
Also you can read about promises and async/await.
They are very cool things, that helps you to struct your code as you would in synchronous environment, but runs asynchronous code.
I understand what a callback is and what asynchronous means, what I don't get is how to run asynchronous functions in node.
For example, how is this
var action = (function(data,callback) {
result = data+1;
callback(result);
});
http.createServer(function (req, res) {
action(5, function(r){
res.end(r.toString());
});
}).listen(80);
different from this
var action = (function(data) {
result = data+1;
return result;
});
http.createServer(function (req, res) {
var r = action(5);
res.end(r.toString());
}).listen(80);
?
I guess in the first example I'm doing it asynchronously, yet I don't know how Node knows when to do it sync or async... is it a matter of the return? or the fact that in the sync mode we're doing var x = func(data);?
And also: when to use sync or async? Because obviously you don't want to use it when adding +1... is it OK to use async just when performing IO tasks, such as reading from DB?
For example, I'm using the library crypto to encrypt a short string (50 chars at most), is this case a good example where I should already be using async?
I guess in the first example I'm doing it asynchronously...
Your first example isn't async :) Merely passing a callback and calling it when you're done doesn't make a function asynchronous.
Asynchronous means that, basically, you're telling Node: "here, do this for me, and let me know when you're done while I continue doing other stuff".
Your example is not handing anything to Node for future completion. It's doing a calculation and calling the callback immediately after that. That's functionally the same as your second example, where you return the result of the calculation.
However, you can change your first example to something that is asynchronous:
var action = (function(data,callback) {
setTimeout(function() {
result = data + 1;
callback(result);
}, 1000);
});
Here, you're telling Node to delay calling the callback for one second, by using setTimeout. In the mean time, Node won't get stuck waiting for a second; it will happily accept more HTTP requests, and each one will be delayed one second before the response is sent.
When to use sync or async?
Asynchronous code is "viral": if you rely on functions that are async, your own code that uses those functions will also have to be async (generally by accepting a callback, or using another mechanism to deal with asynchronicity, like promises).
For example, I'm using the library crypto to encrypt a short string (50 chars at most), is this case a good example where I should already be using async?
This depends on which function you're using. AFAIK, most encryption functions in crypto aren't asynchronous, so you can't "make" them asynchronous yourself.
Both examples will work synchronous. Simple async operations are setTimout and setInterval.
Node actually doesn't care what code are you running. You can block or not (blocking/non-blocking).
In other words - you have event loop. If your process is async he will pass the program control to the event loop, so it can execute any other action node needs to be done. If not - he wont.
if you want a function to work asynchronously, you can do that using promises, look at the code below :
function is_asynch(){
return new Promise((resolve,reject)=>{
resolve( here_your_synch_function() )
})
}