I am writing a node server that reads/deletes/adds/etc a file from the filesystem. Is there any performance advantage to reading asynchronously? I can't do anything while waiting for the file to be read. Example:
deleteStructure: function(req, res) {
var structure = req.param('structure');
fs.unlink(structure, function(err) {
if (err) return res.serverError(err);
return res.ok();
});
}
I am also making requests to another server using http.get. Is there any performance advantage to fetching asynchronously? I can't do anything while waiting for the file to be fetched. Example:
getStructure: function(req, res) {
var structure = urls[req.param('structure')];
http.get(structure).then(
function (response) {
return res.send(response);
},
function (err) {
res.serverError(err)
}
);
}
If there is no performance advantage to reading files asynchronously, I can just use the synchronous methods. However, I am not aware of synchronous methods for http calls, do any built in methods exist?
FYI I am using Sails.js.
Thanks!
I can't do anything while waiting for the file to be read.
I can't do anything while waiting for the file to be fetched.
Wrong; you can handle an unrelated HTTP request.
Whenever your code is in the middle of a synchronous operation, your server will not respond to other requests at all.
This asynchronous scalability is the biggest attraction for Node.js.
Related
Part of my application involves deleting a large folder and some rows that relate to the files in that folder in a database periodically.
To do this I was thinking of using this code:
const { rmdir } = require('fs/promises');
... begin transaction
... remove database rows
await rmdir(dir, { recursive: true });
... end transaction
The reason I want to use the Promises API instead of rmdirSync (which stops the entire server) is that I want the server to keep accepting requests while the operating system is busy deleting the folder. I know I could just use the callback api and just end the transaction before the folder is deleted, but that may lead to inconsistencies and also wouldn't catch any potential errors. (the files might still be served by express.static while the database rows are already gone)
But I read in this documentation:
https://nodejs.org/api/fs.html#fs_callback_example
That the use of the Promise API is not recommended in favour of the callback API.
So here's my questions:
Why is it slower to use the Promise API instead of the Callback API?
How much slower?
Could I just do this instead:
const { unlink } = require('fs');
... begin transaction
... remove database rows
await new Promise((res,rej)=>{
unlink('/tmp/hello', (err) => {
resolve()
})
});
... end transaction
The promises api does exactly the same thing as your included code:
await new Promise((resolve, reject)=>{
unlink('/tmp/hello', (err) => {
if (err) {
return reject(err)
}
resolve()
})
});
The memory and time difference is directly related to the promises. Callback API uses bare c++, but promises require an additional layer of logic.
There is probably no difference until you make a million operations per second.
Hi I'm a newbie in nodejs as far as I'm concerned nodejs is event-driven which is a powerful feature in it.
I have been learning nodejs from last few days and try to build restful apis in it with mongodb, but I'm not able to use its event-driven architecture in apis below is my sudo code
//routes
app.get('/someUrl', SomeClass.executeSomeController);
//controller
class SomeClass {
async executeSomeController(req, res){
let response = awaitSomeHelper.executeQueryAndBusinessLogic(req.body);
res.send(response)
}
}
As per my understanding I have written normal code as I used to write using Ror or PHP The only difference I found that the controller is running asynchronous which does not happens in Ror or Php.
How can I use event-driven architecture to build restful apis
Hope I can cover your question. Basically in some cases 'event-driven architecture' term can be explained differently. In one case it's a basic core NodeJS flow that explains all the async functions. In another case, the root of the question can be related to events, event emitter etc.
But the main idea that you have to wait for all the asynchronous actions. In order to avoid thread blocking it goes further and handles the rest of your code without waiting for heavy requests. And there we have to know how to handle this async functionality.
Basic Async Flow
As I understand, you've got questions related to async operations in NodeJS. That's a root of the technology - all the heavy operations will be handled asynchronously. It's all about V8 and Event Loop.
So in order to work with asynchronous operations, you may use callback functions, promises or async-await syntax.
Callback Functions
function asyncFunction(params, callback) {
//do async stuff
callback(err, result);
}
function callbackFunction(err, result) {
}
asyncFunction(params, callbackFunction);
Promises
promiseFunction()
.then(anotherPromiseFunction)
.then((result) => {
//handle result
})
.catch((err) => {
//handle error
});
async-await
function anotherAsyncFunction() {
//do async stuff
}
const asycnFunction = async (params) => {
const result = await anotherAsyncFunction();
return result;
};
Events/Event Emitter
const fs = require('fs');
const filePath = './path/to/your/file';
const stream = fs.createReadStream(filePath);
stream.on('data', (data) => {
//do something
});
stream.on('end', () => {
//do something;
});
stream.on('error', (err) => {
//do something;
});
You may use these methods depends on the situation and your needs. I recommend skipping callback functions as we have modern ways to work in async flow (promises and async-await). By the way, 'async-await' returns promises as well.
Here is the example of a simple Express JS Server (pretty old syntax), but still valid. Please feel free to check and write questions:
https://github.com/roman-sachenko/express-entity-based
Here is a list of articles I'd recommend you:
https://blog.risingstack.com/node-js-at-scale-understanding-node-js-event-loop/
https://blog.risingstack.com/mastering-async-await-in-nodejs/
I have a method that reads and write a log file, this method is called on every request by all users, then write the log the request path in a file. The questions are two:
Is safe read and then write a file in async mode considering concurrency questions?
If yes for the first question, the code bellow will work considering concurrency questions?
If yes for the first question how I have to do?
Please, disregard exceptions and performance questions, this is a didactic code.
var logFile = '/tmp/logs/log.js';
app.get("/", function(req){
var log = {path: req.path, date: new Date().getTime()};
log(log);
});
function log(data){
fs.exists(logFile, function(exists){
if(exists){
fs.readFile(logFile, function (err, data) {
if (err){
throw err
}
var logData = JSON.parse(data.toString());
logData.push(data);
writeLog(logData);
});
}else{
writeLog([data]);
}
});
}
function writeLog(base){
fs.writeFile(logFile, JSON.stringify(base, null, '\t'), function(err) {
if(err)
throw err;
});
}
I strongly suggest that you don't just "log asynchronously" because you want the log to be ordered based on the order things happened in your app, and there is no guarantee this will happen that way if you don't synchronize it somehow.
You can, for instance, use a promise chain to synchronize it:
var _queue = Promise.resolve();
function log(message){
_queue = _queue.then(function(){ // chain to the queue
return new Promise(function(resolve){
fs.appendFile("/tmp/logs/log.txt", new Date() + message + "\n", function(err, data){
if(err) console.log(err); // don't die on log exceptions
else resolve(); // signal to the queue it can progress
});
});
});
}
You can now call log and it will queue messages and write them some time asynchronously for you. It will never take more than a single file descriptor or exhaust the server either.
Consider using a logging solution instead of rolling your own logger btw.
In you're example you're already using the Asynchronous versions of those functions. If you're concerned about the order of your operations then you should use the synchronous versions of those functions.
readFileSync
writeFileSync
Also to note, JSON.parse() is a synchronous operation.You can make this "asynchronous" using the async module and doing a async.asyncify(JSON.parse(data.toString()));.
As noted by #BenjaminGruenbaum, async.asyncify(); doesn't actually make the operation of JSON.parse(); truly asynchronous but it does provide a more "async" style for the control flow of the operations.
I'm writing a command line script in Node (because I know JS and suck at Bash + I need jQuery for navigating through DOM)… right now I'm reading an input file and I iterate over each line.
How do I go about making one HTTP request (GET) per line so that I can load the resulting string with jQuery and extract the information I need from each page?
I've tried using the NPM httpsync package… so I could make one blocking GET call per line of my input file but it doesn't support HTTPS and of course the service I'm hitting only supports HTTPS.
Thanks!
A good way to handle a large number of jobs in a conrolled manner is the async queue.
I also recommend you look at request for making HTTP requests and cheerio for dealing with the HTML you get.
Putting these together, you get something like:
var q = async.queue(function (task, done) {
request(task.url, function(err, res, body) {
if (err) return done(err);
if (res.statusCode != 200) return done(res.statusCode);
var $ = cheerio.load(body);
// ...
done();
});
}, 5);
Then add all your URLs to the queue:
q.push({ url: 'https://www.example.com/some/url' });
// ...
I would most likely use the async library's function eachLimit function. That will allow you to throttle the number of active connections as well as getting a callback for when all the operations are done.
async.eachLimit(urls, function(url, done) {
request(url, function(err, res, body) {
// do something
done();
});
}, 5, function(err) {
// do something
console.log('all done!');
})
I was worried about making a million simultaneous requests without putting in some kind of throttling/limiting the number of concurrent connections, but it seems like Node is throttling me "out of the box" to something around 5-6 concurrent connections.
This is perfect, as it lets me keep my code a lot simpler while also fully leveraging the inherent asynchrony of Node.
I have the node.js code running on a server and would like to know if it is blocking or not. It is kind of similar to this:
function addUserIfNoneExists(name, callback) {
userAccounts.findOne({name:name}, function(err, obj) {
if (obj) {
callback('user exists');
} else {
// Add the user 'name' to DB and run the callback when done.
// This is non-blocking to here.
user = addUser(name, callback)
// Do something heavy, doesn't matter when this completes.
// Is this part blocking?
doSomeHeavyWork(user);
}
});
};
Once addUser completes the doSomeHeavyWork function is run and eventually places something back into the database. It does not matter how long this function takes, but it should not block other events on the server.
With that, is it possible to test if node.js code ends up blocking or not?
Generally, if it reaches out to another service, like a database or a webservice, then it is non-blocking and you'll need to have some sort of callback. However, any function will block until something (even if nothing) is returned...
If the doSomeHeavyWork function is non-blocking, then it's likely that whatever library you're using will allow for some sort of callback. So you could write the function to accept a callback like so:
var doSomHeavyWork = function(user, callback) {
callTheNonBlockingStuff(function(error, whatever) { // Whatever that is it likely takes a callback which returns an error (in case something bad happened) and possible a "whatever" which is what you're looking to get or something.
if (error) {
console.log('There was an error!!!!');
console.log(error);
callback(error, null); //Call callback with error
}
callback(null, whatever); //Call callback with object you're hoping to get back.
});
return; //This line will most likely run before the callback gets called which makes it a non-blocking (asynchronous) function. Which is why you need the callback.
};
You should avoid in any part of your Node.js code synchronous blocks which don't call system or I/O operations and which computation takes long time (in computer meaning), e.g iterating over big arrays. Instead move this type of code to the separate worker or divide it to smaller synchronous pieces using process.nextTick(). You can find explanation for process.nextTick() here but read all comments too.