I'm writing a small nodejs script to copy files from another folder, modify some text and minify the contents.
As per my research, there seems to be module called async with which functions can be called one by one. But I'm trying how to do the same without installing any modules (as a part of learning).
I'm new to this promises/async world. But I could not get it to work sequentially as expected.
Can someone please help ?
function updateText( textList ){
return new Promise(function(resolve, reject){
var folders = [ 'includes/lr', 'includes/vc' ];
var acceptedFileTypes = ['php', 'js', 'css'];
folders.forEach(function(dir){
files = walk(dir);
files.forEach( function ( file ) {
fs.readFile(file, 'utf8', function (err,data) {
fileType = file.split('.').pop();
if(!acceptedFileTypes.includes(fileType)){
return;
}
console.log('Editing - ' + file);
if (err) {
return console.log(err);
}
for(text in textList){
data = data.replace(new RegExp(text, 'g'), textList[text]);
}
fs.writeFile(file, data, 'utf8', function (err) {
console.log('Written file - ' + file);
if (err) return console.log(err);
});
});
});
});
console.log('Resolving')
resolve();
});
}
function minifyJS(){
return new Promise(function(resolve, reject){
minify('includes/vc/js/script-front.js').then(function(data){
console.log('Minifying file - ');
fs.writeFile('includes/vc/js/script-front.min.js', data, 'utf8', function (err) {
if (err) return console.log(err);
resolve();
});
});
});
}
updateText({
'qwerty': 'hello',
})
.then(function(r){
minifyJS().then(function(){
console.log('# Done');
});
})
Output - things are all over the place and not in sequence, though promise is resolved only after editing the files.
Here I'm expecting, all edits to finish first, writes second and minify at the last.
Resolving
Editing - includes/lr/css/style.css
Editing - includes/vc/css/style.css
Editing - includes/vc/js/script-front.js
Editing - includes/vc/js/script-front.min.js
Minifying file -
Editing - includes/vc/js/script.js
Editing - includes/lr/js/script.js
Editing - includes/vc/index.php
Editing - includes/lr/index.php
Written file - includes/lr/css/style.css
Written file - includes/vc/css/style.css
Written file - includes/vc/js/script-front.js
Written file - includes/vc/js/script.js
Written file - includes/lr/js/script.js
Written file - includes/vc/js/script-front.min.js
# Done
Written file - includes/vc/index.php
Written file - includes/lr/index.php
there seems to be module called async with which functions can be called one by one. But I'm trying how to do the same without installing any modules (as a part of learning).
async/await does not require any module loading and is available as part of the NodeJS runtime starting with NodeJS 8.0.0
here is a post with some examples of how to use
I see the use of fs.readFile and fs.writeFile, both of which make use of the callback function.
That callback functions will be invoked once fs.readFile/fs.writeFile completes their task.
Calls to fs.readFile and fs.writeFile are placed on the event-loop and are not guaranteed to complete in the order in which they were added.
To demonstrate the blocking of the event-loop, you can make use of the synchronous calls from the fs module
fs.readFileSync
fs-writeFileSync
I would suggest restructuring the code to make use of util.promisify which will will allow the fs.readFile and fs.writeFile to be "promisified" and then can be used with async/await.
const listOfFilesRead = await readFiles
await writeFiles(listOfFilesRead)
await minify
readFiles would be a declared async function that makes use of util.promisify on fs.readFile, returning an array of data to be used in writeFiles
writeFiles would be a declared async function that makes use of util.promisify on fs.writeFile to await on each write, before moving onto the next, taking in an array of data of files to be written
minify is an async function that will minify the JS
That will give the order you are wanting of readAll, writeAll, minify
As it is written, it is attempting to do the following
read a file
write that file
repeat for all files
then minify
Since you are learning, the links provided should give examples of how to write callback, promise-based, and async-await of asynchronous functions.
Related
I can read the files using nodejs file system:
const fs = require('fs');
fs.readFile('./assets/test1.txt', (err, data) => {
if(err){
console.log(err)
}
console.log(data.toString())
})
console.log('hello shawn!')
Why console.log('hello shawn!') read first times then read the console.log(data.toString())?
Is there any other things in file system read data first then read below console?
It is because .readFile is a asynchronous operation. Its last parameter is callback function, which is started after operation is done. I recommend read something about callbacks and event loop.
You can use a synchronous version of function readFileSync or use utils.promisify to convert callback function to promise and use async/await then example.
I haven't found anything specific about this, it isn't really a problem but I would like to understand better what is going on here.
Basically, I'am testing some simple NodeJS code , like this :
//Summary : Open a file , write to the file, delete the file.
let fs = require('fs');
fs.open('mynewfile.txt' , 'w' , function(err,file){
if(err) throw err;
console.log('Created file!')
})
fs.appendFile('mynewfile.txt' , 'Depois de ter criado este ficheiro com o fs.open, acrescentei-lhe data com o fs.appendFile' , function(err){
if (err) throw err;
console.log('Added text to the file.')
})
fs.unlink('mynewfile.txt', function(err){
if (err) throw err;
console.log('File deleted!')
})
console.log(__dirname);
I thought this code would be executed in the order it was written from the top to the bottom, but when I look at the terminal I'am not sure that was the case because this is what I get :
$ node FileSystem.js
C:\Users\Simon\OneDrive\Desktop\Portfolio\Learning Projects\NodeJS_Tutorial
Created file!
File deleted!
Added text to the file.
//Expected order would be: Create file, add text to file , delete file , log dirname.
Instead of what ther terminal might make you think, in the end when I look at my folder the code order still seems to have been followed somehow because the file was deleted and I have nothing left on the directory.
So , I was wondering , why is it that the terminal doesn't log in the same order that the code is written from the top to the bottom.
Would this be the result of NodeJS asynchronous nature or is it something else ?
The code is (in princliple) executed from top to bottom, as you say. But fs.open, fs.appendFile, and fs.unlink are asynchronous. Ie, they are placed on the execution stack in the partiticular order, but there is no guarantee whatsoever, in which order they are finished, and thus you can't guarantee, in which order the callbacks are executed. If you run the code multiple times, there is a good chance, that you may encounter different execution orders ...
If you need a specific order, you have two different options
You call the later operation only in the callback of the prior, ie something like below
fs.open('mynewfile.txt' , 'w' , function(err,file){
if(err) throw err;
console.log('Created file!')
fs.appendFile('mynewfile.txt' , '...' , function(err){
if (err) throw err;
console.log('Added text to the file.')
fs.unlink('mynewfile.txt', function(err){
if (err) throw err;
console.log('File deleted!')
})
})
})
You see, that code gets quite ugly and hard to read with all that increasing nesting ...
You switch to the promised based approach
let fs = require('fs').promises;
fs.open("myfile.txt", "w")
.then(file=> {
return fs.appendFile("myfile.txt", "...");
})
.then(res => {
return fs.unlink("myfile");
})
.catch(e => {
console.log(e);
})
With the promise-version of the operations, you can also use async/await
async function doit() {
let file = await fs.open('myfile.txt', 'w');
await fs.appendFile('myfile.txt', '...');
await fs.unlink('myfile.txt', '...');
}
For all three possibilites, you probably need to close the file, before you can unlink it.
For more details please read about Promises, async/await and the Execution Stack in Javascript
It's a combination of 2 things:
The asynchronous nature of Node.js, as you correctly assume
Being able to unlink an open file
What likely happened is this:
The file was opened and created at the same time (open with flag w)
The file was opened a second time for appending (fs.appendFile)
The file was unlinked
Data was appended to the file (while it was already unlinked) and the file was closed
When data was being appended, the file still existed on disk as an inode, but had zero hard links (references) to it. It still takes up space then, but the OS checks the reference count when closing and frees up the space if the count has fallen to zero.
People sometimes run into a similar situation with daemons such as HTTP servers that employ log rotation: if something goes wrong when switching over logs, the old log file may be unlinked but not closed, so it's never cleaned up and it takes space forever (until you reboot or restart the process).
Note that the ordering of operations that you're observing is random, and it is possible that they would be re-ordered. Don't rely on it.
You could write this as (untested):
let fs = require('fs');
const main = async () => {
await fs.open('mynewfile.txt' , 'w');
await fs.appendFile('mynewfile.txt' , 'content');
await fs.unlink('mynewfile.txt');
});
main()
.then(() => console.log('success'()
.catch(console.error);
or within another async function:
const someOtherFn = async () => {
try{
await main();
} catch(e) {
// handle any rejection to your liking
}
}
(The catch block is not mandatory. You can opt to just let them throw to the top. It's just to showcase how async / await allows you to make synchronous code appear as if it was synchronous code without runing into callback hell.)
I have a function that writes data to a file then uploads that file to cloud storage. The file isn't finished writing before it starts uploading so I am getting a partial file in cloud storage. I found that fs.writeFileSync(path, data[, options]) could help, but I am not exactly sure how it works.
It is my understanding that node runs asynchronously and I have several async processes running prior to this portion of code. I understand what synchronous vs asynchronous means, but I am having a little trouble understanding how it plays in this example. Here are my questions if I replace the below code with fs.writeFileSync(path, data[, options])
What do the docs mean by "Synchronously append data to a file"
a. Will the next lines of code be halted until the fs.writeFileSync(path, data) is finished?
b. Are previous asynchronous processes halted by this line of code?
If other async processes are not affected how is writeFileSync different the writeFile?
Is there a callback feature in writeFileSync that I am misunderstanding?
Code for reference
outCsv = "x","y","z"
filename = "file.csv"
fs.writeFile(filename, outCsv, function (err) {
if (err) {
return console.log(err);
}
console.log('The file was saved!');
bucket.upload(filename, (err, file) => {
if (err) {
return console.log(err);
}
console.log('The file was uploaded!');
});
});
Will the next lines of code be halted until the fs.writeFileSync(path, data) is finished?
Yes. It is a blocking operation. Note that you're assuming that fs.writeFileSync does finish.
Are previous asynchronous processes halted by this line of code?
Kinda. Since JavaScript is single-threaded, they will also not be running while the file is writing but will queue up at the next tick of the event loop.
If other async processes are not affected how is writeFileSync different the writeFile?
It blocks any code that comes after it. For an easier example consider the following:
setTimeout(() => console.log('3'), 5);
console.log('1'); // fs.writeFileSync
console.log('2');
vs
setTimeout(() => console.log('3'), 5);
setTimeout(() => console.log('1'), 0); // fs.writeFile
console.log('2');
The first will print 1 2 3 because the call to console.log blocks what comes after. The second will print 2 1 3 because the setTimeout is non-blocking. The code that prints 3 isn't affected either way: 3 will always come last.
Is there a callback feature in writeFileSync that I am misunderstanding?
Don't know. You didn't post enough code for us to say.
This all begs the question of why to prefer fs.writeFile over the alternative. The answer is this:
The sync version blocks.
While it's taking however long it takes your e.g. webserver isn't handling requests.
For a school project I'm creating a portal for KVM using NodeJS and Express.
I need to adjust an XML file and then use that XML File to create an VM.
So i created 2 functions
CreateXML:
function createXML(req, res, next) {
var parses = new xml2js.Parser();
fs.readFile('Debian7.xml', function(err, data){
parser.parseString(data, function (err, result){
result.domain.name = req.body.name;
result.domain.memory[0]['$'].unit = "GB";
result.domain.memory[0]['_'] = req.body.ram;
result.domain.currentMemory[0]['$'].unit = "GB";
result.domain.currentMemory[0]['_'] = req.body.ram;
result.domain.vcpu = req.body.cpus;
var builder = new xml2js.Builder({headless: true});
var xml = builder.buildObject(result);
fs.writeFile('./xmlfiles/' + req.body.name + '.xml', xml, function(err, data){
if(err) console.log(err);
});
});
});
};
CreateDomain:
function createDomain(req, res){
var domainXML = fs.readFileSync('./xmlfiles/' + req.body.name + '.xml', 'utf8');
hypervisor.connect(function(){
hypervisor.createDomainAsync(domainXML).then(function (domain){
console.log('Domain Created');
res.json({success: true, msg: 'succesfully created domain'})
});
});
}
then I call these functions as middleware in my post request
apiRoutes.post('/domainCreate', createXML, createDomain);
But then when I use Postman on the api route I get the following error:
Error: ENOENT: no such file or directory, open './xmlfiles/rickyderouter23.xml'
After the error it still creates the XML file and when I create the XML file before I use postman it works fine. It's like it needs to execute both functions before the creation of the XML file, how do I create the XML file after the first function and then use it in the second function.
The answer is "it's asynchronous" (just like many, many problems in node.js/javascript).
The fs.readFile function is asynchronous: when you call it, you give it a callback function which it will call when it finishes loading the file.
The parser.parseString is asynchronous - it will call your callback function when it finishes parsing the XML.
The fs.writeFile is the same - it will call your callback function when it finishes writing the file.
The hypervisor.connect function is the same - it will call your callback function when it finishes connecting.
The middleware functions are called in order, but they both contain code that may not have completed before they return. So when your code calls createDomain and tries to read the XML file created in createXML, the XML file probably doesn't exist yet. The fs.readFile might not be finished yet; even if it is, the parser.parseString function might not be finished yet; even if that one is finished, the fs.writeFile might not be finished yet.
One way to solve this would be to put the functionality of the createXML and createDomain functions together into one middleware function. That would allow you to rewrite it so that all the function calls that depend on previous asynchronous function calls could actually wait for those calls to complete before executing. A simple way to do it would be this:
function createXML(req, res, next) {
var parses = new xml2js.Parser();
fs.readFile('Debian7.xml', function(err, data){
parser.parseString(data, function (err, result){
result.domain.name = req.body.name;
result.domain.memory[0]['$'].unit = "GB";
result.domain.memory[0]['_'] = req.body.ram;
result.domain.currentMemory[0]['$'].unit = "GB";
result.domain.currentMemory[0]['_'] = req.body.ram;
result.domain.vcpu = req.body.cpus;
var builder = new xml2js.Builder({headless: true});
var xml = builder.buildObject(result);
fs.writeFile('./xmlfiles/' + req.body.name + '.xml', xml, function(err, data){
if(err) console.log(err);
// notice the call to createDomain here - this ensure
// that the connection to the hypervisor is not started
// until the file is written
createDomain(req, res);
});
});
});
};
And change your route to:
apiRoutes.post('/domainCreate', createXML);
Now, that's pretty ugly. I don't like the idea of lumping those two middleware functions into one and I'd prefer to rewrite it to use a promise-based approach, but that's the basic the idea.
So Im trying to use the nodejs express FS module to iterate a directory in my app, store each filename in an array, which I can pass to my express view and iterate through the list, but Im struggling to do so. When I do a console.log within the files.forEach function loop, its printing the filename just fine, but as soon as I try to do anything such as:
var myfiles = [];
var fs = require('fs');
fs.readdir('./myfiles/', function (err, files) { if (err) throw err;
files.forEach( function (file) {
myfiles.push(file);
});
});
console.log(myfiles);
it fails, just logs an empty object. So Im not sure exactly what is going on, I think it has to do with callback functions, but if someone could walk me through what Im doing wrong, and why its not working, (and how to make it work), it would be much appreciated.
The myfiles array is empty because the callback hasn't been called before you call console.log().
You'll need to do something like:
var fs = require('fs');
fs.readdir('./myfiles/',function(err,files){
if(err) throw err;
files.forEach(function(file){
// do something with each file HERE!
});
});
// because trying to do something with files here won't work because
// the callback hasn't fired yet.
Remember, everything in node happens at the same time, in the sense that, unless you're doing your processing inside your callbacks, you cannot guarantee asynchronous functions have completed yet.
One way around this problem for you would be to use an EventEmitter:
var fs=require('fs'),
EventEmitter=require('events').EventEmitter,
filesEE=new EventEmitter(),
myfiles=[];
// this event will be called when all files have been added to myfiles
filesEE.on('files_ready',function(){
console.dir(myfiles);
});
// read all files from current directory
fs.readdir('.',function(err,files){
if(err) throw err;
files.forEach(function(file){
myfiles.push(file);
});
filesEE.emit('files_ready'); // trigger files_ready event
});
As several have mentioned, you are using an async method, so you have a nondeterministic execution path.
However, there is an easy way around this. Simply use the Sync version of the method:
var myfiles = [];
var fs = require('fs');
var arrayOfFiles = fs.readdirSync('./myfiles/');
//Yes, the following is not super-smart, but you might want to process the files. This is how:
arrayOfFiles.forEach( function (file) {
myfiles.push(file);
});
console.log(myfiles);
That should work as you want. However, using sync statements is not good, so you should not do it unless it is vitally important for it to be sync.
Read more here: fs.readdirSync
fs.readdir is asynchronous (as with many operations in node.js). This means that the console.log line is going to run before readdir has a chance to call the function passed to it.
You need to either:
Put the console.log line within the callback function given to readdir, i.e:
fs.readdir('./myfiles/', function (err, files) { if (err) throw err;
files.forEach( function (file) {
myfiles.push(file);
});
console.log(myfiles);
});
Or simply perform some action with each file inside the forEach.
I think it has to do with callback functions,
Exactly.
fs.readdir makes an asynchronous request to the file system for that information, and calls the callback at some later time with the results.
So function (err, files) { ... } doesn't run immediately, but console.log(myfiles) does.
At some later point in time, myfiles will contain the desired information.
You should note BTW that files is already an Array, so there is really no point in manually appending each element to some other blank array. If the idea is to put together the results from several calls, then use .concat; if you just want to get the data once, then you can just assign myfiles = files directly.
Overall, you really ought to read up on "Continuation-passing style".
I faced the same problem, and basing on answers given in this post I've solved it with Promises, that seem to be of perfect use in this situation:
router.get('/', (req, res) => {
var viewBag = {}; // It's just my little habit from .NET MVC ;)
var readFiles = new Promise((resolve, reject) => {
fs.readdir('./myfiles/',(err,files) => {
if(err) {
reject(err);
} else {
resolve(files);
}
});
});
// showcase just in case you will need to implement more async operations before route will response
var anotherPromise = new Promise((resolve, reject) => {
doAsyncStuff((err, anotherResult) => {
if(err) {
reject(err);
} else {
resolve(anotherResult);
}
});
});
Promise.all([readFiles, anotherPromise]).then((values) => {
viewBag.files = values[0];
viewBag.otherStuff = values[1];
console.log(viewBag.files); // logs e.g. [ 'file.txt' ]
res.render('your_view', viewBag);
}).catch((errors) => {
res.render('your_view',{errors:errors}); // you can use 'errors' property to render errors in view or implement different error handling schema
});
});
Note: you don't have to push found files into new array because you already get an array from fs.readdir()'c callback. According to node docs:
The callback gets two arguments (err, files) where files is an array
of the names of the files in the directory excluding '.' and '..'.
I belive this is very elegant and handy solution, and most of all - it doesn't require you to bring in and handle new modules to your script.