I'm adding node.js to an existing site for its "server side Java Script" capabilities, and have been following through several of the tutorials to get a handle on it. ...Of course, as an experienced programmer learning a new thing, I'm experimenting a little beyond what the tutorials might give. So, instead of replacing the effort with every iteration, I've been adding to what the code does in hopes that when I'm done I have one bit of code that exercises all the materials covered.
In doing this I have uncovered a clear example of out-of-order execution that seems to me to be completely unacceptible, but then maybe I'm missing something, which is why I'm posting (though it's also valuable to others who follow that if I'm right, all node.js programmers should know about it).
Here's a link to the tutorial that led me to this. This is the start of it.
When I got to the file-access section, since we were learning how to create and delete files shortly after learning how to populate variable values from the URL, I decided to use a conditional value to delete a newly created or updated file based on what the user had provided in the URL. This seems entirely rational to me!
But, it doesn't work out correctly; even though the code to delete the file occurs AFTER the code to create the file, in fact the deletion occurs before the creation and so, in the end, the file still exists! Perhaps more correctly, an instance of file creation happens after the potential file deletion, even though the order in the code is otherwise. (Note that this is a testing environment, so it's not as if some other user accessed the system causing a creation without the deletion condition being met.) This is easily repeatable.
Here's the code:
var http = require("http");
var dt = require("./firstmodeule.js");
var url = require("url");
var fs = require("fs");
var fn = "index.html";
var log = "nodejs.log";
http.createServer(function (req, res)
{
res.writeHead(200, {"Content-Type": "text/html"});
res.write("<p>New current Date & Time: "+dt.myDateTime()+"<br>");
res.write(req.url+"<br>"); // This captures the right hand of the URL.
var q = url.parse(req.url, true).query;
var txt = q.year + " " + q.month;
res.write(txt+"</p>");
res.write("<p></p>");
fs.appendFile(log, dt.myDateTime()+" "+txt+"\n", function (err)
{
if (err) throw err;
console.log("saved?");
});
fs.readFile(fn, function(err, data)
{
fs.writeFile("delable.index", data, function(err)
{
if (err)
{
throw err;
} else {
console.log("wrote to delable.index");
}
});
res.write(data);
res.end();
});
if (q.month = "Jan")
{
fs.unlink("delable.index", function (err)
{
if (err)
{
throw err;
} else {
console.log("Deleted delable.index");
}
});
}
//res.end(); // End CANNOT be here if we"re doing a res.write(data);
}).listen(8080);
Of course you can get the tiny bit of code for firstmodule.js from the cited tutorial - or just delete it since it has no bearing on this.
Here's an example of the URL:
http://MySite:8080/?year=2020&month=Jan
Here's an example of the output I get:
$ node example.js
Deleted delable.index
saved?
wrote to delable.index
Deleted delable.index
saved?
wrote to delable.index
Contrary to the order of the code, we get the deleted part first!
OK, so, what gives? Does node.js just do things in whatever order it wants?
For anyone curious, this is all extremely current code as of this date. The OS is Fedora 28 downloaded in the last few days, and the node package is nodejs-8.11.4-1.fc28.x86_64.
Change fs.appendFile to fs.appendFileSync and change fs.readFile to fs.readFileSync and then the functions will be synchronous and hence in the order of execution to match the order of your code.
Related
I haven't found anything specific about this, it isn't really a problem but I would like to understand better what is going on here.
Basically, I'am testing some simple NodeJS code , like this :
//Summary : Open a file , write to the file, delete the file.
let fs = require('fs');
fs.open('mynewfile.txt' , 'w' , function(err,file){
if(err) throw err;
console.log('Created file!')
})
fs.appendFile('mynewfile.txt' , 'Depois de ter criado este ficheiro com o fs.open, acrescentei-lhe data com o fs.appendFile' , function(err){
if (err) throw err;
console.log('Added text to the file.')
})
fs.unlink('mynewfile.txt', function(err){
if (err) throw err;
console.log('File deleted!')
})
console.log(__dirname);
I thought this code would be executed in the order it was written from the top to the bottom, but when I look at the terminal I'am not sure that was the case because this is what I get :
$ node FileSystem.js
C:\Users\Simon\OneDrive\Desktop\Portfolio\Learning Projects\NodeJS_Tutorial
Created file!
File deleted!
Added text to the file.
//Expected order would be: Create file, add text to file , delete file , log dirname.
Instead of what ther terminal might make you think, in the end when I look at my folder the code order still seems to have been followed somehow because the file was deleted and I have nothing left on the directory.
So , I was wondering , why is it that the terminal doesn't log in the same order that the code is written from the top to the bottom.
Would this be the result of NodeJS asynchronous nature or is it something else ?
The code is (in princliple) executed from top to bottom, as you say. But fs.open, fs.appendFile, and fs.unlink are asynchronous. Ie, they are placed on the execution stack in the partiticular order, but there is no guarantee whatsoever, in which order they are finished, and thus you can't guarantee, in which order the callbacks are executed. If you run the code multiple times, there is a good chance, that you may encounter different execution orders ...
If you need a specific order, you have two different options
You call the later operation only in the callback of the prior, ie something like below
fs.open('mynewfile.txt' , 'w' , function(err,file){
if(err) throw err;
console.log('Created file!')
fs.appendFile('mynewfile.txt' , '...' , function(err){
if (err) throw err;
console.log('Added text to the file.')
fs.unlink('mynewfile.txt', function(err){
if (err) throw err;
console.log('File deleted!')
})
})
})
You see, that code gets quite ugly and hard to read with all that increasing nesting ...
You switch to the promised based approach
let fs = require('fs').promises;
fs.open("myfile.txt", "w")
.then(file=> {
return fs.appendFile("myfile.txt", "...");
})
.then(res => {
return fs.unlink("myfile");
})
.catch(e => {
console.log(e);
})
With the promise-version of the operations, you can also use async/await
async function doit() {
let file = await fs.open('myfile.txt', 'w');
await fs.appendFile('myfile.txt', '...');
await fs.unlink('myfile.txt', '...');
}
For all three possibilites, you probably need to close the file, before you can unlink it.
For more details please read about Promises, async/await and the Execution Stack in Javascript
It's a combination of 2 things:
The asynchronous nature of Node.js, as you correctly assume
Being able to unlink an open file
What likely happened is this:
The file was opened and created at the same time (open with flag w)
The file was opened a second time for appending (fs.appendFile)
The file was unlinked
Data was appended to the file (while it was already unlinked) and the file was closed
When data was being appended, the file still existed on disk as an inode, but had zero hard links (references) to it. It still takes up space then, but the OS checks the reference count when closing and frees up the space if the count has fallen to zero.
People sometimes run into a similar situation with daemons such as HTTP servers that employ log rotation: if something goes wrong when switching over logs, the old log file may be unlinked but not closed, so it's never cleaned up and it takes space forever (until you reboot or restart the process).
Note that the ordering of operations that you're observing is random, and it is possible that they would be re-ordered. Don't rely on it.
You could write this as (untested):
let fs = require('fs');
const main = async () => {
await fs.open('mynewfile.txt' , 'w');
await fs.appendFile('mynewfile.txt' , 'content');
await fs.unlink('mynewfile.txt');
});
main()
.then(() => console.log('success'()
.catch(console.error);
or within another async function:
const someOtherFn = async () => {
try{
await main();
} catch(e) {
// handle any rejection to your liking
}
}
(The catch block is not mandatory. You can opt to just let them throw to the top. It's just to showcase how async / await allows you to make synchronous code appear as if it was synchronous code without runing into callback hell.)
I am attempting to use nodeJS to do the following:
Start an executable file (executable file then creates a timestamp_Question.txt file), read the created .txt file and return the text to my front end. Accept user input and create a new timestamp_Answer.txt file with a new name containing the user input, and then delete the original txt file.
Everything seems to be working except the deletion of the original file which is getting the following error
Error: EBUSY: resource busy or locked, unlink 'c:\projects\pizzabox\95912_Questi
on.txt'
at Error (native)
Here are the various segments of my server code:
Starting the .exe file:
startProgram: function(req, res){
var date = new Date().toLocaleTimeString();
date = date.replace(/\D/g,'');
exec('C:/projects/pizzabox/server/webserver/Pizza_page_server.exe', [date], function(err, data) {
console.log(err)
});
res.send(date) /// front end needs date value to track file as date is attached to created txtfile
}
Front end requesting the contents of the txt created by the exe:
getDocument: function(req,res){
console.log('file requested', req.params.dateStamp);
var directory = 'C:/projects/pizzabox/'
var fileCode = req.params.dateStamp.toString()
var fileSuffix = "_Question.txt"
var file = directory+fileCode+fileSuffix
console.log('file is', file);
while(!fs.existsSync(file)){};
fs.readFile(file, 'utf8', function(err,data){
if (err) throw err;
console.log('read ', data);
res.send(data)
});
Front end submitting data to create new timestamp_Answer.txt file and deletion of old timestamp_Question.txt file:
sendValue: function(req,res){
console.log('got value', req.body.value, " questionNumber ", req.body.questionNumber, "timestamp ", req.body.timestamp);
fs.appendFile(req.body.timestamp+'_Answer.txt', "Question_Num:"+req.body.questionNumber+"\nAnswer_Val:"+req.body.value, function(err){
if(err) throw err;
console.log('file created!');
fs.unlink(req.body.timestamp+'_Question.txt', function(err){
if (err) throw err;
console.log('question file deleted');
})
})
}
Any reason why unlink wouldn't have access to the file? does fs.readFile() not close/exit the file after it completes?
It appears you have at least one race condition here. Just because fs.fileExistsSync() sees that a file exists, does not mean that the .exe is done writing to that file or has closed that file. So, it's very likely that your fs.readFile() gets called while the file is still open by the other process.
You probably want to rethink how all your operations are sequenced.
First off, the design you have now is a single user design. Your server can only ever be used by one single user at a time since you have no way of differentiating requests that are coming from different users.
Second off, you should move to a design that is more purposefully sequential rather than trying to use the existence of a file in the file system as evidence that some previous operation might now be done.
Third, off while(!fs.existsSync(file)){}; is never used in anything except a one-off script because it halts all processing of any type of request or timer or any event in the server until that file exists. Further, it probably hogs all the CPU. It's never used in a multi-user server.
You haven't really described the overall client/server flow you're trying to design or described whether this is a single-user system or intended to eventually be for multiple users so it's hard for us to make suggestions about exactly where to go for an architectural redesign.
You could probably make one simple improvement by not returning the date to the client until your .exe is done which hopefully will stop it from requesting its next step until the .exe is done:
startProgram: function(req, res){
var date = new Date().toLocaleTimeString();
date = date.replace(/\D/g,'');
exec('C:/projects/pizzabox/server/webserver/Pizza_page_server.exe', [date], function(err, data) {
if (err) {
console.log(err);
res.sendStatus(500);
} else {
// don't send this until the .exe is done
res.send(date) // front end needs date value to track file as date is attached to created txtfile
}
});
}
I am not sure where I am going wrong but I think that the event listener is getting invoked multiple times and parsing the files multiple times.
I have five files in the directory and they are getting parsed. However the pdf file with array 0 gets parsed once and the next one twice and third one three times.
I want the each file in the directory to be parsed once and create a text file by extracting the data from pdf.
The Idea is to parse the pdf get the content as text and convert the text in to json in a specific format.
To make it simple, the plan is to complete one task first then use the output from the below code to perform the next task.
Hope anyone can help and point out where i am going wrong and explain a bit about my mistake so i understand it. (new to the JS and Node)
Regards,
Jai
Using the module from here:
https://github.com/modesty/pdf2json
var fs = require('fs')
PDFParser = require('C:/Users/Administrator/node_modules/pdf2json/PDFParser')
var pdfParser = new PDFParser(this, 1)
fs.readdir('C:/Users/Administrator/Desktop/Project/Input/',function(err,pdffiles){
//console.log(pdffiles)
pdffiles.forEach(function(pdffile){
console.log(pdffile)
pdfParser.once("pdfParser_dataReady",function(){
fs.writeFile('C:/Users/Administrator/Desktop/Project/Jsonoutput/'+pdffile, pdfParser.getRawTextContent())
pdfParser.loadPDF('C:/Users/Administrator/Desktop/Project/Input/'+pdffile)
})
})
})
As mentioned in the comment, just contributing 'work-around' ideas for OP to temporary resolve this issue.
Assuming performance is not an issue then you should be able to asynchronously parse the pdf files in a sequential matter. That is, only parse the next file when the first one is done.
Unfortunately I have never used the npm module PDFParser before so it is really difficult for me to try the code below. Pardon me as it may require some minor tweaks to make it to work, syntactically they should be fine as they were written using an IDE.
Example:
var fs = require('fs');
PDFParser = require('C:/Users/Administrator/node_modules/pdf2json/PDFParser');
var parseFile = function(files, done) {
var pdfFile = files.pop();
if (pdfFile) {
var pdfParser = new PDFParser();
pdfParser.on("pdfParser_dataError", errData => { return done(errData); });
pdfParser.on("pdfParser_dataReady", pdfData => {
fs.writeFile("'C:/Users/Administrator/Desktop/Project/Jsonoutput/" + pdfFile, JSON.stringify(pdfData));
parseFile(files, done);
});
pdfParser.loadPDF('C:/Users/Administrator/Desktop/Project/Input/' + pdfFile);
}
else {
return done(null, "All pdf files parsed.")
}
};
fs.readdir('C:/Users/Administrator/Desktop/Project/Input/',function(err,pdffiles){
parseFile(pdffiles, (err, message) => {
if (err) { console.error(err.parseError); }
else { console.log(message); }
})
});
In the code above, I have isolated out the parsing logic into a separated function called parseFile. In this function it first checks to see if there are still files to process or not, if none then it invokes the callback function done otherwise it will do an array.pop operation to get the next file in queue and starts parsing it.
When parsing is done then it recursively call the parseFile function until the last file is parsed.
I've been trying to pick up NodeJS and learning more for backend development purposes. I can't seem to wrap my mind around Async tasks though and I have an example here that I've spent hours over trying to search for the solution.
app.get('/initialize_all_pictures', function(req, res){
var path = './images/';
fs.readdir(path, function(err, items){
if (err){
console.log("there was an error");
return;
}
console.log(items.length);
for(var i = 0; i<items.length; i++){
var photo = new Photo(path + items[i], 0, 0,Math.floor(Math.random()*1000))
photoArray.push(photo);
}
});
res.json({"Success" : "Done"});
});
Currently, I have this endpoint that is supposed to look through a directory called images and create "Photo" objects and push it into a global array called PhotoArray. It works, except the function for readdir is always being called twice.
console.log would always give output of
2
2
(I have two items in the directory).
Why is this?
Just figured out the problem.
I had a chrome extension that would help me format JSON values from HTTP requests. Unfortunately, the extension actually made an additional call to the endpoint therefore whenever I would point my browser to the endpoint, the function would end up getting called twice!
I keep banging my head against the wall because of tons of different errors. This is what the code i try to use :
fs.readFile("balance.txt", function (err, data) //At the beginning of the script (checked, it works)
{
if (err) throw err;
balance=JSON.parse(data);;
});
fs.readFile("pick.txt", function (err, data)
{
if (err) throw err;
pick=JSON.parse(data);;
});
/*....
.... balance and pick are modified
....*/
if (shutdown)
{
fs.writeFile("balance2.txt", JSON.stringify(balance));
fs.writeFile("pick2.txt", JSON.stringify(pick));
process.exit(0);
}
At the end of the script, the files have not been modified the slightest. I then found out on this site that the files were being opened 2 times simultaneously, or something like that, so i tried this :
var balance, pick;
var stream = fs.createReadStream("balance.txt");
stream.on("readable", function()
{
balance = JSON.parse(stream.read());
});
var stream2 = fs.createReadStream("pick.txt");
stream2.on("readable", function()
{
pick = JSON.parse(stream2.read());
});
/****
****/
fs.unlink("pick.txt");
fs.unlink("balance.txt");
var stream = fs.createWriteStream("balance.txt", {flags: 'w'});
var stream2 = fs.createWriteStream("pick.txt", {flags: 'w'});
stream.write(JSON.stringify(balance));
stream2.write(JSON.stringify(pick));
process.exit(0);
But, this time, both files are empty... I know i should catch errors, but i just don't see where the problem is. I don't mind storing the 2 objects in the same file, if that can helps. Besides that, I never did any javascript in my life before yesterday, so, please give me a simple explanation if you know what failed here.
What I think you want to do is use readFileSync and not use readFile to read your files since you need them to be read before doing anything else in your program (http://nodejs.org/api/fs.html#fs_fs_readfilesync_filename_options).
This will make sure you have read both the files before you execute any of the rest of your code.
Make your like code do this:
try
{
balance = JSON.parse(fs.readFileSync("balance.txt"));
pick = JSON.parse(fs.readFileSync("pick.txt"));
}
catch(err)
{ throw err; }
I think you will get the functionality you are looking for by doing this.
Note, you will not be able to check for an error in the same way you can with readFile. Instead you will need to wrap each call in a try catch or use existsSync before each operation to make sure you aren't trying to read a file that doesn't exist.
How to capture no file for fs.readFileSync()?
Furthermore, you have the same problem on the writes. You are kicking off async writes and then immediately calling process.exit(0). A better way to do this would be to either write them sequentially asynchronously and then exit or to write them sequentially synchronously then exit.
Async option:
if (shutdown)
{
fs.writeFile("balance2.txt", JSON.stringify(balance), function(err){
fs.writeFile("pick2.txt", JSON.stringify(pick), function(err){
process.exit(0);
});
});
}
Sync option:
if (shutdown)
{
fs.writeFileSync("balance2.txt", JSON.stringify(balance));
fs.writeFileSync("pick2.txt", JSON.stringify(pick));
process.exit(0);
}