I have a several processes node.js. I want to only add one string to one file in each request ( like log file in nginx, apache, etc ). What is the best way do it ?
Simple:
fs.open(file, "a", 0744, function (err, fd) {
if (err) throw err;
fs.write(fd, data, null, 'utf8', function (err, written) {
if (err) throw err;
});
});
or else ?
This will work, however it may not be the best solution if it is constantly opening and closing the file. For something with quicker writes I would try benchmarking it against fs.createWriteStream, especially because this gives you a scope you can use in routes.
var fs = require("fs");
//set dummy data as random number
var data = Math.floor(Math.random()*11);
//Set our log file as a writestream variable with the 'a' flag
var logFile = fs.createWriteStream('log.txt', {
flags: "a",
encoding: "encoding",
mode: 0744
})
//call the write option where you need to append new data
logFile.write(new Date().toSting + ': ' data);
logFile.write(new Date().toSting + ': ' data);
Another solution is fs.appendFile. As Sdedelbrock notes though, fs.createWriteStream is much, much faster than this method since it doesn't need to constantly open and close the file. I coded a small benchmark on my machine, and it is about 3 times faster, definitely worth it.
Related
I am attempting to use nodeJS to do the following:
Start an executable file (executable file then creates a timestamp_Question.txt file), read the created .txt file and return the text to my front end. Accept user input and create a new timestamp_Answer.txt file with a new name containing the user input, and then delete the original txt file.
Everything seems to be working except the deletion of the original file which is getting the following error
Error: EBUSY: resource busy or locked, unlink 'c:\projects\pizzabox\95912_Questi
on.txt'
at Error (native)
Here are the various segments of my server code:
Starting the .exe file:
startProgram: function(req, res){
var date = new Date().toLocaleTimeString();
date = date.replace(/\D/g,'');
exec('C:/projects/pizzabox/server/webserver/Pizza_page_server.exe', [date], function(err, data) {
console.log(err)
});
res.send(date) /// front end needs date value to track file as date is attached to created txtfile
}
Front end requesting the contents of the txt created by the exe:
getDocument: function(req,res){
console.log('file requested', req.params.dateStamp);
var directory = 'C:/projects/pizzabox/'
var fileCode = req.params.dateStamp.toString()
var fileSuffix = "_Question.txt"
var file = directory+fileCode+fileSuffix
console.log('file is', file);
while(!fs.existsSync(file)){};
fs.readFile(file, 'utf8', function(err,data){
if (err) throw err;
console.log('read ', data);
res.send(data)
});
Front end submitting data to create new timestamp_Answer.txt file and deletion of old timestamp_Question.txt file:
sendValue: function(req,res){
console.log('got value', req.body.value, " questionNumber ", req.body.questionNumber, "timestamp ", req.body.timestamp);
fs.appendFile(req.body.timestamp+'_Answer.txt', "Question_Num:"+req.body.questionNumber+"\nAnswer_Val:"+req.body.value, function(err){
if(err) throw err;
console.log('file created!');
fs.unlink(req.body.timestamp+'_Question.txt', function(err){
if (err) throw err;
console.log('question file deleted');
})
})
}
Any reason why unlink wouldn't have access to the file? does fs.readFile() not close/exit the file after it completes?
It appears you have at least one race condition here. Just because fs.fileExistsSync() sees that a file exists, does not mean that the .exe is done writing to that file or has closed that file. So, it's very likely that your fs.readFile() gets called while the file is still open by the other process.
You probably want to rethink how all your operations are sequenced.
First off, the design you have now is a single user design. Your server can only ever be used by one single user at a time since you have no way of differentiating requests that are coming from different users.
Second off, you should move to a design that is more purposefully sequential rather than trying to use the existence of a file in the file system as evidence that some previous operation might now be done.
Third, off while(!fs.existsSync(file)){}; is never used in anything except a one-off script because it halts all processing of any type of request or timer or any event in the server until that file exists. Further, it probably hogs all the CPU. It's never used in a multi-user server.
You haven't really described the overall client/server flow you're trying to design or described whether this is a single-user system or intended to eventually be for multiple users so it's hard for us to make suggestions about exactly where to go for an architectural redesign.
You could probably make one simple improvement by not returning the date to the client until your .exe is done which hopefully will stop it from requesting its next step until the .exe is done:
startProgram: function(req, res){
var date = new Date().toLocaleTimeString();
date = date.replace(/\D/g,'');
exec('C:/projects/pizzabox/server/webserver/Pizza_page_server.exe', [date], function(err, data) {
if (err) {
console.log(err);
res.sendStatus(500);
} else {
// don't send this until the .exe is done
res.send(date) // front end needs date value to track file as date is attached to created txtfile
}
});
}
I'm trying to mimic a terminal in node so I need to create a 'touch' function in node.js and I can't find anything that specifically uses touch. How can I set that up?
I've used a couple different things in the past but they keep getting kicked back because I'm not actually using 'fs.touch' or whatever it is.
this was my first attempt.
module.exports.touch = (filename, err) => {
if (err) {
throw err;
} else {
fs.openSync(filename, 'w');
`open filename`
}
};
this was my most recent attempt
module.exports.touch = (filename, callback) => {
open(filename, 'w', (err, fd) => {
err ? callback(err) : close(fd, callback);
});
};
The second one was essentially what they wanted because it did create a touch function but again they want me to actually use fs.touch but I cant find anything about it.
Make time either the current time or the time you want to be set:
fs.utimesSync(filename, time, time);
Just open the path file in write mode, and close it. You will have an empty file, equivalent at touch in command line
I keep banging my head against the wall because of tons of different errors. This is what the code i try to use :
fs.readFile("balance.txt", function (err, data) //At the beginning of the script (checked, it works)
{
if (err) throw err;
balance=JSON.parse(data);;
});
fs.readFile("pick.txt", function (err, data)
{
if (err) throw err;
pick=JSON.parse(data);;
});
/*....
.... balance and pick are modified
....*/
if (shutdown)
{
fs.writeFile("balance2.txt", JSON.stringify(balance));
fs.writeFile("pick2.txt", JSON.stringify(pick));
process.exit(0);
}
At the end of the script, the files have not been modified the slightest. I then found out on this site that the files were being opened 2 times simultaneously, or something like that, so i tried this :
var balance, pick;
var stream = fs.createReadStream("balance.txt");
stream.on("readable", function()
{
balance = JSON.parse(stream.read());
});
var stream2 = fs.createReadStream("pick.txt");
stream2.on("readable", function()
{
pick = JSON.parse(stream2.read());
});
/****
****/
fs.unlink("pick.txt");
fs.unlink("balance.txt");
var stream = fs.createWriteStream("balance.txt", {flags: 'w'});
var stream2 = fs.createWriteStream("pick.txt", {flags: 'w'});
stream.write(JSON.stringify(balance));
stream2.write(JSON.stringify(pick));
process.exit(0);
But, this time, both files are empty... I know i should catch errors, but i just don't see where the problem is. I don't mind storing the 2 objects in the same file, if that can helps. Besides that, I never did any javascript in my life before yesterday, so, please give me a simple explanation if you know what failed here.
What I think you want to do is use readFileSync and not use readFile to read your files since you need them to be read before doing anything else in your program (http://nodejs.org/api/fs.html#fs_fs_readfilesync_filename_options).
This will make sure you have read both the files before you execute any of the rest of your code.
Make your like code do this:
try
{
balance = JSON.parse(fs.readFileSync("balance.txt"));
pick = JSON.parse(fs.readFileSync("pick.txt"));
}
catch(err)
{ throw err; }
I think you will get the functionality you are looking for by doing this.
Note, you will not be able to check for an error in the same way you can with readFile. Instead you will need to wrap each call in a try catch or use existsSync before each operation to make sure you aren't trying to read a file that doesn't exist.
How to capture no file for fs.readFileSync()?
Furthermore, you have the same problem on the writes. You are kicking off async writes and then immediately calling process.exit(0). A better way to do this would be to either write them sequentially asynchronously and then exit or to write them sequentially synchronously then exit.
Async option:
if (shutdown)
{
fs.writeFile("balance2.txt", JSON.stringify(balance), function(err){
fs.writeFile("pick2.txt", JSON.stringify(pick), function(err){
process.exit(0);
});
});
}
Sync option:
if (shutdown)
{
fs.writeFileSync("balance2.txt", JSON.stringify(balance));
fs.writeFileSync("pick2.txt", JSON.stringify(pick));
process.exit(0);
}
In trying to learn node.js/socket.io I have been messing around with creating a simple file uploader that takes data chunks from a client browser and reassembles on server side.
The socket.io event for receiving a chunk looks as follows:
socket.on('sendChunk', function (data) {
fs.appendFile(path + fileName, data.data, function (err) {
if (err)
throw err;
console.log(data.sequence + ' - The data was appended to file ' + fileName);
});
});
The issue is that data chunks aren't necessarily appended in the order they were received due to the async calls. Typical console output looks something like this:
1 - The data was appended to file testfile.txt
3 - The data was appended to file testfile.txt
4 - The data was appended to file testfile.txt
2 - The data was appended to file testfile.txt
My question is, what is the proper way to implement this functionality in a non-blocking way but enforce sequence. I've looked at libraries like async, but really want to be able to process each as it comes in rather than creating a series and run once all file chunks are in. I am still trying to wrap my mind around all this event driven flow, so any pointers are great.
Generally you would use a queue for the data waiting to be written, then whenever the previous append finishes, you try to write the next piece. Something like this:
var parts = [];
var inProgress = false;
function appendPart(data){
parts.push(data);
writeNextPart();
}
function writeNextPart(){
if (inProgress || parts.length === 0) return;
var data = parts.shift();
inProgress = true;
fs.appendFile(path + fileName, data.data, function (err) {
inProgress = false;
if (err) throw err;
console.log(data.sequence + ' - The data was appended to file ' + fileName);
writeNextPart();
});
}
socket.on('sendChunk', function (data) {
appendPart(data);
});
You will need to expand this to keep a queue of parts and inProgress based on the fileName. My example assumes those will be constant for simplicity.
Since you need the appends to be in order or synchronous. You could use fs.appendFileSync instead of fs.appendFile. This is quickest way to handle it, but it hurts performance.
If you want to handle it asynchronously yourself, use streams which deal with this problem using EventEmitter. It turns out that the response (as well as the request) objects are streams. Create a writeable stream with fs.createWriteStream and write all pieces to append the file.
fs.createWriteStream(path, [options])#
Returns a new WriteStream object (See Writable Stream).
options is an object with the following defaults:
{ flags: 'w',
encoding: null,
mode: 0666 }
In your case you would use flags: 'a'
I've written a non-blocking tcp-server with node.js. This server listens on a port and reroutes the request to an other server via a http.request()
To have a back-log of the messages rerouted I want to append every message (single line of information) in a file with the date as filename.
The server is going to be hit by several devices on alternating intervals with small txt strings (800bytes). Writing to the filesystem implicitly calls for a blocking event. Is there a way to prevent this behavior??
If appendFile doesn't work out right, I have myself tested a solution for this using File streams that works with multiple clusters and won't clobber the output
Just use the asynchronous methods of the fs module like appendFile.
http://nodejs.org/api/fs.html#fs_fs_appendfile_filename_data_encoding_utf8_callback
Something like this might help.
var fs = require('fs');
var writer = {
files: {},
appendFile: function(path, data) {
if(this.files[path] === undefined) {
this.files[path] = {open: false, queue: []};
}
this.files[path].queue.push(data);
if(!this.files[path].open) {
this.files[path].open = true;
this.nextWrite(path);
}
},
nextWrite: function(path) {
var data = this.files[path].queue.shift(),
self = this;
if(data === undefined)
return this.files[path].open = false;
fs.appendFile(path, data, function(err) {
if (err) throw err;
self.nextWrite(path);
});
}
}
It requires version 0.8.0 of node for fs.appendFile, but it keeps a queue per file and then appends the things in the order they were added. It works, but I didn't spent very much time on it.. so use it for educational purposes only.
writer.appendFile('test.txt','hello');