I have a file which sometimes gets \00 null characters inside it. So I need to repair it.
Thats why I'm reading it, removing the invalid characters and writing it again. BUT, fs.writeFile is not overwriting its previous contents. The new contents get appended, which is not what i want.
Is is because my write code is inside read code?
fs.readFile('./' + file, function (err, data) {
if (err) {
console.error(err);
return;
}
var str = data.toString();
var repaired = str.slice(0, str.indexOf('\00')) + str.slice(str.lastIndexOf('\00') + 1, str.length);
//console.log(repaired);
fs.writeFile('./' + file, repaired, function (err) {
if (err)
console.error(err);
});
});
I've also tried using {flag:'w'} (which i think fs.writeFile may already have by default)
Thanks to #thefourtheye for pointing me towards me to proper direction.
As there was no \00 character in the file i was testing with, the str.indexOf('\00') and was getting the whole file, and again str.slice(str.lastIndexOf('\00') was getting the whole file. Thats why I thought
Using replace function did the job.
var repaired = str.replace(/\00/g,'');
I had the same or similar problem in that it seems when I called fs.writeFile() with different content but the same file, if the new content was shorter than the existing content then it did not overwrite all of the previous file-content.
I found an explanation why this may be happening and a suggested remedy at:
https://github.com/nodejs/node-v0.x-archive/issues/4965.
According to that "This is not a bug. (Or at least, not one that Node has ever pretended to address...."
The suggested solution is "wait for the callback". I assume that means "wait for the write-callback to be called before trying to read the file". That makes sense of course, you should not try to read what may not have been fully written yet.
But, if you write to the same file several times like I did, then waiting for the (first) write-callback to complete before reading is not enough. Why? Because another 'write' may be in progress when you do the reading, and thus you can get garbled content.
Related
The following "write file", "check hash" in intermediate cases ( only sometimes) fails in an Electron 8.2.1 application. It will display that it has calculated d41d8cd98f00b204e9800998ecf8427e which is the hash over an empty string. However when i look in the folder, the file exists. So my assumption is that somehow sometimes that fs.writeFile under higher load still is not ready writing or something the file returns an empty string instead of the contents.
And note that in 99% of the cases the application runs correctly. Only in some cases (and we think very high load) this fails.
I read node - fs.writeFile creates a blank file which comes closer but it does not provide a reason or "why the hell" nor because of this you need to do that.
fs.writeFile(cPath, body, 'utf-8', (err) => {
if (err) {
errors.handleErrorLocal(err);
reject();
return;
}
log.info(cFile + ' C HASH is:' + hash + ' HASH calculated:' + md5File.sync(cPath))
if (hash && md5File.sync(cPath) !== hash.toLowerCase()) {
log.warn('C failed to download. Wrong Hash. ', cFile)
}
resolve();
});
The answer on node.js readfile woes from panu-logic seems to correspond with this experience (bottom answer) but also does not provide any reason why other than "magical" in his case however he tries to read while being written. "logically" this is not the case but the assumption is that this is for some unknown reason the case.
I am aware that i could rewrite to stream or await but that requiress me to change it , push it out, waiting for some time for bugs then retry. I would rather know before all of this what the reason is and just know for sure that the fix for this works rather than stretching this over weeks or months.
I've been coding just as a side project for a bit, piecing together bits that other people have written (it's for a simple discord bot). I want to split my code to make it easier to problem solve and read, however whenever I try to use the code it comes up with an error saying 'SyntaxError: await is only valid in async function'.
I've tried supposedly loading the code asynchronously, loading it with require() and then making a single command asynchronous, making the entire code in the file asynchronous (it's not just one command I want to load, but a whole file. Also I'm not sure if I tried it correctly or not), using the npm async-require, and maybe some others that have been around on the internet.
//one of the solutions I've tried. This is just copy pasted from the
//answer
//file2.js
var fs = require('fs');
module.exports = function (callback) {
fs.readFile('/etc/passwd', function (err, data) {
callback(err, data);
});
};
//file1.js
require('./passwords')(function (err, passwords) {
// This code runs once the passwords have been loaded.
});
In the first file before I split it, I started it with client.on('message', async message => { and it made me able to use the await function in every command. I want to still be able to do that, but just have it a bit neater and easier to use by splitting it.
I'm trying to get this done so I can move on to a different question I asked and give one of the answers a tick. Any help would be greatly appreciated <3
Fix those awaits so that they are not inside async functions. This is a lexical issue that can be solved just by looking at the location were the error occurs. Just look for the nearest containing function to where the await is and mark it async. Repeat until the error goes away.
excel4node's write to file function catches error and does not propagate to a caller. Therefore, my app cannot determine whether write to file is successful or not.
My current workaround is like below:
let fs = require('fs')
try {
let filePath = 'blahblah'
fs.writeFileSync(filePath, '') // Try-catch is for this statement
excel4nodeWorkbook.write(filePath)
} catch (e) {
console.log('File save is not successful')
}
It works, but I think it's a sort of hack and that it's not a semantically correct way. I also testedfs.access and fs.accessSync, but they only check permission, not the state (busy/lock) of resource.
Is there any suggestion for this to look and behave nicer without modifying excel4node source code?
I think you are asking the wrong question. If you check at time T, then write at time T + 1ms, what would guarantee that the file is still writeable?
If the file is not writeable for whatever reason, the write will fail, period. Nothing to do. Your code is fine, but you can probably also do without the fs.writeFileSync(), which will just erase whatever else was in the file before.
You can also write to a randomly-generated file path to make reasonably sure that two processes are not writing to the same file at the same time, but again, that will not prevent all possible write errors, so what you really, really want is rather some good error handling.
In order to handle errors properly you have to provide a callback!
Something along the lines of:
excel4nodeWorkbook.write(filePath, (err) => {
if (err) console.error(err);
});
Beware, this is asynchronous code, so you need to handle that as well!
You already marked a line in the library's source code. If you look a few lines above, you can see it uses the handler argument to pass any errors to. In fact, peeking at the documentation comment above the function, it says:
If callback is given, callback called with (err, fs.Stats) passed
Hence you can simply pass a function as your second argument and check for err like you've probably already seen elsewhere in the node environment:
excel4nodeWorkbook.write(filepath, (err) => {
if (err) {
console.error(err);
}
});
I've searched here for a long time, but did not get the answer, I just simply want to read 4000 json files in a loop and do something later,
try {
data = JSON.parse(fs.readFileSync(file));
} catch (err) {
console.error(err);
next();
return;
}
this is a so simple problem, why I can't find answer?
I tried graceful-fs, still got the same problem.
Any suggestion?
thanks very much!
I gave up openFileSync and use openFileSync instead.
fs.readFile(file, function read(err, data) {})
I had this same problem where i was traversing folders and uploading files. I was only able to solve it by using the queueing up the files and reading them from a queue. I eventually went with async library
You can use following option to avoid this problem....
var Filequeue = require('filequeue');
var fq = new Filequeue(200); // max number of files to open at once
fq.readdir('/path/to/files/', function(err, files) {
if(err) {
throw err;enter code here
}
files.forEach(function(file) {
fq.readFile('/path/to/files/' + file, function(err, data) {enter code here`
// do something besides crash
}
Make sure you installed filequeue npm here.
Sync functions are not covered by graceful-fs. EMFILE, which means the current process is out of file descriptors, is impossible to deal with during a Sync function. So graceful-fs does not make a difference.
It's weird though: readFileSync is supposed to open the file, read it, then close it. You have probably encountered an fd leak in your version of Node. It has probably been fixed between 2015 and now (2022), but as there is no version information nor the code for the actual looping part it is difficult to tell.
I'm "walking" a hundred thousand JSON files, reading the content and throwing an error if something bad happens:
walk(__dirname + '/lastfm_test', 'json', function (err, files) {
files.forEach(function (filePath) {
fs.readFile(filePath, function (err, data) {
if (err) throw err;
});
});
});
The walk function is largely inspired by this question (chjj answer). After some iteration, the line if (err) throw err gets executed. the error throw is:
Error: OK, open 'path/to/somejsonfile.json'
Any chance to investigate what's happening here? I'm sure that the walk function is ok: in fact replacing the call fs.readFile() with console.log(filePath) shows the paths. without errors.
Some useful info: Windows 7 x64, node.exe x64 .0.10.5. Last.fm dataset downloaded from here.
I recommend using the graceful-fs module for this purpose. It will automatically limit the number of open file descriptors. It's written by Isaac Schlueter, the creator of npm and maintainer of Node, so it's pretty solid. The bare fs module lets you shoot yourself in the foot.
The "foreach-loop" is executing readFile very often. NodeJS starts opening the files in a background thread. But no file is processed in the NodeJS main thread until the foreach loop is finished (and all file open requests are scheduled). For this reason no files are processed (and later closed) while opening all files. At some time point many files are opened and all available handles are used, resulting in the useless error message.
Their are multiple soulutions to your problem:
First you could open all files synchronously after each other. But this would slow down the application and would not match the event based programming model of NodeJS. (But is the easiest solution if you don't mind the performance)
Better would be opening only a specific amount of files at a time (e.g. ~1000 files) and after processing one you could open the next one.
Pseude Code:
1. walk the file system and store all file name in an array
2. fs.readFile for a batch of files from the array
3. In the callback of readFile after processing, start opening more files from the array if not empty.