I implemented big-text file reading using createReadStream. In my implementation i wanted to test for error handling.
So i started process and renamed file, still entire file was read and all data were printed on console. In second try i did same but this time i deleted the file, and results were same, entire file was read and printed on console.
In short, I am reading a file using createReadStream, and after reading started even if i rename/delete the file, it still reads till the end.
Here is a sample implementation,
const fs = require('fs');
const readableStream = fs.createReadStream('mySampleTextFile.txt');
readableStream.on('data', (chunk) => console.log(chunk));
readableStream.on('end', () => console.log('Read till end !'));
mySampleTextFile.txt contains 600k rows with just number on each line, and i verified the last number logged to be exactly same as in the original file.
I am deleting the file before execution ends, i have two terminals open and in one i start execution and in second i fire delete/rename command.
I tried also deleting permanently means directly delete, not in trash.
Am i missing some fundamental of Stream or something? Please help me on this, it's not breaking anything but why it is not breaking ?
At least on Unix-type OS'es, when you remove a file while it's open, the process that opened the file can still read from it (or write to it).
Removing such a file merely removes its directory entry so you can't see it anymore when running ls or a file browser, but the disk space it takes up won't be marked "available" until there are no more processes that are referencing the file (i.e. have it open).
What you could try is truncating the file instead of removing it:
cp /dev/null mySampleTextFile.txt
Related
I have an open std::fs::File, and I want to get it's filename, e.g. as a PathBuf. How do I do that?
The simple solution would be to just save the path used in the call to File::open. Unfortunately, this does not work for me. I am trying to write a program that reads log files, and the program that writes the logs keep changing the filenames as part of it's log rotation. So the file may very well have been renamed since it was opened. This is on Linux, so renaming open files is possible.
How do I get around this issue, and get the current filename of an open file?
On a typical Unix filesystem, a file may have multiple filenames at once, or even none at all. The file metadata is stored in an inode, which has a unique inode number, and this inode number can be linked from any number of directory entries. However, there are no reverse links from the inode back to the directory entries.
Given an open File object in Rust, you can get the inode number using the ino() method. If you know the directory the log file is in, you can use std::fs::read_dir() to iterate over all entries in that directory, and each entry will also have an ino() method, so you can find the one(s) matching your open file object. Of course this approach is subject to race conditions – the directory entry may already be gone again once you try to do anything with it.
On linux, files handles held by the current process can be found under /proc/self/fd. These look and act like symlinks to the original files (though I think they may technically be something else - perhaps someone who knows more can chip in).
You can therefore recover the (possibly changed) file name by constructing the correct path in /proc/self/fd using your file descriptor, and then following the symlink back to the filesystem.
This snippet shows the steps:
use std::fs::read_link;
use std::os::unix::io::AsRawFd;
use std::path::PathBuf;
// if f is your std::fs::File
// first construct the path to the symlink under /proc
let path_in_proc = PathBuf::from(format!("/proc/self/fd/{}", f.as_raw_fd()));
// ...and follow it back to the original file
let new_file_name = read_link(path_in_proc).unwrap();
I'm doing image modifications on a .png and then writing it on my server with
fs.writeFile
Problem is if some clients try to http get the .png from my server while i'm writing it, it will fail.
How can i prevent that ?
Save the modified files under a diferent name then rename it to the original name. The rename operation is an atomic one so it will be done instantly.
Steps:
Make the modification of original.png and write them to original.png.mod
Rename original.png.mod in original.png
Say you are currently writing to example.png.
Write to a different file, and then move it to replace example.png.
That removes the lag between opening the file for writing and finishing writing to it.
I'm writing a desktop app using electron and react. I want to store some information in a JSON file. I've tried both web-fs and browserify-fs to accomplish this, and neither is working as expected. My setup is as follows
project/app/(react files)
project/index.html
project/js/bundle.js
project/main.js
I'm using watchify to compile all the changes in the react files to the bundle.js file (which is read by index.html).
The following is ran from app.js in project/app/ (which is also where the JSON file is stored)
import * as fs from 'browserify-fs';
...
fs.writeFile('./fileData.json', data, function(err){
if(err)console.log(err);
else console.log("success");
});
'success' is always logged to the console, however the contents of the file is not updated, regardless of how I specify the path.
I've tried './fileData.json'
'/fileData.json'
__dirname + '/fileData.json' (which tells me that __dirname couldn't be found)
(absolute path to fileData.json) (which tells me that /Users could not be found)
After doing the above, if I change the writeFile to readFile and log the contents to the console, the updated file is printed. Even if I delete the fileData.json file, the file is successfully read.
This makes me believe that fs.writeFile() is writing to a different directory, not the one the process is being ran from. Despite this, I cannot find any other fileData.json files anywhere on my computer. There's a couple other weird behaviors:
When logging __filename (which should log the entire filepath), the only thing printed is "/app.js" with no leading file path.
Calling "process.cwd()" just gives me "/"
When calling fs.writeFile() with the full file path "/Users/...." I get a folder not found error
Anyone know what could be causing this behavior and how to fix it?
Edit - I also tried getting the absolute path by adding
var path = require('path')
var appDir = path.resolve('./app');
which again only gives me /app when it should be returning an absolute path
Can you confirm the same behavior when not using browserify-fs? Just use plain old fs. (Note you can do this straight from the Chrome dev tools console).
Looking at browserify-fs's page, it looks like it implements a kind of virtual file system using a dependency called level-filesystem (which uses level db). So the files you're expecting to get created aren't. They're being created within a level db database. You could probably find a level db file somewhere that has the information you're trying to write directly to the file system in it.
For simple writing/reading of a JSON file, I'd recommend https://github.com/sindresorhus/electron-config.
I want to create a simple console application that would compare two files based on their filename and output the result into a new file.
My problem is that NodeJS refuses to create a new file if it doesn't exist, and acts like it doesn't exist even if I create it manually.
compare = (data) -> # data is being read from process.stdin
fname = "#{data}_compare.txt"
stdout.write "Attempting to compare #{data}" # stdout = process.stdout
fs.writeFileSync fname, 'A test.'
NodeJS returns Error: ENOENT, no such file or directory in both cases (when I want it to create the file, as well as when the file already exists).
I want the file to be created in the same folder from where the application is run, so path shouldn't be an issue at all (and indeed is correct in the error message).
I tried to specify {flags: 'w'} too, but as far as I know, that's the default value anyways, so it changed noting.
I'm running on Windows 10, tried running command prompt under administrator too, still nothing. Any idea what could be causing this?
The data variable is read from stdin and therefore contains a newline at the end. This is probably what's causing the non-descriptive ENOENT error.
You can remove the newline (and any other whitespace that user might have accidentally entered) with data = data.trim()
This would be better than the substring solution since the newline is 2 characters only on Windows and 1 character elsewhere.
Make sure the path exists (not necessarily the file itself, but the folder structure), and that the process user has write permissions.
An ENOENT error tells you that a component of the specified pathname does not exist -- no entity (file or directory) could be found by the given path.
Make sure you are putting the 'dot':
'./path/to/file'
fs.watch('log-file', function(event, filename) {}); only returns the file name that was changed.
Is it possible to only get what was actually changed? I don't want to read the entire file, and rather want to know what was modified from my file (in my case, there is always addition to a log file. nothing ever gets erased).
It seems that you are looking to solve a similar problem to How to do `tail -f logfile.txt`-like processing in node.js?
As per the first response I would look into the the node-tail module.