Creating a file only if it doesn't exist in Node.js - node.js

We have a buffer we'd like to write to a file. If the file already exists, we need to increment an index on it, and try again. Is there a way to create a file only if it doesn't exist, or should I just stat files until I get an error to find one that doesn't exist already?
For example, I have files a_1.jpg and a_2.jpg. I'd like my method to try creating a_1.jpg and a_2.jpg, and fail, and finally successfully create a_3.jpg.
The ideal method would look something like this:
fs.writeFile(path, data, { overwrite: false }, function (err) {
if (err) throw err;
console.log('It\'s saved!');
});
or like this:
fs.createWriteStream(path, { overwrite: false });
Does anything like this exist in node's fs library?
EDIT: My question isn't if there's a separate function that checks for existence. It's this: is there a way to create a file if it doesn't exist, in a single file system call?

As your intuition correctly guessed, the naive solution with a pair of exists / writeFile calls is wrong. Asynchronous code runs in unpredictable ways. And in given case it is
Is there a file a.txt? — No.
(File a.txt gets created by another program)
Write to a.txt if it's possible. — Okay.
But yes, we can do that in a single call. We're working with file system so it's a good idea to read developer manual on fs. And hey, here's an interesting part.
'w' - Open file for writing. The file is created (if it does not
exist) or truncated (if it exists).
'wx' - Like 'w' but fails if path exists.
So all we have to do is just add wx to the fs.open call. But hey, we don't like fopen-like IO. Let's read on fs.writeFile a bit more.
fs.readFile(filename[, options], callback)#
filename String
options Object
encoding String | Null default = null
flag String default = 'r'
callback Function
That options.flag looks promising. So we try
fs.writeFile(path, data, { flag: 'wx' }, function (err) {
if (err) throw err;
console.log("It's saved!");
});
And it works perfectly for a single write. I guess this code will fail in some more bizarre ways yet if you try to solve your task with it. You have an atomary "check for a_#.jpg existence, and write there if it's empty" operation, but all the other fs state is not locked, and a_1.jpg file may spontaneously disappear while you're already checking a_5.jpg. Most* file systems are no ACID databases, and the fact that you're able to do at least some atomic operations is miraculous. It's very likely that wx code won't work on some platform. So for the sake of your sanity, use database, finally.
Some more info for the suffering
Imagine we're writing something like memoize-fs that caches results of function calls to the file system to save us some network/cpu time. Could we open the file for reading if it exists, and for writing if it doesn't, all in the single call? Let's take a funny look on those flags. After a while of mental exercises we can see that a+ does what we want: if the file doesn't exist, it creates one and opens it both for reading and writing, and if the file exists it does so without clearing the file (as w+ would). But now we cannot use it neither in (smth)File, nor in create(Smth)Stream functions. And that seems like a missing feature.
So feel free to file it as a feature request (or even a bug) to Node.js github, as lack of atomic asynchronous file system API is a drawback of Node. Though don't expect changes any time soon.
Edit. I would like to link to articles by Linus and by Dan Luu on why exactly you don't want to do anything smart with your fs calls, because the claim was left mostly not based on anything.

What about using the a option?
According to the docs:
'a+' - Open file for reading and appending. The file is created if it does not exist.
It seems to work perfectly with createWriteStream

This method is no longer recommended. fs.exists is deprecated. See comments.
Here are some options:
1) Have 2 "fs" calls. The first one is the "fs.exists" call, and the second is "fs.write / read, etc"
//checks if the file exists.
//If it does, it just calls back.
//If it doesn't, then the file is created.
function checkForFile(fileName,callback)
{
fs.exists(fileName, function (exists) {
if(exists)
{
callback();
}else
{
fs.writeFile(fileName, {flag: 'wx'}, function (err, data)
{
callback();
})
}
});
}
function writeToFile()
{
checkForFile("file.dat",function()
{
//It is now safe to write/read to file.dat
fs.readFile("file.dat", function (err,data)
{
//do stuff
});
});
}
2) Or Create an empty file first:
--- Sync:
//If you want to force the file to be empty then you want to use the 'w' flag:
var fd = fs.openSync(filepath, 'w');
//That will truncate the file if it exists and create it if it doesn't.
//Wrap it in an fs.closeSync call if you don't need the file descriptor it returns.
fs.closeSync(fs.openSync(filepath, 'w'));
--- ASync:
var fs = require("fs");
fs.open(path, "wx", function (err, fd) {
// handle error
fs.close(fd, function (err) {
// handle error
});
});
3) Or use "touch": https://github.com/isaacs/node-touch

Todo this in a single system call you can use the fs-extra npm module.
After this the file will have been created as well as the directory it is to be placed in.
const fs = require('fs-extra');
const file = '/tmp/this/path/does/not/exist/file.txt'
fs.ensureFile(file, err => {
console.log(err) // => null
});
Another way is to use ensureFileSync which will do the same thing but synchronous.
const fs = require('fs-extra');
const file = '/tmp/this/path/does/not/exist/file.txt'
fs.ensureFileSync(file)

With async / await and Typescript I would do:
import * as fs from 'fs'
async function upsertFile(name: string) {
try {
// try to read file
await fs.promises.readFile(name)
} catch (error) {
// create empty file, because it wasn't found
await fs.promises.writeFile(name, '')
}
}

Here's a synchronous way of doing it:
try {
await fs.truncateSync(filepath, 0);
} catch (err) {
await fs.writeFileSync(filepath, "", { flag: "wx" });
}
If the file exists it will get truncated, otherwise it gets created if an error is raised.

This works for me.
// Use the file system fs promises
const {access} = require('fs/promises');
// File Exist returns true
// dont use exists which is no more!
const fexists =async (path)=> {
try {
await access(path);
return true;
} catch {
return false;
}
}
// Wrapper for your main program
async function mainapp(){
if( await fexists("./users.json")){
console.log("File is here");
} else {
console.log("File not here -so make one");
}
}
// run your program
mainapp();
Just keep eye on your async - awaits so everthing plays nice.
hope this helps.

You can do something like this:
function writeFile(i){
var i = i || 0;
var fileName = 'a_' + i + '.jpg';
fs.exists(fileName, function (exists) {
if(exists){
writeFile(++i);
} else {
fs.writeFile(fileName);
}
});
}

Related

How to check whether a directory exists in node.js?

I'd like to save files received from json object in a REST API app and here is the code:
router.post('/addphoto', checkAuth, (req, res)=> {
let filename = Math.floor(Math.random() * 100000);
let dir = './uploads/' + req.user.id;
//Not sure about this
if (!fs.existsSync(dir)){
fs.mkdirSync(dir);
}
base64String = req.body.file;
let base64Image = base64String.split(';base64,').pop();
let filePath = dir + "/" + filename
fs.writeFile( filePath, base64Image, {encoding: 'base64'}, function(err) {
console.log('File created');
});
...
It does the job but I've read that existsSync is deprecated, and also I'm not sure if it's a good idea to use sync code inside a router which is async by nature.
So I'm wondering what is the idiomatic way to do so in such circumstances?
You can use access
fs.access(myDir, function(err) {
if (err && err.code === 'ENOENT') {
fs.mkdir(myDir); //Create dir in case not found
}
});
I've read that existsSync is deprecated
It isn't. See the manual:
fs.exists() is deprecated, but fs.existsSync() is not. The callback parameter to fs.exists() accepts parameters that are inconsistent with other Node.js callbacks. fs.existsSync() does not use a callback.
I'm not sure if it's a good idea to use sync code inside a router which is async by nature.
There's nothing intrinsically wrong about doing something synchronous inside something that is asynchronous — most JS is synchronous — but it does mean that the feature would block the event loop while looking at the file system, and looking at the file system is a relatively time-consuming operation, so it wouldn't be good for performance.
Your code might not need that level of performance, but that's a judgement call we can't make for you.
exists is right next to existsSync in the manual and says:
Deprecated: Use fs.stat() or fs.access() instead.
So pick one of those.
access has an example:
// Check if the file exists in the current directory.
fs.access(file, fs.constants.F_OK, (err) => {
console.log(`${file} ${err ? 'does not exist' : 'exists'}`);
});
You can use existsSync as it's not deprecated. it's exists that got deprecated. I've attached a screenshot and link below so you can use it without any problem.
link->
https://nodejs.org/api/fs.html#fs_fs_existssync_path
image->
Modern async/await way
const isDirExist = async path => await fs.promises.access(path).then(()=>true).catch(()=>false);
Using
const isDirExist = async path => await fs.promises.access(path).then(()=>true).catch(()=>false);
(async () => {
console.log(await isDirExist('/my/path/'));
})()
From official docs https://nodejs.org/api/fs.html#fspromisesaccesspath-mode
fs.access will throw an error if file don't exist. So you will not have a boolean to check if file exist or not, like java does from ancient ages. You should use a try/catch:
var isDirExist = false;
try{
await fs.promises.access("/foo/bar");
isDirExist = true;
}catch(e){
isDirExist = false;
}
If this looks problematic, official docs says:
Using fsPromises.access() to check for the accessibility of a file before calling fsPromises.open() is not recommended. Doing so introduces a race condition, since other processes may change the file's state between the two calls. Instead, user code should open/read/write the file directly and handle the error raised if the file is not accessible.
If you use node-fs-extra you can utilize...
fs.ensureDir(dir[,options][,callback])
Which by definition...
Ensures that the directory exists. If the directory structure does not exist, it is created.
See also fs.ensureDirSync
The below code will check if the destination exists. If it doesn't exist, it'll create the destination as a directory. It will also create the parent directories if they don't exist (because of recursive: true). It does not use Sync functions and will not block requests if used in a web server.
const fs = require('fs');
const targetDirectory = 'your/path/here';
fs.mkdir(targetDirectory, { recursive: true }, (error) => {
if (!error) {
console.log('Directory successfully created, or it already exists.');
return;
}
switch (error.code) {
case 'EEXIST':
// Error:
// Requested location already exists, but it's not a directory.
break;
case 'ENOTDIR':
// Error:
// The parent hierarchy contains a file with the same name as the dir
// you're trying to create.
break;
default:
// Some other error like permission denied.
console.error(error);
break;
}
});
See: mkdir docs.

node.js fs.exists() will be deprecated, what to use instead?

According to the documentation node.js fs.exists() will be deprecated.
Their reasoning:
fs.exists() is an anachronism and exists only for historical reasons. There should almost never be a reason to use it in your own code.
In particular, checking if a file exists before opening it is an anti-pattern that leaves you vulnerable to race conditions: another process may remove the file between the calls to fs.exists() and fs.open(). Just open the file and handle the error when it's not there.
fs.exists() will be deprecated.
I am currently using it before moving files, since the fs.rename() seems to quietly overwrite files with the same name in the destination folder.
My question is; what should I use instead to prevent fs.rename() from overwriting the file in the destination folder?
I assume there's a way that I don't know of. Otherwise I don't see a reason for fs.exists() to be deprecated.
Using fs.open() as suggested seems overkill since I don't want to open the file.
Edit, as per #jfriend00's request for more info about what I'm doing.
I'm making an Electron application where the user can sort files into different directories. It's not a server software, it's intended to run on every day users machines, handling their documents. This is the code so far for moving a file:
function moveFile(destIndex){
var from = queue[currentQueueIndex].path;
var to = destinations[destIndex].path + path.sep + path.basename(from);
console.log("[i] Move file (from/to): ");
console.log(from);
console.log(to);
//Check if file exists, if yes: give them the choice to cancel.
fs.stat(to, function (err, stats) {
if (err){
move(from, to);
} else {
var confirmed = confirm("File already exists, will overwrite.");
if (confirmed) {
move(from, to);
}
}
});
next(); //Show the next file to the user
}
function move(from, to){
fs.rename(from, to, function (err) {
if (err) throw err;
console.log('[i] Move successful');
queue[currentQueueIndex].path = to;
queue[currentQueueIndex].moved = true;
});
}
After the first comment, the part starting with fs.stat, I check whether the file I'm about to create with fs.rename already exists. I guess this is subject to race conditions, but I can't find that fs.rename handles duplicates in any way.
Since this application is intended for "home computing", I don't think the scenario where a file disappears between the stat check and the rename is likely to happen. But still, the more potential problems I can avoid, the better.
Use fs.existsSync().
fs.existsSync() has not been deprecated.
https://nodejs.org/api/fs.html#fs_fs_existssync_path
fs.existsSync(path)
Added in: v0.1.21
path |
Synchronous version of fs.exists(). Returns true if the file exists, false otherwise.
Note that fs.exists() is deprecated, but fs.existsSync() is not. (The callback >parameter to fs.exists() accepts parameters that are inconsistent with other >Node.js callbacks. fs.existsSync() does not use a callback.)
The io.js docs mention the use of fs.stat() or fs.access() in place of fs.exists().
Here is an example using fs.access (in older Node versions, use fs.stat instead of fs.access), but handling errors correctly as well:
import { access } from 'node:fs/promises';
async function fileExists(filename) {
try {
await access(filename);
return true;
} catch (err) {
if (err.code === 'ENOENT') {
return false;
} else {
throw err;
}
}
}
Here is example by using fs.stat:-
fs.stat('mycustomfile.csv', function (err, stats) {
console.log(stats);//here we got all information of file in stats variable
if (err) {
return console.error(err);
}
fs.unlink('mycustomfile.csv',function(err){
if(err) return console.log(err);
console.log('file deleted successfully');
});
});
const exists = !!(await fs.promises.stat(filename).catch(() => null))

atomic write/read of a file in nodejs

My nodejs application is built around a "project file".
Several modules (by "module", I mean a simple javascript file of my project) of this application need to load, modify and save this file often, via streams (fs.createReadStream, fs.createWriteStream), and since those modules are executed independently from each other, with sometimes an origin from an websocket events (for instance), I need to make the save/load operations of the project file atomic.
It means the following scenario:
moduleA writes the project file
in the same time, and before moduleA has finished to write the file, moduleB wants to read it => ideally, it should wait for the write operation of moduleA (currently, it reads a partially written file and detect an error) before really read the file
Is nodejs able to do this natively or do I have to build a sort of atomic wrapper over my read/write stream system?
There is to my knowledge nothing built in. There are modules such as redis-lock though, that implement a lock mechanism.
If you run on a single non-clustered server you could probably cope with implementing a simple local lock though.
This might give you an idea:
var Fs = require("fs"),
LOCK = require ("os").tmpdir () + '/foo-lock.';
function transLock(id, cb) {
Fs.open(LOCK + id, "wx", function(err, fd) {
if (err) {
// someone else has created the file
// or something went wrong
cb(err);
} else {
Fs.close(fd, function(err) {
// there should be no error here except weird stuff
// like EINTR which must be ignored on Linux
cb();
});
}
});
}
function transUnlock(id) {
Fs.unlink(LOCK + id, function(err) {
if (err) {
// something is wrong and nothing we can do except
// perhaps log something or do some background cleanup
}
});
}
function main() {
var id = "some-unique-name";
transLock(id, function(err) {
if (err)
console.log(err);
else {
// ... do your stuffs ...
transUnlock(id);
}
});
}
main();

node.js file system problems

I keep banging my head against the wall because of tons of different errors. This is what the code i try to use :
fs.readFile("balance.txt", function (err, data) //At the beginning of the script (checked, it works)
{
if (err) throw err;
balance=JSON.parse(data);;
});
fs.readFile("pick.txt", function (err, data)
{
if (err) throw err;
pick=JSON.parse(data);;
});
/*....
.... balance and pick are modified
....*/
if (shutdown)
{
fs.writeFile("balance2.txt", JSON.stringify(balance));
fs.writeFile("pick2.txt", JSON.stringify(pick));
process.exit(0);
}
At the end of the script, the files have not been modified the slightest. I then found out on this site that the files were being opened 2 times simultaneously, or something like that, so i tried this :
var balance, pick;
var stream = fs.createReadStream("balance.txt");
stream.on("readable", function()
{
balance = JSON.parse(stream.read());
});
var stream2 = fs.createReadStream("pick.txt");
stream2.on("readable", function()
{
pick = JSON.parse(stream2.read());
});
/****
****/
fs.unlink("pick.txt");
fs.unlink("balance.txt");
var stream = fs.createWriteStream("balance.txt", {flags: 'w'});
var stream2 = fs.createWriteStream("pick.txt", {flags: 'w'});
stream.write(JSON.stringify(balance));
stream2.write(JSON.stringify(pick));
process.exit(0);
But, this time, both files are empty... I know i should catch errors, but i just don't see where the problem is. I don't mind storing the 2 objects in the same file, if that can helps. Besides that, I never did any javascript in my life before yesterday, so, please give me a simple explanation if you know what failed here.
What I think you want to do is use readFileSync and not use readFile to read your files since you need them to be read before doing anything else in your program (http://nodejs.org/api/fs.html#fs_fs_readfilesync_filename_options).
This will make sure you have read both the files before you execute any of the rest of your code.
Make your like code do this:
try
{
balance = JSON.parse(fs.readFileSync("balance.txt"));
pick = JSON.parse(fs.readFileSync("pick.txt"));
}
catch(err)
{ throw err; }
I think you will get the functionality you are looking for by doing this.
Note, you will not be able to check for an error in the same way you can with readFile. Instead you will need to wrap each call in a try catch or use existsSync before each operation to make sure you aren't trying to read a file that doesn't exist.
How to capture no file for fs.readFileSync()?
Furthermore, you have the same problem on the writes. You are kicking off async writes and then immediately calling process.exit(0). A better way to do this would be to either write them sequentially asynchronously and then exit or to write them sequentially synchronously then exit.
Async option:
if (shutdown)
{
fs.writeFile("balance2.txt", JSON.stringify(balance), function(err){
fs.writeFile("pick2.txt", JSON.stringify(pick), function(err){
process.exit(0);
});
});
}
Sync option:
if (shutdown)
{
fs.writeFileSync("balance2.txt", JSON.stringify(balance));
fs.writeFileSync("pick2.txt", JSON.stringify(pick));
process.exit(0);
}

nodejs express fs iterating files into array or object failing

So Im trying to use the nodejs express FS module to iterate a directory in my app, store each filename in an array, which I can pass to my express view and iterate through the list, but Im struggling to do so. When I do a console.log within the files.forEach function loop, its printing the filename just fine, but as soon as I try to do anything such as:
var myfiles = [];
var fs = require('fs');
fs.readdir('./myfiles/', function (err, files) { if (err) throw err;
files.forEach( function (file) {
myfiles.push(file);
});
});
console.log(myfiles);
it fails, just logs an empty object. So Im not sure exactly what is going on, I think it has to do with callback functions, but if someone could walk me through what Im doing wrong, and why its not working, (and how to make it work), it would be much appreciated.
The myfiles array is empty because the callback hasn't been called before you call console.log().
You'll need to do something like:
var fs = require('fs');
fs.readdir('./myfiles/',function(err,files){
if(err) throw err;
files.forEach(function(file){
// do something with each file HERE!
});
});
// because trying to do something with files here won't work because
// the callback hasn't fired yet.
Remember, everything in node happens at the same time, in the sense that, unless you're doing your processing inside your callbacks, you cannot guarantee asynchronous functions have completed yet.
One way around this problem for you would be to use an EventEmitter:
var fs=require('fs'),
EventEmitter=require('events').EventEmitter,
filesEE=new EventEmitter(),
myfiles=[];
// this event will be called when all files have been added to myfiles
filesEE.on('files_ready',function(){
console.dir(myfiles);
});
// read all files from current directory
fs.readdir('.',function(err,files){
if(err) throw err;
files.forEach(function(file){
myfiles.push(file);
});
filesEE.emit('files_ready'); // trigger files_ready event
});
As several have mentioned, you are using an async method, so you have a nondeterministic execution path.
However, there is an easy way around this. Simply use the Sync version of the method:
var myfiles = [];
var fs = require('fs');
var arrayOfFiles = fs.readdirSync('./myfiles/');
//Yes, the following is not super-smart, but you might want to process the files. This is how:
arrayOfFiles.forEach( function (file) {
myfiles.push(file);
});
console.log(myfiles);
That should work as you want. However, using sync statements is not good, so you should not do it unless it is vitally important for it to be sync.
Read more here: fs.readdirSync
fs.readdir is asynchronous (as with many operations in node.js). This means that the console.log line is going to run before readdir has a chance to call the function passed to it.
You need to either:
Put the console.log line within the callback function given to readdir, i.e:
fs.readdir('./myfiles/', function (err, files) { if (err) throw err;
files.forEach( function (file) {
myfiles.push(file);
});
console.log(myfiles);
});
Or simply perform some action with each file inside the forEach.
I think it has to do with callback functions,
Exactly.
fs.readdir makes an asynchronous request to the file system for that information, and calls the callback at some later time with the results.
So function (err, files) { ... } doesn't run immediately, but console.log(myfiles) does.
At some later point in time, myfiles will contain the desired information.
You should note BTW that files is already an Array, so there is really no point in manually appending each element to some other blank array. If the idea is to put together the results from several calls, then use .concat; if you just want to get the data once, then you can just assign myfiles = files directly.
Overall, you really ought to read up on "Continuation-passing style".
I faced the same problem, and basing on answers given in this post I've solved it with Promises, that seem to be of perfect use in this situation:
router.get('/', (req, res) => {
var viewBag = {}; // It's just my little habit from .NET MVC ;)
var readFiles = new Promise((resolve, reject) => {
fs.readdir('./myfiles/',(err,files) => {
if(err) {
reject(err);
} else {
resolve(files);
}
});
});
// showcase just in case you will need to implement more async operations before route will response
var anotherPromise = new Promise((resolve, reject) => {
doAsyncStuff((err, anotherResult) => {
if(err) {
reject(err);
} else {
resolve(anotherResult);
}
});
});
Promise.all([readFiles, anotherPromise]).then((values) => {
viewBag.files = values[0];
viewBag.otherStuff = values[1];
console.log(viewBag.files); // logs e.g. [ 'file.txt' ]
res.render('your_view', viewBag);
}).catch((errors) => {
res.render('your_view',{errors:errors}); // you can use 'errors' property to render errors in view or implement different error handling schema
});
});
Note: you don't have to push found files into new array because you already get an array from fs.readdir()'c callback. According to node docs:
The callback gets two arguments (err, files) where files is an array
of the names of the files in the directory excluding '.' and '..'.
I belive this is very elegant and handy solution, and most of all - it doesn't require you to bring in and handle new modules to your script.

Resources