I'm trying to do the following: Read the content of a directory to find all the .xml files (I'm using glob but I'd like to use something like fs.readdir from fs), then I want to read every file using fs.readFile and then I want to convert the xml file to JSON objects. I'm using xml2json for this purpose.
Once I have the json objects, I would like to iterate every one of them to get the one property out of it and push it to an array. Eventually, all the code is wrapped in a function that logs the content of the array (once is completed). This code currently works fine but I'm getting to the famous callback hell.
const fs = require('fs');
const glob = require('glob');
const parser = require('xml2json');
let connectors = []
function getNames(){
glob(__dirname + '/configs/*.xml', {}, (err, files) => {
for (let j=0; j < files.length; j++) {
fs.readFile( files[j], function(err, data) {
try {
let json = parser.toJson(data, {object: true, alternateTextNode:true, sanitize:true})
for (let i=0; i< json.properties.length; i++){
connectors.push(json.properties[i].name)
if (connectors.length === files.length){return console.log(connectors)}
}
}
catch(e){
console.log(e)
}
});
}
})
}
getNames()
However, I'd like to move to a more clean and elegant solution (using promises). I've been reading the community and I've found some ideas in some similar posts here or here.
I'd like to have your opinion on how I should proceed for this kind of situations. Should I go for a sync version of readFile instead? Should I use promisifyAll to refactor my code and use promises everywhere? If so, could you please elaborate on what my code should look like?
I've also learned that there's a promises based version of fs from node v10.0.0 onwards. Should I go for that option? If so how should I proceed with the parser.toJson() part. I've also seen that there's another promise-based version called xml-to-json-promise.
I'd really appreciate your insights into this as I'm not very familiar with promises when there are several asynchronous operations and loops involved, so I end up having dirty solutions for situations like this one.
Regards,
J
I would indeed suggest that you use the promise-version of glob and fs, and then use async, await and Promise.all to get it all done.
NB: I don't see the logic about the connectors.length === files.length check, as in theory the number of connectors (properties) can be greater than the number of files. I assume you want to collect all of them, irrespective of their number.
So here is how the code could look (untested):
const fs = require('fs').promises; // Promise-version (node 10+)
const glob = require('glob-promise'); // Promise-version
const parser = require('xml2json');
async function getNames() {
let files = await glob(__dirname + '/configs/*.xml');
let promises = files.map(fileName => fs.readFile(fileName).then(data =>
parser.toJson(data, {object: true, alternateTextNode:true, sanitize:true})
.properties.map(prop => prop.name)
));
return (await Promise.all(promises)).flat();
}
getNames().then(connectors => {
// rest of your processing that needs access to connectors...
});
As in comments you write that you have problems with accessing properties.map, perform some validation, and skip cases where there is no properties:
const fs = require('fs').promises; // Promise-version (node 10+)
const glob = require('glob-promise'); // Promise-version
const parser = require('xml2json');
async function getNames() {
let files = await glob(__dirname + '/configs/*.xml');
let promises = files.map(fileName => fs.readFile(fileName).then(data =>
(parser.toJson(data, {object: true, alternateTextNode:true, sanitize:true})
.properties || []).map(prop => prop.name)
));
return (await Promise.all(promises)).flat();
}
getNames().then(connectors => {
// rest of your processing that needs access to connectors...
});
I am trying to use util.promisify to convert AWS Document client get function to a promise based utility. But it does not seem to behave as expected;
// This does not work as expected
const docClient = new AWS.DynamoDB.DocumentClient();
let docClientGet = require('util').promisify(docClient.get);
However when i do usual promise conversion like this,
let docClientGet = function (params) {
return new Promise((resolve, reject) => {
docClient.get(params, function (err, data) {
if (err) {
return reject(err);
}
return resolve(data);
})
})
};
And use it in an async function like this:
await docClientGet(params);
It works!.
I wonder where I am wrong in understanding util.promisify
If the method you are promisifying needs to be associated with the object it is on (which it looks like it does in your case, then this code:
let docClientGet = utils.promisify(docClient.get);
will not retain the association with the docClient object. What happens is that the promisified docClient.get() gets called without the this value set to the docClient object and it can't do its job properly.
You can work around that with this:
utils.promisify(docClient.get.bind(docClient));
The promisify doc does not make this clear because it uses an example from the fs library whose methods do not need to be associated with the fs object in order to work properly.
P.S. It's a bit unusual to put the util library into a variable named utils. That is likely to confuse some people reading your code.
I'm trying to read a file, but I always got the error Error: ENOENT: no such file or directory, open \'SB01028A.RET\'. The file name is correct, and exists because I put the file in my Home/sentbox directory.
What I did wrong here ?
Code:
function downloadFile () {
return new Promise((resolve, reject) => {
try {
const testFolder = `${require('os').homedir()}/sentbox`
fs.readdir(testFolder, (err, files) => {
if (err) {
return reject(err)
}
files.forEach(fileRetorno => {
const retorno = fs.readFileSync(fileRetorno, 'UTF8')
return resolve(retorno)
})
})
} catch (err) {
return reject(err)
}
})
}
You have a number of things wrong in your code like
Using synchronous file reading within a promise when you could be making your file reading asynchronous
Using try/catch in an asynchronous context without wrapping that in a async/await function
You're not using the results of fs.readdir() correctly.
Attemping to resolve/reject a promise that could have already been resolved or rejected
Using require() in a loop, and in an asynchronous context
fs.readdir() is going to return all the entry names within that directory, both files and directories, as an array. Before you can call fs.readFile() you'll need to check if the entry is a file or directory and you'll need to join() the path to the directory read with readdir() in this case, testFolder, with the entry name.
When returning using Promises (async/await wraps promises but still uses them), you only can resolve each promise once. So resolving the same promise multiple times to return different values doesn't work, the same is true for rejecting multiple times. Instead, you'll need to return your values in an Array or an Object. For the scenario in the above code, using an Object would be more ideal since you can associate the file contents to a key for reference and better access later on.
I've used async/await to clean up this code, this will give you the synchronously development approach while getting the functionality of promises. You can read more about async/await on MDN. I've also promisified all of the asynchronous version of the needed fs functions using util.promisify()
The code below will
Read all of the entries in testFolder
Filter the entries array to only include files by calling stat() for each entry and checking if it is a file.
stat() will return a fs.Stat object that can tell us if the entry is a file via stat.isFile(). Since Array#filter() is only expecting a boolean result for each entry interated over, the result of stat.isFile() can be returned directly
Iterate over the files array with Array#reduce() and call readFile() for each file, returning the contents on an object with each file name as a key to the contents
const fs = require('fs')
const {promisify} = require('util')
const os = require('os')
const path = require('path')
const readdir = promisify(fs.readdir)
const readFile = promisify(fs.readFile)
const stat = promisify(fs.stat)
const downloadFile = async () => {
const testFolder = `${os.homedir()}/sentbox`
// Get all the entries in the directory async
const entries = await readdir(testFile)
// We only want the file entries returned
const files = entries.filter(async entry => {
let stat = await stat(path.join(testFolder, entry))
return stat.isFile()
})
return Promise.all(files.reduce(async (filesContents, file) => {
let filepath = path.join(testFolder, file)
fileContents[file] = await readFile(filepath, 'utf8')
return fileContents
}, {}))
}
this is my code.
var fs = require('fs')
var test = readafile('file.txt', function(returnValue) {
console.log(returnValue);
test = returnValue;
});
console.log(test);
function readafile(filepath,callback){
var attachment_path = filepath;
fs.readFile(attachment_path, function(err,data){
var attachment_encoded = new Buffer(data, 'binary').toString('base64');
callback(attachment_encoded);
});
}
In that if i need that return value of that function in variable test means how to achieve that ?
In that console.log(test) it says undefined.
since it is a callback function.
How to get it properly ?
You can't really expect getting a synchronous behavior (like getting a return value) with asynchronous code. You can use fs.readFileSync to avoid the asynchronous aspect or just use your value inside your callback.
Otherwise the async module could help you out.
We have a buffer we'd like to write to a file. If the file already exists, we need to increment an index on it, and try again. Is there a way to create a file only if it doesn't exist, or should I just stat files until I get an error to find one that doesn't exist already?
For example, I have files a_1.jpg and a_2.jpg. I'd like my method to try creating a_1.jpg and a_2.jpg, and fail, and finally successfully create a_3.jpg.
The ideal method would look something like this:
fs.writeFile(path, data, { overwrite: false }, function (err) {
if (err) throw err;
console.log('It\'s saved!');
});
or like this:
fs.createWriteStream(path, { overwrite: false });
Does anything like this exist in node's fs library?
EDIT: My question isn't if there's a separate function that checks for existence. It's this: is there a way to create a file if it doesn't exist, in a single file system call?
As your intuition correctly guessed, the naive solution with a pair of exists / writeFile calls is wrong. Asynchronous code runs in unpredictable ways. And in given case it is
Is there a file a.txt? — No.
(File a.txt gets created by another program)
Write to a.txt if it's possible. — Okay.
But yes, we can do that in a single call. We're working with file system so it's a good idea to read developer manual on fs. And hey, here's an interesting part.
'w' - Open file for writing. The file is created (if it does not
exist) or truncated (if it exists).
'wx' - Like 'w' but fails if path exists.
So all we have to do is just add wx to the fs.open call. But hey, we don't like fopen-like IO. Let's read on fs.writeFile a bit more.
fs.readFile(filename[, options], callback)#
filename String
options Object
encoding String | Null default = null
flag String default = 'r'
callback Function
That options.flag looks promising. So we try
fs.writeFile(path, data, { flag: 'wx' }, function (err) {
if (err) throw err;
console.log("It's saved!");
});
And it works perfectly for a single write. I guess this code will fail in some more bizarre ways yet if you try to solve your task with it. You have an atomary "check for a_#.jpg existence, and write there if it's empty" operation, but all the other fs state is not locked, and a_1.jpg file may spontaneously disappear while you're already checking a_5.jpg. Most* file systems are no ACID databases, and the fact that you're able to do at least some atomic operations is miraculous. It's very likely that wx code won't work on some platform. So for the sake of your sanity, use database, finally.
Some more info for the suffering
Imagine we're writing something like memoize-fs that caches results of function calls to the file system to save us some network/cpu time. Could we open the file for reading if it exists, and for writing if it doesn't, all in the single call? Let's take a funny look on those flags. After a while of mental exercises we can see that a+ does what we want: if the file doesn't exist, it creates one and opens it both for reading and writing, and if the file exists it does so without clearing the file (as w+ would). But now we cannot use it neither in (smth)File, nor in create(Smth)Stream functions. And that seems like a missing feature.
So feel free to file it as a feature request (or even a bug) to Node.js github, as lack of atomic asynchronous file system API is a drawback of Node. Though don't expect changes any time soon.
Edit. I would like to link to articles by Linus and by Dan Luu on why exactly you don't want to do anything smart with your fs calls, because the claim was left mostly not based on anything.
What about using the a option?
According to the docs:
'a+' - Open file for reading and appending. The file is created if it does not exist.
It seems to work perfectly with createWriteStream
This method is no longer recommended. fs.exists is deprecated. See comments.
Here are some options:
1) Have 2 "fs" calls. The first one is the "fs.exists" call, and the second is "fs.write / read, etc"
//checks if the file exists.
//If it does, it just calls back.
//If it doesn't, then the file is created.
function checkForFile(fileName,callback)
{
fs.exists(fileName, function (exists) {
if(exists)
{
callback();
}else
{
fs.writeFile(fileName, {flag: 'wx'}, function (err, data)
{
callback();
})
}
});
}
function writeToFile()
{
checkForFile("file.dat",function()
{
//It is now safe to write/read to file.dat
fs.readFile("file.dat", function (err,data)
{
//do stuff
});
});
}
2) Or Create an empty file first:
--- Sync:
//If you want to force the file to be empty then you want to use the 'w' flag:
var fd = fs.openSync(filepath, 'w');
//That will truncate the file if it exists and create it if it doesn't.
//Wrap it in an fs.closeSync call if you don't need the file descriptor it returns.
fs.closeSync(fs.openSync(filepath, 'w'));
--- ASync:
var fs = require("fs");
fs.open(path, "wx", function (err, fd) {
// handle error
fs.close(fd, function (err) {
// handle error
});
});
3) Or use "touch": https://github.com/isaacs/node-touch
Todo this in a single system call you can use the fs-extra npm module.
After this the file will have been created as well as the directory it is to be placed in.
const fs = require('fs-extra');
const file = '/tmp/this/path/does/not/exist/file.txt'
fs.ensureFile(file, err => {
console.log(err) // => null
});
Another way is to use ensureFileSync which will do the same thing but synchronous.
const fs = require('fs-extra');
const file = '/tmp/this/path/does/not/exist/file.txt'
fs.ensureFileSync(file)
With async / await and Typescript I would do:
import * as fs from 'fs'
async function upsertFile(name: string) {
try {
// try to read file
await fs.promises.readFile(name)
} catch (error) {
// create empty file, because it wasn't found
await fs.promises.writeFile(name, '')
}
}
Here's a synchronous way of doing it:
try {
await fs.truncateSync(filepath, 0);
} catch (err) {
await fs.writeFileSync(filepath, "", { flag: "wx" });
}
If the file exists it will get truncated, otherwise it gets created if an error is raised.
This works for me.
// Use the file system fs promises
const {access} = require('fs/promises');
// File Exist returns true
// dont use exists which is no more!
const fexists =async (path)=> {
try {
await access(path);
return true;
} catch {
return false;
}
}
// Wrapper for your main program
async function mainapp(){
if( await fexists("./users.json")){
console.log("File is here");
} else {
console.log("File not here -so make one");
}
}
// run your program
mainapp();
Just keep eye on your async - awaits so everthing plays nice.
hope this helps.
You can do something like this:
function writeFile(i){
var i = i || 0;
var fileName = 'a_' + i + '.jpg';
fs.exists(fileName, function (exists) {
if(exists){
writeFile(++i);
} else {
fs.writeFile(fileName);
}
});
}