Related
I am running a node.js script that culls through a directory and extracts any regex matches it finds, then pushes them onto an array. Once the query has run, I am stringifying the array into a standalone JSON file for safekeeping until it's needed.
I'm running into sporadic errors where the array in the JSON file is malformed and causes an error when I try to parse it out later on. It is sporadic, and I cannot determine a pattern, but every time (so far) the character that is causing the error is a closing square bracket - just like the one between the 2nd and 3rd elements below:
["first string","second string"]"third string"]
It's almost as if my array.push() is just adding another element onto the end of the existing array, and "forgetting" to change the closing square bracket to a comma...
I have searched, but cannot find references to a known error that would cause this. Has anyone seen this before?
Environment:
Node.js v19.0.0 | Ubuntu 22.04.1 LTS | Intel i7 processor
Here's my code for generating the JSON files:
const fs = require('fs');
const path = require('path');
const devPath = '/home/user/directory';
const activeReminder = [];
function buildArray() {
let getFiles = fs.readdirSync(devPath);
getFiles.forEach(file => {
splitMD(file);
});
}
function splitMD(file) {
if (path.extname(file) == ".md" & file != {}) {
let data = fs.readFileSync(file, 'utf8');
const line = data.split(/\r?\n/);
const match = line.find(element => {
extractions(element, file);
});
}
};
function extractions(element, file) {
if (element.includes("- [ ] reminder: ")) { //the - [ ] is markdown for Obsidian
const datedReminder = "- [ ] [["+file+"]]" + element.split('- [ ] reminder:')[1]
activeReminder.push(datedReminder);
buildReminderJSON();
}
}
function buildCcioJSON() {
const reminderJSON = JSON.stringify(activeReminder);
fs.writeFile("reminder.json", reminderJSON, 'utf8', function (err) {
if (err) {
console.log("Error while writing CCIO JSON:");
return console.log(err);
}
})
}
buildArray();
module.exports = { buildArray };
I don't know what's causing the anomaly described above, but I created a script to correct it as a workaround until I can figure out a root cause. I'm running it as a crontab a few minutes after the crontab that creates the original array.
const fs = require('fs')
const path = require('path')
const devPath = '/home/user/directory'
function cleanup() {
let getFiles = fs.readdirSync(devPath);
getFiles.forEach(file => {
splitJSON(file);
});
}
function splitJSON(file) {
if (path.extname(file) == ".json" & file != {}) {
let data = fs.readFileSync(file, 'utf8');
const line = data.split(/\r?\n/);
const match = line.find(element => {
jsonCheck(element, file);
});
}
};
function jsonCheck(element, file) {
if (element.includes("\"]\"")) {
fs.readFile(file, 'utf8', function (err,data) {
if (err) {
return console.log(err);
}
var result = data.replace(/\"\]\"/g, '","');
fs.writeFile(file, result, 'utf8', function (err) {
if (err) return console.log(err);
});
});
}
}
cleanup();
module.exports = { cleanup };
I have a little problem with my function. I would like to get all files in many directories. Currently, I can retrieve the files in the file passed in parameters. I would like to retrieve the html files of each folder in the folder passed as a parameter. I will explain if I put in parameter "test" I retrieve the files in "test" but I would like to retrieve "test / 1 / *. Html", "test / 2 / . /.html ":
var srcpath2 = path.join('.', 'diapo', result);
function getDirectories(srcpath2) {
return fs.readdirSync(srcpath2).filter(function (file) {
return fs.statSync(path.join(srcpath2, file)).isDirectory();
});
}
The result :
[1,2,3]
thanks !
It looks like the glob npm package would help you. Here is an example of how to use it:
File hierarchy:
test
├── one.html
└── test-nested
└── two.html
JS code:
const glob = require("glob");
var getDirectories = function (src, callback) {
glob(src + '/**/*', callback);
};
getDirectories('test', function (err, res) {
if (err) {
console.log('Error', err);
} else {
console.log(res);
}
});
which displays:
[ 'test/one.html',
'test/test-nested',
'test/test-nested/two.html' ]
I've seen many very long answers, and it's kinda a waste of memory space. Some also use packages like glob, but if you don't want to depend on any package, here's my solution.
const Path = require("path");
const FS = require("fs");
let Files = [];
function ThroughDirectory(Directory) {
FS.readdirSync(Directory).forEach(File => {
const Absolute = Path.join(Directory, File);
if (FS.statSync(Absolute).isDirectory()) return ThroughDirectory(Absolute);
else return Files.push(Absolute);
});
}
ThroughDirectory("./input/directory/");
It's pretty self-explanatory. There's an input directory, and it iterates through that. If one of the items is also a directory, go through that and so on. If it's a file, add the absolute path to the array.
Hope this helped :]
Using ES6 yield
const fs = require('fs');
const path = require('path');
function *walkSync(dir) {
const files = fs.readdirSync(dir, { withFileTypes: true });
for (const file of files) {
if (file.isDirectory()) {
yield* walkSync(path.join(dir, file.name));
} else {
yield path.join(dir, file.name);
}
}
}
for (const filePath of walkSync(__dirname)) {
console.log(filePath);
}
I really liked Smally's Solution but didn't like the Syntax.
Same solution but slightly easier to read:
const fs = require("fs");
const path = require("path");
let files = [];
const getFilesRecursively = (directory) => {
const filesInDirectory = fs.readdirSync(directory);
for (const file of filesInDirectory) {
const absolute = path.join(directory, file);
if (fs.statSync(absolute).isDirectory()) {
getFilesRecursively(absolute);
} else {
files.push(absolute);
}
}
};
Here's mine. Like all good answers it's hard to understand:
const isDirectory = path => statSync(path).isDirectory();
const getDirectories = path =>
readdirSync(path).map(name => join(path, name)).filter(isDirectory);
const isFile = path => statSync(path).isFile();
const getFiles = path =>
readdirSync(path).map(name => join(path, name)).filter(isFile);
const getFilesRecursively = (path) => {
let dirs = getDirectories(path);
let files = dirs
.map(dir => getFilesRecursively(dir)) // go through each directory
.reduce((a,b) => a.concat(b), []); // map returns a 2d array (array of file arrays) so flatten
return files.concat(getFiles(path));
};
With modern JavaScript (NodeJs 10) you can use async generator function and loop through them using for-await...of
// ES modules syntax that is included by default in NodeJS 14.
// For earlier versions, use `--experimental-modules` flag
import fs from "fs/promises"
// or, without ES modules, use this:
// const fs = require('fs').promises
async function run() {
for await (const file of getFiles()) {
console.log(file.path)
}
}
async function* getFiles(path = `./`) {
const entries = await fs.readdir(path, { withFileTypes: true })
for (let file of entries) {
if (file.isDirectory()) {
yield* getFiles(`${path}${file.name}/`)
} else {
yield { ...file, path: path + file.name }
}
}
}
run()
Packed into library:
https://www.npmjs.com/package/node-recursive-directory
https://github.com/vvmspace/node-recursive-directory
List of files:
const getFiles = require('node-recursive-directory');
(async () => {
const files = await getFiles('/home');
console.log(files);
})()
List of files with parsed data:
const getFiles = require('node-resursive-directory');
(async () => {
const files = await getFiles('/home', true); // add true
console.log(files);
})()
You will get something like that:
[
...,
{
fullpath: '/home/vvm/Downloads/images/Some/Some Image.jpg',
filepath: '/home/vvm/Downloads/images/Some/',
filename: 'Some Image.jpg',
dirname: 'Some'
},
]
You can also write your own code like below to traverse the directory as shown below :
var fs = require('fs');
function traverseDirectory(dirname, callback) {
var directory = [];
fs.readdir(dirname, function(err, list) {
dirname = fs.realpathSync(dirname);
if (err) {
return callback(err);
}
var listlength = list.length;
list.forEach(function(file) {
file = dirname + '\\' + file;
fs.stat(file, function(err, stat) {
directory.push(file);
if (stat && stat.isDirectory()) {
traverseDirectory(file, function(err, parsed) {
directory = directory.concat(parsed);
if (!--listlength) {
callback(null, directory);
}
});
} else {
if (!--listlength) {
callback(null, directory);
}
}
});
});
});
}
traverseDirectory(__dirname, function(err, result) {
if (err) {
console.log(err);
}
console.log(result);
});
You can check more information about it here : http://www.codingdefined.com/2014/09/how-to-navigate-through-directories-in.html
I needed to so something similar, in an Electron app: get all subfolders in a given base folder, using TypeScript, and came up with this:
import { readdirSync, statSync, existsSync } from "fs";
import * as path from "path";
// recursive synchronous "walk" through a folder structure, with the given base path
getAllSubFolders = (baseFolder, folderList = []) => {
let folders:string[] = readdirSync(baseFolder).filter(file => statSync(path.join(baseFolder, file)).isDirectory());
folders.forEach(folder => {
folderList.push(path.join(baseFolder,folder));
this.getAllSubFolders(path.join(baseFolder,folder), folderList);
});
}
const fs = require('fs');
const path = require('path');
var filesCollection = [];
const directoriesToSkip = ['bower_components', 'node_modules', 'www', 'platforms'];
function readDirectorySynchronously(directory) {
var currentDirectorypath = path.join(__dirname + directory);
var currentDirectory = fs.readdirSync(currentDirectorypath, 'utf8');
currentDirectory.forEach(file => {
var fileShouldBeSkipped = directoriesToSkip.indexOf(file) > -1;
var pathOfCurrentItem = path.join(__dirname + directory + '/' + file);
if (!fileShouldBeSkipped && fs.statSync(pathOfCurrentItem).isFile()) {
filesCollection.push(pathOfCurrentItem);
}
else if (!fileShouldBeSkipped) {
var directorypath = path.join(directory + '\\' + file);
readDirectorySynchronously(directorypath);
}
});
}
readDirectorySynchronously('');
This will fill filesCollection with all the files in the directory and its subdirectories (it's recursive). You have the option to skip some directory names in the directoriesToSkip array.
Speaking of npm packages - another short option is to use fs-readdir-recursive:
const read = require("fs-readdir-recursive");
const foundFiles = read("test");
console.log(foundFiles);
Output:
[ 'one.html', 'test-nested/some_text.txt', 'test-nested/two.html' ]
If you're interested only in files with specific extension (like .html mentioned in the question) you can filter them using .endsWith():
const filteredFiles = read("test").filter(item => item.endsWith(".html"));
The accepted answer needs to install a package.
If you want a native option that is ES6:
import { readdirSync } from 'fs'
import { join } from 'path'
function walk(dir) {
return readdirSync(dir, { withFileTypes: true }).flatMap((file) => file.isDirectory() ? walk(join(dir, file.name)) : join(dir, file.name))
}
This works for me.
Read root directory with readdirSync
Then map over but flatten as we go
if it's a directory, go recursive; else return the filename
If you rather work synchronously with glob, use the glob.sync() function as mentioned in their documentation. Here's the equivalent example provided by #Paul Mougel but written synchronously:
const glob = require("glob");
var getDirectories = function (src) {
return glob.sync(src + '/**/*');
};
var rest = getDirectories('test');
console.log(res);
A solution with Promises based on globby:
import { globby } from 'globby';
(async () => {
const path = '/path/to/dir';
const files = await globby([`${path}/**/*`]);
console.log(files);
// [
// '/path/to/dir/file1.txt',
// '/path/to/dir/subdir/file2.txt',
// ...
// ]
})()
Synchrone method with two option, simple and efficacy.
const path = require('path');const fs = require('fs');
function toHierarchie_files(pathDir, output_normalize=false, removeEmpty=true)
{
var result = {}, enqueue = [pathDir];
//normalize slash separator if output_normalize is true or just return val
output_normalize = output_normalize == false?val => {return val}:val => {return path.normalize(val)};
//allows absolute or relative path with extended resolution. Returns path normalize absolute to work with or 'none' string.
const path_exist = (path_test) => {var tmpTab = fs.existsSync(path.normalize(path.resolve(path_test))) == true?[path.normalize(path.resolve(path_test))]:['', '../', '../../'].map(val => path.normalize(path.resolve(__dirname, val+path_test))).filter((val, index) => fs.existsSync(path.normalize(path.resolve(__dirname, val+path_test))) == true);return tmpTab.length > 0?tmpTab[0]:'none'};
//Check if file exist and return her type or 'none' string
const getType = (path_test) => {path_test = path_exist(path_test);return path_test == 'none'?'none':fs.lstatSync(path_test).isDirectory() == true?'dir':fs.lstatSync(path_test).isFile() == true?'file':'none';};
function recursive()
{
//init new entrie
var parentDir = enqueue.pop();result[parentDir]=[];
//read dir
fs.readdirSync(path_exist(parentDir)).forEach((file, index) =>{
switch(getType(parentDir+'/'+file))
{
//if detect dir push in queue
case 'dir': enqueue.push(output_normalize(parentDir+'/'+file)); break;
//if file, add in entrie
case 'file': result[parentDir].push(file); break;
//else done
default: break;
};
});
//if optional arg remove empty is true, delete entries if not contains files
if(result[parentDir].length == 0 && removeEmpty == true){Reflect.deleteProperty(result, parentDir);}
//if queue is not empty continue processing
if(enqueue.length > 0){recursive();}
};
//if dir renseign exist, go recusive
if(getType(pathDir) == 'dir'){recursive();}
return result;
};
Result:
{
"public/assets": [
"favicon.ico"
],
"public/assets/js": [
"dede.js",
"test.js"
],
"public/assets/js/css/secure": [
"config.json",
"index.js"
],
"public/assets/css": [
"style.css"
]
}
You can use loop through all the files and directories of the root folder, if it's a directory, then get inside it and repeat the process.
Consider the code below:
const fs = require('fs');
const path = require('path');
const target = './'; // choose the directory to target
var result = []
var filePaths = []
var tempFolder = []
const targetPath = fs.readdirSync(target);
function hit(mainPath = targetPath) {
mainPath.forEach((file) => {
let check = fs.statSync(file);
if (!check.isDirectory()) {
filePaths.push(file)
}
else {
if (file[0] != '.') {
tempFolder.push(file)
}
}
});
// get files from folder
if (tempFolder.length > 0) {
tempFolder.forEach((dir) => {
getFiles(dir)
})
}
// filePaths contains path to every file
}
function getFiles(dir) {
var paths = fs.readdirSync(dir);
var files = [];
paths.forEach(function (file) {
var fullPath = dir + '/' + file;
files.push(fullPath);
});
files.forEach((tempFile) => {
let check = fs.statSync(tempFile);
if (check.isDirectory()) {
getFiles(tempFile)
} else {
filePaths.push(tempFile)
}
})
}
hit(); // main function
Although not perfect in some scenarios, it must be helpful in many.
const getAllFilePath = (path: string) => {
const addData = (_paths: string[]) => {
const newFoldersToScrape: string[] = [];
_paths.forEach(_path => {
fs.readdirSync(_path).forEach((file: string) => {
if (file.indexOf(".") === -1) {
newFoldersToScrape.push(`${_path}/${file}`);
} else {
filePaths.push(`${_path}/${file}`);
}
});
});
foldersToScrape = newFoldersToScrape;
};
const baseDirPath = `<YOUR BASE PATH HERE>/${path}`;
let foldersToScrape: string[] = [];
const filePaths: string[] = [];
addData([baseDirPath]);
while (foldersToScrape.length !== 0) {
addData(foldersToScrape);
}
return filePaths;
};
This is how I did it, I think it is similar to yet simpler than most of the other answers here.
const fs = require('fs')
let files = []
const getFiles = (path) => {
if (fs.lstatSync(path).isDirectory()) { // is this a folder?
fs.readdirSync(path).forEach(f => { // for everything in this folder
getFiles(path + '/' + f) // process it recursively
})
} else if (path.endsWith(".ts")) { // is this a file we are searching for?
files.push(path) // record it
}
}
getFiles("src")
It fills the "files" array with every .ts file under the "src/" directory.
Slightly modified version of #Stephen's response (https://stackoverflow.com/a/66083078/4421370) above that returns the files' path relative to the directory you are searching. Or any arbitrary base path you supply to the function call in-place of the default base. If you want the full path just call it as walkSync(dir, dir).
Search Path is: c:\tmp,
File path is c:\tmp\test\myfile.txt,
Result is test\myfile.txt
Hopefully helpful to some.
const fs = require('fs');
const path = require('path');
function *walkSync(dir, base="") {
const files = fs.readdirSync(dir, { withFileTypes: true })
for (const file of files) {
if (file.isDirectory()) {
yield* walkSync(path.join(dir, file.name), path.join(base, file.name));
} else {
yield path.join(base, file.name);
}
}
}
for (const filePath of walkSync(__dirname)) {
console.log(filePath);
}
Here is a compact pure function that returns all the paths (relatives) in the directory.
import path from 'path'
const getFilesPathsRecursively = (directory: string, origin?: string): string[] =>
fs.readdirSync(directory).reduce((files, file) => {
const absolute = path.join(directory, file)
return [
...files,
...(fs.statSync(absolute).isDirectory()
? getFilesPathsRecursively(absolute, origin || directory)
: [path.relative(origin || directory, absolute)]),
]
}, [])
The solution is written in TypeScript.
modern solution with async/await
No external dependencies.
Asynchronous function (non-blocking like other solutions with readdirSync and statSync)
It is extremely fast because multiple processes work in parallel (it is not waiting for a response from every file in the list).
It has also some naive error handling (if something happens with one file or folder it will not blow whole process)
import path from "path";
import fs from "fs/promises"
export default async function readDirectory(directory: string): Promise<string[]> {
const files = await fs.readdir(directory)
const filesPromises = files.map(async (file) => {
try {
const absolutePath = path.join(directory, file);
const fileStat = await fs.stat(absolutePath)
if (fileStat.isDirectory()) {
return await readDirectory(absolutePath);
} else {
return absolutePath;
}
} catch (err) {
// error handling
return [];
}
});
const filesWithArrays = await Promise.all(filesPromises)
const flatArray = filesWithArrays.reduce<string[]>((acc, fileOrArray) => acc.concat(fileOrArray), []);
return flatArray;
}
usage (if this is a separate file please remember to import)
const results = await readDirectory('some/path');
I did mine with typescript works well fairly easy to understand
import * as fs from 'fs';
import * as path from 'path';
export const getAllSubFolders = (
baseFolder: string,
folderList: string[] = []
) => {
const folders: string[] = fs
.readdirSync(baseFolder)
.filter(file => fs.statSync(path.join(baseFolder, file)).isDirectory());
folders.forEach(folder => {
folderList.push(path.join(baseFolder, folder));
getAllSubFolders(path.join(baseFolder, folder), folderList);
});
return folderList;
};
export const getFilesInFolder = (rootPath: string) => {
return fs
.readdirSync(rootPath)
.filter(
filePath => !fs.statSync(path.join(rootPath, filePath)).isDirectory()
)
.map(filePath => path.normalize(path.join(rootPath, filePath)));
};
export const getFilesRecursively = (rootPath: string) => {
const subFolders: string[] = getAllSubFolders(rootPath);
const allFiles: string[][] = subFolders.map(folder =>
getFilesInFolder(folder)
);
return [].concat.apply([], allFiles);
};
I'm not sure what's wrong. I deleted my code and downloaded it then uploaded it again and now I get this error.
Code: https://replit.com/#hi12167pies/webcord#index.js (Click code for code and output for output)
Error:
/home/runner/C8AU9ceLyjc/node_modules/discord.js/src/rest/RESTManager.js:32
const token = this.client.token ?? this.client.accessToken;
^
SyntaxError: Unexpected token '?'
I have no idea whats wrong since it's in the node_modules folder.
If you have problems viewing it here is the code:
const http = require("http")
const discord = require("discord.js")
const client = new discord.Client()
const config = require("./config.json")
const fs = require("fs")
// const readLine = require("readline")
// const rl = readLine.createInterface({
// input: process.stdin,
// output: process.stdout
// })
let msgs = {
"873195510251532348": [],
"873195522633105429": []
}
client.on("ready", () => {
console.log("ready discord")
})
client.on("message", (message) => {
if (message.author.bot) return
if (!config.chats.includes(message.channel.id.toString())) return
msgs[message.channel.id].push({
"username": message.author.tag,
"content": message.content,
"type": "0"
})
})
http.createServer((req,res) => {
const url = req.url.split("?")[0]
let query = {}
req.url.slice(req.url.split("").indexOf("?")).slice(1).split("&").forEach((e) => {
const splited = e.split("=")
query[splited[0]] = splited[1]
})
if (query.q == "messages") {
let msg = []
let i = 0
while (msgs[query.code].length > i) {
const e = msgs[query.code][msgs[query.code].length - (i+1)]
msg.push(e)
i++
}
res.write(JSON.stringify(msg))
res.end()
} else if (query.q == "post") {
let name = query.name.split("%20").join(" ")
let content = query.content.split("%20").join(" ")
client.channels.cache.get(query.code).send(`**${name}**: ${content}`)
msgs[query.code].push({
"username": name,
"content": content,
"type": "1"
})
res.end()
} else if (url == "/robot" && query.istrue == "true") {
res.write("Robot!")
res.end()
} else {
let path
if (!query.code) {
path = "./code.html"
} else {
if (!config.chats.includes(query.code)) {
path = "./invaildcode.html"
} else {
path = "./chat.html"
}
}
fs.readFile(path, (er, da) => {
if (er) res.write("Could not get index.html")
res.write(da)
res.end()
})
}
}).listen(80, (err) => {
if (err) throw err
console.log("listening webserver")
})
client.login(process.env.TOKEN)
I am aware my code is not good right now, I am rewriting it but I still want to know what the error is.
repl.it uses node v12.22.1 but the nullish coalescing operator (??), is relatively new and was added in node v14.
So to use the ?? operator you need to update node in repl.it.
Which you can do by following this repl.it forum post by lukenzy.
Create a file and name it .replit
Inside it, copy and paste the following code:
run = """
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.34.0/install.sh | bash
export NVM_DIR=\"$HOME/.nvm\"
[ -s \"$NVM_DIR/nvm.sh\" ] && \\. \"$NVM_DIR/nvm.sh\"
[ -s \"$NVM_DIR/bash_completion\" ] && \\.\"$NVM_DIR/bash_completion\"
nvm install 14
node index.js
"""
This will install and use the latest Node.js v14 (14.17.4).
If u want to use a different version, change nvm install 14 to any other
number.
Also, change node index.js to the file u want to run.
You are getting this error because you are using an older version of node that didn't support nullable for some packages.
Simply change node version of yours.
You can simply change node versions using 'nvm'. follow this git repo https://github.com/nvm-sh/nvm
Jest has this feature to log the line that outputs to console methods.
In some cases, this can become annoying:
console.log _modules/log.js:37
ℹ login.0 screenshot start
console.time _modules/init.js:409
login.0.screenshot: 0.33ms
console.time _modules/init.js:394
0 | login.0: 0.524ms
console.log _modules/log.js:37
ℹ login.1 screenshot start
Any idea how I can turn it off?
With Jest 24.3.0 or higher, you can do this in pure TypeScript by adding the following to a Jest setup file configured in setupFilesAfterEnv:
import { CustomConsole, LogType, LogMessage } from '#jest/console';
function simpleFormatter(type: LogType, message: LogMessage): string {
const TITLE_INDENT = ' ';
const CONSOLE_INDENT = TITLE_INDENT + ' ';
return message
.split(/\n/)
.map(line => CONSOLE_INDENT + line)
.join('\n');
}
global.console = new CustomConsole(process.stdout, process.stderr, simpleFormatter);
Jest injects custom console implementation that is based on extendable Console class into test global scope. Normally it provides useful debugging information alongside printed message that answers the question where potentially unwanted output comes from.
In case this is undesirable for some reason, a simple way to retrieve default console implementation is to import it from Node built-in module.
Can be done for specific console calls:
let console = require('console');
...
console.log(...)
For many of them that occur inside a range of tests:
const jestConsole = console;
beforeEach(() => {
global.console = require('console');
});
afterEach(() => {
global.console = jestConsole;
});
And so on.
Update: For newer versions of Jest, see Harald Wellmann's answer.
Looking at the source code for Jest, it doesn't seem like there is a neat way to turn those messages off.
However, one possible solution could be to write your own Console. Here I have used Console.js from Jest as a starting ground, and then created SimpleConsole which does what you need (I have removed some terminal coloring features for simplicity, but you could just add them yourself).
Once added to your project, you can overwrite Jest's normal console with your own before running the tests:
const { SimpleConsole } = require('./SimpleConsole');
global.console = new SimpleConsole(process.stdout, process.stderr);
I have made a REPL that shows it in action.
The source code for SimpleConsole:
const path = require('path');
const assert = require('assert');
const {format} = require('util');
const {Console} = require('console');
function simpleFormatter() {
const TITLE_INDENT = ' ';
const CONSOLE_INDENT = TITLE_INDENT + ' ';
return (type, message) => {
message = message
.split(/\n/)
.map(line => CONSOLE_INDENT + line)
.join('\n');
return (
message +
'\n'
);
};
};
class SimpleConsole extends Console {
constructor(stdout, stderr, formatBuffer) {
super(stdout, stderr);
this._formatBuffer = formatBuffer || simpleFormatter();
this._counters = {};
this._timers = {};
this._groupDepth = 0;
}
_logToParentConsole(message) {
super.log(message);
}
_log(type, message) {
if (process.stdout.isTTY) {
this._stdout.write('\x1b[999D\x1b[K');
}
this._logToParentConsole(
this._formatBuffer(type, ' '.repeat(this._groupDepth) + message),
);
}
assert(...args) {
try {
assert(...args);
} catch (error) {
this._log('assert', error.toString());
}
}
count(label = 'default') {
if (!this._counters[label]) {
this._counters[label] = 0;
}
this._log('count', format(`${label}: ${++this._counters[label]}`));
}
countReset(label = 'default') {
this._counters[label] = 0;
}
debug(...args) {
this._log('debug', format(...args));
}
dir(...args) {
this._log('dir', format(...args));
}
dirxml(...args) {
this._log('dirxml', format(...args));
}
error(...args) {
this._log('error', format(...args));
}
group(...args) {
this._groupDepth++;
if (args.length > 0) {
this._log('group', chalk.bold(format(...args)));
}
}
groupCollapsed(...args) {
this._groupDepth++;
if (args.length > 0) {
this._log('groupCollapsed', chalk.bold(format(...args)));
}
}
groupEnd() {
if (this._groupDepth > 0) {
this._groupDepth--;
}
}
info(...args) {
this._log('info', format(...args));
}
log(...args) {
this._log('log', format(...args));
}
time(label = 'default') {
if (this._timers[label]) {
return;
}
this._timers[label] = new Date();
}
timeEnd(label = 'default') {
const startTime = this._timers[label];
if (startTime) {
const endTime = new Date();
const time = endTime - startTime;
this._log('time', format(`${label}: ${time}ms`));
delete this._timers[label];
}
}
warn(...args) {
this._log('warn', format(...args));
}
getBuffer() {
return null;
}
}
module.exports.SimpleConsole = SimpleConsole;
If you want to use Haral Wellmann's answer but avoid typescript, then you can merely do something like:
const JestConsole = require('./node_modules/#jest/console');
global.console = new JestConsole.CustomConsole(process.stdout, process.stderr, (type, message) => {
const TITLE_INDENT = ' ';
const CONSOLE_INDENT = TITLE_INDENT + ' ';
return message.split(/\n/).map(line => CONSOLE_INDENT + line).join('\n');
});
None of the above options worked for me.
The (current) simplest solution is this:
1: Create a file with this code (e.g. config.js)
import console from "console"
global.console = console
2: Add this line to your jest.config.js
setupFilesAfterEnv: ["./config.js"]
Before:
After:
Enjoi!
I have a little problem with my function. I would like to get all files in many directories. Currently, I can retrieve the files in the file passed in parameters. I would like to retrieve the html files of each folder in the folder passed as a parameter. I will explain if I put in parameter "test" I retrieve the files in "test" but I would like to retrieve "test / 1 / *. Html", "test / 2 / . /.html ":
var srcpath2 = path.join('.', 'diapo', result);
function getDirectories(srcpath2) {
return fs.readdirSync(srcpath2).filter(function (file) {
return fs.statSync(path.join(srcpath2, file)).isDirectory();
});
}
The result :
[1,2,3]
thanks !
It looks like the glob npm package would help you. Here is an example of how to use it:
File hierarchy:
test
├── one.html
└── test-nested
└── two.html
JS code:
const glob = require("glob");
var getDirectories = function (src, callback) {
glob(src + '/**/*', callback);
};
getDirectories('test', function (err, res) {
if (err) {
console.log('Error', err);
} else {
console.log(res);
}
});
which displays:
[ 'test/one.html',
'test/test-nested',
'test/test-nested/two.html' ]
I've seen many very long answers, and it's kinda a waste of memory space. Some also use packages like glob, but if you don't want to depend on any package, here's my solution.
const Path = require("path");
const FS = require("fs");
let Files = [];
function ThroughDirectory(Directory) {
FS.readdirSync(Directory).forEach(File => {
const Absolute = Path.join(Directory, File);
if (FS.statSync(Absolute).isDirectory()) return ThroughDirectory(Absolute);
else return Files.push(Absolute);
});
}
ThroughDirectory("./input/directory/");
It's pretty self-explanatory. There's an input directory, and it iterates through that. If one of the items is also a directory, go through that and so on. If it's a file, add the absolute path to the array.
Hope this helped :]
Using ES6 yield
const fs = require('fs');
const path = require('path');
function *walkSync(dir) {
const files = fs.readdirSync(dir, { withFileTypes: true });
for (const file of files) {
if (file.isDirectory()) {
yield* walkSync(path.join(dir, file.name));
} else {
yield path.join(dir, file.name);
}
}
}
for (const filePath of walkSync(__dirname)) {
console.log(filePath);
}
I really liked Smally's Solution but didn't like the Syntax.
Same solution but slightly easier to read:
const fs = require("fs");
const path = require("path");
let files = [];
const getFilesRecursively = (directory) => {
const filesInDirectory = fs.readdirSync(directory);
for (const file of filesInDirectory) {
const absolute = path.join(directory, file);
if (fs.statSync(absolute).isDirectory()) {
getFilesRecursively(absolute);
} else {
files.push(absolute);
}
}
};
Here's mine. Like all good answers it's hard to understand:
const isDirectory = path => statSync(path).isDirectory();
const getDirectories = path =>
readdirSync(path).map(name => join(path, name)).filter(isDirectory);
const isFile = path => statSync(path).isFile();
const getFiles = path =>
readdirSync(path).map(name => join(path, name)).filter(isFile);
const getFilesRecursively = (path) => {
let dirs = getDirectories(path);
let files = dirs
.map(dir => getFilesRecursively(dir)) // go through each directory
.reduce((a,b) => a.concat(b), []); // map returns a 2d array (array of file arrays) so flatten
return files.concat(getFiles(path));
};
With modern JavaScript (NodeJs 10) you can use async generator function and loop through them using for-await...of
// ES modules syntax that is included by default in NodeJS 14.
// For earlier versions, use `--experimental-modules` flag
import fs from "fs/promises"
// or, without ES modules, use this:
// const fs = require('fs').promises
async function run() {
for await (const file of getFiles()) {
console.log(file.path)
}
}
async function* getFiles(path = `./`) {
const entries = await fs.readdir(path, { withFileTypes: true })
for (let file of entries) {
if (file.isDirectory()) {
yield* getFiles(`${path}${file.name}/`)
} else {
yield { ...file, path: path + file.name }
}
}
}
run()
Packed into library:
https://www.npmjs.com/package/node-recursive-directory
https://github.com/vvmspace/node-recursive-directory
List of files:
const getFiles = require('node-recursive-directory');
(async () => {
const files = await getFiles('/home');
console.log(files);
})()
List of files with parsed data:
const getFiles = require('node-resursive-directory');
(async () => {
const files = await getFiles('/home', true); // add true
console.log(files);
})()
You will get something like that:
[
...,
{
fullpath: '/home/vvm/Downloads/images/Some/Some Image.jpg',
filepath: '/home/vvm/Downloads/images/Some/',
filename: 'Some Image.jpg',
dirname: 'Some'
},
]
You can also write your own code like below to traverse the directory as shown below :
var fs = require('fs');
function traverseDirectory(dirname, callback) {
var directory = [];
fs.readdir(dirname, function(err, list) {
dirname = fs.realpathSync(dirname);
if (err) {
return callback(err);
}
var listlength = list.length;
list.forEach(function(file) {
file = dirname + '\\' + file;
fs.stat(file, function(err, stat) {
directory.push(file);
if (stat && stat.isDirectory()) {
traverseDirectory(file, function(err, parsed) {
directory = directory.concat(parsed);
if (!--listlength) {
callback(null, directory);
}
});
} else {
if (!--listlength) {
callback(null, directory);
}
}
});
});
});
}
traverseDirectory(__dirname, function(err, result) {
if (err) {
console.log(err);
}
console.log(result);
});
You can check more information about it here : http://www.codingdefined.com/2014/09/how-to-navigate-through-directories-in.html
I needed to so something similar, in an Electron app: get all subfolders in a given base folder, using TypeScript, and came up with this:
import { readdirSync, statSync, existsSync } from "fs";
import * as path from "path";
// recursive synchronous "walk" through a folder structure, with the given base path
getAllSubFolders = (baseFolder, folderList = []) => {
let folders:string[] = readdirSync(baseFolder).filter(file => statSync(path.join(baseFolder, file)).isDirectory());
folders.forEach(folder => {
folderList.push(path.join(baseFolder,folder));
this.getAllSubFolders(path.join(baseFolder,folder), folderList);
});
}
const fs = require('fs');
const path = require('path');
var filesCollection = [];
const directoriesToSkip = ['bower_components', 'node_modules', 'www', 'platforms'];
function readDirectorySynchronously(directory) {
var currentDirectorypath = path.join(__dirname + directory);
var currentDirectory = fs.readdirSync(currentDirectorypath, 'utf8');
currentDirectory.forEach(file => {
var fileShouldBeSkipped = directoriesToSkip.indexOf(file) > -1;
var pathOfCurrentItem = path.join(__dirname + directory + '/' + file);
if (!fileShouldBeSkipped && fs.statSync(pathOfCurrentItem).isFile()) {
filesCollection.push(pathOfCurrentItem);
}
else if (!fileShouldBeSkipped) {
var directorypath = path.join(directory + '\\' + file);
readDirectorySynchronously(directorypath);
}
});
}
readDirectorySynchronously('');
This will fill filesCollection with all the files in the directory and its subdirectories (it's recursive). You have the option to skip some directory names in the directoriesToSkip array.
Speaking of npm packages - another short option is to use fs-readdir-recursive:
const read = require("fs-readdir-recursive");
const foundFiles = read("test");
console.log(foundFiles);
Output:
[ 'one.html', 'test-nested/some_text.txt', 'test-nested/two.html' ]
If you're interested only in files with specific extension (like .html mentioned in the question) you can filter them using .endsWith():
const filteredFiles = read("test").filter(item => item.endsWith(".html"));
The accepted answer needs to install a package.
If you want a native option that is ES6:
import { readdirSync } from 'fs'
import { join } from 'path'
function walk(dir) {
return readdirSync(dir, { withFileTypes: true }).flatMap((file) => file.isDirectory() ? walk(join(dir, file.name)) : join(dir, file.name))
}
This works for me.
Read root directory with readdirSync
Then map over but flatten as we go
if it's a directory, go recursive; else return the filename
If you rather work synchronously with glob, use the glob.sync() function as mentioned in their documentation. Here's the equivalent example provided by #Paul Mougel but written synchronously:
const glob = require("glob");
var getDirectories = function (src) {
return glob.sync(src + '/**/*');
};
var rest = getDirectories('test');
console.log(res);
A solution with Promises based on globby:
import { globby } from 'globby';
(async () => {
const path = '/path/to/dir';
const files = await globby([`${path}/**/*`]);
console.log(files);
// [
// '/path/to/dir/file1.txt',
// '/path/to/dir/subdir/file2.txt',
// ...
// ]
})()
Synchrone method with two option, simple and efficacy.
const path = require('path');const fs = require('fs');
function toHierarchie_files(pathDir, output_normalize=false, removeEmpty=true)
{
var result = {}, enqueue = [pathDir];
//normalize slash separator if output_normalize is true or just return val
output_normalize = output_normalize == false?val => {return val}:val => {return path.normalize(val)};
//allows absolute or relative path with extended resolution. Returns path normalize absolute to work with or 'none' string.
const path_exist = (path_test) => {var tmpTab = fs.existsSync(path.normalize(path.resolve(path_test))) == true?[path.normalize(path.resolve(path_test))]:['', '../', '../../'].map(val => path.normalize(path.resolve(__dirname, val+path_test))).filter((val, index) => fs.existsSync(path.normalize(path.resolve(__dirname, val+path_test))) == true);return tmpTab.length > 0?tmpTab[0]:'none'};
//Check if file exist and return her type or 'none' string
const getType = (path_test) => {path_test = path_exist(path_test);return path_test == 'none'?'none':fs.lstatSync(path_test).isDirectory() == true?'dir':fs.lstatSync(path_test).isFile() == true?'file':'none';};
function recursive()
{
//init new entrie
var parentDir = enqueue.pop();result[parentDir]=[];
//read dir
fs.readdirSync(path_exist(parentDir)).forEach((file, index) =>{
switch(getType(parentDir+'/'+file))
{
//if detect dir push in queue
case 'dir': enqueue.push(output_normalize(parentDir+'/'+file)); break;
//if file, add in entrie
case 'file': result[parentDir].push(file); break;
//else done
default: break;
};
});
//if optional arg remove empty is true, delete entries if not contains files
if(result[parentDir].length == 0 && removeEmpty == true){Reflect.deleteProperty(result, parentDir);}
//if queue is not empty continue processing
if(enqueue.length > 0){recursive();}
};
//if dir renseign exist, go recusive
if(getType(pathDir) == 'dir'){recursive();}
return result;
};
Result:
{
"public/assets": [
"favicon.ico"
],
"public/assets/js": [
"dede.js",
"test.js"
],
"public/assets/js/css/secure": [
"config.json",
"index.js"
],
"public/assets/css": [
"style.css"
]
}
You can use loop through all the files and directories of the root folder, if it's a directory, then get inside it and repeat the process.
Consider the code below:
const fs = require('fs');
const path = require('path');
const target = './'; // choose the directory to target
var result = []
var filePaths = []
var tempFolder = []
const targetPath = fs.readdirSync(target);
function hit(mainPath = targetPath) {
mainPath.forEach((file) => {
let check = fs.statSync(file);
if (!check.isDirectory()) {
filePaths.push(file)
}
else {
if (file[0] != '.') {
tempFolder.push(file)
}
}
});
// get files from folder
if (tempFolder.length > 0) {
tempFolder.forEach((dir) => {
getFiles(dir)
})
}
// filePaths contains path to every file
}
function getFiles(dir) {
var paths = fs.readdirSync(dir);
var files = [];
paths.forEach(function (file) {
var fullPath = dir + '/' + file;
files.push(fullPath);
});
files.forEach((tempFile) => {
let check = fs.statSync(tempFile);
if (check.isDirectory()) {
getFiles(tempFile)
} else {
filePaths.push(tempFile)
}
})
}
hit(); // main function
Although not perfect in some scenarios, it must be helpful in many.
const getAllFilePath = (path: string) => {
const addData = (_paths: string[]) => {
const newFoldersToScrape: string[] = [];
_paths.forEach(_path => {
fs.readdirSync(_path).forEach((file: string) => {
if (file.indexOf(".") === -1) {
newFoldersToScrape.push(`${_path}/${file}`);
} else {
filePaths.push(`${_path}/${file}`);
}
});
});
foldersToScrape = newFoldersToScrape;
};
const baseDirPath = `<YOUR BASE PATH HERE>/${path}`;
let foldersToScrape: string[] = [];
const filePaths: string[] = [];
addData([baseDirPath]);
while (foldersToScrape.length !== 0) {
addData(foldersToScrape);
}
return filePaths;
};
This is how I did it, I think it is similar to yet simpler than most of the other answers here.
const fs = require('fs')
let files = []
const getFiles = (path) => {
if (fs.lstatSync(path).isDirectory()) { // is this a folder?
fs.readdirSync(path).forEach(f => { // for everything in this folder
getFiles(path + '/' + f) // process it recursively
})
} else if (path.endsWith(".ts")) { // is this a file we are searching for?
files.push(path) // record it
}
}
getFiles("src")
It fills the "files" array with every .ts file under the "src/" directory.
Slightly modified version of #Stephen's response (https://stackoverflow.com/a/66083078/4421370) above that returns the files' path relative to the directory you are searching. Or any arbitrary base path you supply to the function call in-place of the default base. If you want the full path just call it as walkSync(dir, dir).
Search Path is: c:\tmp,
File path is c:\tmp\test\myfile.txt,
Result is test\myfile.txt
Hopefully helpful to some.
const fs = require('fs');
const path = require('path');
function *walkSync(dir, base="") {
const files = fs.readdirSync(dir, { withFileTypes: true })
for (const file of files) {
if (file.isDirectory()) {
yield* walkSync(path.join(dir, file.name), path.join(base, file.name));
} else {
yield path.join(base, file.name);
}
}
}
for (const filePath of walkSync(__dirname)) {
console.log(filePath);
}
Here is a compact pure function that returns all the paths (relatives) in the directory.
import path from 'path'
const getFilesPathsRecursively = (directory: string, origin?: string): string[] =>
fs.readdirSync(directory).reduce((files, file) => {
const absolute = path.join(directory, file)
return [
...files,
...(fs.statSync(absolute).isDirectory()
? getFilesPathsRecursively(absolute, origin || directory)
: [path.relative(origin || directory, absolute)]),
]
}, [])
The solution is written in TypeScript.
modern solution with async/await
No external dependencies.
Asynchronous function (non-blocking like other solutions with readdirSync and statSync)
It is extremely fast because multiple processes work in parallel (it is not waiting for a response from every file in the list).
It has also some naive error handling (if something happens with one file or folder it will not blow whole process)
import path from "path";
import fs from "fs/promises"
export default async function readDirectory(directory: string): Promise<string[]> {
const files = await fs.readdir(directory)
const filesPromises = files.map(async (file) => {
try {
const absolutePath = path.join(directory, file);
const fileStat = await fs.stat(absolutePath)
if (fileStat.isDirectory()) {
return await readDirectory(absolutePath);
} else {
return absolutePath;
}
} catch (err) {
// error handling
return [];
}
});
const filesWithArrays = await Promise.all(filesPromises)
const flatArray = filesWithArrays.reduce<string[]>((acc, fileOrArray) => acc.concat(fileOrArray), []);
return flatArray;
}
usage (if this is a separate file please remember to import)
const results = await readDirectory('some/path');
I did mine with typescript works well fairly easy to understand
import * as fs from 'fs';
import * as path from 'path';
export const getAllSubFolders = (
baseFolder: string,
folderList: string[] = []
) => {
const folders: string[] = fs
.readdirSync(baseFolder)
.filter(file => fs.statSync(path.join(baseFolder, file)).isDirectory());
folders.forEach(folder => {
folderList.push(path.join(baseFolder, folder));
getAllSubFolders(path.join(baseFolder, folder), folderList);
});
return folderList;
};
export const getFilesInFolder = (rootPath: string) => {
return fs
.readdirSync(rootPath)
.filter(
filePath => !fs.statSync(path.join(rootPath, filePath)).isDirectory()
)
.map(filePath => path.normalize(path.join(rootPath, filePath)));
};
export const getFilesRecursively = (rootPath: string) => {
const subFolders: string[] = getAllSubFolders(rootPath);
const allFiles: string[][] = subFolders.map(folder =>
getFilesInFolder(folder)
);
return [].concat.apply([], allFiles);
};