I'm trying to write a content manglement program in Node. I'm an old Ruby / Perl / Shell hand for many years, and I can't seem to get simple code that works in those languages to look similarly, eh, simple, in Node.
Task: Find all the *.md files, read them (in ls order), transform them, and bracket them with a header comment and a footer comment. The files, in order, have a pieces of Markdown that, when assembled and transformed, are a sensible HTML doc. Here's a shell implementation:
echo '<!-- Generate at:' $(date) ' -->' $(ls *.md |xargs cat|markdown)'<!-- Copyright Mumble-demo Inc. -->'
Produces the desired HTML:
<!-- Generate at: Tue Jun 6 08:25:59 EDT 2017 --> <h1>This is a Markdown File</h1> <h2>Heading 1</h2> <p>Inside of markdown we can create many interesting items</p> <ul> <li>such</li> <li>as</li> <li>lists</li> </ul><!-- Copyright Mumble-demo Inc. -->
Ruby is similarly reasonable...
#!/usr/bin/env ruby
require 'kramdown'
HEADER = "<!-- Generated at #{Time.now} -->\n"
FOOTER = "\n<!-- Copyright Mumble-demo Inc. -->"
OUTPUT = File.open("./output", "w")
results = Dir.glob("*.md").map { |f| File.open(f).readlines.join() }.reduce(:+)
OUTPUT.print(HEADER, Kramdown::Document.new(results).to_html, FOOTER)
But I can't figure out how to do this in Node in a Way That Feels Right(™)
A Way That Feels Wrong(™) is with the synchronous interfaces:
const fs = require("fs")
const marked = require("marked")
const HEADER = `<!-- Generated at ${new Date()} -->\n`
const FOOTER = `\n<!-- Copyright Mumble-demo Inc. -->`
fs.readdir(".", (err, files) => {
if (err) throw err;
let markdownFiles = files.filter((f) => f.endsWith(".md"))
let content = markdownFiles.reduce((memo, fileName) => {
return memo + fs.readFileSync(fileName, 'utf8')
}, "")
let contentString = [HEADER, marked(content), FOOTER].reduce((m, i) => m + i, "")
fs.writeFileSync("derp", contentString);
console.log(contentString);
})
A Way That Feels Right But That I Can't Get To Work(™) is:
Build read streams
Pipe them to markdown transform streams
Open an output stream and redirect transformed data to it
Great news is – this approach works until it comes time to put the header comments in at the top and the bottom. They live in the code, not on the filesystem so I can't "just add" them as another file to stream, sans transform, into the output stream. Most approaches wind up producing: header, footer, streamed data
Obviously the pipe()-work works asynchronously and the footer print fires before the read + transform work is done. I've tried horrible (and broken) Promise chains that ultimately did not work.
One alternate approach would be to turn the header and footer into streams (seems weird...) and flow them into the output stream as well (seems really weird).
I've stumped several seasoned developers with this...surely we're missing some common idiom here or is it actually this hard to do this simple task simply in Node?
Thoughts:
For most of my shell scripts, I simply read file contents into strings, synchronously. This approach doesn’t scale, but it usually doesn’t need to. And everything is so much easier with strings.
If you do anything asynchronous: Use async functions and util.promisify().
Long-term, asynchronous iteration and async generators will help with this kind of scenario, too.
You can run it "A Way That Feels Right" via synchronous executor nsynjs. Your code may transform similarly to this working example:
md-cat.js:
var nsynjs = require('nsynjs');
var nsynFs = require('../wrappers/nodeFs'); // part of nsynjs package, needs to be added manually
var synchronousCode = function(nsynFs) {
var HEADER = "<!-- Generated at "+new Date()+" -->\n";
var FOOTER = "\n<!-- Copyright Mumble-demo Inc. -->";
var files = nsynFs.readdir(nsynjsCtx, ".").data;
var content="";
for(var i=0; i<files.length; i++) {
var file = files[i];
if(file.endsWith('.md'))
content+=nsynFs.readFile(nsynjsCtx,file,"utf8").data;
}
nsynFs.writeFile(nsynjsCtx,"derp",HEADER+content+FOOTER);
};
nsynjs.run(synchronousCode, {},nsynFs, function () {
console.log('synchronousCode done')
});
Even though it looks synchronous, it does not use any synchronous functions under the hood, therefore it will not block node's event loop.
Try Gulp, which is a most idiomatic way nowadays.
If can't or don't want, use Promise chains, they feels like shell pipes.
#!/usr/bin/env node
'use strict';
const Promise = require('bluebird');
const fs = Promise.promisifyAll(require('fs'));
const path = require('path');
const marked = require('marked');
const HEADER = `<!-- Generated at ${(new Date()).toISOString()} -->`;
const FOOTER = '<!-- Copyright Mumble-demo Inc. -->';
fs.readdirAsync(process.cwd())
.map((fileName) => Promise.all([fileName, fs.statAsync(fileName)]))
.filter(([fileName, stat]) => stat.isFile() && path.extname(fileName) === '.md')
.call('sort', ([a], [b]) => a.localeCompare(b, 'en-US'))
.map(([mdFileName]) => fs.readFileAsync(mdFileName, 'utf8'))
.then((mdFiles) => {
let out = [HEADER, marked(mdFiles.join('\n')), FOOTER].join('').replace(/\n/g, '');
console.log(out);
return fs.writeFileAsync('out.html', out);
})
.catch((err) => {
console.error(err);
process.exit(1);
});
Thoughts:
Never write sync code in Node, you will always regret.
Promise chains work best for such tasks.
stat.isFile() and sort(), are just safety features, missing in bash and Ruby examples. Removing them you can save two lines of code.
Usage of Date.prototype.toString() should be considered as a bug in most cases, because output is unpredictable, it is platform and locale specific.
Node streams are overkill until you deal with huge files, which is usually not a case for markdown tasks.
Shell pipes are also not using file system streams and load everything in a memory. Efficiency roughly the same.
I'm using fs.writeFileSync(file, data[, options]) to save a file returned from http.get(options[, callback])
This works fine for text files but images, pdfs etc end up being corrupted. From the searching around that I've done, it's apparently because fs.writeFileSync(file, data[, options]) defaults to UTF-8
I've tried setting the encoding to 'binary', the mime-type and the extension to no avail. It feels like something really obvious that I'm overlooking, can anyone point me in the right direction?
Thank you in advance
Update
I'm running this through electron. I didn't think it was worth mentioning as electron is just running node, but I'm not a node or electron expert so I'm not sure
Create a Buffer from the image data and set its encoding to binary. Then pass that data into a stream.PassThrough and pipe that into a stream.Writable.
var fs = require('fs');
var stream = require('stream');
var imgStream = new stream.PassThrough();
imgStream.end(Buffer.from(data, 'binary'));
var wStream = fs.createWriteStream('./<dest>.<ext>');
imgStream.once('end', () => {
console.log('Image Written');
});
imgStream.once('error', (err) => {
console.log(err);
});
imgStream.pipe(wStream);
I have the following simplified gulp task:
gulp.src(...)
.pipe(stuff())
.pipe(moreStuff())
.pipe(imagemin())
.pipe(yetMoreStuff());
I only want the imagemin stream to be called when the file path contains "xyz", but I want the other three streams to always be called.
Called gulp.src() in another place is not appropriate—this example is massively simplified, and duplicating everything would be messy as hell.
So far, I've got this far:
var through = require('through2');
gulp.src(...)
.pipe(stuff())
.pipe(moreStuff())
.pipe(through.obj(function (file, enc, cb) {
console.log(file.path.indexOf('hero') !== -1);
// file has a pipe method but what do I do?!
}))
.pipe(yetMoreStuff());
Doesn't do anything. I don't know vinyl / streams well enough to be able to do this by myself :(
How do I do this?
It sounds like gulp-filter might be what you're looking for.
var Filter = require('gulp-filter');
var filter = Filter(['**xyz**']);
gulp.src(...)
.pipe(stuff())
.pipe(moreStuff())
.pipe(filter)
.pipe(imagemin())
.pipe(filter.restore())
.pipe(yetMoreStuff());
I'm piping a file through a duplex string (courtesy of through) and I'm having trouble printing information to stdout and writing to the file. One or the other works just fine.
var fs = require('fs');
var path = require('path');
var through = require('through'); // easy duplexing, i'm young
catify = new through(function(data){
this.queue(data.toString().replace(/(woof)/gi, 'meow'));
});
var reader = fs.createReadStream('dogDiary.txt'); // woof woof etc.
var writer = fs.createWriteStream(path.normalize('generated/catDiary.txt')); // meow meow etc.
// yay!
reader.pipe(catify).pipe(writer)
// blank file. T_T
reader.pipe(catify).pipe(process.stdout).pipe(writer)
I'm assuming this is because process.stdout is a writeable stream, but I'm not sure how to do what I want (i've tried passing {end: false} to no avail).
Still struggling to wrap my head around streams, so forgive me if i've missed something obvious : )
I think what you want is:
reader.pipe(catify)
catify.pipe(writer)
catify.pipe(process.stdout)
These needed to be separated because pipes return their destinations and not their source.
New to NodeJS.
Yes I know I could use a framework, but I want to get a good grok on it before delving into the myriad of fine fine tools that are out there.
my problem:
var img = fs.readFileSync(path);
the above works;
fs.readFile(path, function (err, data)
{
if (err) throw err;
console.log(data);
});
the above doesn't work;
the input path is : 'C:\NodeSite\chrome.jpg'
oh and working on Windows 7.
any help would be much appreciated.
Fixed
Late night/morning programming, introduces errors that are hard to spot. The path was being set from two different places, and so the source path were different in both cases. Thankyou for your help. I am a complete numpty. :)
If you are not setting an encoding when reading a file, you will get the binary content.
So for example, the following snippet will output the content of the test file using UTF-8 encoding. If you don't use an encoding, you will get an output like "" on your console (raw binary buffer).
var fs = require('fs');
var path = "C:\\tmp\\testfile.txt";
fs.readFile(path, 'utf8', function (err, data) {
if (err) throw err;
console.log(data);
});
Another issue (especially on windows-based OS's) can be the correct escaping of the target path. The above example shows how path's on Windows have to be escaped.
java guys will just use this javascript asynchronous command as if in pure java , troublefreely :
var fs = require('fs');
var Contenu = fs.readFileSync( fILE_FULL_Name , 'utf8');
console.log( Contenu );
That should take care of small & big files.