I developed a package called transpile-md-to-json that transpile multiple markdown files to a single JSON file.
Running transpile-md-to-json --src examples/content --watch transpiles markdown files to JSON and outputs result to stdout as markdown files are created, edited and deleted.
I tried using get-stdin to capture the JSON and process it some more using another node script.
transpile-md-to-json --src src/privacy-guides --blogify --watch | node test.js
Problem is stdin.on('end') is never fired because the pipe isn’t closed by transpile-md-to-json when watch mode is enabled (--watch).
See https://github.com/sindresorhus/get-stdin/blob/master/index.js#L23-L25
How can I work around this?
As pointed out by Mike in the comments, there appears to be no built-in way of achieving this as the pipe remains open until the parent exits, therefore stdin.on('end') is not fired.
The closest we can get is to use some kind of EOF indicator and use that to end a "cycle". An indicator we can hook to isn’t always present, but in the context of JSON, we’re good as each JSON payload ends with }.
const readline = require("readline")
const fs = require("fs")
process.stdin.setEncoding("utf-8")
const rl = readline.createInterface({
input: process.stdin,
})
var json = ""
rl.on("line", function(line) {
json += `${line}\n`
if (line === "}") {
console.log("done", json)
fs.writeFileSync("test.json", json)
json = ""
}
})
Related
I want to get a value from input
const readline = require('node:readline/promises').createInterface({
input: process.stdin,
output: process.stdout
})
const value = await readline.question('enter :');
console.log(value)
readline.close()
And then I get an Error "SyntaxError: await is only valid in async functions and the top level bodies of modules"
On the other hand in the docs example:
import * as readline from 'node:readline/promises';
import { stdin as input, stdout as output } from 'process';
const rl = readline.createInterface({ input, output });
const answer = await rl.question(
'What do you think of Node.js? '
);
console.log(
`Thank you for your valuable feedback: ${answer}`
);
rl.close();
I copy this example and get the same error. I cant use readline synchronously. I want input a value, some variable will equals this value, and THEN it will console.log(). I dont want to use "Prompt" module. TY
Your issue is because you are trying to use top-level-await.
Now this is possible in modern nodejs (16+ I think), but you need to tell explicitly node to use modules, like your in your second code.
Try to upgrade nodeJS to the latest version 18, it should run fine.
(Also you may need to hint node on using modules explicitly, for that either name your file using the extension .mjs or specify "type": "module" in your package.json
Working example using .mjs and current node v18.6 the code is copy pasted from your question, no changes made.
I'd think this would be simple, but the following does not work as expected.
I want to pipe data to a process, say (just an arbitrary command for illustration) wc, from Node.
The docs and other SO questions seem to indicate that passing a Stream should work:
const {spawnSync} = require('child_process')
const {Readable} = require('stream')
const textStream = new Readable()
textStream.push("one two three")
textStream.push(null)
const stdio = [textStream, process.stdout, process.stderr]
spawnSync('wc', ["-c"], { stdio })
Unfortunately this throws an error:
The value "Readable { ... } is invalid for option "stdio"
The relevant bit of code from internal/child_process.js does not immediately reveal what the anticipated valid options are.
To present particular data as stdin data for the child process, you can use the input option:
spawnSync('wc', ['-c'], { input : 'one two three' })
Trying to spawn a process with node and read its output.
I would like the output to be in a file and to be able to read it.
This is the code I have so far, but it throws an error -
const outFile = fs.openSync('out.log', 'a')
const errFile = fs.openSync('err.log', 'a')
const child = childProcess.spawn('node', [pathToJsFile], {
stdio: ['ignore', outFile, errFile],
detached: true
})
child.unref()
console.log(child.stdio)
console.log('waiting for output')
child.stdio[1].on('data', (data)=> { // ==> get error since stdio[1] is null
As mentioned in the comment, when I look in child.stdio I see [null, null, null]
However, when I look at the file, I can see the output is written.
I am using node 4.2.1
What am I doing wrong and how can I make this work?
You are wiring up the child process's output to a filesystem file 'out.log', so it goes there and therefore is not also directly available via stdout. You'll need to directly read the output file by filesystem path using the fs core module. var outBuffer = fs.readSync('out.log');
I use Node.JS to fetch files from my S3 bucket.
The files over there are gzipped (gz).
I know that the contents of each file is composed by lines, where each line is a JSON of some record that failed to be put on Kinesis.
Each file consists of ~12K such records. and I would like to be able to process the records while the file is being downloaded.
If the file was not gzipped, that could be easily done using streams and readline module.
So, the only thing that stopping me from doing this is the gunzip process which, to my knowledge, needs to be executed on the whole file.
Is there any way of gunzipping a partial of a file?
Thanks.
EDIT 1: (bad example)
Trying what #Mark Adler suggested:
const fileStream = s3.getObject(params).createReadStream();
const lineReader = readline.createInterface({input: fileStream});
lineReader.on('line', line => {
const gunzipped = zlib.gunzipSync(line);
console.log(gunzipped);
})
I get the following error:
Error: incorrect header check
at Zlib._handle.onerror (zlib.js:363:17)
Yes. node.js has a complete interface to zlib, which allows you to decompress as much of a gzip file at a time as you like.
A working example that solves the above problem
The following solves the problem in the above code:
const fileStream = s3.getObject(params).createReadStream().pipe(zlib.createGunzip());
const lineReader = readline.createInterface({input: fileStream});
lineReader.on('line', gunzippedLine => {
console.log(gunzippedLine);
})
I try to pipe appjs console to a file with this code:
var fs = require('fs');
var logStream = fs.createWriteStream(__dirname+ '/log.txt', { flags: 'a' });
process.stdout.pipe(logStream);
process.stderr.pipe(logStream);
console.log("test");
It creates an empty file, but nothing more... With node.exe the "test" goes into the console, not into the log file. The platform is win32, but I don't think it counts.
What's the problem with the code?
conclusion:
Stdout, stderr and a file write stream are all sink type endpoints, so I cannot bind them together. I need to replace stdout and stderr with douplex mock streams so I will be able to bind these mock streams both to the original sinks and the log sink. I am not sure whether console.log and console.error will be affected by replacing the streams with the mechanism supernova suggested, I'd rather use a dedicated logger, which uses the console instead of this workaround.
you have to define getters for process.stdin, process.stdout and process.stderr
var fs = require("fs")
, errlog = fs.createWriteStream("./err.log", { flags: 'a' })
process.__defineGetter__("stderr", function(){
return errlog
})
process.stderr.write("test")
this should work