I'm writing a cli in node, I want to open the users $EDITOR to edit data that is read from a stream (an http response IncomingMessage).
How can I send the data to a file descriptor?
In bash I could write this:
$EDITOR <(curl $url)
or
$DIFF <(curl $url_1) <(curl $url_2)
<(curl $url) expands to something like /proc/self/fd/11
echo <(curl $url)
/proc/self/fd/11
But how would I write it in javascript?
import cp from 'child_process'
const fisrt = request(...);
const second = require(...);
const first_fd = ???;
const second_fd = ???;
const proc = cp.spawn(process.env.DIFF, [first_fd, second_fd] { stdio: 'inherit' });
Okay, if stream is backed by a socket or fd you can pass it to options.stdio, but what if it isn't, what if it's a transform stream?
options.stdio
object - Share a readable or writable stream that refers to a tty, file, socket, or a pipe with the child process. The stream's underlying file descriptor is duplicated in the child process to the fd that corresponds to the index in the stdio array. The stream must have an underlying descriptor (file streams do not until the 'open' event has occurred).
Yes I could create a temp file but can I do it without one?
You can stream a downloaded content into vim text editor in your terminal with the following nodejs code:
const { spawn } = require('child_process');
const request = require('request');
//
request({
url: 'https://google.com'
}, function (err, res, body) {
const vi = spawn('vi', ['-'], { stdio: ['pipe', process.stdout, process.stderr] });
vi.stdin.write(body);
vi.stdin.end();
});
Then from your terminal if you execute this code, it will download google's html and let you edit and save it in a file. You can use :w myfile.txt to save to a file in vim.
Further reading on this matter: https://2ality.com/2018/05/child-process-streams.html
const { spawnSync } = require('child_process');
const string_1 = 'foo';
const string_2 = 'foobar';
const command = 'diff';
const args = [
'--unified',
`<(echo "${string_1}")`,
`<(echo "${string_2}")`,
];
const options = {
'shell': '/bin/bash',
};
const diff = spawnSync(command, args, options);
console.log(diff.stdout.toString());
Related
Let say I have this command
somecli -s < "/path/to/file.txt"
How can I convert the above command to NodeJS spawn command ? I did something like this, but seems like it didn't pass the input.
spawn('somecli', ['-s', '<', '"/path/to/file.txt"'], { stdio: 'inherit'}).on('error', function (error) {
// something
});
I can use the exec command below and it's working, but I prefer if we can see the live output.
exec('somecli -s < "/path/to/file.txt"', (e, stdout, stderr) => {
// something
})
something like this should help
const { spawn } = require('child_process');
const fs = require('fs');
const writeStream = fs.createWriteStream("/path/to/file.txt");
const shell = spawn('somecli', ['-s']);
shell.stdout.pipe(writeStream);
To pass file input to command ( STDIN redirection )
$ somecli -s < /path/to/file.txt
We can do it something like this
spawn('somecli', ['-s'], {stdio: [fs.openSync('/path/to/file.txt', 'r'), process.stdout, process.stderr]});
To pass command output to file ( STDOUT redirection )
$ somecli -s > /path/to/file.txt
You may follow Ashish answer
let s = spawn('somecli', ['-s])
s.stdout.pipe(fs.createWriteStream('/path/to/file.txt'))
I need to be able to tar a directory, and send this to a remote endpoint via HTTP PUT.
I could of course create the tar, save it to disk, then read it again and send it.
But I'd rather like to create the tar, then pipe it to some buffer and send it immediately. I haven't been able to achieve this.
Code so far:
var tar = require('tar');
var fs = require("fs");
var path = "/home/me/uploaddir";
function getTar(path, cb) {
var buf = new Buffer('');
var wbuf = fs.createWriteStream(buf);
wbuf.on("finish", function() {
cb(buf);
});
tar.c({file:""},[path]).
pipe(wbuf);
}
getTar(path, function(tar) {
//send the tar over http
});
This code results in:
fs.js:575
binding.open(pathModule._makeLong(path),
^
TypeError: path must be a string
at TypeError (native)
at Object.fs.open (fs.js:575:11)
I've also tried using an array as buffer, no joy.
The following solution
creates the tar, then pipes it to some buffer and sends it immediately
and with great speed thanks to the tar-fs library:
First install the libraries request for simplified requests and tar-fs, which provides filesystem bindings for tar-stream: npm i -S tar-fs request
var tar = require('tar-fs')
var request = require('request')
var fs = require('fs')
// pack specific files in the directory
function packTar (folderName, pathsArr) {
return tar.pack(folderName, {
entries: pathsArr
})
}
// return put stream
function makePutReq (url) {
return request.put(url)
}
packTar('./testFolder', ['test.txt', 'test1.txt'])
.pipe(makePutReq('https://www.example.com/put'))
I have renamed the function names to be super verbose.
I'm trying to make an extension that uses chrome native messaging to communicate with youtube-dl using a node.js host script. I've been able to successfully parse the stdin from the extension & also been able to run a child process (i.e. touch file.dat), but when I try to exec/spawn youtube-dl it hangs on the command. I've tried the host script independently of chrome native input and it works fine. I think the problem may have something to do with 1MB limitations on buffer size of chrome native messaging. Is there a way around reading the buffer?
#! /usr/bin/env node
"use strict";
const fs = require('fs');
const exec = require('child_process').execSync;
const dlPath = '/home/toughluck/Music';
let first = true;
let buffers = [];
process.stdin.on('readable', () => {
let chunk = process.stdin.read();
if (chunk !== null) {
if (first) {
chunk = chunk.slice(4);
first = false;
}
buffers.push(chunk);
}
});
process.stdin.on('end', () => {
const res = Buffer.concat(buffers);
const url = JSON.parse(res).url;
const outTemplate = `${dlPath}/%(title)s.%(ext)s`;
const cmdOptions = {
shell: '/bin/bash'
};
const cmd = `youtube-dl --extract-audio --audio-format mp3 -o \"${outTemplate}\" ${url}`;
// const args = ['--extract-audio', '--audio-format', 'mp3', '-o', outTemplate, url];
// const cmd2 = 'youtube-dl';
process.stderr.write('Suck it chrome');
process.stderr.write('stderr doesnt stop host');
exec(cmd, cmdOptions, (err, stdout, stderr) => {
if (err) throw err;
process.stderr.write(stdout);
process.stderr.write(stderr);
});
process.stderr.write('\n Okay....');
});
The full codebase can be found at https://github.com/wrleskovec/chrome-youtube-mp3-dl
So I was right about what was causing the problem. It had to do with 1 MB limitation on host to chrome message. You can avoid this by redirecting the stdout/stderr to a file.
const cmd = `youtube-dl --extract-audio --audio-format mp3 -o \"${outTemplate}\" ${url} &> d.txt`;
This worked for me. To be honest I'm not entirely why the message is considered > 1 MB and if someone can give a better explanation that would be great.
I use console.log in my node.js: that way I can log to screen
ex:
node myscript.js
If I use
node myscript.js>log.txt then I log to file log.txt
How can I log to screen AND to file ?
Use tee.
node myscript.js | tee log.txt
If you want this behavior to be persistent within your app, you could create a through stream and pipe it to both a writeStream and stdout.
var util = require('util');
var fs = require('fs');
// Use the 'a' flag to append to the file instead of overwrite it.
var ws = fs.createWriteStream('/path/to/log', {flags: 'a'});
var through = require('through2');
// Create through stream.
var t = new through();
// Pipe its data to both stdout and our file write stream.
t.pipe(process.stdout);
t.pipe(ws);
// Monkey patch the console.log function to write to our through
// stream instead of stdout like default.
console.log = function () {
t.write(util.format.apply(this, arguments) + '\n');
};
Now this will write to both stdout (terminal display) and to your log file.
You can also omit the through stream and just write to both streams in the monkey patched function.
console.log = function () {
var text = util.format.apply(this, arguments) + '\n';
ws.write(text);
process.stdout.write(text);
};
The through stream just gives you a single stream you could utilize in other ways around your app and you'd always know that it was piped to both output streams. But if all you want is to monkey patch console.log then the latter example is sufficient :)
If you only want to do this for a single run of your app from the terminal, see #andars' answer and the tee command :)
PS - This is all that console.log actually does in node, in case you were wondering.
Console.prototype.log = function() {
this._stdout.write(util.format.apply(this, arguments) + '\n');
};
when use the following code to tail a file, we can successfully output data.
var spawn = require('child_process').spawn;
var filename = '/logs/error.log';
var tail = spawn("tail", ["-f", filename]);
tail.stdout.on("data", function (data) {
console.log(data);
});
but when i change filename to "/logs/*.log", i don't find anything output. who can tell me what is the reason? Thanks!
When typing tail -f /logs/*.log on the console, the expansion of /logs/*.log is handled by the shell; by the time the tail program gets the arguments, they've already been expanded to tail -f /logs/error.log /logs/other.log. You need to do the expansion yourself for Node:
var fs = require('fs');
var spawn = require('child_process').spawn;
var filename = fs.readdirSync('/logs').map(function(file) { return '/logs/' + file });
var tail = spawn("tail", ["-f"].concat(filename));
tail.stdout.on("data", function (data) {
console.log(data);
});
Because neither tail nor spawn know how to expand file names with wild cards into the set of matching file names. That's normally performed by the shell, so in this case you'll need to do it yourself in code.