The program below simply reads a string and outputs it. When I run this on cmd, the program doesn't print out the string. It keeps reading inputs until I terminate with Ctrl+C. How do I tell the program when my input string is over, so it can print the output?
var concat=require('concat-stream');
var str=[];
process.stdin.pipe(concat(function(buff){
console.log(buff.toString());
}));
concat-stream is waiting to receive a finish event. In your example that will happen when you close stdin. If you’re running this in a shell you can close stdin by pressing Ctrl+D. If you’re piping something to your process, make sure it closes its stdout when it’s done.
If you’re trying to make your script interactive in the shell, try split:
process.stdin
.pipe(require('split')())
.on('data', function (line) {
console.log('got “%s”', line);
});
Obviously the answer by Todd Yandell is the right one, and I have already upvoted it, but I wanted to add that besides split, you may also consider the use of through which creates a sort of transformer and it would also work in a interactive way, since it is not an aggregation pipe.
Like this example in which everything you write in the standard input gets uppercased in standard output interactively:
var through = require('through');
function write(buffer){
var text = buffer.toString();
this.queue(text.toUpperCase());
}
function end(){
this.queue(null);
}
var transform = through(write, end);
process.stdin.pipe(transform).pipe(process.stdout);
You may even combine it with split by doing:
process.stdin
.pipe(split())
.pipe(transform)
.pipe(process.stdout);
Related
I´m using the subprocess module and it works fine, the only thing is that the stdout returns a value "b'" or in some cases longer text like "user config - ignore ...". Is it possible to remove this first part of the stdout without using str.substring() or similar methodes.
output = subprocess.run(['ls', '-l'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
in the abrove example the std.out.decode() function could be used and it will be saved as < str >
decoded_output = nodes.stdout.decode()
And if some type of commands support json output(for example pvesh in proxmox) you could use the string and load it as json.
json_output = json.loads(decoded_output)
I have 2 programs.
The first (which could be written in any language, actually and therefore cannot be altered at all) looks like this:
#!/bin/env python3
import random
while True:
s = input() # get input from stdin
i = random.randint(0, len(s)) # process the input
print(f"New output {i}", flush=True) # prints processed input to stdout
It runs forever, read something from stdin, processes it and writes the result to stdout.
I am trying to write a second program in Python using the asyncio library.
It executes the first program as a subprocess and attempt to feed it input via its stdin and retrieve the result from the its stdout.
Here is my code so far:
#!/bin/env python3
import asyncio
import asyncio.subprocess as asp
async def get_output(process, input):
out, err = await process.communicate(input)
print(err) # shows that the program crashes
return out
# other attempt to implement
process.stdin.write(input)
await process.stdin.drain() # flush input buffer
out = await process.stdout.read() # program is stuck here
return out
async def create_process(cmd):
process = await asp.create_subprocess_exec(
cmd, stdin=asp.PIPE, stdout=asp.PIPE, stderr=asp.PIPE)
return process
async def run():
process = await create_process("./test.py")
out = await get_output(process, b"input #1")
print(out) # b'New output 4'
out = await get_output(process, b"input #2")
print(out) # b''
out = await get_output(process, b"input #3")
print(out) # b''
out = await get_output(process, b"input #4")
print(out) # b''
async def main():
await asyncio.gather(run())
asyncio.run(main())
I struggle to implement the get_output function. It takes a bytestring (as needed by the input parameter of the .communicate() method) as parameter, writes it to the stdin of the program, reads the response from its stdout and returns it.
Right now, only the first call to get_output works properly. This is because the implementation of the .communicate() method calls the wait() method, effectively causing the program to terminate (which it isn't meant to). This can be verified by examining the value of err in the get_output function, which shows the first program reached EOF. And thus, the other calls to get_output return an empty bytestring.
I have tried another way, even less successful, since the program gets stuck at the line out = await process.stdout.read(). I haven't figured out why.
My question is how do I implement the get_output function to capture the program's output in (near) real time and keep it running ? It doesn't have to be using asyncio, but I have found this library to be the best one so far for that.
Thank you in advance !
If the first program is guaranteed to print only one line of output in response to the line of input that it has read, you can change await process.stdout.read() to await process.stdout.readline() and your second approach should work.
The reason it didn't work for you is that your run function has a bug: it never sends a newline to the child process. Because of that, the child process is stuck in input() and never responds. If you add \n at the end of the bytes literals you're passing to get_output, the code works correctly.
Let's say I have a file that I know is going to be updated sometime in the next minute. I want a Python function to fire (roughly) as soon as that happens.
I set up something rudimentary to await that input:
import os, time
modification_time = os.path.getmtime(file)
def wait_for_update(file):
modified = False
while not modified:
if modification_time == os.path.getmtime(file):
time.sleep(1)
else:
modified = True
file_contents = file.read()
Theoretically, that should work fine.
However, I'm coming from a Javascript background. In Javascript, a single-threaded language, the equivalent of that function will stop the entire programme until that while loop finishes - and precious seconds of time that could be spent processing other input is wasted.
In Javascript, there's a handy window.setTimeout() function that queues up a function to be fired at some point in the future when the main thread is clear of processes. It's not truly asynchronous, it just offsets that operation a little bit into the future, leaving other stuff to happen in the meantime.
So that same process written in recursive JS might look something like this:
function waitForUpdate(file) {
var modified = false;
modified = compareFileModificationDates();
if(modified) {
modified = true;
// do other things
} else {
setTimeout(waitForUpdate(file), 1000);
}
}
Do I need to do something similar in Python? Or is time.sleep fine?
How is subprocess.Popen different from os.fork()? The doc just says it creates a child program in a new process.
And what does this code specifically do?
from subprocess import PIPE
subprocess.Popen("ls", shell = True, stdin = PIPE, stdout = PIPE, stderr = PIPE)
I'm specifically confused about the parameters, stdin = PIPE, stdout and stderr = PIPE), again, the doc says that they are special valuesthat can be used to Popen and indicates a pipe to the standard stream should be opened.
Does this mean that it's setting the default standard stream to stdin, stdout or stderr respectively?
Plus, I didn't get the documentation listing something like:
process = Popen(some_command, shell = True, stdin = Pipe....#same code)
process.stdin.write(#some binary data)
We're not accessing the arguments here are we?
How is this code all working?
I'd like to pass a JSON object as a command line argument to node. Something like this:
node file.js --data { "name": "Dave" }
What's the best way to do this or is there another more advisable way to do accomplish the same thing?
if its a small amount of data, I'd use https://www.npmjs.com/package/minimist, which is a command line argument parser for nodejs. It's not json, but you can simply pass options like
--name=Foo
or
-n Foo
I think this is better suited for a command line tool than json.
If you have a large amount of data you want to use you're better of with creating a json file and only pass the file name as command line argument, so that your program can load and parse it then.
Big objects as command line argument, most likely, aren't a good idea.
this works for me:
$ node foo.js --json-array='["zoom"]'
then in my code I have:
import * as _ from 'lodash';
const parsed = JSON.parse(cliOpts.json_array || []);
_.flattenDeep([parsed]).forEach(item => console.log(item));
I use dashdash, which I think is the best choice when it comes to command line parsing.
To do the same thing with an object, just use:
$ node foo.js --json-object='{"bar": true}'
This might be a bit overkill and not appropriate for what you're doing because it renders the JSON unreadable, but I found a robust way (as in "works on any OS") to do this was to use base64 encoding.
I wanted to pass around lots of options via JSON between parts of my program (a master node routine calling a bunch of small slave node routines). My JSON was quite big, with annoying characters like quotes and backslashes so it sounded painful to sanitize that (particularly in a multi-OS context).
In the end, my code (TypeScript) looks like this:
in the calling program:
const buffer: Buffer = new Buffer(JSON.stringify(myJson));
const command: string = 'node slave.js --json "' + buffer.toString('base64') + '" --b64';
const slicing: child_process.ChildProcess = child_process.exec(command, ...)
in the receiving program:
let inputJson: string;
if (commander.json) {
inputJson = commander.json;
if (commander.b64) {
inputJson = new Buffer(inputJson, 'base64').toString('ascii');
}
}
(that --b64 flag allows me to still choose between manually entering a normal JSON, or use the base64 version, also I'm using commander just for convenience)