I want to (hopefully easily) write from a bash script to any arbitrary program that is already running via that program's stdin.
Say I have some arbitrary program "Sum", that constantly takes user input from the terminal. Every integer it receives from stdin it adds to the current sum and outputs the new sum. Here's example terminal text of what I mean:
$: ./Sum
Sum: 0
Give me an integer: 2
Sum: 2
Give me an integer: 5
Sum: 7
How would automate this process in a bash script? If I had control of Sum's source code I could let it accept integer arguments. But if I don't have said control of the program, how can I automate the interaction with Sum? Here's a psuedo code of a bash snippet of what I want to do:
In example.sh:
#!/bin/bash
my_program_input_point=./Sum &
echo 2 > my_program_input_point
echo 5 > my_program_input_point
Thus on my terminal screen it would still look like this:
$: ./example.sh
Sum: 0
Give me an integer: 2
Sum: 2
Give me an integer: 5
Sum: 7
The difference is I wouldn't have typed any of it.
It feels like this task should be really easy, because basically anything you can do in a terminal, you can also easily do in a script. Except, apparently, directly interact with an arbitrary process once its started.
This topic answers some aspects of my question by using pipes. However the accepted answer only works if I have control of both programs on each side of the pipe, the "recipient" program is under no obligation to read from the pipe automatically.
This is assuming I can't modify the program/script in question. I also can't write a "pipe reader" script to wrap around the program (Sum), because that wrapper script would still be required to interact with the running process Sum.
The second answer to this question on serverfault.com (written by jfgagne) seems much closer, but I can't seem to get it working. Is there any easier way to do this that I just don't know about?
For information on how to capture and read an arbitrary program's output, see my next question for more information
The most straightforward way is to use a pipeline. The trick is that you can make the lefthand side of the pipeline a compound statement, or a function call. It doesn't have to be just a single command.
{
echo 2
echo 5
} | ./Sum
or
numbers() {
echo 2
echo 5
}
numbers | ./Sum
This lets you do whatever you want to generate the input. You don't have to have it all ahead of time. If you generate the input bit by bit, it'll be fed to ./Sum bit by bit.
You can use a named pipe. Whatever starts Sum is responsible for creating the named pipe some place convenient, then starting Sum with the named pipe as its standard input.
mkfifo /tmp/pipe
Sum < /tmp/pipe
Then your script could take the name of the pipe as an argument, then treat it as a file it can write to.
#!/bin/bash
p=$1
echo 2 > "$p"
echo 5 > "$p"
Then you could call your script with client /tmp/pipe.
You can write to the standard input file descriptor of the running process. Here is the same question: https://serverfault.com/questions/178457/can-i-send-some-text-to-the-stdin-of-an-active-process-running-in-a-screen-sessi
You need to write to /proc/*pid of the program*/fd/0 which is the file descriptor for the standard input of the process with that pid. Make sure you have access to do this.
Related
Ok this is merely to satisfy my curiosity and anyone else that might have a similar question. Please bear with the ignorance and lengthy question as its partially a case of "i dont know what i dont know".
Section 1
Assume a fileToFollow.txt that has some arbitrary content.
row1
row2
row3
Executing tail fileToFollow.txt | cat yields the contents of the file as expected.
Executing tail -f fileToFollow.txt | cat will keep on outputing anything that is written into fileToFollow.txt
I imagined piping as getting the output of one program and feeding it into the input of another ( so for example if cat was a C program it would be able to access that input through the main() arguments ).
Question 1: Is that whats going on here, is cat just called everytime tail has an output?
Section 2
I decided to throw grep into the mix setting up the following:
tail -f fileToFollow.txt | grep "whatever" | cat
Clearly cat is not needed here as grep itself does output to the terminal anyway. But given the idea that piping is output from one program to input in another, i was assuming it would. However, in this case no output is displayed in the terminal.
The following of course works fine:
tail -f fileToFollow.txt | grep "whatever"
I have a hunch i am little bit confused as to how piping might actually work and why the cases i presented aren't behaving as i would expect them to.
Any kind of enlightenment is welcome. Thanks a bunch for taking the time.
Answer to question 1: No, cat is running as a process all the time but it's blocked reading stdin when there is nothing available. When the process before it in the pipeline (tail) writes new bytes to the pipe, the read call will return and cat is able to process the new data. After that it will read again and block until new data is available.
When you pipe into a program, the stdout of the source will usually switch into buffered mode (see man setvbuf()), which means that a certain amount of data (2KiB or 4KiB or so) must be generated before it will be given to write(2).
Giving it out to tty uses line buffered mode so that buffers are flushed after \n.
There exists a stdbuf tool to modify this behavior.
I have a program that accepts input on one FIFO and emits output to another FIFO. I want to write a small script to control this program. The script needs to listen both to standard input (so I can input commands to adjust things in real time) and the program's output FIFO (so it can respond to events happening there as well).
Essentially my control program needs to select between standard input and a file (my FIFO).
I like learning how to figure out how to develop simple and elegant bash-based solutions to complex problems, and after a little headscratching I remembered that that tail -f will happily select on multiple files and tell you when one of them changes in real time, so I initially tried
tail -f <(od -An -vtd1 -w1) <(cat fifo)
to read both standard input (I'd previously run stty icanon min 1; this od invocation shows each stdin character on a separate line alongside its ASCII code, and is great for escape sequence parsing) and my FIFO. This failed epically (as does cat <(cat)): od gets run here as a backgrounded task, so it doesn't get access to the controlling TTY, and fails with a cryptic "I/O error" that was explained incredibly well here.
So now I'm a bit stumped. I realize that I can use any scripting language like Perl/Python/Ruby/Tcl to solve this; my compsci/engineering question is whether/how I might be able to solve this using (Linux) shell scripting.
Good afternoon everyone,
I have created a resource monitoring tool that works fairly well.
Pulls CPU average usage
Pulls average Memory usage
Even calculates % of NIC throughput (if you have a 1Gb NIC, it'll show percent that is being processed at a time)... and yes I know this is more of a rough estimate/theoretical max limit of a NIC.
I am having one issue with my script though. Portion of the code I am experiencing my issue with is below (I converted some to pseudo code for simplicity).
COUNT=1
read -rsp "When you are ready to begin, please press any key" -n1
echo "processing"
sleep 3
while [ ${COUNT} = "1" ; do
read -t 1 -n 1
if [$? = 0 ] ; then
exit 0
else
`Resource command` > ${cpulog} file for future graphs
`Resource command` > ${memlog} file for future graphs
`Resource command` > ${network} log file for future graphs
`etc`
fi
done
Basically, you hit any key to start the program, and whenever you press any key on the keyboard after the program has started (While loop), the program stops recording information and moves on.
Now this script works and does everything I need it to do. The issue I have come across is when you "press any key".
Note that there are two points in the script waiting for a key press.
If I were to press any key more than once at the first point, the second key input would get processed by my read -t 1 -n 1 command (at the second point), and thus fail to run my resource pulls. Since that happens immediately, the script fails.
Basically, I am trying to figure out if there is a way I can shutdown input after that first key stroke for a limited time while I retrieve a limited amount of data, or flush any input that was given prior to hitting my read -t 1 command. Thank you.
The script can call a small program, written in C, perl or similar, which calls the FIONREAD ioctl on stdin.
Then read the unexpected extra characters to be thrown away with a read call, see Perl Cookbook to determine unread bytes
You can actually enter the perl code on command line with perl -e. to keep it all within the bash script.
At the "processing line," add the following loop:
# Eat any remaining input
while read -t 1 -n 1
do
# Do nothing here
:
done
# Continue processing now that all input has been consumed...
It will add about 1 second delay to startup (more if the user is sitting there pressing keys), but otherwise does what you want.
I have a script that produces a lot of output. The script pauses for a few seconds at point T.
Now I am using the less command to analyze the output of the script.
So I execute ./script | less. I leave it running for sufficient time so that the script would have finished executing.
Now I go through the output of the less command by pressing Pg Down key. Surprisingly while scrolling at the point T of the output I notice the pause of few seconds again.
The script does not expect any input and would have definitely completed by the time I start analyzing the output of less.
Can someone explain how the pause of few seconds is noticable in the output of less when the script would have finished executing?
Your script is communicating with less via a pipe. Pipe is an in-memory stream of bytes that connects two endpoints: your script and the less program, the former writing output to it, the latter reading from it.
As pipes are in-memory, it would be not pleasant if they grew arbitrarily large. So, by default, there's a limit of data that can be inside the pipe (written, but not yet read) at any given moment. By default it's 64k on Linux. If the pipe is full, and your script tries to write to it, the write blocks. So your script isn't actually working, it stopped at some point when doing a write() call.
How to overcome this? Adjusting defaults is a bad option; what is used instead is allocating a buffer in the reader, so that it reads into the buffer, freeing the pipe and thus letting the writing program work, but shows to you (or handles) only a part of the output. less has such a buffer, and, by default, expands it automatically, However, it doesn't fill it in the background, it only fills it as you read the input.
So what would solve your problem is reading the file until the end (like you would normally press G), and then going back to the beginning (like you would normally press g). The thing is that you may specify these commands via command line like this:
./script | less +Gg
You should note, however, that you will have to wait until the whole script's output loads into memory, so you won't be able to view it at once. less is insufficiently sophisticated for that. But if that's what you really need (browsing the beginning of the output while the ./script is still computing its end), you might want to use a temporary file:
./script >x & less x ; rm x
The pipe is full at the OS level, so script blocks until less consumes some of it.
Flow control. Your script is effectively being paused while less is paging.
If you want to make sure that your command completes before you use less interactively, invoke less as less +G and it will read to the end of the input, you can then return to the start by typing 1G into less.
For some background information there's also a nice article by Alexander Sandler called "How less processes its input"!
http://www.alexonlinux.com/how-less-processes-its-input
Can I externally enforce line buffering on the script?
Is there an off the shelf pseudo tty utility I could use?
You may try to use the script command to turn on line-buffering output mode.
script -q /dev/null ./script | less # FreeBSD, Mac OS X
script -c "./script" /dev/null | less # Linux
For more alternatives in this respect please see: Turn off buffering in pipe.
I have two programs A and B. I can't change the program A - I can only run it with some parameters, but I have written the B myself, and I can modify it the way I like.
Program A runs for a long time (20-40 hours) and during that time it produces output to the file, so that its size increases constantly and can be huge at the end of run (like 100-200 GB). The program B then reads the file and calculates some stuff. The special property of the file is that its content is not correlated: I can divide the file in half and run calculations on each part independently, so that I don't need to store all the data at once: I can calculate on the first part, then throw it away, calculate on the second one, etc.
The problem is that I don't have enough space to store such a big files. I wonder if it is possible to pipe somehow the output of the A to B without storing all the data at once and without making huge files. Is it possible to do something like that?
Thank you in advance, this is crucial for me now, Roman.
If program A supports it, simply pipe.
A | B
Otherwise, use a fifo.
mkfifo /tmp/fifo
ls -la > /tmp/fifo &
cat /tmp/fifo
EDIT: Adjust buffer sizes with ulimit -p and then:
cat /tmp/fifo | B
It is possible to pipeline output of one program into another.
Read here to know the syntax and know-hows of Unix pipelining.
you can use socat which can take stdout and feed it to network and get from network and feed it to stdin
named or unnamed pipe have a problem of small ( 4k ? ) buffer .. that means too many process context switches if you are writing multi gb ...
Or if you are adventurous enough .. you can LD_PRELOAD a so in process A, and trap the open/write calls to do whatever ..