Control executable in bash script - linux

First of all, I don't know how to search what I want to do.
I have one exec that produces outputs in a terminal (Linux).
Let's take a simple C program a.out:
#include <stdio.h>
int main (int argc, char *argv[]) {
int i=0;
float j=0;
for(i=0; i<=10000000;i++)
{
j = i*-1e-5;
printf (" %d 2.0 %f 4.0 5.0\n",i,j);
}
}
Outputs produced are like :
0 2.0 -0.000000 4.0 5.0
1 2.0 -0.000010 4.0 5.0
2 2.0 -0.000020 4.0 5.0
3 2.0 -0.000030 4.0 5.0
...
Depending on this outputs I want to :
Launch this exec
"Capture" outputs
If 3rd column value reach -0.5, stop/kill exec
How will you do this ?
For instance, exec is not stopped with this script exec.sh:
#/bin/sh
PROG=./a.out
$PROG > output &
progpid=$!
(tail -fn 0 output & echo $! > tailpid ) | awk -v progpid=$progpid '{
if($3<=-0.5){
system("kill "progpid)
# system( ##update other file )
system("kill $(<tailpid)")
}
}'
Any ideas ?
Thanks in advance

I think this sort of construct addresses all of your points:
programname > output &
progpid=$!
(tail -fn 0 output & echo $! > tailpid ) | awk -v progpid=$progpid '{
if( condition ) {
system("kill "progpid)
system( ##update other file )
system("kill $(<tailpid)")
}
}'
We run the program in the background and redirect output to output. Then we monitor output as it is updated using the tail -f option, which reads lines from the end of the file as they are added. Then we pipe this into awk, which can run a system command to kill the program process if condition is met, then run another command to update your parameters in your separate text file, then run another command to kill tail so that it doesn't hang in the background forever (awk will also exit once tail is killed).

Related

how to redirect this perl script's output to file?

I don't have much experience with perl, and would appreciate any/all feedback....
[Before I start: I do not have access/authority to change the existing perl scripts.]
I run a couple perl scripts several times a day, but I would like to begin capturing their output in a file.
The first perl script does not take any arguments, and I'm able to "tee" its output without issue:
/asdf/loc1/rebuild-stuff.pl 2>&1 | tee $mytmpfile1
The second perl script hangs with this command:
/asdf/loc1/create-site.pl --record=${newsite} 2>&1 | tee $mytmpfile2
FYI, the following command does NOT hang:
/asdf/loc1/create-site.pl --record=${newsite} 2>&1
I'm wondering if /asdf/loc1/create-site.pl is trying to process the | tee $mytmpfile2 as additional command-line arguments? I'm not permitted to share the entire script, but here's the beginning of its main routine:
...
my $fullpath = $0;
$0 =~ s%.*/%%;
# Parse command-line options.
...
Getopt::Long::config ('no_ignore_case','bundling');
GetOptions ('h|help' => \$help,
'n|dry-run|just-print' => \$preview,
'q|quiet|no-mail' => \$quiet,
'r|record=s' => \$record,
'V|noverify' => \$skipverify,
'v|version' => \$version) or exit 1;
...
Does the above code provide any clues? Other than modifying the script, do you have any tips for allowing me to capture its output in a file?
It's not hanging. You are "suffering from buffering". Like most programs, Perl's STDOUT is buffered by default. Like most programs, Perl's STDOUT is flushed by a newline when connected to a terminal, and block buffered otherwise. When STDOUT isn't connected to a terminal, you won't get any output until 4 KiB or 8 KiB of output is accumulated (depending on your version of Perl) or the program exits.
You could add $| = 1; to the script to disable buffering for STDOUT. If your program ends with a true value or exits using exit, you can do that without changing the .pl file. Simply use the following wrapper:
perl -e'
$| = 1;
$0 = shift;
do($0);
my $e = $# || $! || "$0 didn\x27t return a true value\n";
die($e) if $e;
' -- prog args | ...
Or you could fool the program into thinking it's connected to a terminal using unbuffer.
unbuffer prog args | ...

Two way communication with process

I have given some compiled program. I want to communicate with it from my bash script by program stdin and stdout. I need two way communication. Program cannot be killed between exchange of information. How I can do that?
Simple example:
Let that program be compiled partial summation (C++) and script results will be squares of that sums. Program:
int main() {
int num, sum = 0;
while(true) {
std::cin >> num;
sum += num;
std::cout << sum << std::endl;
}
}
My script should looks like that:
for i in 1 2 3 4; do
echo "$i" > program
read program to line;
echo $((line * line))
done
If in program I have for(int i = 1; i <= 4; ++i), then I can do something like that:
exec 4< <(./program); # Just read from program
for i in 1 2 3 4; do
read <&4 line;
echo "sh: $((line * line))";
done
For more look here. From the other hand, if in program I have std::cout << sum * sum;, then solution could be:
exec &3> >(./program); # Write to program
for i in 1 2 3 4; do
echo "$i" > &3
done
My problem is two way communication with other process / program. I don't have to use exec. I cannot install third party software. Bash-only solution, without files, will be nice.
If I run other process, it will be nice to know pid to kill that at the end of script.
I think about communication with two or maybe three processes in the future. Output of firs program may dependents on output of second program and also in second side. Like communicator of processes.
However, I cannot recompile programs and change something. I have only stdin and stdout communication in programs.
If you have bash which is newer than 4.0, you can use coproc.
However, don't forget that the input/output of the command you want to communicate might be buffered.
In that case you should wrap the command with something like stdbuf -i0 -o0
Reference: How to make output of any shell command unbuffered?
Here's an example
#!/bin/bash
coproc mycoproc {
./a.out # your C++ code
}
# input to "std::cin >> num;"
echo "1" >&${mycoproc[1]}
# get output from "std::cout << sum << std::endl;"
# "-t 3" means that it waits for 3 seconds
read -t 3 -u ${mycoproc[0]} var
# print it
echo $var
echo "2" >&${mycoproc[1]}
read -t 3 -u ${mycoproc[0]} var
echo $var
echo "3" >&${mycoproc[1]}
read -t 3 -u ${mycoproc[0]} var
echo $var
# you can get PID
kill $mycoproc_PID
output will be
1
3
6
If your bash is older than 4.0, using mkfifo can do the same thing like:
#!/bin/bash
mkfifo f1 f2
exec 4<> f1
exec 5<> f2
./a.out < f1 > f2 &
echo "1" >&4
read -t 3 -u 5 var
echo $var
rm f1 f2
Considering that your C++ program reads from standard output, and prints to standard output, it's easy to put it inside a simple chain of pipes:
command_that_writes_output | your_cpp_program | command_that_handle_output
In your specific case you probably need to modify the program to only handle one single input and writing one single output, and remove the loop. Because then you can do it very simple, like this:
for i in 1 2 3 4; do
result=`echo $i | ./program`
echo $((result * result))
done

merge sorted files, with minimal buffering

I have two log files that are prefixed with a sortable timesetamp.
I'd like to see them, in order, while the processes generating the log files are still running. This is a pretty faithful simulation of the situation:
slow() {
# print stdout at 30bps
exec pv -qL 30
}
timestamp() {
# prefix stdin with a sortable timestamp
exec tai64n
}
# Simulate two slowly-running batch jobs:
seq 000 099 | slow | timestamp > seq.1 &
seq1=$!
seq 100 199 | slow | timestamp > seq.2 &
seq2=$!
# I'd like to see the combined output of those two logs, in timestamp-sorted order
try1() {
# this shows me the output as soon as it's available,
# but it's badly interleaved and not necessarily in order
tail -f seq.1 --pid=$seq1 &
tail -f seq.2 --pid=$seq2 &
}
try2() {
# this gives the correct output,
# but outputs nothing till both jobs have stopped
sort -sm <(tail -f seq.1 --pid=$seq1) <(tail -f seq.2 --pid=$seq2)
}
try2
wait
The solution using tee (to write the files so that the standard output still goes to the console) won't work because tee introduces unwanted latency and doesn't solve the problem. Similarly, I couldn't get solutions using tail -f -s 0.01 (which alters the polling to 100/s), and/or some kind of call like split --filter='sort -sm' to sort small batches.
I also don't have tai64n, so my test code actually used this functionally identical perl code:
tai64n() {
perl -MTime::HiRes=time -pe '
printf "\#4%015x%x%n", split(/\./,time), $c; print 0 x(25-$c) . " "'
}
After failing to solve this with sh and bash, I exercised my standard failover, perl:
slow() {
# print stdout at 30bps
pv -qL 30
}
tai64n_and_tee() {
# prefix stdin with a sortable timestamp and copy to given file
perl -MTime::HiRes=time -e '
$_ = shift;
open(TEE, "> $_") or die $!;
while (<>) {
$_ = sprintf("\#4%015x%x%n", split(/\./,time), $c) . 0 x(25-$c) . " $_";
print TEE $_;
print $_;
}
' "$1"
}
# Simulate two slowly-running batch jobs:
seq 000 099 | slow | tai64n_and_tee seq.1 &
seq 100 199 | slow | tai64n_and_tee seq.2 &
wait
This was convenient for me because I had already used perl for the timestamp. I failed to do this with perl acting as tai64n and a separate perl call to act as tee, but it might work with the real tai64n.

prstat in Ubuntu or Centos

As the Java Performance said:
Solaris prstat has additional capabilities
such as reporting both user and kernel or system CPU utilization along with other
microstate information using the prstat -m and -L options. The -m option prints
microstate information, and -L prints statistics on per lightweight process.
There is any tool available like prstat in Centos or Ubuntu ?
I believe the Linux commands you are looking for are top and pstree .
Here is ptree for Linux,
#!/bin/sh
# Solaris style ptree
[ -x /usr/bin/ptree ] && exec /usr/bin/ptree "$#"
# Print process tree
# $1 = PID : extract tree for this process
# $1 = user : filter for this (existing) user
# $1 = user $2 = PID : do both
PATH=/bin:/usr/bin:/usr/sbin:/sbin
export PATH
psopt="-e"
case $1 in
[a-z]*) psopt="-u $1";shift;;
esac
[ -z "$1" ] &&
exec ps $psopt -Ho pid=,args=
#some effort to add less to the ps list
tmp=/tmp/ptree.$$
trap 'rm $tmp' 0 HUP INT TERM
ps $psopt -Ho pid=,args= >$tmp
<$tmp awk '
{ ci=index(substr($0,7),$2); o[ci]=$0 }
ci>s[a] { s[++a]=ci }
$1==pid {
for(i=1;i<=a;i++) {
si=s[i]; if(si<=ci) print o[si]
}
walkdown=ci
next
}
ci<walkdown { exit }
walkdown!=0 { print }
' pid="$1"
There is no prstat "equivalent" tool in Linux. You can use a combination of top and ps (or /proc/$pid/ resources) to get some useful result; maybe writing a shell script (using grep, sed and awk) which collects results from above commands and files.
Just for reference I found this link about top command and kernel, user and idle CPU utilization intresting
http://blog.scoutapp.com/articles/2015/02/24/understanding-linuxs-cpu-stats
Hope this helps.

TAP::Harness perl tests tee output

I am running my tests using TAP::Harness , when I run the tests from command line on a Linux system I get the test results on STDOUT as it is run but when i try to capture the output to a file as well as STDOUT using perl harness.pl | tee out.tap the results are buffered and displayed only at the end, I tried passing in a file handle to the new but the results are still buffered before being written to a file , Is there a way not to buffer the output, I have a long running suite and would like to look at the results while the tests are running as well as capture the output.
TAP::Harness version 3.22 and perl version 5.8.8
here is the sample code
harness.pl
#!/usr/bin/perl
use strict;
use warnings;
use TAP::Harness;
$|++;
my #tests = ('del.t',);
my $harness = TAP::Harness->new( {
verbosity => 1,
} );
$harness->runtests(#tests);
and the test del.t
use Test::More qw /no_plan/;
$|++;
my $count =1;
for (1 ..20 ) {
ok ( $count ++ == $_, "Pass $_");
sleep 1 if ( $count % 5 == 0 ) ;
}
Using script instead of tee does what you want:
script -c 'perl harness.pl' file
Found a simple change to make tee work as well: Specify a formatter_class:
my $harness = TAP::Harness->new( {
verbosity => 1,
formatter_class => 'TAP::Formatter::Console',
} );
This is because TAP::Harness normally uses a different default one if the output is not a tty, which is what is causing the buffering you're seeing.

Resources