I have a process I start on my shell that prints out data and I have a python script that calculates it and should return that to the process. How can I do that?
I know using pipe or > could help me with one side, but how can I achieve a two-way redirection?
Is something like: my_process | my_script | my_process legal?
What I want to happen:
start process from shell
process sends data to my script (instead of printing it to STDOUT)
my script returns an answer to process (instead of printing it to
STDOUT)
return to step 2
It's emulatable with FIFOs (or if you're using python, you can create your own pipes with the pipe system call and redirect them according to your arrangement).
Some example code:
#!/bin/sh
#note: the resource management and error handling here is not very clean or robust
mkfifo fifo0 fifo1
exec 3<>fifo0
exec 4<>fifo1
rm -f fifo0 fifo1
many()
{
i=0
while [ $i -lt $1 ]; do
echo -n x
i=$((i+1))
done
echo
}
proc1()
{
ulimit_p=$(ulimit -p)
while :; do
echo proc1_msg
echo >&2 proc1_written=proc1_msg
read var
echo >&2 proc1_read=$var
sleep 1
done
}
proc2()
{
while :; do
read var
echo >&2 proc2_read=$var
echo proc2_msg
echo >&2 proc2_written=proc2_msg
done
}
pid=0
trap 'echo kill $pid; kill $pid; exit 130' INT
proc1 <&3 >&4 & pid=$!
proc2 <&4 >&3
Related
I have a script that performs many different operations and displays the status of their completion in a clear way for the user. I need a function so that some strings can be retrieved as variables for further processing.
This is a highly simplified example:
#!/bin/bash
echo "Test script."
echo -n "1) cat file "
cat ./testfile.f &> /dev/null
if [ $? -eq 0 ]
then
echo "$(tput hpa $(tput cols))$(tput cub 8)[OK]"
else
echo "$(tput hpa $(tput cols))$(tput cub 8)[FAIL]"
fi
echo -n "2) make subfolder "
mkdir ./testdir &> /dev/null
if [ $? -eq 0 ]
then
echo "$(tput hpa $(tput cols))$(tput cub 8)[OK]"
else
echo "$(tput hpa $(tput cols))$(tput cub 8)[FAIL]"
fi
It take some as:
$./test.sh
Test script.
1) cat file [FAIL]
2) make subfolder [OK]
How can I get the last line (ideally, any string) during script execution? Ideally it would be using a function (so I could work with the resulting string) This string will be processed in the same script.
So far, I see only one solution: redirect the output of each echo command using tee.
Is there any way to read the already outputted data!?
Suppose that you have strings {1..5} that you need to process:
process() {
while read -r input; do
sleep 2
echo "Processed ${input}."
done
}
printf "line %s\n" {1..5} | process
In this situation you might want to see the numbers before they are being processed.
You can do this by duplicating stdout to fd 3 and use a function.
display() {
while read -r input; do
echo "=== ${input} ===" >&3
echo "${input}"
done
}
process() {
while read -r input; do
sleep 2
echo "Processed ${input}."
done
}
exec 3>&1
printf "line %s\n" {1..5} | display | process
I'm running a bash script with multiple simultaneous commands (python scripts).
I'm trying to kill all the processes if one of them has failed.
The thing is that the python scripts are still running in the background, and if one of them has failed, my bash script doesn't know about.
Here's a snippet from my script:
set -a
trap cleanup_children SIGTERM
MY_PID=$$
function thread_listener () {
to_execute="$1"
echo "Executing $to_execute ..."
$to_execute &
PID=$!
trap 'echo killing $PID; kill $PID' SIGTERM
echo "Waiting for $PID ($to_execute) ..."
wait $PID || if `kill -0 $MY_PID &> /dev/null`; then kill $MY_PID; fi
}
function cleanup_children () {
for job in `jobs -p`
do
if `kill -0 $job &> /dev/null`; then
echo "Killing child number $job"
ps -p $job
kill $job
fi
done
}
function create_app1 () {
cd ${GIT_DIR}
python ./create-app.py -myapp
exit_code=$?
echo "Create app1 ISO result: ${exit_code}"
[ "${exit_code}" == "1" ] && exit 1
mv ${ISO_OUTPUT_DIR}/rhel-7.1.iso ${ISO_OUTPUT_DIR}/${ISO_NAME}.iso
}
function create_app2 () {
cd ${GIT_DIR}
python ./create-app.py -do-something
exit_code=$?
echo "Create app1 ISO result: ${exit_code}"
[ "${exit_code}" == "1" ] && exit 1
mv ${ISO_OUTPUT_DIR}/rhel-7.1.iso ${ISO_OUTPUT_DIR}/${ISO_NAME}.iso
}
export -f create_app1
export -f create_app2
echo "MY_PID=$MY_PID"
thread_listener create_app1 &
PID_APP1=$!
thread_listener create_app2 &
PID_APP2=$!
wait
kill $PID_APP1 2> /dev/null
kill $PID_APP2 2> /dev/null
Hm, this looks quite advanced ;). Do I assume correctly that you never see the "Create app1 ISO result" output then because the python script does not terminate? It might be an issue with the signal not being properly dispatched to bash background jobs. It might also be related to your python code not properly reacting to the signal. Have you checked out https://docs.python.org/2/library/signal.html? Sure you'd have to figure out the exact steps how to interrupt you python code while executing. I'd suggest to first make sure that the python code reacts to signals the way you want.
How to write an infinite loop echoing numbers from 1 to infinite in bash.
I am using a for loop but it is being killed by bash on using value more than 100000000.
#!/bin/bash
for a in {1..100000000..1}
do
echo "$a"
done
any alternative for it?
Have you tried doing a while loop?
#!/bin/bash
num=0;
while :
do
num=$((num+1))
echo "$num"
done
This should work in all POSIX shells:
i=0; while :; do echo "$((i+=1))"; done
: is interchangeable with the true builtin (which you can use instead): it's a no-op that always succeeds (=returns 0).
If integer overflow bothers you, and you want arbitrary precision with standard tools:
nocontinuation(){ sed ':x; /\\$/ { N; s/\\\n//; tx }'; }
i=99999999999999999999999999999999999999999999999999999999999999999999;
while : ; do i=`echo "$i + 1" | bc | nocontinuation`; echo "$i"; done
This would be quite slow, because it spawns in each iteration.
To avoid that, you could reuse one bc instance and communicate with it over pipes:
#!/usr/bin/bash
set -e
nocontinuation(){ sed -u ':x; /\\$/ { N; s/\\\n//; tx }'; }
trap 'rm -rf "$tmpdir"' exit
tmpdir=`mktemp -d`
cd "$tmpdir"
mkfifo p n
<p bc | nocontinuation >n &
exec 3>p
exec 4<n
i=99999999999999999999999999999999999999999999999999999999999999999999;
while : ; do
echo "$i + 1" >&3
read i <&4
echo "$i"
done
Can't you just do a while true;?
a=0
while true;
do
a=$((a+1))
# $[$a+1] also works.
echo "$a"
done
I need to the following things to make sure my application server is
Tail a log file for a specific string
Remain blocked until that string is printed
However if the string is not printed for about 20 mins quit and throw and exception message like "Server took more that 20 mins to be up"
If string is printed in the log file quit the loop and proceed.
Is there a way to include time outs in a while loop ?
#!/bin/bash
tail -f logfile | grep 'certain_word' | read -t 1200 dummy_var
[ $? -eq 0 ] && echo 'ok' || echo 'server not up'
This reads anything written to logfile, searches for certain_word, echos ok if all is good, otherwise after waiting 1200 seconds (20 minutes) it complains.
You can do it like this:
start_time=$(date +"%s")
while true
do
elapsed_time=$(($(date +"%s") - $start_time))
if [[ "$elapsed_time" -gt 1200 ]]; then
break
fi
sleep 1
if [[ $(grep -c "specific string" /path/to/log/file.log) -ge 1 ]]; then
break
fi
done
You can use signal handlers from shell scripts (see http://www.ibm.com/developerworks/aix/library/au-usingtraps/index.html).
Basically, you'd define a function to be called on, say, signal 17, then put a sub-script in the background that will send that signal at some later time:
timeout(pid) {
sleep 1200
kill -SIGUSR1 $pid
}
watch_for_input() {
tail -f file | grep item
}
trap 'echo "Not found"; exit' SIGUSR1
timeout($$) &
watch_for_input
Then if you reach 1200 seconds, your function is called and you can choose what to do (like signal your tail/grep combo that is watching for your pattern in order to kill it)
time=0
found=0
while [ $time -lt 1200 ]; do
out=$(tail logfile)
if [[ $out =~ specificString ]]; then
found=1
break;
fi
let time++
sleep 1
done
echo $found
The accepted answer doesn't work and will never exit (because althouth read -t exits, the prior pipe commands (tail -f | grep) will only be notified of read -t exit when they try to write to output, which never happens until the string matches).
A one-liner is probably feasible, but here are scripted (working) approaches.
Logic is the same for each one, they use kill to terminate the current script after the timeout.
Perl is probably more widely available than gawk/read -t
#!/bin/bash
FILE="$1"
MATCH="$2"
# Uses read -t, kill after timeout
#tail -f "$FILE" | grep "$MATCH" | (read -t 1 a ; kill $$)
# Uses gawk read timeout ability (not available in awk)
#tail -f "$FILE" | grep "$MATCH" | gawk "BEGIN {PROCINFO[\"/dev/stdin\", \"READ_TIMEOUT\"] = 1000;getline < \"/dev/stdin\"; system(\"kill $$\")}"
# Uses perl & alarm signal
#tail -f "$FILE" | grep "$MATCH" | perl -e "\$SIG{ALRM} = sub { `kill $$`;exit; };alarm(1);<>;"
I am calling another shell script testarg.sh within my main script.
the logfiles of testarg.sh are stored in $CUSTLOGS in the below format
testarg.DDMONYY.PID.log
example: testarg.09Jun10.21165.log
In the main script after the testarg process gets completed i need to grep the log file for the text "ERROR" and "COMPLETED SUCCESSFULLY".
How do i get the PID of the process and combine with DDMONYY for grepping. Also i need to check whether file
exists before grepping
$CUSTBIN/testarg.sh
$CUSTBIN/testarg.sh
rc=$?
if [ $rc -ne 0 ]; then
return $CODE_WARN
fi
You may background testarg.sh, which puts its pid into $!, and then wait for it:
#! /bin/bash
...
$CUSTBIN/testarg.sh &
LOGFILE=testarg.$(date +%d%b%y).$!.log # testarg.09Jun10.12345.log
wait $!
# ... $? is set as you expect ...
[ -f $LOGFILE ] && grep {pattern} $LOGFILE
...
If you can modify testarg.sh and it doesn't otherwise output anything, just change it to output its log file with a line like:
echo testarg.$(date +%blah).$$.log
then use:
fspec=$($CUSTBIN/testarg.sh)
in your parent.
Alternatively, you can provide a wrapper function to do the work:
#!/bin/bash
function fgpid() {
"$#" &
pid=$!
ps -ef | grep ${pid} | sed 's/^/DEBUG:/' >&2 # debugging
wait ${pid}
echo ${pid}
}
fspec=testarg.$(date +%d%b%y).$(fgpid sleep 5).log
echo ${fspec}
This produces:
pax> ./qq.sh
DEBUG:pax 2656 2992 con 15:27:00 /usr/bin/sleep
testarg.09Jun10.2656.log
as expected.
Or this if you think your executable may output something. This one stores the PID into a variable:
#!/bin/bash
function fgpid() {
"$#" &
pid=$!
ps -ef | grep ${pid} | sed 's/^/DEBUG:/' >&2 # debugging
wait ${pid}
}
fgpid sleep 5
fspec=testarg.$(date +%d%b%y).${pid}.log
echo ${fspec}
There are two simple ways to get the PID of some process you've just spawned.
One would be to modify the program being spawned (the subprocess) to have it write its PID to a file. You'd then read it therefrom with something like:
$CUSTBIN/testarg.sh
TSTARGSPID=$(cat /var/run/custbin.testarg.pid)
Another more elegant method would be:
$CUSTBIN/testarg.sh &
TSTARGSPID=$!
wait
# Do stuff with PID and output files