Im trying to execute a program and send it commands based on a file that Im reading but as soon as I send the command the loop stop..it must be related to pipes but I would appreciate if someone could explain how they work
Code:
function setValue{
echo "$1" > &9
}
mkfifo program.fifo
./program
exec 9 > program.fifo
while read value
do
setValue $value
done < file.csv
file.csv has 8000 rows, but the program stops after the first line...and finishes the execution without looping on any more lines of the csv
By leaving the pipe open for reading the loop is not finished after reading the first line and it can continue to the end of the file:
exec 9 <> program.fifo
What is reading from program.fifo? If there is no process that is reading it, then your script is blocked trying to write. That write will not complete until some process is reading the fifo.
My guess is that the code you aren't showing us in the while loop is consuming all of its stdin, which is the same as the input of the while loop. The cleanest way to avoid that is to ensure that the commands in the loop are not reading from the same file as the loop. There are many ways to do that; I like:
while read value; do {
commands
} < /dev/null
done < file.csv
Related
I need 3 commands to be run and their (single-line) outputs assigned to 3 different variables, which then I use to write to a file. I want to wait till the variable assignment is complete for all 3 before I echo the variables to the file. I am running these in a loop within a bash script.
This is what I have tried -
var1=$(longRunningCommand1) &
var2=$(longRunningCommand2) &
var3=$(longRunningCommand3) &
wait %1 %2 %3
echo "$var1,$var2,$var3">>$logFile
This gives no values at all, for the variables. I get -
,,
,,
,,
However, if I try this -
var1=$(longRunningCommand1 &)
var2=$(longRunningCommand2 &)
var3=$(longRunningCommand3 &)
wait %1 %2 %3
echo "$var1,$var2,$var3">>$logFile
I get the desired output,
o/p of longRunningCommand1, o/p of longRunningCommand2, o/p of longRunningCommand3
o/p of longRunningCommand1, o/p of longRunningCommand2, o/p of longRunningCommand3
o/p of longRunningCommand1, o/p of longRunningCommand2, o/p of longRunningCommand3
but the nohup.out for this shell script indicates that there was no background job to wait for -
netmon.sh: line 35: wait: %1: no such job
netmon.sh: line 35: wait: %2: no such job
netmon.sh: line 35: wait: %3: no such job
I would not have bothered much about this, but I definitely need to make sure that my script is waiting for all the 3 variables to be assigned before attempting the write. Whereas, the nohup.out tells me otherwise! I think I want to know if the 2nd approach is the right way when I run into a situation where any of those 3 commands are running for more than a few seconds. I have not yet been able to get a really long running command or a resource contention on the box to actually resolve this doubt of mine.
Thank you very much for any helpful thoughts.
-MT
Your goal of writing the output of echo "$var1,$var2,$var3">>$logFile while backgrounding actual processes of longRunningCommand1, ..2, ..3 can be accomplished using a list and redirection. As #that_other_guy notes, you cannot assign the result of a command substitution to a variable in the background to begin with. However, for a shell that provides process substitution like bash, you can write the output of a process to a file in the background and separating your processes and redirections by a ';' will insure the sequential write of command1, ..2, ..3 to the log file, e.g.:
Commands that are separated by a <semicolon> ( ';' )
shall be executed sequentially.
POSIX Specification - lists
Putting those pieces together, you would sequentially write the results of your comment to $logfile with something similar to the following,
( (longRunningCommand1) >> $logfile; (longRunningCommand2) >> $logfile; \
(longRunningCommand3) >> $logfile) &
(note: the ';' between commands writing to $logfile)
While not required, if you wanted to wait until all commands had been written to $logfile within your script (and your script supports $! as the PID for the last backgrouded process), you could simply wait $!, though that is not required to insure the write to the file completes.
I am having a problem getting bash to do exactly what I want, it's not a major issue, but annoying.
1.) I have a third party software I run that produces some output as stderr. Some of it is useful, some of it is regularly stuff I don't care about and I don't want this dumped to screen, however I do want the useful parts of the stderr dumped to screen. I figured the best way to achieve this was to pass stderr to a function, then use conditions in that function to either show the stderr or not.
2.) This works fine. However the solution I have implemented dumped out my errors at the right time, but then returns a bash prompt and I want to summarise the status of the errors at the end of the function, but echo-ing here prints the text after the prompt meaning that I have to press enter to get back to a clean prompt. It shall become clear with the example below.
My error stream generator:
./TestErrorStream.sh
#!/bin/bash
echo "test1" >&2
My function to process this:
./Function.sh
#!/bin/bash
function ProcessErrors()
{
while read data;
do
echo Line was:"$data"
done
sleep 5 # This is used simply to simulate the processing work I'm doing on the errors.
echo "Completed"
}
I source the Function.sh file to make ProcessErrors() available, then I run:
2> >(ProcessErrors) ./TestErrorStream.sh
I expect (and want) to get:
user#user-desktop:~/path$ 2> >(ProcessErrors) ./TestErrorStream.sh
Line was:test1
Completed
user#user-desktop:~/path$
However what I really get is:
user#user-desktop:~/path$ 2> >(ProcessErrors) ./TestErrorStream.sh
Line was:test1
user#user-desktop:~/path$ Completed
And no clean prompt. Of course the prompt is there, but "Completed" is being printed after the prompt, I want to printed before, and then a clean prompt to appear.
NOTE: This is a minimum working example, and it's contrived. While other solutions to my error stream problem are welcome I also want to understand how to make bash run this script the way I want it to.
Thanks for your help
Joey
Your problem is that the while loop stay stick to stdin until the program exits.
The release of stdin occurs at the end of the "TestErrorStream.sh", so your prompt is almost immediately available compared to what remains to process in the function.
I suggest you wrap the command inside a script so you'll be able to handle the time you want before your prompt is back (I suggest 1sec more than the suspected time needed for the function to process the remaining lines of codes)
I successfully managed to do this like that :
./Functions.sh
#!/bin/bash
function ProcessErrors()
{
while read data;
do
echo Line was:"$data"
done
sleep 5 # simulate required time to process end of function (after TestErrorStream.sh is over and stdin is released)
echo "Completed"
}
./TestErrorStream.sh
#!/bin/bash
echo "first"
echo "firsterr" >&2
sleep 20 # any number here
./WrapTestErrorStream.sh
#!/bin/bash
source ./Functions.sh
2> >(ProcessErrors) ./TestErrorStream.sh
sleep 6 # <= this one is important
With the above you'll get a nice "Completed" before your prompt after 26 seconds of processing. (Works fine with or without the additional "time" command)
user#host:~/path$ time ./WrapTestErrorStream.sh
first
Line was:firsterr
Completed
real 0m26.014s
user 0m0.000s
sys 0m0.000s
user#host:~/path$
Note: the process substitution ">(ProcessErrors)" is a subprocess of the script "./TestErrorStream.sh". So when the script ends, the subprocess is no more tied to it nor to the wrapper. That's why we need that final "sleep 6"
#!/bin/bash
function ProcessErrors {
while read data; do
echo Line was:"$data"
done
sleep 5
echo "Completed"
}
# Open subprocess
exec 60> >(ProcessErrors)
P=$!
# Do the work
2>&60 ./TestErrorStream.sh
# Close connection or else subprocess would keep on reading
exec 60>&-
# Wait for process to exit (wait "$P" doesn't work). There are many ways
# to do this too like checking `/proc`. I prefer the `kill` method as
# it's more explicit. We'd never know if /proc updates itself quickly
# among all systems. And using an external tool is also a big NO.
while kill -s 0 "$P" &>/dev/null; do
sleep 1s
done
Off topic side-note: I'd love to see how posturing bash veterans/authors try to own this. Or perhaps they already did way way back from seeing this.
My Perl script outputs and logs many lines of text, and it does some cleanup and compresses some logs in an END block.
The problem is when you do something like this on the command line:
perl myscript.pl | head -n 10
This causes the script to die as soon as it outputs 10 lines, so the END block doesn't get executed and the logs don't get compressed. Is there a way to get around this and make sure the code in my END block is still executed?
When the reading end of a pipe is closed, and the writing process tries to write something to a pipe, then the writing process receives a SIGPIPE. The pipe is called broken.
We can capture this event like
local $SIG{PIPE} = sub {
# This is our event handler.
warn "Broken pipe, will exit\n";
exit 1;
};
This would gracefully exit your program. Instead of installing a sub as event handler, you could give the string IGNORE. This would let your script carry on as if nothing happened.
# print will now return false with $!{EPIPE} true instead of dying
local $SIG{PIPE} = 'IGNORE';
I've got a function that does (in short):
my $file = IO::File->new("| some_command >> /dev/null 2>&1")
or die "cannot open some_command for writing: $!\n";
...
undef $file;
Right now I'm not even writing anything to $file. Currently there are no other operations on $file at all. When I run the program, it doesn't exit properly. I see that handle is closed, but my program is still waiting for the process to close. Captured with strace:
close(6) = 0
rt_sigaction(SIGHUP, {SIG_IGN}, {SIG_DFL}, 8) = 0
rt_sigaction(SIGINT, {SIG_IGN}, {SIG_DFL}, 8) = 0
rt_sigaction(SIGQUIT, {SIG_IGN}, {SIG_DFL}, 8) = 0
wait4(16861, ^C <unfinished ...>
I don't see this problem if I open the same process for reading.
What do I have to do to make the program to exit?
Edit: Suggestions so far were to use the Expect library or to finish the input stream via ctrl+d. But I do not want to interact with the program in any way at this point. I want it to finish exactly now without any more IO going on. Is that possible?
undef $file removes a reference count from the filehandle and makes it eligible for garbage collection. If $file is a handle to a regular file and there are no other references to the filehandle anywhere else, it should work as documented in IO::File. In this case $file is a handle to a shell command, and there may be some other internal references to the filehandle that keep it from getting destroyed. Using $file->close is safer and makes your intent much clearer.
To kill off the command when closing the filehandle doesn't work, you need the process ID. If you invoked the command like
my ($file,$pid);
$pid = open($file, "| some_command >> /dev/null 2>&1");
then you could
kill 'TERM',$pid;
at the end of your program. I don't know how to extract the process ID from the return value of IO::File::new though.
If some_command is waiting for input, it will likely sit there forever doing just that, waiting for input.
From what the docs say, I don't think it makes any difference, but I always do $file->close() instead of/before undef'ing the handle.
EDIT: Send it Control D?
Perhaps some_command is reading tty's instead of stdin, like passwd does. If you are in that realm, I'd suggest looking up Expect.
Control D simply duplicates the zero byte read that close should do anyway for a command line program.
Have you tried using $file->close() instead of the undef?
Does some_command slurp all input and process it? Such as grep?
Or does it prompt? Like, say... chfn?
Does it return any useful information? Like an indication that it's finished?
If it's the latter, you might want to read up on Expect so you that you can interact with it.
This ugly, ugly hack will cause some_command to be parented to init instead of staying in your perl's process tree. Perl no longer has any process to wait for, and the pipe still works -- yay UNIX.
my $file = IO::File->new("| some_command >> /dev/null 2>&1 &")
Cons: The shell will succeed at & even if some_command fails, so you won't get any errors back.
or die "cannot open some_command for writing: $!\n"; # now useless
If some_command exited as soon as it got an EOF on stdin (and never stops reading from stdin), though, I'd expect this wouldn't be necessary.
$ cat | some_command
^D
Does that hang, and can you fix that?
Ok, I was running POV-Ray on all the demos, but POV's still single-threaded and wouldn't utilize more than one core. So, I started thinking about a solution in BASH.
I wrote a general function that takes a list of commands and runs them in the designated number of sub-shells. This actually works but I don't like the way it handles accessing the next command in a thread-safe multi-process way:
It takes, as an argument, a file with commands (1 per line),
To get the "next" command, each process ("thread") will:
Waits until it can create a lock file, with: ln $CMDFILE $LOCKFILE
Read the command from the file,
Modifies $CMDFILE by removing the first line,
Removes the $LOCKFILE.
Is there a cleaner way to do this? I couldn't get the sub-shells to read a single line from a FIFO correctly.
Incidentally, the point of this is to enhance what I can do on a BASH command line, and not to find non-bash solutions. I tend to perform a lot of complicated tasks from the command line and want another tool in the toolbox.
Meanwhile, here's the function that handles getting the next line from the file. As you can see, it modifies an on-disk file each time it reads/removes a line. That's what seems hackish, but I'm not coming up with anything better, since FIFO's didn't work w/o setvbuf() in bash.
#
# Get/remove the first line from FILE, using LOCK as a semaphore (with
# short sleep for collisions). Returns the text on standard output,
# returns zero on success, non-zero when file is empty.
#
parallel__nextLine()
{
local line rest file=$1 lock=$2
# Wait for lock...
until ln "${file}" "${lock}" 2>/dev/null
do sleep 1
[ -s "${file}" ] || return $?
done
# Open, read one "line" save "rest" back to the file:
exec 3<"$file"
read line <&3 ; rest=$(cat<&3)
exec 3<&-
# After last line, make sure file is empty:
( [ -z "$rest" ] || echo "$rest" ) > "${file}"
# Remove lock and 'return' the line read:
rm -f "${lock}"
[ -n "$line" ] && echo "$line"
}
#adjust these as required
args_per_proc=1 #1 is fine for long running tasks
procs_in_parallel=4
xargs -n$args_per_proc -P$procs_in_parallel povray < list
Note the nproc command coming soon to coreutils will auto determine
the number of available processing units which can then be passed to -P
If you need real thread safety, I would recommend to migrate to a better scripting system.
With python, for example, you can create real threads with safe synchronization using semaphores/queues.
sorry to bump this after so long, but I pieced together a fairly good solution for this IMO
It doesnt work perfectly, but it will limit the script to a certain number of child tasks running, and then wait for all the rest at the end.
#!/bin/bash
pids=()
thread() {
local this
while [ ${#} -gt 6 ]; do
this=${1}
wait "$this"
shift
done
pids=($1 $2 $3 $4 $5 $6)
}
for i in 1 2 3 4 5 6 7 8 9 10
do
sleep 5 &
pids=( ${pids[#]-} $(echo $!) )
thread ${pids[#]}
done
for pid in ${pids[#]}
do
wait "$pid"
done
it seems to work great for what I'm doing (handling parallel uploading of a bunch of files at once) and keeps it from breaking my server, while still making sure all the files get uploaded before it finishes the script
I believe you're actually forking processes here, and not threading. I would recommend looking for threading support in a different scripting language like perl, python, or ruby.