I have two shell scripts. In one of them I have:
export FOO="yes"
sh another.sh &
# ops ...
export FOO=0
In the another.sh I have:
while [[ $FOO -eq "yes" ]]
do
# something ...
done
The thing is, when the first script finishes, setting FOO=0, the value of FOO in another.sh continues being "yes". I want to know how to get the updated FOO so I can figure out when the first script (the caller) has finished.
A child process receives a copy of its parent's environment, not references to the original. If the parent changes while the child is still running, the child still has the original values in its memory space.
As a workaround, you could have the parent create an empty file in a known location before it exits, and have the child look for that file to be created. It's not very robust, but it's a simple form of interprocess communication that may be sufficient.
This is not possible: changes to the parent's environment are not passed on to child processes.
Have you considered inverting the parent and child relationship? If the child is waiting for the parent to finish, why not call what was the parent script from what was the child script? Assuming your first script was called "first.sh", something like
another.sh:
sh first.sh &
CHILDPID=$!
while ! (ps -p $CHILDPID | grep $CHILDPID)
do
[...]
done
Using job control (using set -m in bash) may be another solution.
Piped `while-read' loop starts subshell
Try to use other loop mechanism instead
Something like:
while read fileName
do
.........
done < <(ls -1tr $SOURCE_DIR)
Related
I'm sure this is probably a shamefully daft question, but it feels like I've done two laps of the web reading things about process management, output redirection etc. but I'm struggling to make sense of it all enough to achieve what I'd like.
In essence:
I want to run a shell script (with some arguments passed when it's called) in the background; and as it runs, write the pid of that instance to a specified file.
./offtimer.sh device1 10 &
of course runs the script in the background, and outputs that instance's pid to the screen; I'd just like to get that number into a file (eg. device1.pid).
I know (by reading as well as trial & error, I promise!) that
./offtimer.sh device1 10 & > device1.pid
isn't valid syntax; but despite thread after thread I've read, I can't figure out a way to do it.
Sincerely grateful for any help!
R
You can access the last child process ID using $!.
./offtimer.sh device1 10 &
echo $! > device1.pid
$! is the last background process's ID.
./offtimer.sh device1 10 &
echo $! > device1.pid
./offtimer.sh device1 10 > device1.pid &
Alternatively you could write that to the file within the script rather than output it to stdout.
I have 100 files in a directory, and want to process each one with several steps, while step1 is time-consuming. So the pseudocode is like:
for filename in ~/dir/*; do
run_step1 filename >${filename}.out &
done
for outfile in ~/dir/*.out; do
run_step2 outfile >${outfile}.result
done
My question is how can I check if step1 is complete for a given input file. I used to use threads.join in C#, but not sure if bash shell has equivalent.
It looks like you want:
for filename in ~/dir/*
do
(
run_step1 $filename >${filename}.out
run_step2 ${filename}.out >${filename}.result
) &
done
wait
This processes each file in a separate sub-shell, running first step 1 then step 2 on each file, but processing multiple files in parallel.
About the only issue you'll need to worry about is ensuring you don't try running too many processes in parallel. You might want to consider GNU parallel.
You might want to write a trivial script (doit.sh, perhaps):
run_step1 "$1" > "$1.out"
run_step2 "$1.out" > "$1.result"
and then invoke that script from parallel, one file per invocation.
Try this:
declare -a PROCNUMS
ITERATOR=0
for filename in ~/dir/*; do
run_step1 filename >${filename}.out &
PROCNUMS[$ITERATOR]=$!
let "ITERATOR=ITERATOR+1"
done
ITERATOR=0
for outfile in ~/dir/*.out; do
wait ${PROCNUMS[$ITERATOR]}
run_step2 outfile >${outfile}.result
let "ITERATOR=ITERATOR+1"
done
This will make an array of the created processes then wait for them in order as they need to be completed, not it relies on the fact there is a 1 to 1 relationship between in and out files and the directory is not changed while it is running.
Not for a small performance boost you can now run the second loop asynchronously too if you like assuming each file is independant.
I hope this helps, but if you have any questions please comment.
The Bash builtin wait can wait for a specific background job or all background jobs to complete. The simple approach would be to just insert a wait in between your two loops. If you'd like to be more specific, you could save the PID for each background job and wait PID directly before run_step2 inside the second loop.
After the loop that executes step1 you could write another loop that executes fg command which moves last process moved to background into foreground.
You should be aware that fg could return an error if a process already finished.
After the loop with fgs you are sure that all steps1 have finished.
I have a cron task that runs periodically. This task depends on a condition to be valid in order to complete its processing. In case it matters this condition is just a SELECT for specific records in the database. If the condition is not satisfied (i.e the SELECT does not return the result set expected) then the script exits immediately.
This is bad as the condition would be valid soon enough (don't know how soon but it will be valid due to the run of another script).
So I would like somehow to make the script more robust. I thought of 2 solutions:
Put a while loop and sleep constantly until the condition is
valid. This should work but it has the downside that once the script
is in the loop, it is out of control. So I though to additionally
after waking up to check is a specific file exists. If it does it
"understands" that the user wants to "force" stop it.
Once the script figures out that the condition is not valid yet it
appends a script in crontab and stops. That seconds script
continually polls for the condition and if the condition is valid
then restart the first script to restart its processing. This solution to me it seems to work but I am not sure if it is a good solution. E.g. perhaps programatically modifying the crontab is a bad idea?
Anyway, I thought that perhaps this problem is common and could have a standard solution, much better than the 2 I came up with. Does anyone have a better proposal? Which from my ideas would be best? I am not very experienced with cron tasks so there could be things/problems I could be overseeing.
instead of programmatically appending the crontab, you might want to consider using at to schedule the job to run again at some time in the future. If the script determines that it cannot do its job now, it can simply schedule itself to run again a few minutes (or a few hours, as it may) later by way of an at command.
Following up from our conversation in comments, you can take advantage of conditional execution in a cron entry. Supposing you want to branch based on time of day, you might use the output from date.
For example: this would always invoke the first command, then invoke the second command only if the clock hour is currently 11:
echo 'ScriptA running' ; [ $(date +%H) == 11 ] && echo 'ScriptB running'
More examples!
To check the return value from the first command:
echo 'ScriptA' ; [ $? == 0 ] echo 'ScriptB'
To instead check the STDOUT, you can use as colon as a noop and branch by capturing output with the same $() construct we used with date:
: ; [ $(echo 'ScriptA') == 'ScriptA' ] && echo 'ScriptB'
One downside on the last example: STDOUT from the first command won't be printed to the console. You could capture it to a variable which you echo out, or write it to a file with tee, if that's important.
I have a bash script similar to the following:
function test
{
running=$(( $running - 1 ))
}
running=0
test &
echo $running
Because the test function is run in a sub shell it doesn't affect the running variable and I get 0 echoed to the screen. I need the sub shell to be able to change the parent shells variables, how can this be done? I have tried export but to no avail.
EDIT Thanks for all the helpful answers, The reason I want to run this function in the background is to allow the running of multiple of functions simultaneously. I need to be able to call back to the parent script to tell it when all the functions are finished. I had been using pids to do this but I don't like having to check if multiple processes are alive constantly in a loop.
You can't really. Each shell has a copy of the environment.
see Can a shell script set environment variables of the calling shell?
But for what you are doing in your example, try this as your script:
#!/bin/bash
function testSO
{
running=$(( $running - 1 ));
return $running;
}
and invoke it as:
running=$(testSO)
If you only want to effectively return a value, then just return the value from the function.
Use aliases instead of functions. Or write a script with the function body and execute it with source.
Both solutions avoid creating a sub-shell.
I'm writing a program that forks multiple child processes and I'd like for all of these child processes to be able to write lines to STDERR and STDOUT without the output being garbled. I'm not doing anything fancy, just emitting lines that end with a new line (that, at least in my understanding would be an atomic operation for Linux). From perlfaq it says:
Both the main process and the backgrounded one (the "child" process) share the same STDIN, STDOUT and STDERR filehandles. If both try to access them at once, strange things can happen. You may want to close or reopen these for the child. You can get around this with opening a pipe (see open) but on some systems this means that the child process cannot outlive the parent.
It says I should "close or reopen" these filehandles for the child. Closing is simple, but what does it mean by "reopen"? I've tried something like this from within my child processes and it doesn't work (the output still gets garbled):
open(SAVED_STDERR, '>&', \*STDERR) or die "Could not create copy of STDERR: $!";
close(STDERR);
# re-open STDERR
open(STDERR, '>&SAVED_STDERR') or die "Could not re-open STDERR: $!";
So, what am I doing wrong with this? What would the pipe example it alludes to look like? Is there a better way to coordinate output from multiple processes together to the console?
Writes to a filehandle are NOT atomic for STDOUT and STDIN. There are special cases for things like fifos but that's not your current situation.
When it says re-open STDOUT what that means is "create a new STDOUT instance" This new instance isn't the same as the one from the parent. It's how you can have multiple terminals open on your system and not have all the STDOUT go to the same place.
The pipe solution would connect the child to the parent via a pipe (like | in the shell) and you'd need to have the parent read out of the pipe and multiplex the output itself. The parent would be responsible for reading from the pipe and ensuring that it doesn't interleave output from the pipe and output destined to the parent's STDOUT at the same time. There's an example and writeup here of pipes.
A snippit:
use IO::Handle;
pipe(PARENTREAD, PARENTWRITE);
pipe(CHILDREAD, CHILDWRITE);
PARENTWRITE->autoflush(1);
CHILDWRITE->autoflush(1);
if ($child = fork) { # Parent code
chomp($result = <PARENTREAD>);
print "Got a value of $result from child\n";
waitpid($child,0);
} else {
print PARENTWRITE "FROM CHILD\n";
exit;
}
See how the child doesn't write to stdout but rather uses the pipe to send a message to the parent, who does the writing with its stdout. Be sure to take a look as I omitted things like closing unneeded file handles.
While this doesn't help your garbleness, it took me a long time to find a way to launch a child-process that can be written to by the parent process and have the stderr and stdout of the child process sent directly to the screen (this solves nasty blocking issues you may have when trying to read from two different FD's without using something fancy like select).
Once I figured it out, the solution was trivial
my $pid = open3(*CHLD_IN, ">&STDERR", ">&STDOUT", 'some child program');
# write to child
print CHLD_IN "some message";
close(CHLD_IN);
waitpid($pid, 0);
Everything from "some child program" will be emitted to stdout/stderr, and you can simply pump data by writing to CHLD_IN and trust that it'll block if the child's buffer fills. To callers of the parent program, it all just looks like stderr/stdout.