I am using two scripts say script1 and script2. I call script2 from script1 which generates lots of text data. I am redirecting this text to a file using tee. But tee is stuck and doesn't move forward.
I cannot give the exact example as scripts are way too long to be included here. So giving a snippet of what's going on.
# script1
# Do some task
/bin/bash script2.sh 2>&1 | tee script2.log
capture=$(cat script2.log | grep "Successfully completed" | wc -l)
# Do some more work
#script2
# Execute some code
# Some python scripts
echo "Script End here"
Now when I execute script1.sh I am getting Script End here indicating second script completed successfully. But script1 is stuck at tee. When I removed tee and didn't capture output it worked out fine.
Did anyone face this before. Any idea what might be going on.
# System details
$ uname -a
Linux jetson-nx-jetpack461 4.9.253-tegra #1 SMP PREEMPT Sat Feb 19 08:58:27 PST 2022 aarch64 aarch64 aarch64 GNU/Linux
# Bash version
$ bash --version
GNU bash, version 4.4.20(1)-release (aarch64-unknown-linux-gnu)
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
From script2
When the script has a large number of lines (1000/2000/...), it is difficult to debug using set -x.
Either proceed based on comment from Barmar, or you include:
echo read1
read
command12
echo read2
read
command23
View the location where it is hanging.
If it is hanging exactly due to tee, use the following command:
echo read5
read
echo type tee
type tee
echo read6
read
Using this we can identify exactly where the script is hanging.
You can also use read1.1 read1.0 read1.-1 ... when debugging.
Related
I wrote a simple bash alias to create a sort of daily tmp directory to work in:
alias datdir="mkdir $(date +'%m_%d_%Y')___$RANDOM"
When I call it repeatedly on the same day, I get this:
mkdir: cannot create directory ‘02_04_2022___24499’: File exists
However, when I simply run echo $RANDOM on the terminal, I get different numbers.
I'm running Git Bash on Microsoft Windows 10 Business, Version 10.0.19044 Build 19044
$ bash --version
GNU bash, version 4.4.23(1)-release (x86_64-pc-msys)
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Parallel seems to be installed properly as I am able to run basic parallel check commands.
# parallel --version
GNU parallel 20161222
Copyright (C) 2007,2008,2009,2010,2011,2012,2013,2014,2015,2016
Ole Tange and Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
GNU parallel comes with no warranty.
Web site: http://www.gnu.org/software/parallel
When using programs that use GNU Parallel to process data for publication
please cite as described in 'parallel --citation'.
But when I'm trying to execute a query on remote servers it doesn't give any output also doesn't returns the prompt.. eg :-
parallel ssh {} hostname :::: hosts
Academic tradition requires you to cite works you base your article on.
When using programs that use GNU Parallel to process data for publication
please cite:
O. Tange (2011): GNU Parallel - The Command-Line Power Tool,
;login: The USENIX Magazine, February 2011:42-47.
This helps funding further development; AND IT WON'T COST YOU A CENT.
If you pay 10000 EUR you should feel free to use GNU Parallel without citing.
To silence this citation notice: run 'parallel --citation'.
in above eg , hosts file contains a list of remote server IP's
Even a simple parallel echo ::: A B C command doens's gives output..
I tried running with /usr/bin/parallel but issue remains..
when I ran the command using bash -x I get the following error :-
bash -x parallel ssh {} hostname :::: hosts
+ use IPC::Open3
/usr/bin/parallel: line 22: use: command not found
/usr/bin/parallel: parallel: line 24: syntax error near unexpected token `('
/usr/bin/parallel: parallel: line 24: `use POSIX qw(:sys_wait_h setsid ceil :errno_h);'
I've checked the file /usr/bin/parallel seems fine to me.
Not able to find solution for this. Any help is really appreciated.
Thanks.
UPDATE_1 :-
root#sg-server:[~]:# parallel -Dall echo ::: 1
shell? perl /usr/bin/parallel -Dall echo ::: 1
shell? -bash
which -bash => shell path /bin/bash
Academic tradition requires you to cite works you base your article on.
When using programs that use GNU Parallel to process data for publication
please cite:
O. Tange (2011): GNU Parallel - The Command-Line Power Tool,
;login: The USENIX Magazine, February 2011:42-47.
This helps funding further development; AND IT WON'T COST YOU A CENT.
If you pay 10000 EUR you should feel free to use GNU Parallel without citing.
To silence this citation notice: run 'parallel --citation'.
Open file descriptors: 2 1
echo Context: Non: 4 Ctxgrp: NonCtxGrp: 1
echo ▒<▒> Context: Non: 4 Ctxgrp: 1 NonCtxGrp: 1
Wanted procs: 16
MultifileQueue->empty RecordQueue->empty CommandLineQueue->empty JobQueue->empty MultifileQueue->empty RecordQueue->empty eval 1
noncontext + command: 5
context+noncontext + command: 5
cmd_line->number_of_args 1
Replace echo ▒<▒>
Target: echoTarget: ▒<▒>Replacing in ▒<▒>
Return echo 1
noncontext + command: 5
context+noncontext + command: 5
noncontext + command: 5
context+noncontext + command: 5
6 == 6 echo 1
Time to fork 1 procs: 0 (processes so far: 1)
MultifileQueue->empty 1 RecordQueue->empty 1CommandLineQueue->empty 1JobQueue->empty 1 RecordQueue-unget ''
JobQueue->empty Limited to procs: 1
JobQueue->empty Running jobs before on ::
: has out of 1 jobs running. Start another.
JobQueue->empty Command to run on ':': 'echo 1'
processes . Starting (1): echo 1
bash -c testfun\(\)\ \{\ rm\ /tmp/parTMasE.tst\;\ \}\;\ export\ -f\ testfun\;\ perl\ -MIPC::Open3\ -e\ if\\\(\\\$pid\\\=::open3\\\(\\\$i,\\\$o,\\\$e,\\\"-\\\"\\\)\\\)\\\ \\\{\\\ wait\\\;\\\ \\\}\\\ else\\\ \\\{\\\ exec\\\(\\\"bash\\\",\\\"-c\\\",\\\"testfun\\\ \\\&\\\&\\\ true\\\"\\\)\\\;\\\ \\\} 2>/dev/null
^Ckill_sleep TERM
kill_sleep TERM
kill_sleep TERM
kill_sleep KILL
Even on running parallel -Dall echo ::: 1 command, I had to CTRL+c to get the prompt back.
As we debugged in the comments your (/usr/local)/bin/rm is not the normal (/usr/local)/bin/rm.
I am running:
» make --version
GNU Make 4.1
Built for x86_64-pc-linux-gnu
Copyright (C) 1988-2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
This works, from bash:
$ echo $RANDOM
14522
This does not work:
$ make echo-random
With Makefile:
echo-random:
echo $(RANDOM)
Some questions:
Does make use a shell to run commands?
Is it possible to tell shell to use bash?
Can make somehow access bash special env variables?
You can invoke bash with the -c argument (that tells it the next argument is a command it has to run and exit):
echo-random:
#bash -c 'echo $$RANDOM'
This way, each invocation of make echo-random starts a new bash instance that runs the command echo $RANDOM and it produces the outcome you expect.
The answer by #axiac is good. This is an alternative:
SHELL = /bin/bash
random := $(shell echo $$RANDOM)
echo-random:
echo $$RANDOM
echo $(random)
Output:
» make
echo $RANDOM
18826
echo 16300
16300
See here and here
I have a bash script called test.sh which, for the sake of simplicity, prints one line to stdout and one line to stderr.
test.sh:
#!/bin/bash
echo "this is to stdout"
echo "this is to stderr" 1>&2
I want to run the script test.sh at 7:00 PM, but only if certain conditions are met. To this end, I have another bash script called schedule.sh, which checks some stuff and then submits the command to at to be run later.
I want the output of test.sh (both stdout and stderr) to be sent to me in an email. I use mailx to do this so I can get a nice subject name.
Furthermore, I want at to shut up. No output from at because it always sends me ugly emails (no subject line) if at produces any output.
schedule.sh:
#!/bin/bash
my_email="me#example.com" # Email is a variable
# Check some stuff, exit if certain conditions not met
echo "~/test.sh 2>&1 | mailx -s\"Cool title\" $my_email" | at 7:00 PM &> /dev/null
What's interesting is that when I run schedule.sh from cron (which runs the script with sh), it works perfectly. However, when I manually run schedule.sh from the terminal (NB: I'm using tcsh), at (not mailx) sends me an email saying
Ambiguous output redirect.
I'm not sure why the shell I run schedule.sh from makes a difference, when schedule.sh is a bash script.
Here is my thinking in looking at schedule.sh. Everything within the quotation marks "~/test.sh 2>&1 | mailx -s\"Cool title\" me#email.com" should be an argument to at, and at runs that argument as a command using sh. The redirection 2>&1 | is in the style of sh for this reason.
When I remove 2>&1 and only pipe the stdout of test.sh to mailx, it does work; however, I receive 2 emails: one with stdout from mailx and another from stderr from at.
What gives? How can I make this work regardless of the shell I'm calling it from?
Thanks.
edit:
uname -o says my OS is GNU/Linux
Here is uname -a if it helps:
Linux [hostname censored] 2.6.9-89.ELlargesmp #1 SMP Mon Jun 22 12:46:58 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux
When I check the at contents using at -c, here's what I see:
#!/bin/sh
# atrun uid=xxxxx gid=xxxxx
# mail username 0
# ...
SHELL=/bin/tcsh; export SHELL
# ...
${SHELL:-/bin/sh} << `(dd if=/dev/urandom count=200 bs=1 2>/dev/null|LC_ALL=C tr -d -c '[:alnum:]')`
~/test.sh 2>&1 | mailx -s"Cool title" me#example.com
I'm having a hard time understanding the second to last line... is this going to execute using $SHELL or /bin/sh?
The command executed via at is:
~/test.sh 2>&1 | mailx -s\"Cool title\" $my_email
The behavior of at command varies from one system to another. On Linux, the command is executed using /bin/sh. In fact, on my system (Linux Mint 14), it prints a warning message:
$ echo 'printenv > at.env' | at 19:24
warning: commands will be executed using /bin/sh
On Solaris, the command is executed by the shell specified by the current value of the $SHELL environment variable. Using an account where my default shell is /bin/tcsh on Solaris 9, I get:
% echo 'printenv > at.env' | at 19:25
commands will be executed using /bin/tcsh
job 1397874300.a at Fri Apr 18 19:25:00 2014
% echo 'printenv > at.env' | env SHELL=/bin/sh at 19:28
commands will be executed using /bin/sh
job 1397874480.a at Fri Apr 18 19:28:00 2014
Given that at's behavior is inconsistent (and frankly confusing), I suggest having it execute just a single command, with any I/O redirection being performed inside that command. That's the best way to ensure that the command will be executed correctly regardless of which shell is used to execute it.
For example (untested code follows):
echo '#!/bin/bash' > tmp.bash
echo "~/test.sh 2>&1 | mailx -s\"Cool title\" $my_email" >> tmp.bash
chmod +x tmp.bash
echo "./tmp.bash" | at 7:00 PM
What will happen when I a script file and save it when it's still running, and will it print my needed results.
Let's test it.
Create a script test.sh:
#!/usr/bin/env bash
sleep 1
echo 'echo "executed overwritten"' >> "$0" # append to self
sleep 1
echo 'executed original'
and execute it:
$ bash --version
GNU bash, version 4.2.24(1)-release (i686-pc-linux-gnu)
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
$ chmod +x test.sh
$ ./test.sh
executed original
executed overwritten
$
Notice that bash continued reading the modified file. It maintains its current position in the file (in bytes) when the file changes.
As a demonstration, the script
#!/usr/bin/env bash
sleep 1
dd if=/dev/urandom bs=1024 count=1 of="$0" &>/dev/null # overwrite self
sleep 1
echo 'executed original'
gives the output
$ ./test.sh
./test.sh: line 6: syntax error near unexpected token `$'\311\262\203''
./test.sh: line 6: `��z�eп9)�v��▒y�a��44'{�d��4\:�A����˷���&�$�����l�
#(ɲ��4��OϹI�n>��7��P�M�a��X.�S�a���V�m�~O<��{}������J��$��TOtRd��Nw�&��B�Dz�▒��-��<`�P<?N��▒rT�Jq�L����JY�*hz���M�����i�⫣��S+�����\��c���m�NKV�8|��xvX}�V����PTd䊄�9��7���|��/��X��
��0¤k��_�R���e�*���(qu:UUɭp/j��n��bŇ_�UR?3▒�▒�%Rn�|DE$8�QbaK)A�{ ��O>9��A�����lt�����g)s��O��M��#���w��|�����N��,W'
Notice that it attempted to execute the random gibberish.
(This is Ubuntu 12.04. Behavior may vary with other shells.)