I am trying to make a script where I strictly want to control the standard output. so I am redirecting everything to /dev/null and printing my own messages.
Example:
user#root> tar zxvf samplefile.tar;> /dev/null 2>&1 ;echo $?
gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error exit delayed from previous errors
0
tar command failed but $? is storing exit status of redirection. I want exit status of tar command.
Query:
How can I get the status of tar command also redirect all std output to /dev/null?
As mentioned by Subbeh and Ploutox, I should not have used ";" in between the two commands.after removing ";" ,redirection became part of the orignal command and gave the correct return code.
However I am still not sure what could be used if we need to get exit status of ($?-1).
Related
I am new to using Linux/Unix command. I have a command that creates a Jmeter file and then in log checks if the generated Jmeter file (If there is any error in Jmeter file it will show error in percentage). If the log has some errors it provides Failure. During this process I get outcome as Failure or Success but the file does not get generated. How can I modify the command so that file gets generated as well as able to check error.
if jmeter -n -t $JMX_FILE -Jurl=$TEST_URL -Jthreads=1 -Jrampup=1 -Jcount=1 -Jsla=$SLA -f -l jmeter-results.jtl | grep "0 (0.00%)$"
then
echo Success
exit 0
else
echo Failure
exit 1
fi
I would like to conduct analysis using program Arlsumstat_64bit with thousand of input files.
Arlsumstat_64bit reads input files (.arp) and write result file (sumstat.out).
Each input will append new line on the result file (sumstat.out) based on the argument "0 1"
Therefore, I wrote a shell script to execute all the input (*.arp) in the same folder.
However, if the input files contain error, the shell script will be stuck without any subsequently process. Therefore, I found a command with "timeout" to deal my issue.
I made a shell script as following
#!/bin/bash
for sp in $(ls *.arp) ;
do
echo "process start: $sp"
timeout 10 arlsumstat_64bit ${sp}.arp sumstat.out 1 0
rm -r ${sp}.res
echo "process done: $sp"
done
However, I still need to know which input files failed.
How could make a list telling me which input files are "timeout"?
See the man page for the timeout command http://man7.org/linux/man-pages/man1/timeout.1.html
If the command times out, and --preserve-status is not set, then exit
with status 124. Otherwise, exit with the status of COMMAND. If no
signal is specified, send the TERM signal upon timeout. The TERM
signal kills any process that does not block or catch that signal.
It may be necessary to use the KILL (9) signal, since this signal
cannot be caught, in which case the exit status is 128+9 rather than
124.
You should find out which exit codes are possible for the program arlsumstat_64bit. I assume it should exit with status 0 on success. Otherwise the script below will not work. If you need to distinguish between timeout and other errors it should not use exit status 124 or which is used by timeout to indicate a timeout. So you can check the exit status of your command to distinguish between success, error or timeout as necessary.
To keep the script simple I assume you don't need to distingish between timeout and other errors.
I added some comments where I modified your script to improve it or to show alternatives.
#!/bin/bash
# don't parse the output of ls
for sp in *.arp
do
echo "process start: $sp"
# instead of using "if timeout 10 arlsumstat_64bit ..." you could also run
# timeout 10 arlsumstat_64bit... and check the value of `$?` afterwards,
# e.g. if you want to distinguish between error and timeout.
# $sp will already contain .arp so ${sp}.arp is wrong
# use quotes in case a file name contains spaces
if timeout 10 arlsumstat_64bit "${sp}" sumstat.out 1 0
then
echo "process done: $sp"
else
echo "processing failed or timeout: $sp"
fi
# If the result for foo.arp is foo.res, the .arp must be removed
# If it is foo.arp.res, rm -r "${sp}.res" would be correct
# use quotes
rm -r "${sp%.arp}.res"
done
Below code should work for you:
#!/bin/bash
for sp in $(ls *.arp) ;
do
echo "process start: $sp"
timeout 10 arlsumstat_64bit ${sp}.arp sumstat.out 1 0
if [ $? -eq 0 ]
then
echo "process done sucessfully: $sp"
else
echo "process failed: $sp"
fi
echo "Deleting ${sp}.res"
rm -r ${sp}.res
done
This question already has answers here:
Exit when one process in pipe fails
(2 answers)
Closed 4 years ago.
Even if the mycode.sh has non-0 exit code this command returns 0 as ssh connection was successful. How to get the actual return code of the .sh on remote server?
/home/mycode.sh '20'${ODATE} 1 | ssh -L 5432:localhost:5432 myuser#myremotehost cat
This is not related to SSH, but to how bash handles the exit status in pipelines. From the bash manual page:
The return status of a pipeline is the exit status of the last command, unless the pipefail option is enabled. If pipefail is enabled, the pipeline's return status is the value of the last (rightmost) command to exit with a non-zero status, or zero if all commands exit successfully. If the reserved word ! precedes a pipeline, the exit status of that pipeline is the logical negation of the exit status as described above. The shell waits for all commands in the pipeline to terminate before returning a value.
If you want to check that there was an error in the pipeline due to any of the commands involved, just set the pipefail option:
set -o pipefail
your_pipeline_here
echo $? # Prints non-zero if something went wrong
It is not possible to actually send the exit status to the next command in the pipeline (in your case, ssh) without additional steps. If you really want to do that, the command will have to be split like this:
res="$(/home/mycode.sh '20'${ODATE} 1)"
if (( $? == 0 )); then
echo -n "$res" | ssh -L 5432:localhost:5432 myuser#myremotehost cat
else
# You can do anything with the exit status here - even pass it on as an argument to the remote command
echo "mycode.sh failed" >&2
fi
You may want to save the output of mycode.sh to a temporary file instead of the $res variable if it's too large.
/home/mycode.sh is located onto the local host.
the ssh command is running cat on the remote server.
All text printed to the standard output of the /home/mycode.sh is redirected to the cat standard input.
The man ssh reads:
EXIT STATUS
ssh exits with the exit status of the remote command or with 255 if an error occurred.
Conclusion: the ssh exists with the EXIT STATUS of the cat or 255 if an error occurred.
if /home/mycode.sh script prints commands to the standard input, they can be run on the remote server when the cat is not present:
/home/mycode.sh '20'${ODATE} 1 | ssh -L 5432:localhost:5432 myuser#myremotehost
In my test, the EXIT STATUS of the last command executed on the remote server is returned by ssh:
printf "%s\n" "uname -r" date "ls this_file_does_not_exist" |\
ssh -L 5432:localhost:5432 myuser#myremotehost ;\
printf "EXIT STATUS of the last command, executed remotely with ssh is %d\n" $?
4.4.0-119-generic
Wed Aug 29 02:55:04 EDT 2018
ls: cannot access 'this_file_does_not_exist': No such file or directory
EXIT STATUS of the last command, executed remotely with ssh is 2
I am trying to run a simple one-rule snakemake file as following:
resources_dir='resources'
rule downloadReference:
output:
fa = resources_dir+'/human_g1k_v37.fasta',
fai = resources_dir+'/human_g1k_v37.fasta.fai',
shell:
('mkdir -p '+resources_dir+'; cd '+resources_dir+'; ' +
'wget ftp://ftp-trace.ncbi.nih.gov/1000genomes/ftp/technical/reference/human_g1k_v37.fasta.gz; gunzip human_g1k_v37.fasta.gz; ' +
'wget ftp://ftp-trace.ncbi.nih.gov/1000genomes/ftp/technical/reference/human_g1k_v37.fasta.fai;')
But I get an error as :
Error in job downloadReference while creating output files
resources/human_g1k_v37.fasta, resources/human_g1k_v37.fasta.fai.
RuleException:
CalledProcessError in line 10 of
/lustre4/home/masih/projects/NGS_pipeline/snake_test:
Command 'mkdir -p resources; cd resources; wget ftp://ftp-
trace.ncbi.nih.gov/1000genomes/ftp/technical/reference/human_g1k_v37.fasta.gz; gunzip human_g1k_v37.fasta.gz; wget ftp://ftp-trace.ncbi.nih.gov/1000genomes/ftp/technical/reference/human_g1k_v37.fasta.fai;' returned non-zero exit status 2.
File "/lustre4/home/masih/projects/NGS_pipeline/snake_test", line 10, in __rule_downloadReference
File "/home/masih/miniconda3/lib/python3.6/concurrent/futures/thread.py", line 55, in run
Removing output files of failed job downloadReference since they might be corrupted:
resources/human_g1k_v37.fasta
Will exit after finishing currently running jobs.
Exiting because a job execution failed. Look above for error message
I am not using the threads option in snakemake. I can not figure out how this is related with thread.py. Anybody has experience with this error?
When a shell command fails, it has an exit status which is not 0.
This is what "returned non-zero exit status 2" indicates.
One of your shell command fails, and the failure is propagated to snakemake. I suppose that snakemake uses threads and that the failure manifests itself at the level of some code in the threads.py file1.
In order to better understand what is happening, we can capture the first error using the || operator followed by a function issuing an error message:
# Define functions to be used in shell portions
shell.prefix("""
# http://linuxcommand.org/wss0150.php
PROGNAME=$(basename $0)
function error_exit
{{
# ----------------------------------------------------------------
# Function for exit due to fatal program error
# Accepts 1 argument:
# string containing descriptive error message
# ----------------------------------------------------------------
echo "${{PROGNAME}}: ${{1:-"Unknown Error"}}" 1>&2
exit 1
}}
""")
resources_dir='resources'
rule downloadReference:
output:
fa = resources_dir+'/human_g1k_v37.fasta',
fai = resources_dir+'/human_g1k_v37.fasta.fai',
params:
resources_dir = resources_dir
shell:
"""
mkdir -p {params.resources_dir}
cd {params.resources_dir}
wget ftp://ftp-trace.ncbi.nih.gov/1000genomes/ftp/technical/reference/human_g1k_v37.fasta.gz || error_exit "fasta download failed"
gunzip human_g1k_v37.fasta.gz || error_exit "fasta gunzip failed"
wget ftp://ftp-trace.ncbi.nih.gov/1000genomes/ftp/technical/reference/human_g1k_v37.fasta.fai || error_exit "fai download failed"
"""
When I run this, I get the following message after the messages of the first download:
gzip: human_g1k_v37.fasta.gz: decompression OK, trailing garbage ignored
bash: fasta gunzip failed
It turns out that gzip uses a non-zero exit code in case of warnings:
Exit status is normally 0; if an error occurs, exit status is 1. If a warning occurs, exit status is 2.
(from the DIAGNOSTICS section of man gzip)
If I remove the error-capturing || error_exit "fasta gunzip failed", the workflow is able to complete. So I don't understand why you had this error in the first place.
I'm surprised that gzip authors decided to use a non-zero status in case of a simple warning. They added a -q option to turn off this specific warning, due to the presence of trailing zeroes, but strangely, the exit status is still non-zero when this option is used.
1 According to Johannes Köster, author of snakemake:
Sorry for the misleading thread.py thing, this is just the place where snakemake detects the problem. The real issue is that your command exits with exit code 2, which indicates an error not related to Snakemake
What does tar exit code of -1 means? I am using the command
system("tar xzf DirName/tarFile.tar DirNameInsideTar/FileName")
to untar from perl script. I am trying to untar around 1000 tar files but randomly after untarring sometimes 100 or sometimes 500 tar files the system command fails and returns -1. I checked both the memory usage of the code and disk space available and both seems fine. Does anyone know what the error could be?
When system returns -1, $! contains the error message.
system("tar xzf DirName/tarFile.tar DirNameInsideTar/FileName");
die("Can't execute command: $!\n") if $? == -1;
die("Child killed by signal ".($? & 0x7F)."\n") if $? & 0x7F;
die("Child exited with error ".($? >> 8)."\n") if $? >> 8;
(The value returned by system is also available as $?.)