I have a makefile like this:
default:
%:
#$(MAKE) -i -C subdir1 $*
#$(MAKE) -i -C subdir2 $*
#$(MAKE) -i -C subdir3 $*
#$(MAKE) -i -C subdir4 $*
#$(MAKE) -i -C subdir5 $*
The basic concept is that I have 5 (or more) sub projects which I will call make on sequentially. I use "-i" flag so that the make can continue to the end and the "-C dir" flag to call make in a sub-directory.
So, lets say that sub project 2 and 5 are failing, then at the end I want to be able to print something like:
3 projects built ok, 2 projects have errors.
So I think I want a counter of some sort, but I have no idea how I can set/increment it on an error. Any ideas?
As every call to $(MAKE) spawns its own subprocess, I can't think of a way to record these numbers easily with an ordinary make variable. You can, however, log the return value of each invocation to a (possibly hidden) file and then grep for your build stats like this:
errLog = .errLog
default:
%:
#$(MAKE) -i -C subdir1 $*; echo $$? > $(errLog)
#$(MAKE) -i -C subdir2 $*; echo $$? >> $(errLog)
#$(MAKE) -i -C subdir3 $*; echo $$? >> $(errLog)
#$(MAKE) -i -C subdir4 $*; echo $$? >> $(errLog)
#$(MAKE) -i -C subdir5 $*; echo $$? >> $(errLog)
#echo "`grep -c '^0' $(errLog)` built ok, `grep -c '^[^0]' $(errLog)` have errors."
Note that the first output redirection must be a single > to overwrite previous return codes in the file, while all others should be two > to not overwrite the file content.
Related
Hello I'm running this bash script here and the command "bwa index .." will generate several files, and this command doesn't have the "-o" flag for specifying output path. I want to redirect these files into a new folder. I've searched online and all the answers I came across are redirecting files generated to a specific file name, but how to generally just redirect the output files into a new folder? Thanks very much!
#!/bin/bash
### qsub file.name to run from anywhere
### first step of BMW, making indexed fasta file
#PBS -N bwa_index
#PBS -S /bin/bash
#PBS -l walltime=24:00:00
#PBS -l nodes=1:ppn=8
#PBS -l mem=64gb
#PBS -o /gpfs/data/mcnerney-lab/liuweihan/chip_seq/becker_lab/bwa_index.out
#PBS -e /gpfs/data/mcnerney-lab/liuweihan/chip_seq/becker_lab/bwa_index.err
date
module load gcc/6.2.0
module load bwa/0.7.17
bwa index -p mm10_bwa_idx -a bwtsw /gpfs/data/mcnerney-lab/liuweihan/chip_seq/becker_lab/mm10.fa
date
echo END
You can change to the target directory and then run the bwa command. Using pushd and popd will ensure that rest of the script runs in correct directory.
pushd some_directory
bwa index -p mm10_bwa_idx -a bwtsw /gpfs/data/mcnerney-lab/liuweihan/chip_seq/becker_lab/mm10.fa
popd
First
You can define a variable and use it in your script.
Second
You can pass arguments to your script and capture them via $1, $2, $3 and so on.
Third if you wanted to be fully dynamic by using option ( e.g. --output | -o ) you can use a simple parse as bellow:
#!/bin/bash
################################################################################
# main flags, both longs and shorts
################################################################################
ARGS=`getopt -o "o::" -l "output:" -- "$#"`
eval set -- "$ARGS"
declare -A _output;
_output['flag']=0;
_output['path']=0;
################################################################################
# extract options and their arguments into variables.
################################################################################
while true ; do
case "$1" in
-o | --output )
_output['flag']=1;
_output['path']=$2;
shift 2;
;;
--)
shift;
break;
;;
* )
echo "Internal error!" ; exit 1
;;
esac
done
echo "output: ${_output['path']}";
And usage is super simple, you can use --output <YOUR-PATH> or -o <YOUR-PATH>
# test
./cli.sh --output /path/to/dir
output: /path/to/dir
./cli.sh -o /path/to/dir
output: /path/to/dir
I am trying to run a Makefile from under WSL that contains the following lines:
debug: create_soft_links
#mkdir -p Debug64
#echo -e 'all: bld' > Debug64/Makefile
#echo >> Debug64/Makefile
#echo -e '%.o: ../../%.c' >> Debug64/Makefile
#echo -e '\tgcc -g $$(CFLAGS) $$(INCLUDE) $$< -o $$#' >> Debug64/Makefile
Problem is that the resulting Debug64/Makefile file looks like this:
-e all: bld
-e %.o: ../../%.c
-e gcc -O3 $(CFLAGS) $(INCLUDE) $< -o $#
A colleague just showed me on an actual Linux machine that the make command works correctly there, and the preceding -e flag is not printed in the generated Debug64/Makefile. What am I doing wrong?
Use instead of echo the printf(1) command. So your last line would be
#printf "\tgcc -g %s %s $$< -o $$#\n" $$(CFLAGS) $$(INCLUDE)
BTW, if you generate your build automation script, consider switching to ninja. You might use Guile or Python or GNU awk as such a generator.
I declare three variables.
$1=`ssh <server_1> cat /etc/passswd|cut -f -d:|grep -e $IID -e $EID`
$2=`ssh <server_2> cat /etc/shadow|cut -f -d:|grep -e $IID -e $EID`
$3=`ssh <server_3> cat /etc/passwd}|cut -f -d:|grep -i $CID`
The above three variables are created by taking ssh to servers and checking the presence of the IDs which I give as input. If the ID doesn't exist already, the the variable is going to be null.
Now, how do I verify if all the three variables are null. I wanted to use the OR condition specified within an IF.
I tried,
if [ -s "$1" -o -s "$2" -o -s "$3"];then
echo -$1 $2 $3 "already exist(s)"
It didnt work. Please advise.
PS: I have just begun my career in Unix and correct me If am wrong anywhere.
Several points.
When you assign to a variable, don't use the dollar sign:
foo=xxx
Variables $1, $2 etc are already used for your command line arguments. Pick other names. But not $4please. :-)
When you specify a command for ssh, and it has arguments, it has to be quoted, because the command needs to be a single argument for ssh. In your case use double quotes, as you want variable expansion for $IID etc.
Most Unix utils are able to open input files themselves, so you don't need to start your pipeline with cat.
foo=`ssh <server_1> "cut -f -d: /etc/passwd | grep -e $IID -e $EID"`
Or something like that.
It was a typo in my question. I had actually declared it as,
1=`ssh <server_1> cat /etc/passswd|cut -f -d:|grep -e $IID -e $EID`
2=`ssh <server_2> cat /etc/shadow|cut -f -d:|grep -e $IID -e $EID` and so on.
And I tried it as ,
if [ -s "$1" -o -s "$2" -o -s "$3"];then
echo -e $1 $2 $3 "already exist(s)"
Since I had to Deliver my script today, I used the conventional method of,
ssh <server_1> "cat /etc/passswd|cut -f -d:|grep -e $IID -e $EID" > file1
ssh <server_2> "cat /etc/shadow|cut -f -d:|grep -e $IID -e $EID" > file2
ssh <server_3> "cat /etc/passwd|cut -f -d:|grep -ix $CID" > file3
if [ -s file1 -o -s file2 -o -s file3]; then
for i in `cat file1 file2 file3`
do
echo $i "already exists"
done
else
And I have now learnt from my first post, that -s to ensure that a file is not empty and -z is to ensure string is empty.
I have written a Makefile which works fine; I am just posting the part of it under investigation:
BUILD_PRINT = #echo -e "\e[1;34mBuilding $<\e[0m"
COMPILE_cpp = $(CXX) $(CFLAGS) -o $# -c $< $(MAKEDEP) $(INCLUDES)
%.o : %.cpp
$(BUILD_PRINT)
$(COMPILE_cpp)
.SUFFIXES: .o .cpp
I would like to highlight the warnings and errors given by the compiler without using external tools (such as colorgcc or CMake); I thought that a good way to hack it was via "bash script tricks". Looking at the solution posted in How Can I highlight the warning and error lines in the make output? I have tried the following:
pathpat="(/[^/]*)+:[0-9]+"
ccred=$(echo -e "\033[0;31m")
ccyellow=$(echo -e "\033[0;33m")
ccend=$(echo -e "\033[0m")
BUILD_PRINT = #echo -e "\e[1;34mBuilding $<\e[0m"
COMPILE_cpp = $(CXX) $(CFLAGS) -o $# -c $< $(MAKEDEP) $(INCLUDES)
%.o : %.cpp
$(BUILD_PRINT)
$(COMPILE_cpp) 2>&1 | sed -e "/[Ee]rror[: ]/ s%$pathpat%$ccred&$ccend%g" -e "/[Ww]arning[: ]/ s%$pathpat%$ccyellow&$ccend%g" echo "${PIPESTATUS[0]}"
.SUFFIXES: .o .cpp
but it is not working. I get the following output
Building main.cpp
g++ -o main.o -c main.cpp 2>&1 | sed -e "/[Ee]rror[: ]/ s%athpat%cred&cend%g" -e "/[Ww]arning[: ]/ s%athpat%cyellow&cend%g" echo ""
sed: can't read echo: No such file or directory
sed: can't read : No such file or directory
Thanks in advance!
In make context the following lines do not behave the way they would in the shell:
ccred=$(echo -e "\033[0;31m")
ccyellow=$(echo -e "\033[0;33m")
ccend=$(echo -e "\033[0m")
In the shell those would put the echoed output into those variables. In make that tries to run the echo command, which doesn't exist, and ends up creating empty variables.
Those lines should either be
ccred=$(shell echo -e "\033[0;31m") to run the commands through the shell and store the output and then used as $(ccred) in the body
ccred=\033[0;31m to store the string in the variable and then used as $$(echo -e '$(ccred)') in the body
ccred=echo -e "\033[0;31m" to store the command in the variable and then used as $$($(ccred) in the body
Either of the first or second options is likely fine. (Use := instead of = in the first option to have make only run the echo command once, at make parse time, instead of every time ccred is used.)
make and shell variables share a prefix sigil $. As such to use shell variables in make contexts requires escaping the $ in shell contexts by doubling it to $$. As such s%$pathpat%$ccred&$ccend%g needs to be s%$$pathpat%$$ccred&$$ccend%g, etc.
Each line of a make rule body is executed as a distinct shell command, as such the commands cannot interact with each other. In order to use constructs like echo "${PIPESTATUS[0]}" meaningfully therefore requires that they be on the same command line in make. As such the compile line in that pattern rule would need to be $(COMPILE_cpp) 2>&1 | sed ...; echo "${PIPESTATUS[0]}".
However, even that isn't going to do what you want since you don't need to echo the exit status from the compilation you need to exit with it, so you probably want ; exit "${PIPESTATUS[0]}" there instead.
Got it working.
First of all thanks #EtanReisner and #rici. Here's the code:
BUILD_PRINT = \e[1;34mBuilding $<\e[0m
COMPILE_cpp = $(CXX) $(CFLAGS) -o $# -c $< $(MAKEDEP) $(INCLUDES)
COMPILE_cpp_OUT=$$($(COMPILE_cpp) 2>&1 | sed -e 's/error/\\\e[1;31merror\\\e[0m/g' -e s/warning/\\\e[1;33mwarning\\\e[0m/g')
%.o : %.cpp
#echo -e "$(BUILD_PRINT)\n$(COMPILE_cpp)\n$(COMPILE_cpp_OUT)"
.SUFFIXES: .o .cpp
All the commands are invoked by only one echo because I want all the outputs (command string and warnings/errors) coherently grouped for each file built when I launch a parallel build with make -j.
$(BUILD_PRINT) just prints out the path of the file currently being built.
$(COMPILE_cpp) prints out the string of the compiler, so that I can see the command with all the flags/dependencies/etc...
$(COMPILE_cpp_OUT) stores the output of the compiler and change some relevant word colour via sed command.
#!/bin/sh
LOCAL=/var/local
TMP=/var/tmp
URL=http://um10.eset.com/eset_upd
USER=""
PASSWD=""
WGET="wget --user=$USER --password=$PASSWD -t 15 -T 15 -N -nH -nd -q"
UPDATEFILE="update.ver"
cd $LOCAL
CMD="$WGET $URL/$UPDATEFILE"
eval "$CMD" || exit 1;
if [ -n "`file $UPDATEFILE|grep -i rar`" ]; then
(
cd $TMP
rm -f $TMP/$UPDATEFILE
unrar x $LOCAL/$UPDATEFILE ./
)
UPDATEFILE=$TMP/$UPDATEFILE
URL=`echo $URL|sed -e s:/eset_upd::`
fi
TMPFILE=$TMP/nod32tmpfile
grep file=/ $UPDATEFILE|tr -d \\r > $TMPFILE
FILELIST=`cut -c 6- $TMPFILE`
rm -f $TMPFILE
echo "Downloading updates..."
for FILE in $FILELIST; do
CMD="$WGET \"$URL$FILE\""
eval "$CMD"
done
cp $UPDATEFILE $LOCAL/update.ver
perl -i -pe 's/\/download\/\S+\/(\S+\.nup)/\1/g' $LOCAL/update.ver
echo "Done."
So I have this code to download definitions for my antivirus. The only problem is that, it downloads all files everytime i run script. Is it possible to implement some sort file checking ?, let's say for example,
"if that file is present and have same filesize skip it"
Bash Linux
The -nc argument to wget will not re-fetch files that already exist. It is, however, not compatible with the -N switch. So you'll have to change your WGET line to:
WGET="wget --user=$USER --password=$PASSWD -t 15 -T 15 -nH -nd -q -nc"