How to pass a Makefile target to another - linux

so I have a large Makefile that runs all of my tests for my particular project. Each target is a different group of tests. The script will run the target, store its output into a temporary file.
Currently the target looks like this:
count:
# USE: make count test=<name of test to run>
# Save output to target
$(MAKE) $(test) > last_output.txt
cat last_output.txt
# Print Passed
#cat last_output.txt | { grep -E -w "SUCCESS|RELAX-PASS" || true; }
# Print Failed
#cat last_output.txt | { grep -E -w "FAILED" || true; }
# Failed Count
#echo "\e[1;31mFAILED:\e[1;37m"
#cat last_output.txt | { grep -c "FAILED" || true; }
# Passed Count
#echo "\e[1;32mPASSED:\e[1;37m"
#cat last_output.txt | grep -E -c "SUCCESS|RELAX-PASS"
# Count all
#echo "TOTAL: "
#cat last_output.txt | { grep -E -c "FAILED|SUCCESS|RELAX-PASS" || true; }
And the instruction to execute it looks like:
make count test=add
What I was wondering was if I could not specify test= when I'm running the command so that it would look like this:
make count add
and then the add target will execute which looks like:
add:
clear && run.pl add_0.asm
clear && run.pl add_1.asm
clear && run.pl add_2.asm
clear && run.pl add_3.asm
ect.

Every command line argument to make is either an option, a variable assignment, or a target. There's no possible way to treat an argument as anything else. So when you run make check add, the add will be a target that make will attempt to build and there's no way to have it be considered any other way.
As #Beta suggests if you are willing to embed to the test name into the target, like make count_add, then you can do this:
SHOW = \
cat last_output.txt; \
grep -E -w "SUCCESS|RELAX-PASS" < last_output.txt; \
grep -E -w "FAILED" < last_output.txt; \
echo "\e[1;31mFAILED:\e[1;37m"; \
grep -c "FAILED" < last_output.txt; \
echo "\e[1;32mPASSED:\e[1;37m"; \
grep -E -c "SUCCESS|RELAX-PASS" < last_output.txt; \
echo "TOTAL: "; \
grep -E -c "FAILED|SUCCESS|RELAX-PASS" < last_output.txt;
true
count:
$(MAKE) $(test) > last_output.txt
#$(SHOW)
count_%:
$(MAKE) $* > last_output.txt
#$(SHOW)
If you don't want to do that, the only possible solution is to use ifeq combined with $(MAKECMDGOALS) to break your makefile up into two sections: one that has a .DEFAULT target that does nothing (to ignore the extra targets like add etc.) and a second that runs those targets.

Related

exit shell script from subshell if files are not present on remote server or disk is full?

I am using below script to copy files in parallel from remote server:
#!/bin/bash
primary=/bat/primary
secondary=/bat/secondary
do_copy() {
el=$1
primsec=$2
(scp -C -o StrictHostKeyChecking=no \
goldy#"$remote":"$pod1"/abc_187_"$el"_111_8.data "$primsec"/. > /dev/null 2>&1)\
|| \
(scp -C -o StrictHostKeyChecking=no \
goldy#"$remote":"$pod2"/abc_187_"$el"_111_8.data "$primsec"/. > /dev/null 2>&1)\
|| \
(scp -C -o StrictHostKeyChecking=no \
goldy#"$remote":"$pod3"/abc_187_"$el"_111_8.data "$primsec"/. > /dev/null 2>&1)\
|| \
(scp -C -o StrictHostKeyChecking=no \
goldy#"$remote":"$pod4"/abc_187_"$el"_111_8.data "$primsec"/. > /dev/null 2>&1)
}
export -f do_copy
parallel -j "$5" do_copy {} $primary ::: ${pri[#]} &
parallel -j "$5" do_copy {} $secondary ::: ${snd[#]} &
wait
echo "all files copied"
On the remote server I have four folders represented by $pod1, $pod2, $pod3, $pod4. If files are not there in $pod1, then it should be there in $pod2, or it will be there in $pod3 or it will be there in $pod4 folder.
Now what I want to do is:
If files are not there in any of those four folders then I want to exit from shell script with nonzero status code immediately.
If for whatever reason let's say disk is full and file were not copied fully then I want to exit as well with nonzero status code. Each file size is around 15GB and sometimes because of disk full, I see only 3GB copied out of 15GB and my shell script don't report any issues and passes successfully.
I tried adding exit command next to last scp command but it just exits from the subshell not from whole script. Is there any way to do these things?
I will assume that the file exists on only one pod, so trying all pods is not a problem.
#!/bin/bash
primary=/bat/primary
secondary=/bat/secondary
# Activate parset and env_parallel if not already done
. `which env_parallel.bash`
# Make array with $primary as many times as elements in $pri
parset prim -N0 echo $primary ::: ${pri[#]}
# Make array with $secondary as many times as elements in $pri
parset seco -N0 echo $secondary ::: ${sec[#]}
export remote
do_copy() {
el=$1
primsec=$2
pod=$3
scp -C -o StrictHostKeyChecking=no \
goldy#"$remote":"$pod"/abc_187_"$el"_111_8.data "$primsec"/
}
export -f do_copy
copy_one_file() {
parallel do_copy $1 $2 ::: pod{1..4}
if [ $? == 4 ] ; then
return 1
else
return 0
fi
}
export -f copy_one_file
parallel --halt now,fail=1 copy_one_file ::: ${pri[#]} ${sec[#]} :::+ ${prim[#]} ${seco[#]}
parset makes an array with $primary as long as $pri.
:::+ links the elements from $pri to the elements in $prim.
You will need parallel version > 20170422 for parset.

how to do if statements with "and" in sh

I try to solve this problem with the sh and not the bash.
All i want is a if statement that check some regex AND something else. Normally in bash this would be an easy job but with the sh i only find solutions online that wont work for me
First thing i want to check:
if echo "$1"| grep -E -q '^(\-t|\-\-test)$';
Than i want to check:
if echo "$#"| grep -E -q '^(1|2)$';
Combined:
if [ \(echo "$1"| grep -E -q '^(\-h|\-\-help)$'\) -a \(echo "$#"| grep -E -q '^(1|\2)$'\) ];
ERROR:
grep: grep: ./analysehtml.sh: 41: [: missing ]
Invalid back reference
(echo: No such file or directory
grep: 1: No such file or directory
I also try many diffrent combinations with this brackets but non of them worked for me. Maybe someone can help me here :)
logical and between commands is &&
if
echo "$1"| grep -E -q '^(\-h|\-\-help)$' &&
echo "$#"| grep -E -q '^(1|\2)$';
By default the exit status of a pipe is the exit status of last command.
set -o pipefail the exit status is fail if if any command of pipe has a fail exit status.
when only the exit status of the last command of a sequence must be checked
if { command11; command12;} && { command21; command22;};
However to check parameters there is no need to launch another process grep with a pipe there's an overhead.
Consider using following constructs work with any POSIX sh.
if { [ "$1" = -h ] || [ "$1" = --help ];} &&
{ [ $# -eq 1 ] || [ $# -eq 2 ];};
EDIT: Following are not POSIX but may work with many shell
if [[ $1 = -h || $1 = --help ]] && [[ $# = 1 || $# = 2 ]];
Works also with bash with set -o posix
Perhaps for your particular case, pattern matching might be better:
if [[ $1 =~ ^(\-h|\-\-help)$ && $# =~ ^(1|\2)$ ]]; then
The problem with your command is that the part within test or [ command is expression, not commands list.
So when you run [ echo 'hello' ] or [ \( echo 'hello' \) ] complains error in spite of sh or Bash. Refer to the classic test usage: The classic test command
And the syntax of if is:
if list; then list; fi
So you can just combine command with && operator in if statements:
if echo "$1"| grep -E -q '^(\-h|\-\-help)$' && echo "$#"| grep -E -q '^(1|\2)$';

Makefile not executing the commands as expected

I'm using the following Makefile, which should check whether the files are in some directory *.rpm, if the files hasn't been found i'm going to execute some command (run a script or make the rpm)
Here's a snippet from my Makefile
include /home/user/workspace/test/exec_recipe.mk
export BUILD_LOGS_DIR = $(CURDIR)/build_logs
.PHONY: my_rpm
libpdutil_rpm:
#echo "Making target 'libpdutil_rpm'. Time: $$(date +%T)"
cd /home/user/workspace/test/build/test && $(call exec_recipe,$(ls /home/user/workspace/test/build/test/rpmbuild/RPMS/x86_64d/*.rpm) || $(./test.sh),test.log,)
#echo "Finished making target 'my_rpm'. Time: $$(date +%T)"
And here's the exec_recipe.mk
SHELL:=/bin/bash
exec_recipe = \
echo "The logs dir is: $$BUILD_LOGS_DIR"; \
log_name="$(2)"; \
echo "The log name is $$log_name"; \
cmd="$(1)"; \
eval "$$cmd" 2>&1 | tee -a "$$BUILD_LOGS_DIR/$$log_name"; rc="$${PIPESTATUS[0]}"; \
if [ $$rc = 0 ]; then \
res="PASS"; \
else \
res="FAIL"; \
fi; \
flock $(SUMMARY) echo "Making target '$#': $$res" >> $(SUMMARY); \
exit $$rc
So the problem is when it's reaching the execution of the exec_recipe.mk it giving me some errors:
/bin/bash: eval: line 0: syntax error near unexpected token||'
/bin/bash: eval: line 0: || '
I'm sure that i'm doing something wrong..
I need the helper make file, because i'm using it for other purposes as well
You have to escape all the dollar signs that you don't want make to interpret. You've forgotten to escape the dollar signs in the arguments to $(call exec_recipe,...):
cd /home/user/workspace/test/build/test && $(call exec_recipe,$$(ls /home/user/workspace/test/build/test/rpmbuild/RPMS/x86_64d/*.rpm) || $$(./test.sh),test.log,)
I'm not sure what the final comma is for but whatever.

Chained && and || operators in bash scripting

I want to execute a series of commands in a bash script.
If any fail, the script should echo a line describing which line failed, and then exit the function.
Currently, my script echos the output adequately, but doesn't exit the script (because the echo command returns 0).
I have
COMMAND_A || echo "command A failed" | tee -a $SUMMARY_FILE
COMMAND_B || echo "command B failed" | tee -a $SUMMARY_FILE
COMMAND_C || echo "command C failed" | tee -a $SUMMARY_FILE
but these have no dependence on the previous line - I think enclosing with bash -e would also not help, since then either COMMAND_A failing would exit without the echo, or else the echo would succeed and nothing would exit.
Since the echo will succeed, I can't just add && to each line.
I could perhaps use
{ COMMAND_A && \
{ COMMAND_B && \
{ COMMAND_C || echo "C failed" | tee -a $SUM } \
|| echo "B failed" | tee -a $SUM } \
|| echo "A failed" | tee -a $SUM }
but this seems very clunky.
Is there a better syntax/style/method?
For a concrete example:
cmake -DCMAKE_INSTALL_PREFIX=$PREFIX -DCMAKE_BUILD_TYPE=Release .. || \
echo "$(date +%d/%m/%y-%H:%M:%S): cmake failed for $1" | tee -a $SUMFILE
make -j16 || \
echo "$(date +%d/%m/%y-%H:%M:%S): make failed for $1" | tee -a $SUMFILE
sudo make install && \
echo "$(date +%d/%m/%y-%H:%M:%S): Installed $1" | tee -a $SUMFILE || \
echo "$(date +%d/%m/%y-%H:%M:%S): make install failed for $1" | tee -a $SUMFILE
Edit
I can perhaps get some improvement with
COMMAND_A || { echo "A failed" && false; } && \
COMMAND_B || { echo "B failed" && false; } && \
COMMAND_C && echo "C succeeded" || { echo "C failed" && false; }
but this will still print that A, B and C failed if A fails (even though the commands are short-circuited, hence the improvement)
Since the series of commands is executed within a function, the return statement can be used.
If this were not the case, exit may be the appropriate choice.
Consequently, I can use the idea in the question's edit to write:
standardInstall() {
# Should be passed the directory (ie application/library) name as
# an argument.
if [ -z "$1" ]; then
echo "No argument was passed to standardInstall()"
echo "Exiting"
exit 2
fi
pushd $1/build/
cmake -DCMAKE_INSTALL_PREFIX=$KF5 -DCMAKE_BUILD_TYPE=Release .. || \
{ echo "$(date +%d/%m/%y-%H:%M:%S): cmake failed for $1" \
| tee -a $SUMFILE && return 1; } && \
make -j16 || \
{ echo "$(date +%d/%m/%y-%H:%M:%S): make failed for $1" \
| tee -a $SUMFILE && return 1; } && \
sudo make install && \
echo "$(date +%d/%m/%y-%H:%M:%S): Installed $1" | tee -a $SUMFILE || \
{ echo "$(date +%d/%m/%y-%H:%M:%S): make install failed for $1" \
| tee -a $SUMFILE && return 1; }
popd
}

How to stop xargs on first error?

I have an pages.txt file with 100 URLs inside. I want to check them one by one and fail on the first problem. This is what I'm doing:
cat pages.txt | xargs -n 1 curl --silent \
--output /dev/null --write-out '%{url_effective}: %{http_code}\n'; echo $?
Exit code is 1, but I see it only when the entire file is done. How to stop earlier, on the first problem?
General method
xargs -n 1 sh -c '<your_command> $0 || exit 255' < input
Specific case
xargs -n 1 sh -c 'curl --silent --output /dev/null \
--write-out "%{url_effective}: %{http_code}\n" $0 || exit 255' < pages.txt
Explanation
For every URL in pages.txt, executes sh -c 'curl ... $0 || exit 255' one by one (-n 1) forcing to exit with 255 if the command fails.
From man xargs:
If any invocation of the command exits with a status of 255, xargs will stop immediately without reading any further input. An error message is issued on stderr when this happens.
I haven't found a way to do what you ask for with xargs, but a loop with read might be what you are looking for.
while read URL; do
curl --silent \
--output /dev/null --write-out '%{url_effective}: %{http_code}\n' $URL;
RET=$?;
echo $RET;
if [ $RET -ne 0 ]; then break; fi
done < pages.txt

Resources