I can multi-thread a make with make -jN
Can I dictate multi-threading within the Makefile so that just make from the command-line runs multiple threads. Here's my makefile:
BIN_OBJS = $(wildcard *.bin)
HEX_OBJS = $(subst .bin,.hex,$(BIN_OBJS))
all: $(HEX_OBJS)
$(HEX_OBJS): %.hex: %.bin
python ../../tools/bin2h.py $< > $#
First, to be clear, make is not multi-threaded. Using -j just tells make to run multiple commands at the same time (in the background, basically).
Second, no, it's not possible to enable multiple jobs from within the makefile. You don't want to do that, in general, anyway because other systems will have different numbers of cores and whatever value you choose won't work well on those systems.
You don't have to write multiple makefiles, though, you can just use:
BIN_OBJS = $(wildcard *.bin)
HEX_OBJS = $(subst .bin,.hex,$(BIN_OBJS))
.PHONY: all multi
multi:
$(MAKE) -j8 all
all: $(HEX_OBJS)
$(HEX_OBJS): %.hex: %.bin
python ../../tools/bin2h.py $< > $#
Be careful using -j if the filesystem where the make is occurring is an nfs share. I have seen odd results and had it mentioned to me that nfs mounted directories operate differently (some sort of file lock issue?)
I ran my multi makes from a script and checked cpuinfo to find out how many processors the build box had (was running same script against multiple architectures/build machines)
CPUCOUNT=$(grep -c "^processor" /proc/cpuinfo)
if [ ${CPUCOUNT} -lt 1 -o ${CPUCOUNT} -gt 4 ]
then
echo "Unexpected value for number of cpus, ${CPUCOUNT}, exiting ..."
exit 16
fi
echo "This machine has ${CPUCOUNT} cpus, will use make -j${CPUCOUNT} where possible"
Related
I'm often creating png files out of dot (graphviz format) files. The command to do so is the following:
$ dot my_graph.dot -o my_graph.png -Tpng
However, I would like to be able to have a shorter command format like $ make my_graph.dot to automatically generate my png file.
For the moment, I'm using a Makefile in which I've defined the following rule, but the recipe is only available in the directory containing the Makefile
%.eps: %.dot
dot $< -o $# -Teps
Is it possible to define custom implicit GNU Make recipes ? Which would allow the above recipe to be available system-wide
If not, what solution do you use to solve those kind of problem ?
Setup:
Fedora Linux with ZSH/Bash
You could define shell functions in your shell's startup files, e.g.
dotpng()
{
echo dot ${1%.dot}.dot -o ${1%.dot}.png -Tpng;
}
This function can be called like
dotpng my_graph.dot
or
dotpng my_graph
The code ${1%.dot}.dot strips .dot from the file name if present and appends it (again) to allow both my_graph.dot and my_graph as function argument.
Is it possible to define custom implicit GNU Make recipes ?
Not without modifying the source code of GNU Make.
If not, what solution do you use to solve those kind of problem ?
I wouldn't be a fan o modyfying the system globally, but you could do:
Create a file /usr/local/lib/make/myimplicitrules.make with the content
%.eps: %.dot
dot $< -o $# -Teps
Use include /usr/local/lib/make/myimplicitrules.make in your Makefile.
I would rather use a git submodule or similar to share common configuration between projects, rather than depending on global configuration. Depending on global environment will make your program hard to test and non-portable.
I would rather go with a shell function, something along:
mymake() {
make -f <(cat <<'EOF'
%.eps: %.dot
dot $< -o $# -Teps
EOF
) "$#"
}
mymake my_graph.dot
GNU Make lets you specify extra makefiles to read using the MAKEFILES
environment variable. Quoting from info '(make)MAKEFILES Variable':
the default goal is never taken from one of these makefiles (or any
makefile included by them) and it is not an error if the files listed
in 'MAKEFILES' are not found
if you are running 'make' without a specific makefile, a makefile
in 'MAKEFILES' can do useful things to help the built-in implicit
rules work better
As an example, with no makefile in the current directory and the
following .mk files in make's include path (e.g. via
MAKEFLAGS=--include-dir="$HOME"/.local/lib/make/) you can create
subdir gen/ and convert my_graph.dot or dot/my_graph.dot by
running:
MAKEFILES=dot.mk make gen/my_graph.png
To further save some typing it's tempting to add MAKEFILES=dot.mk
to a session environment but defining MAKEFILES in startup files
can make things completely nontransparent. For that reason I prefer
seeing MAKEFILES=… on the command line.
File: dot.mk
include common.mk
genDir ?= gen/
dotDir ?= dot/
dotFlags ?= $(if $(DEBUG),-v)
Tvariant ?= :cairo:cairo
vpath %.dot $(dotDir)
$(genDir)%.png $(genDir)%.svg $(genDir)%.eps : %.dot | $(genDir).
dot $(dotFlags) $< -o $# -T'$(patsubst .%,%,$(suffix $#))$(Tvariant)'
The included common.mk is where you'd store general definitions to
manage directory creation, diagnostics etc., e.g.
.PRECIOUS: %/. ## preempt 'unlink: ...: Is a directory'
%/. : ; $(if $(wildcard $#),,mkdir -p -- $(#D))
References:
?= = := … - info '(make)Reading Makefiles'
vpath - info '(make)Selective Search'
order-only prerequisites (e.g. | $(genDir).) - info '(make)Prerequisite Types'
.PRECIOUS - info '(make)Chained Rules'
SO
I can't find out why these lines are not called for the first time I run 'make' but are called the next time:
sb_path = sb
sb_src := $(sb_path)/src
sb_build := $(sb_path)/build
ifndef DO_NOT_GENERATE_COMMIT_INFO
commit_sb: | $(sb_bin)
#$(sb_build)/generate-commit-info $(sb_path)
$(sb_src)/last_git_commit_info.h: | commit_sb ;
endif
I'm just curious because there is no file generate-commit-info file and make crashes when I call it for the second time, but it compiles my program ok for the first try.
I use script on my local machine to copy sources over ssh to another machine and to run compile.sh script there:
...
scp -r $sbfolder/build $sbfolder/Makefile "$buildserver:$root/$curdate"
check_retcode
scp -r $sbfolder/sb/Makefile "$buildserver:$root/$curdate/sb/"
...
ssh $buildserver "$root/compile.sh $curdate $debug"
compile.sh:
# fix Makefile: we don't have git installed here
#DO_NOT_GENERATE_COMMIT_INFO=true
#now we can compile sb
curdir="/home/tmp/kamyshev/sb_new/$1"
cd $curdir
check_retcode
t_path=$curdir
debug=$2
config=RELEASE
if [[ debug -eq 1 ]]; then
config=DEBUG
fi
echo "building sb... CONFIG=$config"
make -j2 CONFIG=$config
check_retcode
As you see DO_NOT_GENERATE_COMMIT_INFO=true is commented out. So I just don't see a reason why the code is not run when I call a make or the script for the first time (either from the remote script or myselft from command line).
Do you have any clues?
UPDATE on Etan Reisner comment:
commit_sb target is checked, it does not exist, so it's rule is being run and it updates last_git_commit_info.h. Thus it forces to update the .h file. It also gives me a .PHONY target commit_sb so I could do it directly by calling make commit_sb.
The generate-commit-info also creates a file in a $(sb_bin) folder.
My another guess is that you are talking about a better way to organize this code.
I can update last_git_commit_info.h directly with a such rule:
commit_sb $(sb_src)/last_git_commit_info.h: FORCE | $(sb_bin)
#$(sb_build)/generate-commit-info $(sb_path)
FORCE:
Thanks to the commenters on my question I've done some additional research: I've tried to make a minimal complete example. And this led me to the answer.
My code generates dependency files (look at -MMD command in SB_CXXFLAGS):
# just example - in real Makefile these are calculated on the fly
sb_deps := file1.d file2.d [...]
# rules with dependances of .o files against .h files
-include $(sb_deps)
SB_CXXFLAGS = $(CXXFLAGS) [...] -MMD
# compile and generate dependency info;
$(sb_obj)/%.o:$(sb_src)/%.cpp
$(CXX) $(SB_CXXFLAGS) $< -o $#
And when I run make for the first time there no *.d files, so no *.cpp depends on last_git_commit_info.h file and the rule is not applied.
On the subsequent runs the dependency rule appears in one of *.d files, the rule is executed and I get the error.
UPDATE: This does not concern the question directly, but this is the better way to write these rules:
ifndef DO_NOT_GENERATE_COMMIT_INFO
commit_sb $(sb_src)/last_git_commit_info.h: FORCE | $(sb_bin)
#$(sb_build)/generate-commit-info $(sb_path)
FORCE:
endif
I have a very simple Makefile that isn't doing what I expect it would do. The ultimate goal is that it should call itself recursively, including the appropriate file each time, resulting in a build specific to what was included (I'm building several projects that all share the same code base, but utilize different combinations of the source files). I've never really dealt with recursive calls to make, so I must be missing something obvious. At the moment, I only have one .mk file in the same folder as my Makefile. It's a simple one-liner just for the purposes of this test. It will eventually contain various per-project settings.
Makefile:
SHELL = /bin/sh
ifdef MYFILE
include $(MYFILE)
PROGRAM = $(basename $(MYFILE))
endif
all: $(wildcard *.mk)
dummy:
#echo -- Entering dummy stub ... why do I need this?
%.mk: dummy
#echo Calling $(MAKE) MYFILE=$# $*
$(MAKE) MYFILE=$# $*
$(PROGRAM): objs
#echo Time to link!
objs:
#echo Building objs!
test.mk
SOMEVAR = SomeValue
I have the following two problems:
Problem 1
If I remove the dummy prerequisite from my pattern rule, the pattern rule never gets called (I get the dreaded 'Nothing to be done for all' error). Is there a way I can get the recipes under the %.mk rule to run without needing that dummy prerequisite?
Problem 2
Given the two aforementioned files, I would expect make to do the following:
make[1] starts and hit the all rule
make[1] jumps down to the %.mk pattern rule
make[1] calls itself recursively (the call would look like make MYFILE=test.mk test)
make[2] starts, includes the test.mk file, and sets up the PROGRAM variable
make[2] jumps down to the $(PROGRAM) rule (since we were explicitly called with that target)
make[2] jumps to the objs rule, runs the recipes, and returns back up the chain
In actuality, make gets stuck on the %.mk pattern rule and enters an infinite loop. I don't understand why it's insisting on hitting the pattern rule, when I explicitly told it to build test in my first recursive call (which should correspond to the $(PROGRAM) target). What am I missing here?
Problem 0:
This is overdesigned. You don't need to use recursive Make here.
Problem 1:
The reason Make doesn't try to rebuild test.mk (without a dummy preq) is that test.mk is up to date. A better approach is to switch to a static pattern rule and use PHONY:
MKS = $(wildcard *.mk)
.PHONY: $(MKS)
$(MKS): %.mk:
#echo Calling $(MAKE) MYFILE=$# $*
$(MAKE) MYFILE=$# $*
An even better approach is not to use the name of a real file as a target of a rule that doesn't rebuild (or even "touch") that file.
Problem 2:
In make[2], the makefile includes test.mk. If a makefile includes another file, Make will attempt to rebuild that file before doing anything else. If there is a rule for that file (which there is) and if it succeeds (which it does) Make then reinvokes itself.
You should reconsider this design from the ground up. There are many ways to get the behavior you're looking for, depending on the specifics (how many variable will be defined in a foo.mk? do you really want to manage the build by manually moving those files around? and so on).
P.S. Here's one kludge that springs to mind. Whether it suits your case depends on the specifics:
makefile:
# includes nothing
%.mk: dummy
#echo Calling $(MAKE) MYFILE=$# -f $# $*
$(MAKE) MYFILE=$# -f $# $*
test.mk:
SOMEVAR = SomeValue
include makefile
what does "make -j$TOTAL_PROCESSORS" means?
Say if I have a two core processor, It will execute "make -j2". What exactly it does?
I am adding a small example below
For compiling my toolchain the script file uses -
pushd toolchaindir
export TARGET=powerpc-linux-gnu
export LINUX_ARCH=powerpc
TOTAL_PROCESSORS=$(grep processor /proc/cpuinfo | wc -l)
make -j$TOTAL_PROCESSORS
if [ "$?" = "0" ]; then
echo "built toolchain successfully"
else
echo "failed during build"
exit 1
fi
popd
exit 0
How it builds toolchain?
make -j2 tells make that it can run two shell commands at once. Make determines whether it can do this from your makefile, so you had better write your makefiles correctly!
Consider this noddy makefile:
1.o: 1.c
gcc -c 1.c -o 1.o
2.o: 2.c
gcc -c 2.c -o 2.o
prog: 1.o 2.o
gcc 1.o 2.o -o prog
If you say make -j2 prog, then make cleverly decides that the production of 1.o is entirely independent of 2.o. Thus it can run the two compiles at the same time without error. So it does. Make waits for both these compiles to finish before combining both object files into prog in the final link step.
Unspeakably clever, so long as you get your makefiles right (if they don't work under -jn then they are bad bad bad!).
In one word: yes
It authorizes make to start $TOTAL_PROCESSORS compilations in parallel.
It expands the environment variable TOTAL_PROCESSORS, presumably to a number which indicates how many CPUs/cores you have, and then runs make with this amount of parallel jobs.
You'll need to look at what sets TOTAL_PROCESSORS to a value.
It reads whatever you shell variable $TOTAL_PROCESSORS is and runs that many jobs. I'm guessing that variable is set to the number of processors or cores on your machine. You can echo it's value in a shell just to be sure.
I have a lot of source directories in common directory. When I start make by issuing command:
make -j 4
I receive a lot of strings from make's threads along with invoked gcc compiler instances.
For parsing errors I have to run make twice, second time with one thread:
make -j 1
so I can correctly parse make's output.
Is there a way for running multithreaded make one time and correctly decide which error related to which project (source directory)?
Thank you!
If you are using recursive make (naughty boy) together with -j, then you can wrap Make with a shell script which prefixes each line of output with a unique per-make-invocation string.
$ cat M
#!/bin/bash
PREFIX=$$:
exec -a $0 make "$#" 2>&1 | sed "s/^/$PREFIX/"
Now, assuming your makefiles correctly use ${MAKE} to indicate recursion, we can use ./M instead of make.
$ ./M -j --no-print-directory target
28720:/home/user/M -fa.mak
28720:/home/user/M -fb.mak
28720:/home/user/M -fc.mak
28720:/home/user/M -fd.mak
28720:/home/user/M -fe.mak
28720:32484:gcc blah...
28720:31936:/home/user/M -fanother.mak
28720:32484:gcc blah...
28720:31936:gcc blah...
28720:31936:gcc blah...
28720:31936:56781:echo blah...
∶
In this case, each line is prefixed with a list of process IDs (good for debugging recursive make). For your use case, you may prefer M to mangle source file names so that they become absolute pathnames in error messages.
May be it is possible to switch to pmake?
PMake is set up to handle the output from multiple jobs in a graceful fashion (source)
If you're using some kind of meta-build system (eg CMake) try using Ninja to actually run the build.
It solves this problem as well as being quite a lot faster.