I have a lot of source directories in common directory. When I start make by issuing command:
make -j 4
I receive a lot of strings from make's threads along with invoked gcc compiler instances.
For parsing errors I have to run make twice, second time with one thread:
make -j 1
so I can correctly parse make's output.
Is there a way for running multithreaded make one time and correctly decide which error related to which project (source directory)?
Thank you!
If you are using recursive make (naughty boy) together with -j, then you can wrap Make with a shell script which prefixes each line of output with a unique per-make-invocation string.
$ cat M
#!/bin/bash
PREFIX=$$:
exec -a $0 make "$#" 2>&1 | sed "s/^/$PREFIX/"
Now, assuming your makefiles correctly use ${MAKE} to indicate recursion, we can use ./M instead of make.
$ ./M -j --no-print-directory target
28720:/home/user/M -fa.mak
28720:/home/user/M -fb.mak
28720:/home/user/M -fc.mak
28720:/home/user/M -fd.mak
28720:/home/user/M -fe.mak
28720:32484:gcc blah...
28720:31936:/home/user/M -fanother.mak
28720:32484:gcc blah...
28720:31936:gcc blah...
28720:31936:gcc blah...
28720:31936:56781:echo blah...
∶
In this case, each line is prefixed with a list of process IDs (good for debugging recursive make). For your use case, you may prefer M to mangle source file names so that they become absolute pathnames in error messages.
May be it is possible to switch to pmake?
PMake is set up to handle the output from multiple jobs in a graceful fashion (source)
If you're using some kind of meta-build system (eg CMake) try using Ninja to actually run the build.
It solves this problem as well as being quite a lot faster.
Related
How do I get the first n lines of the output of a makefile (specifically, my complier is g++). Either a script in linux or in the makefile would work (if you could provide both, that'll be even better).
I have tried
make | head -n 5
but it's not working.
Currently, the process I go through is tedious; I'm piping the output to a text file before using head on it (then having to delete the file).
Given that the messages from the compiler appear on standard error rather than standard output, you need to redirect both:
make 2>&1 | head -n 20
(I think 5 lines will be too small to be useful.)
I use scons (V1.1.0) for a project that contains a build step that involves the flex tool.
The definition for the flex command in the scons default rules is:
env["LEX"] = env.Detect("flex") or "lex"
env["LEXFLAGS"] = SCons.Util.CLVar("")
env["LEXCOM"] = "$LEX $LEXFLAGS -t $SOURCES > $TARGET"
which I don't want to change.
However, since -t causes #line directives to be created in the output file that refer to the file "<stdout>", this confuses the subsequent gcov processing.
As a solution, I found that -o can be used to override the file name flex produces into the #line directives (it still produces its output on stdout due to the -t option which apparently has precedence).
To achieve that, I added this in the project's SConscript file:
env.AppendUnique(LEXFLAGS = ['-o $TARGET','-c'],delete_existing=1)
I added the -c option (which does nothing) only to show the difference between how it is treated compared to -o.
An according debug print in the SConscript file results in the following (as expected):
repr(env["LEXFLAGS"]) = ['-o $TARGET', '-c']
This results in the following command line, according to the scons log:
flex "-o build/myfile.cpp" -c -t src/myfile.ll > build/myfile.cpp
So the -c option gets into the command line as desired, but the -o option and its filename parameter has double quotes around it, that must have been created by scons when expanding the LEXFLAGS variable.
When I use this definition for LEXFLAGS instead:
env.AppendUnique(LEXFLAGS = ['--outfile=$TARGET','-c'],delete_existing=1)
the resulting command line works as desired:
flex --outfile=build/myfile.cpp -c -t src/myfile.ll > build/myfile.cpp
So one could speculate that the blank in the -o case caused the double quotes to be used, maybe in an attempt to bind the content together into one logical parameter for the command.
So while my immediate problem is solved by using --outfile, my question is still is it possible to rid of the double quotes in the -o case?
Thanks,
Andy
SCons 1.1.0 is extremely old at this point. I'd recommend trying 2.3.0. But your analysis is correct; if an option (a single option, that is) has a space in it, SCons will quote it so it stays a single option. But you don't have a single option; you really have two, '-o' and '$TARGET'. Just break it up like that and it'll work.
I'm writing a program where the command line usage should be something like:
mkblueprint FILE FILE FILE -o <output name> -s <string> -r <number> -p pOPT1 pOPT2 pOPT3
I'm currently using CmdLib and I can't figure out a way to handle this; a flag is required for each input(so I can't just have FILEs sitting alone) and there doesn't appear to be a way to pass multiple arguments to a flag, as with -p. These are extremely common in command line programs so I figure I'm just misunderstanding the documentation, but it's not mentioned in any command line library I look at for Haskell.
After some more work with CmdLib I was able to handle the bare FILE input via the Extra tag and then checking that each string is a valid file, which seems to be the standard way to handle it despite the name. -p pOPT1 pOPT2 pOPT3 is apparently not allowed under the POSIX standard, which is why I'm not finding libraries that will do it.
You might consider the GetOpt bindings that come with base. They're not as sexy as some of the more modern alternatives, but they support bare arguments and final options well.
I have a very simple Makefile that isn't doing what I expect it would do. The ultimate goal is that it should call itself recursively, including the appropriate file each time, resulting in a build specific to what was included (I'm building several projects that all share the same code base, but utilize different combinations of the source files). I've never really dealt with recursive calls to make, so I must be missing something obvious. At the moment, I only have one .mk file in the same folder as my Makefile. It's a simple one-liner just for the purposes of this test. It will eventually contain various per-project settings.
Makefile:
SHELL = /bin/sh
ifdef MYFILE
include $(MYFILE)
PROGRAM = $(basename $(MYFILE))
endif
all: $(wildcard *.mk)
dummy:
#echo -- Entering dummy stub ... why do I need this?
%.mk: dummy
#echo Calling $(MAKE) MYFILE=$# $*
$(MAKE) MYFILE=$# $*
$(PROGRAM): objs
#echo Time to link!
objs:
#echo Building objs!
test.mk
SOMEVAR = SomeValue
I have the following two problems:
Problem 1
If I remove the dummy prerequisite from my pattern rule, the pattern rule never gets called (I get the dreaded 'Nothing to be done for all' error). Is there a way I can get the recipes under the %.mk rule to run without needing that dummy prerequisite?
Problem 2
Given the two aforementioned files, I would expect make to do the following:
make[1] starts and hit the all rule
make[1] jumps down to the %.mk pattern rule
make[1] calls itself recursively (the call would look like make MYFILE=test.mk test)
make[2] starts, includes the test.mk file, and sets up the PROGRAM variable
make[2] jumps down to the $(PROGRAM) rule (since we were explicitly called with that target)
make[2] jumps to the objs rule, runs the recipes, and returns back up the chain
In actuality, make gets stuck on the %.mk pattern rule and enters an infinite loop. I don't understand why it's insisting on hitting the pattern rule, when I explicitly told it to build test in my first recursive call (which should correspond to the $(PROGRAM) target). What am I missing here?
Problem 0:
This is overdesigned. You don't need to use recursive Make here.
Problem 1:
The reason Make doesn't try to rebuild test.mk (without a dummy preq) is that test.mk is up to date. A better approach is to switch to a static pattern rule and use PHONY:
MKS = $(wildcard *.mk)
.PHONY: $(MKS)
$(MKS): %.mk:
#echo Calling $(MAKE) MYFILE=$# $*
$(MAKE) MYFILE=$# $*
An even better approach is not to use the name of a real file as a target of a rule that doesn't rebuild (or even "touch") that file.
Problem 2:
In make[2], the makefile includes test.mk. If a makefile includes another file, Make will attempt to rebuild that file before doing anything else. If there is a rule for that file (which there is) and if it succeeds (which it does) Make then reinvokes itself.
You should reconsider this design from the ground up. There are many ways to get the behavior you're looking for, depending on the specifics (how many variable will be defined in a foo.mk? do you really want to manage the build by manually moving those files around? and so on).
P.S. Here's one kludge that springs to mind. Whether it suits your case depends on the specifics:
makefile:
# includes nothing
%.mk: dummy
#echo Calling $(MAKE) MYFILE=$# -f $# $*
$(MAKE) MYFILE=$# -f $# $*
test.mk:
SOMEVAR = SomeValue
include makefile
I need to find out which library will be loaded given in the information returned from /sbin/ldconfig. I came up with the following:
#!/bin/bash
echo $(dirname $(/sbin/ldconfig -p | awk "/$1/ {print \$4}" | head -n 1))
Running this results with:
$ whichlib libGL.so
/usr/X11R6/lib
This a two part question:
Will this produce a reliable result across platform?
Is there a slicker way to parse the output of ldconfig?
Thanks,
Paul
There're several ways the library is loaded by executeable:
1.
Using $LD_LIBRARY_PATH
Using ld cache
Libary with full path compiled into binary (-rpath gcc flag)
You're using option 2, while option 1 and 3 are not considered.
Depending on what exactly you're doing you may want to run ldd directly on the executable you're planning to run rather than the general case ldconfig.
Since you asked, you could write your script like this:
dirname "$(/sbin/ldconfig -p | awk "\$1 == "$1" {print \$4; exit}")"
It's a little more precise and has one less pipe. Also echo $(cmd) is redundant; you can just write cmd.