I have a reasonably large project (4272 .o files) and I can't get it to link with GNU Make. I run into make: /bin/sh: Argument list too long. This is a Qt 5 project that uses qmake to generate the makefile.
I know there are lots of questions about this, but I don't know how to apply any of the solutions to my problem. I'm also not totally sure why I'm running into this at the linking step. The error I get is:
make: /bin/sh: Argument list too long
The makefile entry for linking my project looks like this:
build/debug/my_target/my_target: $(OBJECTS)
#test -d build/debug/my_target/ || mkdir -p build/debug/my_target/
$(LINK) $(LFLAGS) -o $(TARGET) $(OBJECTS) $(OBJCOMP) $(LIBS)
which expands to something like:
#echo linking /build/debug/my_target/my_target && clang++ -ccc-gcc-name g++ -lc++ -L/path/to/licensing/lib -Wl,-rpath,/path/to/qt/lib -Wl,-rpath-link,/path/to/qt/lib -o build/debug/my_target/my_target build/debug/my_target/obj/object1.o build/debug/my_target/obj/object2.o ... build/debug/my_target/obj/object4272.o ... [ a bunch of moc_X.o ] ... [ a bunch of libs ] -lGL -lpthread -no-pie
This is pretty long. But here's where it gets weird: when I put the expanded command after the #echo linking build/debug/my_target/my_target && into a shell script, and it runs. The shell script is 202,420 characters (including the #!/bin/sh line). Also, if I get rid of the #echo ... && part of the command I can run make and linking works.
Another workaround: if I manually edit my makefile so that the linking command contains build/debug/my_target/*.o instead of $(OBJECTS) it works:
build/debug/my_target/my_target: $(OBJECTS)
#test -d build/debug/my_target/ || mkdir -p build/debug/my_target/
$(LINK) $(LFLAGS) -o $(TARGET) build/debug/my_target/*.o $(OBJCOMP) $(LIBS)
I don't think I can get qmake to do this, though, so I'm stuck manually editing my makefile unless I can find another solution.
Answers to similar problems seem to focus on line breaks and how they're handled in makefiles. My shell script only has two lines (one after #!/bin/sh and one after the actual command). Also, one solution that people have come up with (for example this one) uses a for loop to iteratively run a command on each argument. I'm not sure how I could apply this here, since (I think) I need all those object files in my linker command.
How does #echo cause the max argument length to be exceeded?
Questions I originally asked that aren't really relevant:
(Note: as originally posted this question missed the #echo at the beginning of the linking command. That seems to be the answer to "why is this happening" and as such I don't really need to know the answer to the second question, which is answered in the first comment in any case).
Why is this happening? How is it that make is running into this error with a command that I can apparently run in a shell script?
How can I get around this if there's no way to run my command as an iterative series of shorter commands?
Various details about my system that might be relevant:
I'm running a fairly up-to-date Arch Linux system, kernel 5.8.10
ARG_MAX value is 2097152, the output from xargs --show-limits is:
Your environment variables take up 2343 bytes
POSIX upper limit on argument length (this system): 2092761
POSIX smallest allowable upper limit on argument length (all systems): 4096
Maximum length of command we could actually use: 2090418
Size of command buffer we are actually using: 131072
Maximum parallelism (--max-procs must be no greater): 2147483647
ulimit -s output: 8192 (I've tried setting this to much larger values, e.g. ulimit -s 65536 without success, which maybe isn't surprising since ARG_MAX appears to be much larger than the linker command).
GNU Make version is 4.3
clang/clang++ version is 10.0.1
Qt version is 5.15.1 (I'm fairly certain this isn't relevant, we've just switched our project over from 5.9.6 and I had the same problem then as well).
Just FYI, the reason removing the echo fixes the problem (this is what I was going to suggest as well) is that when you remove the special shell operator && and just have a simple command invocation with no shell features like multiple commands, special quoting, globbing, etc. then make uses the "fast path" to invoke your command.
That is, if make can determine that the shell would do nothing special with your command, other than run it, make will skip invoking the shell and instead run your command directly.
In that case you will not run up against the single-argument limit because it doesn't use the /bin/sh -c '...' form.
Of course, this can be a little magical and inflexible since you have to be careful to ensure no special shell operations are ever included in your link line. But if you can ensure this then it should solve your problem.
Why is this happening? How is it that make is running into this error with a command that I can apparently run in a shell script?
Because make is running the shell commands (recipes) by passing them as a single argument to /bin/sh -c, and not only will that run into the OS's limit on command line arguments + environment variables, but also into the much lower limit that Linux is imposing on a single string from the command line or environment, which is usually 128k bytes.
How can I get around this if there's no way to run my command as an iterative series of shorter commands?
As suggested by #ephemient can use the #arglist argument of gcc or ld (which directs it to take its arguments from a file), and use the file function of GNU make to create that arglist file, which being internal to make, will not run into any that OS limit.
Related
For example, in Linux, gonna compile 3 files "test1.c", "test2.c" and "test3.c" separately to "test1", "test2" and "test3", is there a way using wildcard or pipe to do it by one line bash instruction?
There's not really such a thing as a "bash instruction", and even if you say "bash command", then gcc isn't really such a thing either because it is not intrinsically related to, or a part of bash. It is a separate binary/program that can be run under any shell, such as sh, ksh, ash, dash or even tcsh.
Assuming you mean one line, you can do:
gcc prog1.c -o prog1 & gcc prog2.c -o prog2 & gcc -o prog3 prog3.c
You can equally do that in a loop as William was suggesting:
for f in test?.c; do gcc "$f" -o "${f%.c}" & done
IMHO though, the easiest is with GNU Parallel:
parallel gcc {} -o {.} ::: test?.c
There are those who will rightly say that GNU Parallel is not part of POSIX, but, to be fair, you didn't mention POSIX in your question. So, if POSIX-compliance is an issue, you may choose to avoid this suggestion.
As suggested in the comments, a Makefile would be even better, So if you create a file called Makefile with the following contents:
all: test1 test2 test3
You can simply run:
make
and all three programs will be compiled, only if necessary, i.e. if you have changed the source file since last compiling. Or, if you want to use all those cores that you paid Intel so handsomely for, you can do them in parallel with:
make -j
I had written a simple shell script (called test.sh) to compile a test C++ file using two different compilers (g++ and clang++) and put some echo statements in to compare the output. On the command line, I accidentally typed make test, even though there was no Makefile in that directory. Instead of complaining about no Makefile or no target defined, it executed the following commands (my system is running the 64-bit Debian stretch OS with GNU Make 4.1 ):
user#hostname test_dir$ make test
cat test.sh >test
chmod a+x test
user#hostname test_dir$
Curious about that, I made another shell script (other.sh) and did the same thing.
Here is my other.sh file:
#!/bin/bash
echo ""
echo "This is another test script!"
echo ""
Command line:
user#hostname test_dir$ make other
cat other.sh >other
chmod a+x other
user#hostname test_dir$
My question is why does make automatically create an executable script (without the .sh extension) when running the make command in the terminal? Is this normal/expected/standard behavior? Can I rely on this behavior on all Linux machines?
Side question/note: Is there a list of supported "implicit suffixes" for which make will automatically create an executable?
This is one of a number of "implicit rules" which are built into Gnu make. (Every make implementation will have some implicit rules, but there is no guarantee that they are the same.)
Why does make automatically create an executable script without the .sh extension?
There is an old source repository system called Source Code Control System (SCCS). Although it no longer has much use, it was once the most common way of maintaining source code repositories. It had the quirk that it did not preserve file permissions, so if you kept an (executable) shell script in SCCS and later checked it out, it would no longer be executable. Gnu make could automatically extract files from an SCCS repository; to compensate for the disappearing executable permission issue, it was common to use the .sh extension with shell scripts; make could then do a two-step extraction, where it first extracted foo.sh from the repository and then copied it to foo, adding the executable permission.
Is this normal/expected/standard behavior? Can I rely on this behavior on all Linux machines?
Linux systems with a development toolset installed tend to use Gnu make, so you should be able to count on this behaviour on Linux systems used for development.
BSD make also comes with a default rule for .sh, but it only copies the file; it doesn't change permissions (at least on the bsdmake distribution on my machine). So the behaviour is not universal.
Is there a list of supported "implicit suffixes" for which make will automatically create an executable?
Yes, there is. You'll find it in the make manual:
The default suffix list is: .out, .a, .ln, .o, .c, .cc, .C, .cpp, .p, .f, .F, .m, .r, .y, .l, .ym, .lm, .s,
.S, .mod, .sym, .def, .h, .info, .dvi, .tex, .texinfo, .texi, .txinfo, .w, .ch,
.web, .sh, .elc, .el.
For a more accurate list of implicit rules, you can use the command
make -p -f/dev/null
# or, if you like typing, make --print-data-base -f /dev/null
as described in the make options summary.
From the make man page:
The purpose of the make utility is to determine automatically
which
pieces of a large program need to be recompiled, and issue the commands
to recompile them. The manual describes the GNU implementation of
make, which was written by Richard Stallman and Roland McGrath, and is
currently maintained by Paul Smith. Our examples show C programs,
since they are most common, but you can use make with any programming
language whose compiler can be run with a shell command. In fact, make
is not limited to programs. You can use it to describe any task where
some files must be updated automatically from others whenever the others change.
make really is more than most people make it out to be...
This is a reduced example of a makefile which illustrates my problem:
exec:
time (ls > ls.txt; echo $$? > code) 2> time.txt
make exec runs fine under one Linux installation:
Linux-2.6.32-642.4.2.el6.x86_64-x86_64-with-centos-6.8-Final
but it fails under my Ubuntu installation:
Linux-4.4.0-64-generic-x86_64-with-Ubuntu-16.04-xenial
and produces the message:
/bin/sh: 1: Syntax error: word unexpected (expecting ")")
No problems if I run the command time directly from the terminal.
Are there different versions of the command in different Linux installations? I need the version which allows a sequence of commands.
Make always invokes /bin/sh to run the recipe. On some systems, /bin/sh is an alias for bash which has a lot of extra extensions to the standard POSIX shell (sh). On other systems (like Ubuntu), /bin/sh is an alias for dash which is a smaller, simpler, closer to plain POSIX shell.
Bash has a built-in time operation which accepts an entire pipeline and shows the time taken for it (run help time at a bash shell command prompt to see documentation). Other shells like dash don't have a built-in time, so when you run it you get the program /usr/bin/time; run man time to see documentation. As a separate program it of course cannot time an entire pipeline (because a pipeline is a feature of the shell); it can only time one individual command.
You have various options:
You can force your makefile to always use bash as its shell by adding:
SHELL := /bin/bash
to it. I recommend adding a comment there as well describing why bash specifically is needed.
Or you can modify your rule to work in a portable way by making the shell invocation explicit so that time only has one command to invoke:
exec:
time /bin/sh -c 'ls > ls.txt; echo $$? > code' 2>/time.txt
Put a semicolon in front of "time". As is, make is trying to parse your command as a list of dependencies.
The only suggestion that worked is to force bash in my makefile:
SHELL := /bin/bash
I checked: on my Ubuntu machine, /bin/sh is really /bin/dash whereas on the CentOS machine it is /bin/bash!
Thanks!
first i used command : which gcc
If it shows location other than /usr/bin, then how to set the right path to compile the C program
It depends upon your $PATH. And that could be set to something starting with a directory containing some gcc command. Run echo $PATH to find out what is your current $PATH.
You could either type exactly /usr/bin/gcc, or add some alias to your interactive shell configuration (often ~/.bashrc which you might edit with great care), or change your PATH setting, or, assuming which gcc gives something like /home/zaid/bin/gcc (i.e. your $HOME/bin/gcc if $HOME/bin appears early in your $PATH), add a symbolic link ln -sv /usr/bin/gcc $HOME/bin/.
If you compile a program made of several translation units, you should use some build automation tool, probably GNU make. Try once make -p to understand the builtin rules known to your make and take advantage of these. Then, edit your Makefile, perhaps by adding near its beginning lines like
CC=/usr/bin/gcc
CFLAGS+= -Wall -g
The first line (with CC=) sets your C compiler in your Makefile. The second one (with CFLAGS+=) asks for all warnings (-Wall) & debug info (-g). Because you'll use the gdb debugger.
I've got a problem with ANSI escape codes in my terminal on OpenSuse 13.2.
My Makefile use to display pretty colors on OSX at work but at home when I use it I get the litteral termcaps such as \033[1;30m ... \033[0m
I know close to nothing about termcaps, I just found these escape characters that seemed to be working fine ! The strangest is that both my OSX and Linux terminal are configured with TERM=xterm-256color so I really don't know where to look for the correct setting I'm currently missing on Linux.
TL;DR: How to get escape codes such as \033[1;30m working in Konsole with xterm-256color ?
Edit: Here's a snippet of the Makefile I am talking about:
\Here's a snippet of the Makefile I am talking about:
# Display settings
RED_L = \033[1;31m
GREEN_L = \033[1;32m
GREEN = \033[0;32m
BLUE = \033[0;34m
RED = \033[0;31m
all: $(OBJ_DIR) $(NAME)
$(OBJ_DIR):
#mkdir -p $(OBJ_DIR)
$(NAME): $(OBJ)
#echo "$(BLUE)Linking binary $(RED)$(NAME)$(BLUE).\n"
#$(CC) -o $# $^ $(LFLAGS)
#echo "\t✻ $(GRAY)$(CC) -o $(RED)$(NAME)$(GRAY) object files:$(GREEN) OK! √\n$(NC)
The example which you gave does not rely upon the setting of TERM (unless it is going someplace other than the terminal, e.g., via some program which interprets it such as the ls program, which has its own notion about colors). It would help if you quoted the section of the makefile which uses the escape sequences. Without that, we can offer only generic advice, e.g,. by assuming you have an echo command in the makefile.
The place to start looking is at the shell which your makefile uses. One would expect bash to be the default shell on OpenSUSE. But suppose you are actually using some other shell which happens to not recognize the syntax you are using, and trying to do something like
echo '\033[1;34mhello\033[m'
To help ensure that you are using the expected shell, you can put an assignment in your makefile, e.g.,
SHELL = /bin/sh
This assumes that /bin/sh itself is going to work as intended. However, that is commonly a symbolic link (for Linux) to the real shell. If so, one possible solution would be to change the real shell using OpenSUSE's update-alternatives feature to change the shell to bash (or zsh).
For additional information, see the discussion of SHELL in the GNU make manual.
Reflecting comments on the version of make -- GNU make 4.0 is known to have incompatible changes versus 3.81, as noted in the thread GNU Make 4.0 released on LWN.net. In particular, there are several comments relating to your problem, starting here.
However, checking a recent Fedora, it seems that the problem really is that the default behavior for echo has changed. As noted in other discussions (such as Why doesn't echo support “\e” (escape) when using the -e argument in MacOSX), this was done to improve POSIX compatibility. You can get your colors back by adding a -e option to the echo commands.
I finally found the solution:
the problem was I used echo instead of echo -e which seems to be the default behaivour on Mac OSX.
Thanks for your help though, it lead me to good lectures :)