I was planning something like:
URLS=www.host.com/file1.tar.gz www.host2.com/file2.tar.gz
$(somefunc $URLS): #somefunc produces downloads/file1.tar.gz downloads/file2.tar.gz
mkdir -P downloads
wget whatever # I can't get the real url here because the targets don't contain the full url anymore
myproject: $(somefunc URLS)
#Files should already be in downloads/ here
The problem I have is that if I convert URLS with somefunc I lose the urls, if I don't I can't use it as a target to avoid being downloaded when it is already there.
Any ideas?
If somefunc only modifies the path, not the actual filename, and there are no duplicates, you could try searching $(URLS) for the original filename.
Maybe something like this? (untested)
$(somefunc $URLS): #somefunc produces downloads/file1.tar.gz downloads/file2.tar.gz
mkdir -p $(dir $#)
wget $(filter $(addprefix %/,$(notdir $#)),$(URLS)) -O $#
$(notdir $#) evaluates to file1.tar.gz for one of the targets.
$(addprefix %/,file1.tar.gz) evaluates to %/file1.tar.gz
$(filter %/file1.tar.gz,www.host.com/file1.tar.gz www.host2.com/file2.tar.gz) evaluates to www.host.com/file1.tar.gz.
(Presumably you want an http:// on there too?)
At this point, I don't think you can do it directly with make. But here's a solution if you are willing to use some scripting:
$(URLS):
#for url in ${URLS}; do \
if [ ! -e $$(somefunc $${url}) ]; then \
echo wget $${url}; \
wget $${url}; \
fi \
done
Related
I've written a snippet of Makefile that checks if file exists before executing rm upon it.
clean:
echo "Attempt to remove $(exec) file"
if test -f "${exec}" ; then \
echo "Removing file ${exec}" ; \
rm ${exec} ; \
else \
echo "No ${exec} file to remove" ; \
fi
if "$(wildcard *.o)" = "" ; then \
echo "No files found" ; \
else \
echo " Found $(wildcard *.o) " ; \
fi
First if statement works fine
Attempt to remove hello file
No hello file to remove
while the second produces this:
/bin/sh: 1: main.o factorial.o: not found
Found main.o factorial.o
My question is: How come that make recipe produces valid output ( these files truly exist ) whilst the shell does not? And why shell even tries to find them?
Thank you for your time reading this question. That's my first one here so if I did something inappropriately please let me know
if main.o is very different than if test -f main.o. The former attempts to run a command main.o, while the latter runs the command test. if "main.o factorial.o" is similar, in that it attempts to run a command named main.o factorial.o which the shell is correctly complaining that it cannot find. It is not likely that you have a file named main.o factorial.o (that's a single file with a space in its name) in your PATH.
But don't do this. There is absolutely no point in ever checking whether or not a file exists before you unlink it. There's an inherent race condition. Just attempt to remove the file, and deal with errors that may occur if the file didn't exist.
It's much easier to write rm $(wildcard *.o) and just let rm emit error messages for files that don't exist. Or rm -f $(wildcard *.o) to suppress errors. If you really insist on iterating over the files and checking, you could do something like:
for f in $(wildcard *.o); do \
if ! test -f "$$f"; then echo "$$f" does not exist; \
else rm "$$f"; fi
but it's really not worth it. Also, it seems pretty pointless since the wildcard is only going to expand to files that exist. (But note that this exhibits the same race condition: the wildcard might expand to a list of files, but new files might be created between the time the wildcard is expanded and the rm is run.) Don't use wildcard like this. Explicitly list the files that you want to work with.
As part of a make install rule for a testing suite, I'd like to move all binary executables in one directory (a src directory) to a bin directory. I thought an easy way to do this would be to simply loop over each file in the src directory and then use patsubst to replace src with bin in each path. Unfortunately, I can't get it to work because I can't get make to evaluate the name of the current FILE in each loop iteration. All I have access to is the bash shell variable $$FILE, but when I use this with the make patsubst function, it doesn't actually evaluate the shell variable $$FILE... rather, the patsubst function seems to just see the string "$FILE".
So, here is what I'm trying:
install :
-- irrelevant stuff snipped --
for FILE in $(BINARY_TARGETS); do \
if [ -f $$FILE ]; then mv -f $$FILE $(patsubst %/src/,%/bin/,$$FILE); fi \
done
This results in an error for each file:
mv: ‘./src/foo/bar’ and ‘./src/foo/bar’ are the same file
This error leads me to understand that the patsubst function in make is not actually evaluating shell variables, but just sees $FILE, and so the result is that it doesn't find the substitution pattern, and the final command passed to mv has the source and destination path as the same string.
So, is there a way to get patsubst to evaluate the value of a shell variable? Or is there a better way in general to accomplish what I'm trying to achieve here?
make processing has a precedence over passing commands to shell. And, once passed, they are executed by shell. So, at first make, processes the command and in:
$(patsubst %/src/,%/bin/,$$FILE)
$$FILE is substituted by $FILE and then treated literally. So, no pattern is matched and in effect patsubst returns $FILE. Please see following example:
bar:
echo $(patsubst %/src/,%/bin/,$$whatever)
It gives:
arturcz#szczaw:/tmp/m$ make bar
echo $whatever
arturcz#szczaw:/tmp/m$
As a result of your makefile rule bash is given following command to execute:
for FILE in src/a src/b src/c; do \
if [ -f $FILE ]; then mv -f $FILE $FILE; fi \
done
and that's why you got your result.
Solution
You can rely on bash to do a proper substitution, but you need to enforce it as a shell (by default it is sh, which lacks some required features):
SHELL=bash
install:
for FILE in $(BINARY_TARGETS); do \
if [ -f $$FILE ]; then echo $$FILE $${FILE/\/src\//\/bin\/}; fi \
done
You can also ask make to do a loop and substitution. There are few ways you can achieve that. This one is doing all the replacement and prepares command on the fly.
install:
$(foreach d,$(BINARY_TARGETS),if [ -f $(d) ]; then mv -f $(d) $(d:./src/%=./bin/%);fi;)
You can cease checking existence of files to make too by using `$(wildcard) function:
install:
$(foreach d,$(wildcard $(BINARY_TARGETS)),mv -f $(d) $(d:./src/%=./bin/%);)
And, finally, solution which I personally prefer - do it in a make way using a proper dependencies and rules.
install: $(BINARY_TARGETS:./src/%=./bin/%)
bin/%: src/%
mv -f $< $#
If existence any of files in BINARY_TARGET is optional, you may want to use the $(wildcard) trick again:
install: $(patsubst ./src/%,./bin/%,$(wildcard $(BINARY_TARGETS)))
bin/%: src/%
mv -f $< $#
I have directory structure like this
containers/con1
containers/con2
containers/con3
Now every folder like con1, con2 has Makefile in it with targets like build, run
I run it like make run and make build
But i have to go inside that folder.
Is it possible that i have another Makefile in containers/Makefile
and i can run like
Make con1.run Make con2.run
Yes, you can do that. Something like the following should do what you want.
$ cat containers/Makefile
%.run: %
$(MAKE) -C $#
That being said as you can see the command to do what you want is trivial enough to make such a makefile not really necessary (and a simple shell script is as useful here as a makefile).
$ cat run.sh
[ -d "$1" ] || { echo 'No such directory.' >&2; exit 1; }
#make -C "$1"
# OR
#cd "$1" && make
If you wanted to be able to build all the sub-directory projects at once then a makefile could help you with that but even that is a simple enough shell one-liner.
$ for mkfile in */Makefile; do make -C "$(dirname "$mkfile"); done
$ for mkfile in */Makefile; do (cd "$(dirname "$mkfile") && make); done
As far as I understand you want this:
-C dir, --directory=dir
Change to directory dir before reading the makefiles or doing anything else. If multiple -C options are specified, each is interpreted relative to the previous one: -C / -C etc is equivalent to -C /etc. This is typi‐
cally used with recursive invocations of make.
Add -C option like this: make -C con1/
Recursive makes are evil, but if you want that:
# con1.run is a phony target...
.PHONY: con1.run
con1.run:
$(MAKE) -C con1
I can't wget while there is no path already to save. I mean, wget doens't work for the non-existing save paths. For e.g:
wget -O /path/to/image/new_image.jpg http://www.example.com/old_image.jpg
If /path/to/image/ is not previously existed, it always returns:
No such file or directory
How can i make it work to automatically create the path and save?
Try curl
curl http://www.site.org/image.jpg --create-dirs -o /path/to/save/images.jpg
mkdir -p /path/i/want && wget -O /path/i/want/image.jpg http://www.com/image.jpg
To download a file with wget, into a new directory, use --directory-prefix without -O:
wget --directory-prefix=/new/directory/ http://www.example.com/old_image.jpg
Using -O new_file in conjunction with --directory-prefix, will not create the new directory structure, and will save the new file in the current directory.
It may even fail with "No such file or directory" error, if you specify -O /new/directory/new_file
I was able to create folder if it doesn't exists with this command:
wget -N http://www.example.com/old_image.jpg -P /path/to/image
wget is only getting a file NOT creating the directory structure for you (mkdir -p /path/to/image/), you have to do this by urself:
mkdir -p /path/to/image/ && wget -O /path/to/image/new_image.jpg http://www.example.com/old_image.jpg
You can tell wget to create the directory (so you dont have to use mkdir) with the parameter --force-directories
alltogether this would be
wget --force-directories -O /path/to/image/new_image.jpg http://www.example.com/old_image.jpg
After searching a lot, I finally found a way to use wget to download for non-existing path.
wget -q --show-progress -c -nc -r -nH -i "$1"
=====
Clarification
-q
--quiet --show-progress
Kill annoying output but keep the progress-bar
-c
--continue
Resume download if the connection lost
-nc
--no-clobber
Overwriting file if exists
-r
--recursive
Download in recursive mode (What topic creator asked for!)
-nH
--no-host-directories
Tell wget do not use the domain as a directory (for e.g: https://example.com/what/you/need
- without this option, it will download to "example.com/what/you/need")
-i
--input-file
File with URLs need to be download (in case you want to download a lot of URLs,
otherwise just remove this option)
Happy wget-ing!
I have a project where the directory structure is like this:
$projectroot
|
+---------------+----------------+
| | |
part1/ part2/ part3/
| | |
+------+-----+ +---+----+ +---+-----+
| | | | | | |
data/ src/ inc/ src/ inc/ src/ inc/
How should I write a makefile that would be in part/src (or wherever really) that could comple/link on the c/c++ source files in part?/src ?
Can I do something like
-I$projectroot/part1/src -I$projectroot/part1/inc -I$projectroot/part2/src ...
If that would work, is there an easier way to do it. I've seen projects where there is a makefile in each of the corresponding part? folders. [in this post I used the question mark like in bash syntax]
The traditional way is to have a Makefile in each of the subdirectories (part1, part2, etc.) allowing you to build them independently. Further, have a Makefile in the root directory of the project which builds everything. The "root" Makefile would look something like the following:
all:
+$(MAKE) -C part1
+$(MAKE) -C part2
+$(MAKE) -C part3
Since each line in a make target is run in its own shell, there is no need to worry about traversing back up the directory tree or to other directories.
I suggest taking a look at the GNU make manual section 5.7; it is very helpful.
If you have code in one subdirectory dependent on code in another subdirectory, you are probably better off with a single makefile at top-level.
See Recursive Make Considered Harmful for the full rationale, but basically you want make to have the full information it needs to decide whether or not a file needs to be rebuilt, and it won't have that if you only tell it about a third of your project.
The link above seems to be not reachable. The same document is reachable here:
aegis.sourceforge.net (archived)
lcgapp.cern.ch
The VPATH option might come in handy, which tells make what directories to look in for source code. You'd still need a -I option for each include path, though. An example:
CXXFLAGS=-Ipart1/inc -Ipart2/inc -Ipart3/inc
VPATH=part1/src:part2/src:part3/src
OutputExecutable: part1api.o part2api.o part3api.o
This will automatically find the matching partXapi.cpp files in any of the VPATH specified directories and compile them. However, this is more useful when your src directory is broken into subdirectories. For what you describe, as others have said, you are probably better off with a makefile for each part, especially if each part can stand alone.
You can add rules to your root Makefile in order to compile the necessary cpp files in other directories. The Makefile example below should be a good start in getting you to where you want to be.
CC=g++
TARGET=cppTest
OTHERDIR=../../someotherpath/in/project/src
SOURCE = cppTest.cpp
SOURCE = $(OTHERDIR)/file.cpp
## End sources definition
INCLUDE = -I./ $(AN_INCLUDE_DIR)
INCLUDE = -I.$(OTHERDIR)/../inc
## end more includes
VPATH=$(OTHERDIR)
OBJ=$(join $(addsuffix ../obj/, $(dir $(SOURCE))), $(notdir $(SOURCE:.cpp=.o)))
## Fix dependency destination to be ../.dep relative to the src dir
DEPENDS=$(join $(addsuffix ../.dep/, $(dir $(SOURCE))), $(notdir $(SOURCE:.cpp=.d)))
## Default rule executed
all: $(TARGET)
#true
## Clean Rule
clean:
#-rm -f $(TARGET) $(OBJ) $(DEPENDS)
## Rule for making the actual target
$(TARGET): $(OBJ)
#echo "============="
#echo "Linking the target $#"
#echo "============="
#$(CC) $(CFLAGS) -o $# $^ $(LIBS)
#echo -- Link finished --
## Generic compilation rule
%.o : %.cpp
#mkdir -p $(dir $#)
#echo "============="
#echo "Compiling $<"
#$(CC) $(CFLAGS) -c $< -o $#
## Rules for object files from cpp files
## Object file for each file is put in obj directory
## one level up from the actual source directory.
../obj/%.o : %.cpp
#mkdir -p $(dir $#)
#echo "============="
#echo "Compiling $<"
#$(CC) $(CFLAGS) -c $< -o $#
# Rule for "other directory" You will need one per "other" dir
$(OTHERDIR)/../obj/%.o : %.cpp
#mkdir -p $(dir $#)
#echo "============="
#echo "Compiling $<"
#$(CC) $(CFLAGS) -c $< -o $#
## Make dependancy rules
../.dep/%.d: %.cpp
#mkdir -p $(dir $#)
#echo "============="
#echo Building dependencies file for $*.o
#$(SHELL) -ec '$(CC) -M $(CFLAGS) $< | sed "s^$*.o^../obj/$*.o^" > $#'
## Dependency rule for "other" directory
$(OTHERDIR)/../.dep/%.d: %.cpp
#mkdir -p $(dir $#)
#echo "============="
#echo Building dependencies file for $*.o
#$(SHELL) -ec '$(CC) -M $(CFLAGS) $< | sed "s^$*.o^$(OTHERDIR)/../obj/$*.o^" > $#'
## Include the dependency files
-include $(DEPENDS)
If the sources are spread in many folders, and it makes sense to have individual Makefiles then as suggested before, recursive make is a good approach, but for smaller projects I find it easier to list all the source files in the Makefile with their relative path to the Makefile like this:
# common sources
COMMON_SRC := ./main.cpp \
../src1/somefile.cpp \
../src1/somefile2.cpp \
../src2/somefile3.cpp \
I can then set VPATH this way:
VPATH := ../src1:../src2
Then I build the objects:
COMMON_OBJS := $(patsubst %.cpp, $(ObjDir)/%$(ARCH)$(DEBUG).o, $(notdir $(COMMON_SRC)))
Now the rule is simple:
# the "common" object files
$(ObjDir)/%$(ARCH)$(DEBUG).o : %.cpp Makefile
#echo creating $# ...
$(CXX) $(CFLAGS) $(EXTRA_CFLAGS) -c -o $# $<
And building the output is even easier:
# This will make the cbsdk shared library
$(BinDir)/$(OUTPUTBIN): $(COMMON_OBJS)
#echo building output ...
$(CXX) -o $(BinDir)/$(OUTPUTBIN) $(COMMON_OBJS) $(LFLAGS)
One can even make the VPATH generation automated by:
VPATH := $(dir $(COMMON_SRC))
Or using the fact that sort removes duplicates (although it should not matter):
VPATH := $(sort $(dir $(COMMON_SRC)))
I think it's better to point out that using Make (recursive or not) is something that usually you may want to avoid, because compared to today tools, it's difficult to learn, maintain and scale.
It's a wonderful tool but it's direct use should be considered obsolete in 2010+.
Unless, of course, you're working in a special environment i.e. with a legacy project etc.
Use an IDE, CMake or, if you're hard cored, the Autotools.
(edited due to downvotes, ty Honza for pointing out)
I was looking for something like this and after some tries and falls i create my own makefile, I know that's not the "idiomatic way" but it's a begining to understand make and this works for me, maybe you could try in your project.
PROJ_NAME=mono
CPP_FILES=$(shell find . -name "*.cpp")
S_OBJ=$(patsubst %.cpp, %.o, $(CPP_FILES))
CXXFLAGS=-c \
-g \
-Wall
all: $(PROJ_NAME)
#echo Running application
#echo
#./$(PROJ_NAME)
$(PROJ_NAME): $(S_OBJ)
#echo Linking objects...
#g++ -o $# $^
%.o: %.cpp %.h
#echo Compiling and generating object $# ...
#g++ $< $(CXXFLAGS) -o $#
main.o: main.cpp
#echo Compiling and generating object $# ...
#g++ $< $(CXXFLAGS)
clean:
#echo Removing secondary things
#rm -r -f objects $(S_OBJ) $(PROJ_NAME)
#echo Done!
I know that's simple and for some people my flags are wrong, but as i said this is my first Makefile to compile my project in multiple dirs and link all of then together to create my bin.
I'm accepting sugestions :D
RC's post was SUPER useful. I never thought about using the $(dir $#) function, but it did exactly what I needed it to do.
In parentDir, have a bunch of directories with source files in them: dirA, dirB, dirC. Various files depend on the object files in other directories, so I wanted to be able to make one file from within one directory, and have it make that dependency by calling the makefile associated with that dependency.
Essentially, I made one Makefile in parentDir that had (among many other things) a generic rule similar to RC's:
%.o : %.cpp
#mkdir -p $(dir $#)
#echo "============="
#echo "Compiling $<"
#$(CC) $(CFLAGS) -c $< -o $#
Each subdirectory included this upper-level makefile in order to inherit this generic rule. In each subdirectory's Makefile, I wrote a custom rule for each file so that I could keep track of everything that each individual file depended on.
Whenever I needed to make a file, I used (essentially) this rule to recursively make any/all dependencies. Perfect!
NOTE: there's a utility called "makepp" that seems to do this very task even more intuitively, but for the sake of portability and not depending on another tool, I chose to do it this way.
Hope this helps!
Recursive Use of Make
all:
+$(MAKE) -C part1
+$(MAKE) -C part2
+$(MAKE) -C part3
This allows for make to split into jobs and use multiple cores
I suggest to use autotools:
//## Place generated object files (.o) into the same directory as their source files, in order to avoid collisions when non-recursive make is used.
AUTOMAKE_OPTIONS = subdir-objects
just including it in Makefile.am with the other quite simple stuff.
Here is the tutorial.