Make
The workspace tool (dws) deals with inter-project dependency. Once it has generated a topological ordering of projects to be rebuilt, it invokes the unix make utility for each project in sequence. Thus intra-project dependencies are handled and updated through regular makefiles.
The GNU Make Reference Manual is very well written and contains the detailed information on how make works, thus the place to look when debugging Makefiles. Of course, as a reference manual, it has very little on writing makefiles for large maintainable source code bases.
Makefile patterns
We will focus here on patterns and anti-patterns and the art of writing makefiles. Building packages and managing project dependencies is left for another section.
Path variables
installTopwhere distributed executables, libraries, etc. are installed
where intermediate files are created
where the source files under revision control live
Those paths relate to the following commands issued at the shell prompt.
cd *objDir*
*srcDir*/configure --prefix=*installTop*
make
make install
Writing dependencies
Makefiles dependencies are written in terms of file path names. A first decision when writing the build infrastructure is thus to decide between using absolute paths, relative paths and relative to which directory.
Makefile to be executed from anywhere
$(objDir)/hello.o: $(srcDir)/hello.c
Makefile to be executed from srcDir
$(objDir)/hello.o: hello.c
Makefile to be executed from objDir
hello.o: $(srcDir)/hello.c
or
vpath %.c $(srcDir)
hello.o: hello.c
Trampoline
In case the intermediate directory is implicit, we need a trampoline to change the directory to objDir before executing make.
Create a Makefile in the objDir through a configure script
cd ${objDir} && ${srcDir}/configure && make
Use a pre-existing Makefile from the source tree
cd ${objDir} && make -f ${srcDir}/Makefile
Invoke make through a wrapper
cd ${srcDir} && cmake
Use of a trampoline target. This one is high voltige!
# $(srcDir)/Makefile
ifeq ($(MAKECMDGOALS),)
MAKECMDGOALS := all
endif
ifeq ($(MAKELEVEL),0)
$(MAKECMDGOALS): trampoline
endif
trampoline:
cd $(objDir) && $(MAKE) -f $(srcDir)/Makefile $(MAKECMDGOALS)
process=`ps | grep make | grep -v grep | cut -d ' ' -f 1` \
&& kill -3 $$process > /dev/null 2>&1
cd ${srcDir} && make
Makefile fragments
A Makefile can be divided from top to bottom in four sections
- default variables
- overridden variables
- specific rules
- default rules
The first overall organization consists of putting the default variables and rules in a common.mk fragment file included in all subsequent Makefiles. The structure of such Makefile then looks like:
# $(srcDir)/Makefile
APPNAME := hello
SRCS := hello.c
include releng/common.mk
# releng/common.mk
CC := gcc
OBJS := $(SRCS:.c=.o)
$(APPNAME): $(OBJS)
$(CC) -o $@ $^
This structure works well for projects where all Makefiles builds the same number and kinds of targets. Since the overridden variables appear before the included common.mk fragment, it is thus very important to understand the semantics of the assignment operators (=, :=, ?=). Otherwise some variables might always end up with the default value.
A second overall organization consists of putting the default variables and rules in a single common Makefile and include a specifics.mk fragment from the source directory.
# $(srcDir)/specifics.mk
APPNAME := hello
SRCS := hello.c
# releng/Makefile
CC := gcc
include $(srcDir)/specifics.mk
OBJS := $(SRCS:.c=.o)
$(APPNAME): $(OBJS)
$(CC) -o $@ $^
This scheme is very close to the previous one but does not suffer from the default vs. overridden variables order. There is also no need to have an include statement in each source directory fragment since the inclusion order has been reversed in this case.
As it turns out, in a way or another, we really need two separate fragments, a prefix.mk and suffix.mk. Then it becomes quickly a very complex game of variable expansion and substitution when different Makefiles require to make slightly different targets. The third approach thus leaves the specification of dependencies to the source directory Makefile and gathers the default rules into the common suffix.mk fragment.
# $(srcDir)/Makefile
include releng/prefix.mk
bins := hello
hello: hello.o
include releng/suffix.mk
# releng/prefix.mk
CC := gcc
# releng/suffix.mk
install: $(bins)
$(installExecs) $^ $(binDir)
%: %.o
$(CC) -o $@ $^
Default variables and rules
There are already a lot of built-in variables and rules in make and these can be printed out with a command line option.
$ make -p
...
COMPILE.cc = $(CXX) $(CXXFLAGS) $(CPPFLAGS) $(TARGET_ARCH) -c
LINK.o = $(CC) $(LDFLAGS) $(TARGET_ARCH)
...
%.o: %.cc
# commands to execute (built-in):
$(COMPILE.cc) $(OUTPUT_OPTION) $<
%: %.o
# commands to execute (built-in):
$(LINK.o) $^ $(LOADLIBES) $(LDLIBS) -o $@
...
Missing dependencies on headers
All dependencies need to be specified. If there is a missing or implicit dependency, make will fail to update the target when the prerequisite is updated.
hello.o: hello.c
rewrite as
CFLAGS += -MMD
hello.o: hello.c
-include *.d
Targets with side effects
When a command updates multiple files, it creates an implicit dependency on the target which is not specified on the dependency line.
A: B
prog --update STATE -o A B
rewrite as
STATE: A
A: B
prog --update STATE -o A B
As a corollary, updating the same file multiple times will break the expected static single assignment Makefile structure and definitely confuse make.
A: B
prog --update STATE -o A B
B: C
prog --update STATE -o B C
rewrite as
STATE: A
A: B STATE_B
cp STATE_B STATE
prog --update STATE -o A B
STATE_B: B
B: C
prog --update STATE_B -o B C
Prerequisites and commands
When a command input is not derived from the prerequisites, it is possible to get a drift over time and introduce implicit dependencies. As a matter of fact, the best way to prevent such issues popping up is to always specify commands in terms of prerequisites.
hello.o: hello.c
$(CXX) -o hello.o -c hello.c
rewrite as
hello.o: hello.c
$(CXX) -o $@ -c $<
Another example
monthly_results: rent.sub bills.sub sales.add interests.add
print -o $@ -col1 rent.sub bills.sub -col2 sales.add interests.add
rewrite as
monthly_results: rent.sub bills.sub sales.add interests.add
print -o $@ -col1 $(filter %.sub,$^) -col2 $(filter %.add,$^)
Yet another example
test_unit.log: query clients $(wildcard *.idx)
./$< -o $@ --keys=clients $(filter %.idx,$^)
query: query.c
rewrite as
test_unit.log: query clients $(wildcard *.idx)
./$< -o $@ --keys=$(word 2,$^) $(filter %.idx,$^)
query: query.c
Bailing out with targets partially built
Some commands might start to produce a target and fail in the middle of execution, leaving a partial built target file. This is usually the case with commands redirecting output to $@. The first time make bails out with an error but on the second execution make finds an up-to-date file and keeps going happily with that partially built target. It is thus important to delete those partially built targets on errors.
test_unit.log: test_unit
./$< > $@
rewrite as
test_unit.log: test_unit
./$< > $@ || rm $@
Variables substitution and ifeq
Variable substitutions are very powerful in make. Most times you do need conditional code to set variables based on enumerated values.
ifeq ($(DEBUGSET),1)
BLOCKS := A B C
else
BLOCKS := D E F
endif
rewrite as
BLOCKS_1 := A B C
BLOCKS_0 := D E F
BLOCKS := $(BLOCKS_$(DEBUGSET))
For loops as shell scripts
Makefiles specify dependencies in static single assignment form. Often times a command written as a for loop shell script is unnecessary and counter-productive in make.
test_unit.log: test_unit1 test_unit2
(for i in $^ ; do \
./$$i ; \
done) > $@ || rm $@
rewrite as
test_unit.log: test_unit1.log test_unit2.log
cat $^ > $@ || rm $@
test_unit1.log: test_unit1
test_unit2.log: test_unit2
%.log: %
./$< > $@ || rm $@
Substituting commas
Commas are used as separator in make function calls. Substituting a comma is thus a challenge. Hopefully we can use make variable substitution to still perform the substitution operation.
hello.h: block1 block2 block3
echo "int blocks[] = { 1, 2, 3 };" > $@ || rm $@
rewrite as
comma=,
hello.h: block1 block2 block3
echo "int blocks[] = { $(patsubst %,%$(comma),$^) };" > $@ || rm $@
drop prefix and suffix fragments
As seen before, common practise to factor makefiles is to devide them in three parts: a prefix that contains some definitions, project-specific declarations and a suffix that contains implicit rules.
Drop's prefix.mk and suffix.mk define variables and rules to provide compilation of executables and libraries, installation of different files on the local system, packaging of a project for various distributions and running unit tests.
Using drop to write a project makefile requires to include drop's prefix.mk at the begining of the project makefile and drop's suffix at the end of the project makefile. In order to convieniently support multiple workspaces, it is also required to include dws.mk through a include $(shell dws context) statement as the first line of the project's makefile.
The install target defined in prefix.mk depends on the content of the bins, scripts, libs and includes variables. The install target also defines the rules to install the generated files in the appropriate system directories.
| bins | install -s -p -m 755 | binDir |
| scripts | install -p -m 755 | binDir |
| libs | install -p -m 644 | libDir |
| includes | install -p -m 644 | includeDir |
A typical Makefile using the drop prefix and suffixes would thus look like:
# -*- Makefile -*-
include $(shell dws context)
include $(shareDir)/dws/prefix.mk
bins := example
...
example: example.c
include $(shareDir)/dws/suffix.mk
Drop is especially relevent in large cross-platform build systems. By including crossprefix.mk and crosssuffix.mk instead of the usual prefix.mk and suffix.mk, each predefined variable (ie. CC, LD, etc.) is doubled with a host equivalent (ie. CC.host, LD.host, etc.) and the associated rules. The cross*.mk fragments enable building both target and host executables in the same makefile, a feature often required for development of virtual machines.
The dws integrate command is particularly useful to integrate a patch (under source control) into a source package fetched from a remote machine. Accordingly, dws upstream is used to easily generate patches that can be submitted to upstream maintainers of third-party packages.