Have you ever been CONFUSED by creating make files? Have you ever WONDERED which files had to be rebuild because of changes in make files or header files? Have you ever thought about how CONCURRENT your make files actually are? Have you ever been disturbed by DISTRIBUTING IMPLEMENTATION DETAILS about a particular .c- or .h-file over multiple makefiles? Have you ever thought how concurrency could really work while invoking make MULTIPLE (or many) times and how that puts a lot of OVERHEAD into a build process? Have you ever found it hard to MODULARIZE your make files? Have you ever QUESTIONED the sense of make files in general?

THAN THIS BLOG POSTING IS FOR you  :-)

Warning: When coming down to creating, maintaining  or compiling stuff, I know I am different, so be warned  :*) I did briefly explain to others how I felt things should work ... and did hear from some that I am odd, while others seemed to like it ... anyway, I do believe in and successfully tried the below approach in my home projects and am currently playing with it while experimenting with packaging ... but judge yourself :-)

A Start: Let's start with simple things, in my little world I mostly have some .c- and .h-files, while I typically want to create some .o-, .bin- and .so- respectively lib*.so-files. Some files need special flags, e.g. for optimization or debugging. To understand how we can automate this - thus abandoning the makefile burden ideally completely - we may want to take a look at a typical C include.

A simple include looks like this:

foo.c:
    #include "a.h"

Compiling: For the corresponding .o-file (foo.o) this means, that it needs to be remade in case foo.c, a.h or any of the transitive includes of a.h changes.

Linking: To be able to link foo.o into an executable or shared library, we also need to link-in any objects or shared libraries implementing a.h (or its includes respectively).

Sample Code: Let's say, a.h only declares what's implemented in a.c, such that a.h does not include anything else. Unfortunately a.c includes (despite a.h) b.h as well as x.h .

a.c:
    #include "a.h"
    #include "b.h"
    #include "x.h"

Luckily, b.h does not include anything itself, but b.c does, additionally to b.h it also includes c.h, which is implemented in c.c:

b.c:
    #include "b.h"
    #include "c.h"

c.c:
    #include "c.h"

And if this had not been enough, a.c respectively a.o does also need the implementation of x.h to be properly linkable, which is not directly implemented in a .o-file but a linkable shared library named libx.so.

Looking into x.c

x.c:
    #include "x.h"
    #include "y.h"
    #include "d.h"

we see, that itself needs another library (liby.so) and at least one other .o-file (d.o). And foo.bin is not the only executable we would like to link, but bar.bin as well. bar.bin is partly based on the same objects as foo.bin, but links differently, I leave the details out here, please have a look at the bottom for all sample files.

And it's Makefile: Expressing this in GNU make looks like this:

============= handcraft.mk =============:
LDFLAGS+=-Wl,-rpath='$$ORIGIN/' -L.

a.o: a.c a.h b.h c.h x.h
	$(COMPILE.c) $(OUTPUT_OPTION) $<

b.o: b.c b.h c.h 
	$(COMPILE.c) $(OUTPUT_OPTION) $<

c.o: c.c c.h
	$(COMPILE.c) $(OUTPUT_OPTION) $<

x.o: x.c x.h y.h d.h
	$(COMPILE.c) $(OUTPUT_OPTION) $<

y.o: y.c y.h
	$(COMPILE.c) $(OUTPUT_OPTION) $<

libx.so: x.o liby.so
	$(LINK.o) $(LOADLIBES) $(LDLIBS) -shared $^ -L. -ly -o $@

liby.so: y.o
	$(LINK.o) $(LOADLIBES) $(LDLIBS) -shared $^ -o $@

foo.o: foo.c a.h
	$(COMPILE.c) $(OUTPUT_OPTION) $<

foo.bin: foo.o a.o b.o c.o libx.so
	$(LINK.o) $(LOADLIBES) $(LDLIBS) $^ -o $@

bar.o: bar.c b.h c.h d.h
	$(COMPILE.c) $(OUTPUT_OPTION) $<

bar.bin: bar.o b.o c.o d.o
	$(LINK.o) $(LOADLIBES) $(LDLIBS) $^ -o $@

Now we can build the concrete targets by invoking make as:

  > make -f handcraft.mk foo.bin
  > make -f handcraft.mk bar.o
  > make -f handcraft.mk libx.so
  > make -f handcraft.mk -j foo.bin bar.bin
  > make -f handcraft.mk ...

Automation: By increasing the number of targets and prerequisites such a make file can become pretty fast pretty complicated - after all building software properly is not an easy task.

Automatic dependency generation for include files is well known and has been done for long, what's missing to become fully automatic are compilation flags in particular and linking support in general. The point is, that only the moment building becomes fully automatic, make can be modularized reasonably and we can build a whole project invoking make once only (which is desirable :-).

So, let's see what information regarding linking has been implicitly expressed in the above makefile:

• If you include a.h, than you need to link-in a.o as well.
• If you link-in a.o, than you need to link-in b.o and c.o as well, and you have to link against libx.so .
• If you ...
  

The Rule: In principle, the rule seems to be: If you include a header, ensure that you link-in the implementing .o-files and shared libraries, as well as the required .o-files and shared libraries of these .o-files etc.

Cohesion: The hints regarding required .o-files, shared libraries and c-flags should be placed in the appropriate files and need to be available for make during building, otherwise this information gets repeated in multiple targets and makefiles again and again - which to my experience is typical ;*)

Concrete: If we have a header file which requires a particular object file to be linked-in, we can (initially hackily :-) express this directly in-line, prefixing it with a "magic" word, e.g. as:

//§ req_objs:=a.o

We may want to do the same with libraries:

//§ req_libs:=libx.so

And even compilation flags may be expressed this way:

//§ cflags:=-g

Applying the schema to the above files, we get the following additional lines:

foo.c:
  //§ cflags=-g

a.h:
  //§ req_objs:=a.o

b.h:
  //§ req_objs:=b.o

x.h:
  //§ req_libs=libx.so

...

Now, the only think left is to write some generic make "modules" to utilize this now explicit available information ... this is so obvious, that I leave that as a homework for the patient reader ... just joking :-) please have a look below.

Conclusion: By now we have developed a simple schema for linking binaries and shared libraries based on C files, which allows us to build any final or intermediate target fully automatically, expressing linking relevant information in a cohesive way, while eliminating the need to hassle with make-files ever again.

Scalability:  I have to admit that I did not yet test how scalable this approach is, in principle it should scale quite well ... and after all we are only talking about some thousand files for typical projects. At least for my home projects with some hundred files it works quite nicely.

Reasoning: The reason I brought this up is, that I believe that anything can be build fully automatically this way - even installation packages I am currently experimenting with.

In Principle: The overall rule set to build any kind of derived file fully automatically (and not only compiling C sources :-), seems to be something along the following lines:

1. We have prerequisite files and derived files.
2. Every derivation has exactly one primary prerequisite and n secondary prerequisites.
3. A derivation explicitly depends on its primary prerequisite.
4. A derivation implicitly depends on its secondary prerequisites, which need to be determined during runtime out of the primary prerequisite.
5. A prerequisite may specify transformation flags.
6. For every type of prerequisite-to-derivation transformation exactly one explicit rule is defined.
7. For any type of derivation there may be n+ explicit rules.
   

Note on -7-: As a derivation may not depend on its primary prerequisite only, in practice some functions need to be executed at runtime to determine all secondary prerequisites. Actually, the concrete function to be calculated depends on the actual explicit rule selected for the creation of the derivation. As ordinary makes (e.g. GNU make), do not offer a way to add prerequisites to a derivation after an explicit rule has been selected, statement -7- has to be adapted for these to work:

-7- For any type of derivation there is exactly one explicit rule.

Therefore, to still enable deriving one type from different types of prerequisites, we need to mangle in the prerequisites type.

Finally: If there is interest and people think that this general approach seems to be reasonable, I am going to explain in more detail how it actually works and what else could be done to make a builders life easier ... and I may even try it out on a concrete OOo module, may be SAL :-)

Last but not least I have to admit, that I am a descent GNU make fan :-)

Please find sample files etc.  below.

Best regards

Kay

Samples and make files for GNU make, tried on Debian Linux, just copy&paste the stuff into the named files ...

============= foo.c =============:
//§ cflags:=-g

#include "a.h"

int main(void) {
  return 0;
}

============= bar.c =============:
//§ cflags:=-O

#include "b.h"
#include "c.h"
#include "d.h"

int main(void) {
  return 0;
}

============= a.h =============:
//§ req_objs=a.o

============= a.c =============:
#include "a.h"
#include "b.h"
#include "x.h"

============= b.h =============:
//§ req_objs=b.o

============= b.c =============:
#include "b.h"
#include "c.h"

============= c.h =============:
//§ req_objs:=c.o

============= c.c =============:
#include "c.h"

============= d.h =============:
//§ req_objs=d.o

============= d.c =============:
#include "d.h"

============= x.h =============:
//§ req_libs=libx.so

============= x.c =============:
#include "x.h"
#include "y.h"
#include "d.h"

============= y.h =============:
//§ req_libs:=liby.so

============= y.c =============:
#include "y.h"

============= generic.mk =============:
# The master makefile.

# Preserve intermediate files.
.PRECIOUS: %.o

# To find linked libraries relatively.
LDFLAGS+=-Wl,-rpath='$$ORIGIN/' -L.

include functions.mk   # Some helpers.

include info_c.mk      # Create C infos.
include o_c.mk         # Create C objects.
include bin_o.mk       # Create object binaries.
include libso_o.mk     # Create object shared libraries.

$(foreach target,$(MAKECMDGOALS),$(call $(suffix $(target))_spreqs,$(target)))

============= functions.mk =============:
# Guard includes.
define ginc_t
  ifndef $(1)_inc_def
    $(1)_inc_def=:1 

    -include $(1) 
  endif
endef

ginc=$(eval $(ginc_t))

# Calculate the referential transitive closure.
define closure_t
  ifndef $(1)$(2)_cls_def
    $(1)$(2)_cls_def:=1

    ifdef $(suffix $(1))_info
      $$(call $(suffix $(1))_info,$(1))
    else
      $$(call ginc,$(1).info) 
    endif

    $(1)$(2)_cls:=$$(strip $$($(1)$(2)) $$(foreach file,$$($(1)$(2)),$$(call closure,$$(file),$(2))))
  endif
endef

# $(1) = file name
# $(2) = variable postfix
closure=$(eval $(closure_t)) $($(1)$(2)_cls)

============= info_c.mk =============:
# Rules and prerequisits for .info files .

define info_script
  echo Making $@
  echo "`grep ^//§ $<|sed s0//§\ 0$<_0g)`" > $@
  echo "$<_includes:=`echo \`grep \#include $<|sed s0\#include00g|sed s0\\\"00g\``" >> $@
endef

%.info: %
	@$(info_script)

============= o_c.mk =============:
# Rules and prerequisits for .o files, as well as .o.info .

define t_o_info
  ifndef $(1)_info_def
    $(1)_info_def:=1

    t_o_info_ppreq_$(1):=$(1:.o=.c)

    $$(call ginc,$$(t_o_info_ppreq_$(1)).info)

    t_o_info_all_includes:=$$(call closure,$$(t_o_info_ppreq_$(1)),_includes)
    $(1)_req_objs:=$$(subst $(1),,$$(foreach inc,$$(t_o_info_all_includes),$$($$(inc)_req_objs)))
    $(1)_req_libs:=$$(foreach inc,$$(t_o_info_all_includes),$$($$(inc)_req_libs))
  endif
endef

.o_info= $(eval $(t_o_info)) 


define t_o_spreqs
  ifndef t_o_spreqs_$(1)_def
    t_o_spreqs_$(1)_def:=1

    t_o_spreqs_ppreq_$(1):=$(1:.o=.c)

    $(1): $$(call closure,$$(t_o_spreqs_ppreq_$(1)),_includes)
    $(1): cflags_:=$$($$(t_o_spreqs_ppreq_$(1))_cflags)
  endif
endef

.o_spreqs=$(eval $(t_o_spreqs))



%.o: %.c
	$(COMPILE.c) $(OUTPUT_OPTION) $(cflags_) $<

============= bin_o.mk =============:
# Rules and prerequisits for .bin files .

define t_bin_spreqs
  ifndef t_bin_spreqs_$(1)_def
    t_bin_spreqs_$(1)_def:=1
    t_bin_spreqs_$(1)_ppreq:=$(1:.bin=.o)

    $$(call .o_spreqs,$$(t_bin_spreqs_$(1)_ppreq))

    t_bin_spreqs_$(1)_objs:=$$(call closure,$$(t_bin_spreqs_$(1)_ppreq),_req_objs)
    $$(foreach obj,$$(t_bin_spreqs_$(1)_objs),$$(call .o_spreqs,$$(obj)))

    t_bin_spreqs_$(1)_libs:=$$(subst $(1),,$$(foreach obj,$$(t_bin_spreqs_$(1)_ppreq) $$(t_bin_spreqs_$(1)_objs),$$($$(obj)_req_libs)))
    $$(foreach lib,$$(t_bin_spreqs_$(1)_libs),$$(call .so_spreqs,$$(lib)))

    $(1): objs_:=$$(t_bin_spreqs_$(1)_objs)
    $(1): libs_:=$$(t_bin_spreqs_$(1)_libs)
    $(1): $$(t_bin_spreqs_$(1)_objs)
    $(1): $$(t_bin_spreqs_$(1)_libs)
  endif
endef

.bin_spreqs=$(eval $(t_bin_spreqs))

%.bin: %.o
	$(LINK.o) $(LOADLIBES) $(LDLIBS) $< $(objs_) $(patsubst lib%.so,-l%,$(libs_)) -o $@

============= libso_o.mk =============:
# Rules and prerequisits for lib*.so files .

define t_so_spreqs
  ifndef t_bin_spreqs_$(1)_def
    t_so_spreqs_$(1)_def:=1
    t_so_spreqs_$(1)_ppreq:=$(patsubst lib%.so,%.o,$(1))

    $$(call .o_spreqs,$$(t_so_spreqs_$(1)_ppreq))

    t_so_spreqs_$(1)_objs:=$$(call closure,$$(t_so_spreqs_$(1)_ppreq),_req_objs)
    $$(foreach obj,$$(t_so_spreqs_$(1)_objs),$$(call .o_spreqs,$$(obj)))

    t_so_spreqs_$(1)_libs:=$$(subst $(1),,$$(foreach obj,$$(t_so_spreqs_$(1)_ppreq) $$(t_so_spreqs_$(1)_objs),$$($$(obj)_req_libs)))
    $$(foreach lib,$$(t_so_spreqs_$(1)_libs),$$(call .so_spreqs,$$(lib)))

    $(1): objs_:=$$(t_so_spreqs_$(1)_objs)
    $(1): libs_:=$$(t_so_spreqs_$(1)_libs)
    $(1): $$(t_so_spreqs_$(1)_objs)
    $(1): $$(t_so_spreqs_$(1)_libs)
  endif
endef

.so_spreqs=$(eval $(t_so_spreqs))

lib%.so: %.o
	$(LINK.o) $(LOADLIBES) $(LDLIBS) -shared $< $(objs_) $(patsubst lib%.so,-l%,$(libs_)) -o $@

> make -f generic.mk foo.bin
> make -f generic.mk bar.o
> make -f generic.mk libx.so
> make -f generic.mk -j foo.bin bar.bin
> make -f generic.mk ...

tags: automatic build building gnumake make openoffice openoffice.org packaging