The current setup of websites for downloading OpenOffice.org installation sets provides builds for several languages and operating systems. But nowadays a lot more are built. Unfortunately they are not that easy to find as they are on mirror servers. The most users do not know this or how to come to these mirrors.

Furthermore there was no comfortable way to download language packs (currently for 96 languages !).

Another reason is that many native language teams have only a small staff or do not have the time to test all available Release Candidate (RC) builds on all platforms for their language. However, these are very close to a final release but have not got the latest tests. But why not offering these to the users with a hint to be carefully when using?

To improve this situation and to deliver more choice we have created a new download website layout.

Main Download Page

http://download.openoffice.org

This website was enhanced to download easily the build you want. Of course the well-known (green) One-Click download remains the same easy way to get your favorite version. The same for the orange button for Developer Snapshots. The new thing is the yellow button that will guide you to the website for Release Candidates.

Full Installations and Language Packs

The improvements were done here to provide all available stable builds:

http://download.openoffice.org/other.html

The first table provides all full installation sets as stable release of the current OpenOffice.org version. The second table provides all stable language packs.

BTW:
A language pack contains only resource files for a specific language and platform to show, e.g., menus, dialogs and error messages in your language. If translated it may contain also the help content. It's a comfortable way to get several languages without to install the applications double and triple. After installation change the languages via menu "Tools - Options - Language Settings - Languages - User Interface".

Release Candidates

http://download.openoffice.org/all_rc.html

Some days ago the 100 millionth download of an OpenOffice.org build was announced. We hope to increase this impressive number with the new download websites.

Happy downloading. :-)

tags: beta build candidate download languagepack openoffice.org qa release snapshot

tags: build gullfoss linux openoffice.org parallelization scalability

... some weeks ago I did blog about cross compiling OOo for ARM, it seemed that others were interested in doing so as well, especially as the available hardware still seems to be somewhat slow, so I do blog again about it :-)

Please find some guidance for cross compiling OOo for ARM below and have fun with it .... I may give details regarding the "afterburner" in one of my next blog postings :-)

And not to forget, many many thanks go to (Jens-)Heiner (Rechtien), who helped tremendously especially with user space qemu and figuring out which patches we needed etc. Thank you very much Heiner :-)

Sincerely yours

   Kay

Cross Compiling OOo for ARM

Key to build OOo for ARM reasonably fast is the user space QEMU, leading to build times of about 24h, which may be pushed to under 5h, if applying some "afterburner" :-). Only left problem so far is with Java technology not successfully being executable in user space QEMU, but in system QEMU only, means that we need both.

Hosting

Ideally you have a fast ARM machine, where you install the distro of choice, and where you build OOo just as usual. Unfortunately I did not had any hardware, so I had to build in emulation.  The host as well as the target environment were both Ubuntu Jaunty Jackalope, this made it easy to improve build times later, after the basic approach did work.

ARM Root File System

The latest Ubuntu (9.04 - Jaunty Jackalope) is available for ARM CPUs. Just follow the instructions here and you get a tarred ARM root file system (e.g. armel-rootfs-200903170957.tgz). Untar that to a convenient location ...

> mkdir /local/jaunty_arm_rfs
> cd /local/jaunty_arm_rfs
> tar -xvzf armel-rootfs-200903170957.tgz

... and have a first look at it:

> file bin/ls
bin/ls: ELF 32-bit LSB executable, ARM, version 1 (SYSV), 
dynamically linked (uses shared libs), for GNU/Linux 2.6.18, stripped

QEMU

"QEMU is a generic and open source machine emulator and virtualizer."

QEMU does not only allow system emulation, but also supports process granularity (user space) emulation. The former means, that a whole computer is emulated, including BIOS, user-land, kernel-land, hardware and everything. The latter means, that a program compiled e.g. for ARM (a "foreign" binary) may seamlessly be executed on a host system, e.g. x86 or x64. Important difference for building OOo in user space emulation and not in system emulation is the execution speed and available memory. Whole system emulation is slow and especially for ARM memory size is limited to 256MB, while user-space (or in-process) emulation is much faster, as it only emulates user land, the kernel and everything else run natively. This is interesting as it allows to mix foreign binaries with native ones, see "afterburner" in my next blog posting.

User Space Qemu

To utilize user space qemu, we need the target distribution as a directory on the host system (a "root file system"), so that we can enter it with "chroot". We need to register user space qemu (e.g. qemu-arm) for the foreign binaries and need to make it available in the root file system, as this is now executable we call that a "chroot environment". Unfortunately the user space qemu from the repository does not work, means we have to build our own, but see below ...

Building and Patching

Get the latest QEMU sources from the QEMU site:

> wget http://download.savannah.gnu.org/releases/qemu/qemu-0.10.3.tar.gz

... unpack it:

> tar -xvzf qemu-0.10.3.tar.gz

Next we need to get the ARM EABI patches from SourceForge and apply them, Heiner helped me with that, actually we had to delete some of the patches from the patch series, as they did not apply or disturbed otherwise:

> tar -xvjf qemu-arm-eabi-0.3.tar.bz2
> mv qemu-arm-eabi-0.3/patches .
> export QUILT_PATCHES=../patches
> cd qemu-0.10.3
> for patch in 02_fix_page_range_check.patch 04_shmat_strace.patch \
  10_signal_jobs.patch 19_zero_null.4.acct.patch 19_zero_null.2.read.patch \
  22_IPCOP_msg.patch 23_msg_syscalls.patch 24_IPCOP_sem.patch 25_sem_syscalls.patch \
  26_IPCOP_shm.patch 27_shm_syscalls.patch 33_fix_getdents_syscalls.patch \
  36_fix_iovec.patch 38_fix_recvmsg_return_value.patch 39_fix_exit_syscall.patch \
  99_sbox_proc.patch ; do quilt delete -r $patch ; done

Unfortunately utimensat has a bug in user space qemu, so we need another patch to fix this:

> cat syscall_utimensat.patch 

--- qemu/linux-user/syscall.c.org	2009-05-08 16:04:11.908901048 +0200
+++ qemu/linux-user/syscall.c	2009-05-08 16:05:49.229947790 +0200
@@ -6203,9 +6203,15 @@
 #if defined(TARGET_NR_utimensat) && defined(__NR_utimensat)
     case TARGET_NR_utimensat:
         {
-            struct timespec ts[2];
-            target_to_host_timespec(ts, arg3);
-            target_to_host_timespec(ts+1, arg3+sizeof(struct target_timespec));
+            struct timespec * ts = NULL;
+
+            if (arg3) {
+              struct timespec ts_[2];
+              ts = ts_;
+              target_to_host_timespec(ts, arg3);
+              target_to_host_timespec(ts+1, arg3+sizeof(struct target_timespec));
+            }
+
             if (!arg2)
                 ret = get_errno(sys_utimensat(arg1, NULL, ts, arg4));
             else {

... add this patch to the quilt patch series:

> quilt import syscall_utimensat.patch

... and apply all patches:

> quilt push -fa

If quilt stops complaining that a refresh is needed, just do

> quilt refresh

.. and start again

> quilt push -fa

.. until it says, that all patches have been applied:

> quilt applied

Next we configure and build the user space qemu ...

> ./configure --target-list=arm-linux-user --static
> make -j 4
> ./arm-linux-user/qemu-arm -v
qemu-arm version 0.10.3, Copyright (c) 2003-2008 Fabrice Bellard
usage: qemu-arm [options] program [arguments...]
Linux CPU emulator (compiled for arm emulation)
... 

Phew!, that was quite some work :-)

Register User Space Qemu

Now we need to register user space qemu ("qemu-arm"), which effectively is the "interpreter" or "CPU" for all ARM binares, to be invoked in case an ARM binary becomes invoked. The following line has been copied from the qemu-binfmt-conf.sh script, which is part of QEMU 0.10.3. Important difference is the appended "C", which ensures that the ARM binaries are executed on behalf of the executing user, respectively taking any suid into account:

> echo ':arm:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-arm:C' \
  >> /proc/sys/fs/binfmt_misc/register

We need to workaround a common problem with user space qemus mmap schema:

> echo 4 > /proc/sys/vm/mmap_min_addr

From now on, we may execute any ARM binary by simply invoking it - unfortunately library resolution and placement has not been designed with mixed binaries in mind, so we likely get an error for non matching libraries if we try so. This should be better in the chroot environment ...

Chroot Environment

Your distribution may bring a user space qemu, though I recommend to build one from scratch (please see above), as we need to patch it slightly. Copy this (ideally statically) linked qemu-arm into your brand new ARM root file system:

> cp qemu-arm /local/jaunty_arm_rfs/usr/bin
copying ..

...  and see what happens when we chroot into it now:

> chroot /local/jaunty_arm_rfs/ /bin/su - ubuntu
> uname -m
armv5tel

... so now we do have a "chroot environment" :-)

System Qemu

To use system qemu, we may create a disk image and everything, which may than be booted by system qemu ... or we try to re-use our chroot environment and to boot a kernel in system qemu with the chroot environment as its root file system, mounted via NFS.

So, let's export the chroot environment via NFS ...

> echo '/local/jaunty_arm_rfs/ *(rw,no_root_squash,async,insecure)' >> /etc/exports
> exportfs -ra
...
> showmount -e localhost
Export list for localhost:
/local/jaunty_arm_rfs      *

... make the network available:

> cp /etc/resolv.conf /local/jaunty_arm_rfs/etc

... install a kernel:

> mount -o bind /proc /local/jaunty_arm_rfs/proc
> chroot /local/jaunty_arm_rfs
> apt-get update
> apt-get install linux-image-2.6.28-11-versatile

... prepare the initrd.img to enable NFS root by replacing "BOOT=local" with "BOOT=nfs" in /etc/initramfs-tools/initramfs.conf and update it:

> update-initramfs -uv

... disable the network manager as it interferes with NFS root:

> apt-get purge network-manager

I did experience problems with gdm, so I disabled it:

> update-rc.d -f gdm remove

Finally boot the kernel in system qemu:

> qemu-system-arm -M versatilepb -m 256 \
  -kernel /local/jaunty_arm_rfs/boot/vmlinuz-2.6.28-11-versatile \
  -initrd /local/jaunty_arm_rfs/boot/initrd.img-2.6.28-11-versatile \
  -append "root=/dev/nfs ip=dhcp nfsroot=10.0.2.2:/local/jaunty_arm_rfs,timeo=14 rw mem=256M"

... and enjoy Ubuntu Jaunty form ARM :-)

Building OOo

Utilizing the chroot environment, OOo is now mostly buildable as usual, just check out the sources and configure them. Problematic is everything relying on Java technology and Mozilla. Java technology currently does not work well in user space qemu, but in system qemu only. You may want to build the related modules in system qemu, or for a first try just disable Java technology support during configuration. The used Mozilla version (1.7.5) is not buildable under ARM, AFAIK this will not change. So, you may either want to use system Mozilla or to disable Mozilla completely. Using the system Mozilla works fine as long as you don't need Address Book integration. This is actually how I did configure my build:

> ./configure --with-system-mozilla --with-openldap --without-java

And not to forget, you certainly need to install the prerequisites, so slow I do recommend doing that in system qemu, as you otherwise likely end up with system daemons running twice, one from the host and one from the chroot environment :^):

> apt-get update
> apt-get dist-upgrade
> apt-get install gnome-devel libcups2-dev tcsh

Caolan McNamara did fix sal/typesconfig/typesconfig.c for ARMel in cmcfixes56, for convenience I give that patch here:

> cat typeconfig.c.patch
Index: typesconfig.c
===================================================================
--- typesconfig.c	(revision 270430)
+++ typesconfig.c	(working copy)
@@ -163,20 +163,37 @@
 |*	Letzte Aenderung
 |*
 *************************************************************************/
-static int dummy(void* unused);
+#if defined(IA64) || defined(ARM32)
 
+int forceerror()
+{
+#if defined(ARM32)
+// workaround for qemu-user
+    hit = 1;
+#else
+    raise (SIGBUS);
+#endif
+    return 1;
+}
+
 int GetAtAddress( Type eT, void* p )
 {
-#if defined(IA64) || defined(ARM32)
   switch ( eT )
   {
   case t_char:		return *((char*)p);
-  case t_short:		if ((long)p % sizeof(short)) abort(); else return *((short*)p);
-  case t_int:		if ((long)p % sizeof(int)) abort(); else return *((int*)p);
-  case t_long:		if ((long)p % sizeof(long)) abort(); else return *((long*)p);
-  case t_double:	if ((long)p % sizeof(double)) abort(); else return *((double*)p);
+  case t_short:		if ((long)p % sizeof(short)) return forceerror(); else return *((short*)p);
+  case t_int:		if ((long)p % sizeof(int)) return forceerror(); else return *((int*)p);
+  case t_long:		if ((long)p % sizeof(long)) return forceerror(); else return *((long*)p);
+  case t_double:	if ((long)p % sizeof(double)) return forceerror(); else return *((double*)p);
   }
+  abort();
+}
+
 #else
+static int dummy(void* unused);
+
+int GetAtAddress( Type eT, void* p )
+{
   switch ( eT )
   {
   case t_char: { char x = *(char*)p; return dummy(&x); }
@@ -185,7 +202,6 @@
   case t_long: { long x = *(long*)p; return dummy(&x); }
   case t_double: { double x = *(double*)p; return dummy(&x); }
   }
-#endif
   abort();
 }
 
@@ -195,6 +211,7 @@
     return 0;
 }
 
+#endif
 /*************************************************************************
 |*
 |*	SetAtAddress()

Property changes on: typesconfig.c
___________________________________________________________________
Added: svn:mergeinfo
   Merged /cws/cmcfixes56/sal/typesconfig/typesconfig.c:r269782-269948

The gcc has problems in ICU, dying with some "internal compiler" error, the workaround is to disable optimization, just apply:

> cat icu_makefile.mk.patch
Index: makefile.mk
===================================================================
--- makefile.mk	(revision 270947)
+++ makefile.mk	(working copy)
@@ -68,7 +68,7 @@
 # add SYSBASE libraries and make certain that they are found *after* the 
 # icu build internal libraries - in case that icu is available in SYSBASE
 # as well
-icu_LDFLAGS+= -L../lib  -L../../lib -L../stubdata -L../../stubdata  -L$(SYSBASE)$/usr$/lib
+icu_LDFLAGS+= -L../../lib -L../../stubdata  -L$(SYSBASE)$/usr$/lib
 .ENDIF			# "$(SYSBASE)"!=""
 
 .IF "$(OS)"=="MACOSX"
@@ -85,8 +85,10 @@
 icu_LDFLAGS+=-Wl,-z,noexecstack
 .ENDIF
 
-icu_CFLAGS+=-O $(ARCH_FLAGS) $(EXTRA_CDEFS)
-icu_CXXFLAGS+=-O $(ARCH_FLAGS) $(EXTRA_CDEFS)
+icu_CFLAGS+= $(ARCH_FLAGS) $(EXTRA_CDEFS)
+#icu_CFLAGS+=-O $(ARCH_FLAGS) $(EXTRA_CDEFS)
+icu_CXXFLAGS+= $(ARCH_FLAGS) $(EXTRA_CDEFS)
+#icu_CXXFLAGS+=-O $(ARCH_FLAGS) $(EXTRA_CDEFS)
 
 BUILD_ACTION_SEP=;
 # remove conversion and transliteration data to reduce binary size.
@@ -116,7 +118,7 @@
 # note the position of the single quotes.
 
 BUILD_DIR=$(CONFIGURE_DIR)
-BUILD_ACTION=$(GNUMAKE)
+BUILD_ACTION=$(GNUMAKE) -j 4
 OUT2LIB= \
 	$(BUILD_DIR)$/lib$/libicudata$(DLLPOST).$(ICU_MAJOR)$(ICU_MINOR).$(ICU_MICRO) \
 	$(BUILD_DIR)$/lib$/libicudata$(DLLPOST).$(ICU_MAJOR)$(ICU_MINOR) \

Another patch is needed for xmlsecurity/source/xmlsec/nss/seinitializer_nssimpl.cxx as it has problems finding an include in case of building --with-system-mozilla:

cat /usr/local/kr93794/xmlsecurity_source_xmlsec_nss.patch 
Index: seinitializer_nssimpl.cxx
===================================================================
--- seinitializer_nssimpl.cxx	(revision 270947)
+++ seinitializer_nssimpl.cxx	(working copy)
@@ -68,7 +68,8 @@
 #include "pk11func.h"
 #ifdef SYSTEM_MOZILLA
 #include "nssrenam.h"
-#include "secmod.h"        
+#include "secmod.h"  
+#include "nss/nss.h"      
 #endif
 #include "cert.h"
 #include "cryptohi.h"

Last but not least we have the problem, that the generated .deb have the wrong architecture, so here is a patch fixing that for the set_soenv.in script, for ARMel and AMD64:

# cat /usr/local/kr93794/set_soenv.in.patch
--- /usr/local/kr93794/Asus/xandros/SO_OOO310_m10/set_soenv.in	2009-04-16 12:45:37.000000000 +0200
+++ /usr/local/kr93794/set_soenv.in	2009-04-28 16:30:06.000000000 +0200
@@ -32,6 +32,19 @@
 use strict;                        # pragma
 use File::Basename;
 
+sub dpkg_architecture {
+  my @lines=split(/\n/,`/usr/bin/dpkg-architecture`);
+
+  my %data;
+  my @name_value;
+
+  foreach (@lines){
+    @name_value=split(/=/,$_);
+    $data{$name_value[0]}=$name_value[1];
+  }
+
+  return %data
+}
 
 #
 #--------------------------------------------------------
@@ -63,7 +76,7 @@
       $GVER, $OS, $OSVERSION, $OUTPATH, $INPATH, $PATH_SEPERATOR,
       $DYNAMIC_CRT, $SET_EXCEPTIONS, $use_shl_versions, $CDPATHx, $JRELIBDIR,
       $JREEXTRALIBDIR, $JRETOOLKITDIR, $JRETHREADDIR,
-      $FLIPCMD );
+      $FLIPCMD, $ABI );
 #
 #-------------------------------------------
 # IIc. Declaring the environment variables.
@@ -373,6 +386,12 @@
       $OS             = "LINUX";
       $PATH_SEPERATOR = $ps;
 
+      if (-e "/etc/debian_version")
+      {
+        my %data=dpkg_architecture();
+        $ABI=$data{"DEB_HOST_ARCH"};
+      }
+
 #Set platform specific values: 
    if ($platform =~ m/^i[3456]86/)
    {  print "Setting Linux x86 specific values... ";
@@ -409,11 +428,20 @@
    }
    elsif ($platform =~ m/^x86_64/)
    {  print "Setting Linux x86-64 specific values... ";
-      $outfile        = "LinuxX86-64Env.Set";
+
       $CPU            = "X";
       $CPUNAME        = "X86_64";
       $CVER           = "C341";
-      $OUTPATH        = "unxlngx6";
+      if ($ABI =~ "") {
+        $outfile        = "LinuxX86-64Env.Set";
+        $OUTPATH        = "unxlngx6";
+      }
+      else {
+        $outfile        = "Linux" . $ABI . "Env.Set";
+        $OUTPATH        = "unxlng" . $ABI;
+        $EPM_FLAGS      = "-a $ABI";
+      }
+
       # Blackdown.org JDK porting project uses `amd64' and `server' in JDK 1.4.2 RC1
       $JRELIBDIR      = '$JAVA_HOME'.$ds."jre".$ds."lib".$ds."amd64";
       # has both server and client
@@ -533,15 +561,24 @@
    }
    elsif ($platform =~ m/^arm.*?l-/)
    {  print "Setting Linux ARM specific values... ";
-      $outfile        = "LinuxARMEnv.Set"; 
+
+      if ($ABI =~ "") {
+        $ABI="arm";
+        $outfile = "LinuxARMEnv.Set"; 
+        $OUTPATH = "unxlngr";
+      }
+      else {
+        $outfile = "Linux" . $ABI . "Env.Set"; 
+        $OUTPATH = "unxlng" . $ABI;
+      }
+
       $CPU            = "R";
       $CPUNAME        = "ARM";
-      $OUTPATH        = "unxlngr";
       $JRELIBDIR      = '$JAVA_HOME'.$ds."jre".$ds."lib".$ds."arm";
       $JRETOOLKITDIR  = '$JAVA_HOME'.$ds."jre".$ds."lib".$ds."arm".$ds."server";
       $JRETHREADDIR   = '$JAVA_HOME'.$ds."jre".$ds."lib".$ds."arm".$ds."native_threads";
       $JREEXTRALIBDIR = '$JAVA_HOME'.$ds."jre".$ds."lib".$ds."arm".$ds."xawt";
-      $EPM_FLAGS      = "-a arm";
+      $EPM_FLAGS      = "-a $ABI";
    }
    elsif ($platform =~ m/^mips/)
    {  print "Setting Linux MIPS specific values... ";

After some clean ups I am going to put that into CWS as soon as I find the time ... that's it for now :-)

tags: arm build building compile compiling cross openoffice openoffice.org qemu quilt

Hi OOo Folks,

some of you may have noticed that ARM is en vogue, though analysts deeply disagree over it's chances e.g. in the netbook market.

Nevertheless I thought it would be a good idea to see how OOo behaves and actually builds on ARM. Unfortunately, with the exception of my iPod Touch, which I received as price from Sun Learning eXchange for this little video about ODF@WWW, and by the way Dmitri Popov just reminded me to continue on that, I didn't have any hardware to build or run it on.

So suddenly I entered the wonderful world of QEMU. QEMU is an emulator, which can, amongst many other architectures, emulate various ARM CPUs and systems. While looking at it I stumbled over a really nice and unique feature, namely the QEMU user space emulator. In user space emulation, QEMU does not emulate a whole system, but allows to execute an alien binary (such as a binary build for ARM) under the host (e.g. x64) directly, dynamically mapping system calls etc. to the hosts architecture, avoiding a bigger part of emulation, which is typically done by whole system emulation.

I already did compile OOo on various flavours of Linux, Mac OS X and on Windows, using different CPUs etc., such as x86, x64 and PPC, but I never did cross compile it (though the term emu(-lator) compile may be more appropriate :-). Still I felt well equipped.

Days (or weeks?) later I actually can report success, just an hour ago I finished my first cross compilation of OOo for ARM CPUs successfully, and surprisingly the installed application seems to work as expected :-)

Initially I planned to utilize the QEMU system mode, were building should just have been like any native build ... unfortunately it was unbearable slow, so I switched to user space mode, which had it's own pitfalls, but which was by factors faster.

Building OOo with QEMU in system mode seemed to take weeks, while utilizing user space mode in principle allowed to finish in about 24 hours, I think there is potential to even push that down to may be 8 hours. By the way, the machine I used had 4 cores.

If there is interest, I may post more details regarding the QEMU user space build, and how I think it can be accelerated further.

Have a nice weekend

Kay

tags: arm build building compile compiling cross openoffice openoffice.org qemu

Have you ever been CONFUSED by creating make files? Have you ever WONDERED which files had to be rebuild because of changes in make files or header files? Have you ever thought about how CONCURRENT your make files actually are? Have you ever been disturbed by DISTRIBUTING IMPLEMENTATION DETAILS about a particular .c- or .h-file over multiple makefiles? Have you ever thought how concurrency could really work while invoking make MULTIPLE (or many) times and how that puts a lot of OVERHEAD into a build process? Have you ever found it hard to MODULARIZE your make files? Have you ever QUESTIONED the sense of make files in general?

THAN THIS BLOG POSTING IS FOR you  :-)

Warning: When coming down to creating, maintaining  or compiling stuff, I know I am different, so be warned  :*) I did briefly explain to others how I felt things should work ... and did hear from some that I am odd, while others seemed to like it ... anyway, I do believe in and successfully tried the below approach in my home projects and am currently playing with it while experimenting with packaging ... but judge yourself :-)

A Start: Let's start with simple things, in my little world I mostly have some .c- and .h-files, while I typically want to create some .o-, .bin- and .so- respectively lib*.so-files. Some files need special flags, e.g. for optimization or debugging. To understand how we can automate this - thus abandoning the makefile burden ideally completely - we may want to take a look at a typical C include.

A simple include looks like this:

foo.c:
    #include "a.h"

Compiling: For the corresponding .o-file (foo.o) this means, that it needs to be remade in case foo.c, a.h or any of the transitive includes of a.h changes.

Linking: To be able to link foo.o into an executable or shared library, we also need to link-in any objects or shared libraries implementing a.h (or its includes respectively).

Sample Code: Let's say, a.h only declares what's implemented in a.c, such that a.h does not include anything else. Unfortunately a.c includes (despite a.h) b.h as well as x.h .

a.c:
    #include "a.h"
    #include "b.h"
    #include "x.h"

Luckily, b.h does not include anything itself, but b.c does, additionally to b.h it also includes c.h, which is implemented in c.c:

b.c:
    #include "b.h"
    #include "c.h"

c.c:
    #include "c.h"

And if this had not been enough, a.c respectively a.o does also need the implementation of x.h to be properly linkable, which is not directly implemented in a .o-file but a linkable shared library named libx.so.

Looking into x.c

x.c:
    #include "x.h"
    #include "y.h"
    #include "d.h"

we see, that itself needs another library (liby.so) and at least one other .o-file (d.o). And foo.bin is not the only executable we would like to link, but bar.bin as well. bar.bin is partly based on the same objects as foo.bin, but links differently, I leave the details out here, please have a look at the bottom for all sample files.

And it's Makefile: Expressing this in GNU make looks like this:

============= handcraft.mk =============:
LDFLAGS+=-Wl,-rpath='$$ORIGIN/' -L.

a.o: a.c a.h b.h c.h x.h
	$(COMPILE.c) $(OUTPUT_OPTION) $<

b.o: b.c b.h c.h 
	$(COMPILE.c) $(OUTPUT_OPTION) $<

c.o: c.c c.h
	$(COMPILE.c) $(OUTPUT_OPTION) $<

x.o: x.c x.h y.h d.h
	$(COMPILE.c) $(OUTPUT_OPTION) $<

y.o: y.c y.h
	$(COMPILE.c) $(OUTPUT_OPTION) $<

libx.so: x.o liby.so
	$(LINK.o) $(LOADLIBES) $(LDLIBS) -shared $^ -L. -ly -o $@

liby.so: y.o
	$(LINK.o) $(LOADLIBES) $(LDLIBS) -shared $^ -o $@

foo.o: foo.c a.h
	$(COMPILE.c) $(OUTPUT_OPTION) $<

foo.bin: foo.o a.o b.o c.o libx.so
	$(LINK.o) $(LOADLIBES) $(LDLIBS) $^ -o $@

bar.o: bar.c b.h c.h d.h
	$(COMPILE.c) $(OUTPUT_OPTION) $<

bar.bin: bar.o b.o c.o d.o
	$(LINK.o) $(LOADLIBES) $(LDLIBS) $^ -o $@

Now we can build the concrete targets by invoking make as:

  > make -f handcraft.mk foo.bin
  > make -f handcraft.mk bar.o
  > make -f handcraft.mk libx.so
  > make -f handcraft.mk -j foo.bin bar.bin
  > make -f handcraft.mk ...

Automation: By increasing the number of targets and prerequisites such a make file can become pretty fast pretty complicated - after all building software properly is not an easy task.

Automatic dependency generation for include files is well known and has been done for long, what's missing to become fully automatic are compilation flags in particular and linking support in general. The point is, that only the moment building becomes fully automatic, make can be modularized reasonably and we can build a whole project invoking make once only (which is desirable :-).

So, let's see what information regarding linking has been implicitly expressed in the above makefile:

• If you include a.h, than you need to link-in a.o as well.
• If you link-in a.o, than you need to link-in b.o and c.o as well, and you have to link against libx.so .
• If you ...
  

The Rule: In principle, the rule seems to be: If you include a header, ensure that you link-in the implementing .o-files and shared libraries, as well as the required .o-files and shared libraries of these .o-files etc.

Cohesion: The hints regarding required .o-files, shared libraries and c-flags should be placed in the appropriate files and need to be available for make during building, otherwise this information gets repeated in multiple targets and makefiles again and again - which to my experience is typical ;*)

Concrete: If we have a header file which requires a particular object file to be linked-in, we can (initially hackily :-) express this directly in-line, prefixing it with a "magic" word, e.g. as:

//§ req_objs:=a.o

We may want to do the same with libraries:

//§ req_libs:=libx.so

And even compilation flags may be expressed this way:

//§ cflags:=-g

Applying the schema to the above files, we get the following additional lines:

foo.c:
  //§ cflags=-g

a.h:
  //§ req_objs:=a.o

b.h:
  //§ req_objs:=b.o

x.h:
  //§ req_libs=libx.so

...

Now, the only think left is to write some generic make "modules" to utilize this now explicit available information ... this is so obvious, that I leave that as a homework for the patient reader ... just joking :-) please have a look below.

Conclusion: By now we have developed a simple schema for linking binaries and shared libraries based on C files, which allows us to build any final or intermediate target fully automatically, expressing linking relevant information in a cohesive way, while eliminating the need to hassle with make-files ever again.

Scalability:  I have to admit that I did not yet test how scalable this approach is, in principle it should scale quite well ... and after all we are only talking about some thousand files for typical projects. At least for my home projects with some hundred files it works quite nicely.

Reasoning: The reason I brought this up is, that I believe that anything can be build fully automatically this way - even installation packages I am currently experimenting with.

In Principle: The overall rule set to build any kind of derived file fully automatically (and not only compiling C sources :-), seems to be something along the following lines:

1. We have prerequisite files and derived files.
2. Every derivation has exactly one primary prerequisite and n secondary prerequisites.
3. A derivation explicitly depends on its primary prerequisite.
4. A derivation implicitly depends on its secondary prerequisites, which need to be determined during runtime out of the primary prerequisite.
5. A prerequisite may specify transformation flags.
6. For every type of prerequisite-to-derivation transformation exactly one explicit rule is defined.
7. For any type of derivation there may be n+ explicit rules.
   

Note on -7-: As a derivation may not depend on its primary prerequisite only, in practice some functions need to be executed at runtime to determine all secondary prerequisites. Actually, the concrete function to be calculated depends on the actual explicit rule selected for the creation of the derivation. As ordinary makes (e.g. GNU make), do not offer a way to add prerequisites to a derivation after an explicit rule has been selected, statement -7- has to be adapted for these to work:

-7- For any type of derivation there is exactly one explicit rule.

Therefore, to still enable deriving one type from different types of prerequisites, we need to mangle in the prerequisites type.

Finally: If there is interest and people think that this general approach seems to be reasonable, I am going to explain in more detail how it actually works and what else could be done to make a builders life easier ... and I may even try it out on a concrete OOo module, may be SAL :-)

Last but not least I have to admit, that I am a descent GNU make fan :-)

Please find sample files etc.  below.

Best regards

Kay

Samples and make files for GNU make, tried on Debian Linux, just copy&paste the stuff into the named files ...

============= foo.c =============:
//§ cflags:=-g

#include "a.h"

int main(void) {
  return 0;
}

============= bar.c =============:
//§ cflags:=-O

#include "b.h"
#include "c.h"
#include "d.h"

int main(void) {
  return 0;
}

============= a.h =============:
//§ req_objs=a.o

============= a.c =============:
#include "a.h"
#include "b.h"
#include "x.h"

============= b.h =============:
//§ req_objs=b.o

============= b.c =============:
#include "b.h"
#include "c.h"

============= c.h =============:
//§ req_objs:=c.o

============= c.c =============:
#include "c.h"

============= d.h =============:
//§ req_objs=d.o

============= d.c =============:
#include "d.h"

============= x.h =============:
//§ req_libs=libx.so

============= x.c =============:
#include "x.h"
#include "y.h"
#include "d.h"

============= y.h =============:
//§ req_libs:=liby.so

============= y.c =============:
#include "y.h"

============= generic.mk =============:
# The master makefile.

# Preserve intermediate files.
.PRECIOUS: %.o

# To find linked libraries relatively.
LDFLAGS+=-Wl,-rpath='$$ORIGIN/' -L.

include functions.mk   # Some helpers.

include info_c.mk      # Create C infos.
include o_c.mk         # Create C objects.
include bin_o.mk       # Create object binaries.
include libso_o.mk     # Create object shared libraries.

$(foreach target,$(MAKECMDGOALS),$(call $(suffix $(target))_spreqs,$(target)))

============= functions.mk =============:
# Guard includes.
define ginc_t
  ifndef $(1)_inc_def
    $(1)_inc_def=:1 

    -include $(1) 
  endif
endef

ginc=$(eval $(ginc_t))

# Calculate the referential transitive closure.
define closure_t
  ifndef $(1)$(2)_cls_def
    $(1)$(2)_cls_def:=1

    ifdef $(suffix $(1))_info
      $$(call $(suffix $(1))_info,$(1))
    else
      $$(call ginc,$(1).info) 
    endif

    $(1)$(2)_cls:=$$(strip $$($(1)$(2)) $$(foreach file,$$($(1)$(2)),$$(call closure,$$(file),$(2))))
  endif
endef

# $(1) = file name
# $(2) = variable postfix
closure=$(eval $(closure_t)) $($(1)$(2)_cls)

============= info_c.mk =============:
# Rules and prerequisits for .info files .

define info_script
  echo Making $@
  echo "`grep ^//§ $<|sed s0//§\ 0$<_0g)`" > $@
  echo "$<_includes:=`echo \`grep \#include $<|sed s0\#include00g|sed s0\\\"00g\``" >> $@
endef

%.info: %
	@$(info_script)

============= o_c.mk =============:
# Rules and prerequisits for .o files, as well as .o.info .

define t_o_info
  ifndef $(1)_info_def
    $(1)_info_def:=1

    t_o_info_ppreq_$(1):=$(1:.o=.c)

    $$(call ginc,$$(t_o_info_ppreq_$(1)).info)

    t_o_info_all_includes:=$$(call closure,$$(t_o_info_ppreq_$(1)),_includes)
    $(1)_req_objs:=$$(subst $(1),,$$(foreach inc,$$(t_o_info_all_includes),$$($$(inc)_req_objs)))
    $(1)_req_libs:=$$(foreach inc,$$(t_o_info_all_includes),$$($$(inc)_req_libs))
  endif
endef

.o_info= $(eval $(t_o_info)) 


define t_o_spreqs
  ifndef t_o_spreqs_$(1)_def
    t_o_spreqs_$(1)_def:=1

    t_o_spreqs_ppreq_$(1):=$(1:.o=.c)

    $(1): $$(call closure,$$(t_o_spreqs_ppreq_$(1)),_includes)
    $(1): cflags_:=$$($$(t_o_spreqs_ppreq_$(1))_cflags)
  endif
endef

.o_spreqs=$(eval $(t_o_spreqs))



%.o: %.c
	$(COMPILE.c) $(OUTPUT_OPTION) $(cflags_) $<

============= bin_o.mk =============:
# Rules and prerequisits for .bin files .

define t_bin_spreqs
  ifndef t_bin_spreqs_$(1)_def
    t_bin_spreqs_$(1)_def:=1
    t_bin_spreqs_$(1)_ppreq:=$(1:.bin=.o)

    $$(call .o_spreqs,$$(t_bin_spreqs_$(1)_ppreq))

    t_bin_spreqs_$(1)_objs:=$$(call closure,$$(t_bin_spreqs_$(1)_ppreq),_req_objs)
    $$(foreach obj,$$(t_bin_spreqs_$(1)_objs),$$(call .o_spreqs,$$(obj)))

    t_bin_spreqs_$(1)_libs:=$$(subst $(1),,$$(foreach obj,$$(t_bin_spreqs_$(1)_ppreq) $$(t_bin_spreqs_$(1)_objs),$$($$(obj)_req_libs)))
    $$(foreach lib,$$(t_bin_spreqs_$(1)_libs),$$(call .so_spreqs,$$(lib)))

    $(1): objs_:=$$(t_bin_spreqs_$(1)_objs)
    $(1): libs_:=$$(t_bin_spreqs_$(1)_libs)
    $(1): $$(t_bin_spreqs_$(1)_objs)
    $(1): $$(t_bin_spreqs_$(1)_libs)
  endif
endef

.bin_spreqs=$(eval $(t_bin_spreqs))

%.bin: %.o
	$(LINK.o) $(LOADLIBES) $(LDLIBS) $< $(objs_) $(patsubst lib%.so,-l%,$(libs_)) -o $@

============= libso_o.mk =============:
# Rules and prerequisits for lib*.so files .

define t_so_spreqs
  ifndef t_bin_spreqs_$(1)_def
    t_so_spreqs_$(1)_def:=1
    t_so_spreqs_$(1)_ppreq:=$(patsubst lib%.so,%.o,$(1))

    $$(call .o_spreqs,$$(t_so_spreqs_$(1)_ppreq))

    t_so_spreqs_$(1)_objs:=$$(call closure,$$(t_so_spreqs_$(1)_ppreq),_req_objs)
    $$(foreach obj,$$(t_so_spreqs_$(1)_objs),$$(call .o_spreqs,$$(obj)))

    t_so_spreqs_$(1)_libs:=$$(subst $(1),,$$(foreach obj,$$(t_so_spreqs_$(1)_ppreq) $$(t_so_spreqs_$(1)_objs),$$($$(obj)_req_libs)))
    $$(foreach lib,$$(t_so_spreqs_$(1)_libs),$$(call .so_spreqs,$$(lib)))

    $(1): objs_:=$$(t_so_spreqs_$(1)_objs)
    $(1): libs_:=$$(t_so_spreqs_$(1)_libs)
    $(1): $$(t_so_spreqs_$(1)_objs)
    $(1): $$(t_so_spreqs_$(1)_libs)
  endif
endef

.so_spreqs=$(eval $(t_so_spreqs))

lib%.so: %.o
	$(LINK.o) $(LOADLIBES) $(LDLIBS) -shared $< $(objs_) $(patsubst lib%.so,-l%,$(libs_)) -o $@

> make -f generic.mk foo.bin
> make -f generic.mk bar.o
> make -f generic.mk libx.so
> make -f generic.mk -j foo.bin bar.bin
> make -f generic.mk ...

tags: automatic build building gnumake make openoffice openoffice.org packaging

With a new major release of OpenOffice.org we finally get the opportunity to update some of our build tools. Especially compilers could use some refreshing, the default ones used by Sun OOo release engineering date from 2003/2004. A pretty complete list of the compilers used for production builds of OOo for the major platforms and distributions can be found here.

The planned changes in detail:

• Solaris SPARC, Solaris x86: We (Sun OOo release engineering) plan to switch to SunStudio 12 (09/07). This release has better support for boost, which is becoming more prominent in our code base. More important, SunStudio 12 is freely available (registration required) and thus is already the compiler of choice for most Solaris developers. It looks like we can safely drop support for older SunStudio compilers after the switch, please notify me if this is not the case.
• Linux: Most likely the current gcc-4.2.2 will be the future default Linux compiler for Sun OOo developers. We will still support older compilers, way back to gcc-3.3.x, they are needed for MacOSX 10.3 and OS/2. As soon as these two platforms have upgraded to gcc-4.x we will be able to drop support for gcc-3.x completely as everyone else seems to use gcc-4.x.
• Windows: Most probably we will switch to .Net 2008 once it is released, for now we'll use .Net 2005. We will still support .Net 2003 for a while to protect the investment some OOo developers have made into this compiler, but this can't last forever.
  

tags: build compiler openoffice.org

With milestone SRC680 m218 SunStudio 12 enters the list of supported compilers for building OOo. Actually the only change needed was to add a SunStudio 12 test to configure. This is amazing, considering the amount of new features and changes which went into the latest SunStudio version on one hand and the diversity of the OOo code base on the other hand.

Currently supported compilers:

• Solaris (sparc, x86)
  
• SunStudio: all versions from SunStudio 8 up to SunStudio 12
• Linux (x86, sparc, ppc)
  
• gcc: all versions from gcc-3.3.x up to gcc-4.2.0
• Linux (x86_64)
• gcc: gcc-4.0.1 up to gcc-4.2.0
  
• Windows
• Microsoft Visual Studio .NET 2003 and 2005
• Microsoft Visual Studio Express 2005
• mingw-gcc: gcc-3.4.4
• MacOSX (ppc)
  
• gcc-3.3
• MacOSX (intel)
  
• gcc-4.0.1
  
• FreeBSD
  
• gcc: all versions from gcc-3.3.x up to gcc-4.2.0
• OS/2
• gcc-3.3.5

Not all platform/compiler combinations are tested on a regular basis so some build breakage might still occur. If this happens, release engineering will gladly accept reasonable patches to get things going again.

Two platforms - MacOSX ppc and OS/2 - depend on gcc-3.3.x support, otherwise we would drop gcc-3.3.x from the list. This version exhibits a long standing bug in the old gcc parser which is triggered very frequently in the OOo code base. The new parser which was introduced with gcc-3.4 does not exhibit this bug. For new ports, gcc-3.3.x should be avoided.

tags: build compiler openoffice.org tools

tags: build openoffice.org tools windows