... some weeks ago I did blog about cross compiling OOo for ARM, it seemed that others were interested in doing so as well, especially as the available hardware still seems to be somewhat slow, so I do blog again about it :-)

Please find some guidance for cross compiling OOo for ARM below and have fun with it .... I may give details regarding the "afterburner" in one of my next blog postings :-)

And not to forget, many many thanks go to (Jens-)Heiner (Rechtien), who helped tremendously especially with user space qemu and figuring out which patches we needed etc. Thank you very much Heiner :-)

Sincerely yours

   Kay

Cross Compiling OOo for ARM

Key to build OOo for ARM reasonably fast is the user space QEMU, leading to build times of about 24h, which may be pushed to under 5h, if applying some "afterburner" :-). Only left problem so far is with Java technology not successfully being executable in user space QEMU, but in system QEMU only, means that we need both.

Hosting

Ideally you have a fast ARM machine, where you install the distro of choice, and where you build OOo just as usual. Unfortunately I did not had any hardware, so I had to build in emulation.  The host as well as the target environment were both Ubuntu Jaunty Jackalope, this made it easy to improve build times later, after the basic approach did work.

ARM Root File System

The latest Ubuntu (9.04 - Jaunty Jackalope) is available for ARM CPUs. Just follow the instructions here and you get a tarred ARM root file system (e.g. armel-rootfs-200903170957.tgz). Untar that to a convenient location ...

> mkdir /local/jaunty_arm_rfs
> cd /local/jaunty_arm_rfs
> tar -xvzf armel-rootfs-200903170957.tgz

... and have a first look at it:

> file bin/ls
bin/ls: ELF 32-bit LSB executable, ARM, version 1 (SYSV), 
dynamically linked (uses shared libs), for GNU/Linux 2.6.18, stripped

QEMU

"QEMU is a generic and open source machine emulator and virtualizer."

QEMU does not only allow system emulation, but also supports process granularity (user space) emulation. The former means, that a whole computer is emulated, including BIOS, user-land, kernel-land, hardware and everything. The latter means, that a program compiled e.g. for ARM (a "foreign" binary) may seamlessly be executed on a host system, e.g. x86 or x64. Important difference for building OOo in user space emulation and not in system emulation is the execution speed and available memory. Whole system emulation is slow and especially for ARM memory size is limited to 256MB, while user-space (or in-process) emulation is much faster, as it only emulates user land, the kernel and everything else run natively. This is interesting as it allows to mix foreign binaries with native ones, see "afterburner" in my next blog posting.

User Space Qemu

To utilize user space qemu, we need the target distribution as a directory on the host system (a "root file system"), so that we can enter it with "chroot". We need to register user space qemu (e.g. qemu-arm) for the foreign binaries and need to make it available in the root file system, as this is now executable we call that a "chroot environment". Unfortunately the user space qemu from the repository does not work, means we have to build our own, but see below ...

Building and Patching

Get the latest QEMU sources from the QEMU site:

> wget http://download.savannah.gnu.org/releases/qemu/qemu-0.10.3.tar.gz

... unpack it:

> tar -xvzf qemu-0.10.3.tar.gz

Next we need to get the ARM EABI patches from SourceForge and apply them, Heiner helped me with that, actually we had to delete some of the patches from the patch series, as they did not apply or disturbed otherwise:

> tar -xvjf qemu-arm-eabi-0.3.tar.bz2
> mv qemu-arm-eabi-0.3/patches .
> export QUILT_PATCHES=../patches
> cd qemu-0.10.3
> for patch in 02_fix_page_range_check.patch 04_shmat_strace.patch \
  10_signal_jobs.patch 19_zero_null.4.acct.patch 19_zero_null.2.read.patch \
  22_IPCOP_msg.patch 23_msg_syscalls.patch 24_IPCOP_sem.patch 25_sem_syscalls.patch \
  26_IPCOP_shm.patch 27_shm_syscalls.patch 33_fix_getdents_syscalls.patch \
  36_fix_iovec.patch 38_fix_recvmsg_return_value.patch 39_fix_exit_syscall.patch \
  99_sbox_proc.patch ; do quilt delete -r $patch ; done

Unfortunately utimensat has a bug in user space qemu, so we need another patch to fix this:

> cat syscall_utimensat.patch 

--- qemu/linux-user/syscall.c.org	2009-05-08 16:04:11.908901048 +0200
+++ qemu/linux-user/syscall.c	2009-05-08 16:05:49.229947790 +0200
@@ -6203,9 +6203,15 @@
 #if defined(TARGET_NR_utimensat) && defined(__NR_utimensat)
     case TARGET_NR_utimensat:
         {
-            struct timespec ts[2];
-            target_to_host_timespec(ts, arg3);
-            target_to_host_timespec(ts+1, arg3+sizeof(struct target_timespec));
+            struct timespec * ts = NULL;
+
+            if (arg3) {
+              struct timespec ts_[2];
+              ts = ts_;
+              target_to_host_timespec(ts, arg3);
+              target_to_host_timespec(ts+1, arg3+sizeof(struct target_timespec));
+            }
+
             if (!arg2)
                 ret = get_errno(sys_utimensat(arg1, NULL, ts, arg4));
             else {

... add this patch to the quilt patch series:

> quilt import syscall_utimensat.patch

... and apply all patches:

> quilt push -fa

If quilt stops complaining that a refresh is needed, just do

> quilt refresh

.. and start again

> quilt push -fa

.. until it says, that all patches have been applied:

> quilt applied

Next we configure and build the user space qemu ...

> ./configure --target-list=arm-linux-user --static
> make -j 4
> ./arm-linux-user/qemu-arm -v
qemu-arm version 0.10.3, Copyright (c) 2003-2008 Fabrice Bellard
usage: qemu-arm [options] program [arguments...]
Linux CPU emulator (compiled for arm emulation)
... 

Phew!, that was quite some work :-)

Register User Space Qemu

Now we need to register user space qemu ("qemu-arm"), which effectively is the "interpreter" or "CPU" for all ARM binares, to be invoked in case an ARM binary becomes invoked. The following line has been copied from the qemu-binfmt-conf.sh script, which is part of QEMU 0.10.3. Important difference is the appended "C", which ensures that the ARM binaries are executed on behalf of the executing user, respectively taking any suid into account:

> echo ':arm:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-arm:C' \
  >> /proc/sys/fs/binfmt_misc/register

We need to workaround a common problem with user space qemus mmap schema:

> echo 4 > /proc/sys/vm/mmap_min_addr

From now on, we may execute any ARM binary by simply invoking it - unfortunately library resolution and placement has not been designed with mixed binaries in mind, so we likely get an error for non matching libraries if we try so. This should be better in the chroot environment ...

Chroot Environment

Your distribution may bring a user space qemu, though I recommend to build one from scratch (please see above), as we need to patch it slightly. Copy this (ideally statically) linked qemu-arm into your brand new ARM root file system:

> cp qemu-arm /local/jaunty_arm_rfs/usr/bin
copying ..

...  and see what happens when we chroot into it now:

> chroot /local/jaunty_arm_rfs/ /bin/su - ubuntu
> uname -m
armv5tel

... so now we do have a "chroot environment" :-)

System Qemu

To use system qemu, we may create a disk image and everything, which may than be booted by system qemu ... or we try to re-use our chroot environment and to boot a kernel in system qemu with the chroot environment as its root file system, mounted via NFS.

So, let's export the chroot environment via NFS ...

> echo '/local/jaunty_arm_rfs/ *(rw,no_root_squash,async,insecure)' >> /etc/exports
> exportfs -ra
...
> showmount -e localhost
Export list for localhost:
/local/jaunty_arm_rfs      *

... make the network available:

> cp /etc/resolv.conf /local/jaunty_arm_rfs/etc

... install a kernel:

> mount -o bind /proc /local/jaunty_arm_rfs/proc
> chroot /local/jaunty_arm_rfs
> apt-get update
> apt-get install linux-image-2.6.28-11-versatile

... prepare the initrd.img to enable NFS root by replacing "BOOT=local" with "BOOT=nfs" in /etc/initramfs-tools/initramfs.conf and update it:

> update-initramfs -uv

... disable the network manager as it interferes with NFS root:

> apt-get purge network-manager

I did experience problems with gdm, so I disabled it:

> update-rc.d -f gdm remove

Finally boot the kernel in system qemu:

> qemu-system-arm -M versatilepb -m 256 \
  -kernel /local/jaunty_arm_rfs/boot/vmlinuz-2.6.28-11-versatile \
  -initrd /local/jaunty_arm_rfs/boot/initrd.img-2.6.28-11-versatile \
  -append "root=/dev/nfs ip=dhcp nfsroot=10.0.2.2:/local/jaunty_arm_rfs,timeo=14 rw mem=256M"

... and enjoy Ubuntu Jaunty form ARM :-)

Building OOo

Utilizing the chroot environment, OOo is now mostly buildable as usual, just check out the sources and configure them. Problematic is everything relying on Java technology and Mozilla. Java technology currently does not work well in user space qemu, but in system qemu only. You may want to build the related modules in system qemu, or for a first try just disable Java technology support during configuration. The used Mozilla version (1.7.5) is not buildable under ARM, AFAIK this will not change. So, you may either want to use system Mozilla or to disable Mozilla completely. Using the system Mozilla works fine as long as you don't need Address Book integration. This is actually how I did configure my build:

> ./configure --with-system-mozilla --with-openldap --without-java

And not to forget, you certainly need to install the prerequisites, so slow I do recommend doing that in system qemu, as you otherwise likely end up with system daemons running twice, one from the host and one from the chroot environment :^):

> apt-get update
> apt-get dist-upgrade
> apt-get install gnome-devel libcups2-dev tcsh

Caolan McNamara did fix sal/typesconfig/typesconfig.c for ARMel in cmcfixes56, for convenience I give that patch here:

> cat typeconfig.c.patch
Index: typesconfig.c
===================================================================
--- typesconfig.c	(revision 270430)
+++ typesconfig.c	(working copy)
@@ -163,20 +163,37 @@
 |*	Letzte Aenderung
 |*
 *************************************************************************/
-static int dummy(void* unused);
+#if defined(IA64) || defined(ARM32)
 
+int forceerror()
+{
+#if defined(ARM32)
+// workaround for qemu-user
+    hit = 1;
+#else
+    raise (SIGBUS);
+#endif
+    return 1;
+}
+
 int GetAtAddress( Type eT, void* p )
 {
-#if defined(IA64) || defined(ARM32)
   switch ( eT )
   {
   case t_char:		return *((char*)p);
-  case t_short:		if ((long)p % sizeof(short)) abort(); else return *((short*)p);
-  case t_int:		if ((long)p % sizeof(int)) abort(); else return *((int*)p);
-  case t_long:		if ((long)p % sizeof(long)) abort(); else return *((long*)p);
-  case t_double:	if ((long)p % sizeof(double)) abort(); else return *((double*)p);
+  case t_short:		if ((long)p % sizeof(short)) return forceerror(); else return *((short*)p);
+  case t_int:		if ((long)p % sizeof(int)) return forceerror(); else return *((int*)p);
+  case t_long:		if ((long)p % sizeof(long)) return forceerror(); else return *((long*)p);
+  case t_double:	if ((long)p % sizeof(double)) return forceerror(); else return *((double*)p);
   }
+  abort();
+}
+
 #else
+static int dummy(void* unused);
+
+int GetAtAddress( Type eT, void* p )
+{
   switch ( eT )
   {
   case t_char: { char x = *(char*)p; return dummy(&x); }
@@ -185,7 +202,6 @@
   case t_long: { long x = *(long*)p; return dummy(&x); }
   case t_double: { double x = *(double*)p; return dummy(&x); }
   }
-#endif
   abort();
 }
 
@@ -195,6 +211,7 @@
     return 0;
 }
 
+#endif
 /*************************************************************************
 |*
 |*	SetAtAddress()

Property changes on: typesconfig.c
___________________________________________________________________
Added: svn:mergeinfo
   Merged /cws/cmcfixes56/sal/typesconfig/typesconfig.c:r269782-269948

The gcc has problems in ICU, dying with some "internal compiler" error, the workaround is to disable optimization, just apply:

> cat icu_makefile.mk.patch
Index: makefile.mk
===================================================================
--- makefile.mk	(revision 270947)
+++ makefile.mk	(working copy)
@@ -68,7 +68,7 @@
 # add SYSBASE libraries and make certain that they are found *after* the 
 # icu build internal libraries - in case that icu is available in SYSBASE
 # as well
-icu_LDFLAGS+= -L../lib  -L../../lib -L../stubdata -L../../stubdata  -L$(SYSBASE)$/usr$/lib
+icu_LDFLAGS+= -L../../lib -L../../stubdata  -L$(SYSBASE)$/usr$/lib
 .ENDIF			# "$(SYSBASE)"!=""
 
 .IF "$(OS)"=="MACOSX"
@@ -85,8 +85,10 @@
 icu_LDFLAGS+=-Wl,-z,noexecstack
 .ENDIF
 
-icu_CFLAGS+=-O $(ARCH_FLAGS) $(EXTRA_CDEFS)
-icu_CXXFLAGS+=-O $(ARCH_FLAGS) $(EXTRA_CDEFS)
+icu_CFLAGS+= $(ARCH_FLAGS) $(EXTRA_CDEFS)
+#icu_CFLAGS+=-O $(ARCH_FLAGS) $(EXTRA_CDEFS)
+icu_CXXFLAGS+= $(ARCH_FLAGS) $(EXTRA_CDEFS)
+#icu_CXXFLAGS+=-O $(ARCH_FLAGS) $(EXTRA_CDEFS)
 
 BUILD_ACTION_SEP=;
 # remove conversion and transliteration data to reduce binary size.
@@ -116,7 +118,7 @@
 # note the position of the single quotes.
 
 BUILD_DIR=$(CONFIGURE_DIR)
-BUILD_ACTION=$(GNUMAKE)
+BUILD_ACTION=$(GNUMAKE) -j 4
 OUT2LIB= \
 	$(BUILD_DIR)$/lib$/libicudata$(DLLPOST).$(ICU_MAJOR)$(ICU_MINOR).$(ICU_MICRO) \
 	$(BUILD_DIR)$/lib$/libicudata$(DLLPOST).$(ICU_MAJOR)$(ICU_MINOR) \

Another patch is needed for xmlsecurity/source/xmlsec/nss/seinitializer_nssimpl.cxx as it has problems finding an include in case of building --with-system-mozilla:

cat /usr/local/kr93794/xmlsecurity_source_xmlsec_nss.patch 
Index: seinitializer_nssimpl.cxx
===================================================================
--- seinitializer_nssimpl.cxx	(revision 270947)
+++ seinitializer_nssimpl.cxx	(working copy)
@@ -68,7 +68,8 @@
 #include "pk11func.h"
 #ifdef SYSTEM_MOZILLA
 #include "nssrenam.h"
-#include "secmod.h"        
+#include "secmod.h"  
+#include "nss/nss.h"      
 #endif
 #include "cert.h"
 #include "cryptohi.h"

Last but not least we have the problem, that the generated .deb have the wrong architecture, so here is a patch fixing that for the set_soenv.in script, for ARMel and AMD64:

# cat /usr/local/kr93794/set_soenv.in.patch
--- /usr/local/kr93794/Asus/xandros/SO_OOO310_m10/set_soenv.in	2009-04-16 12:45:37.000000000 +0200
+++ /usr/local/kr93794/set_soenv.in	2009-04-28 16:30:06.000000000 +0200
@@ -32,6 +32,19 @@
 use strict;                        # pragma
 use File::Basename;
 
+sub dpkg_architecture {
+  my @lines=split(/\n/,`/usr/bin/dpkg-architecture`);
+
+  my %data;
+  my @name_value;
+
+  foreach (@lines){
+    @name_value=split(/=/,$_);
+    $data{$name_value[0]}=$name_value[1];
+  }
+
+  return %data
+}
 
 #
 #--------------------------------------------------------
@@ -63,7 +76,7 @@
       $GVER, $OS, $OSVERSION, $OUTPATH, $INPATH, $PATH_SEPERATOR,
       $DYNAMIC_CRT, $SET_EXCEPTIONS, $use_shl_versions, $CDPATHx, $JRELIBDIR,
       $JREEXTRALIBDIR, $JRETOOLKITDIR, $JRETHREADDIR,
-      $FLIPCMD );
+      $FLIPCMD, $ABI );
 #
 #-------------------------------------------
 # IIc. Declaring the environment variables.
@@ -373,6 +386,12 @@
       $OS             = "LINUX";
       $PATH_SEPERATOR = $ps;
 
+      if (-e "/etc/debian_version")
+      {
+        my %data=dpkg_architecture();
+        $ABI=$data{"DEB_HOST_ARCH"};
+      }
+
 #Set platform specific values: 
    if ($platform =~ m/^i[3456]86/)
    {  print "Setting Linux x86 specific values... ";
@@ -409,11 +428,20 @@
    }
    elsif ($platform =~ m/^x86_64/)
    {  print "Setting Linux x86-64 specific values... ";
-      $outfile        = "LinuxX86-64Env.Set";
+
       $CPU            = "X";
       $CPUNAME        = "X86_64";
       $CVER           = "C341";
-      $OUTPATH        = "unxlngx6";
+      if ($ABI =~ "") {
+        $outfile        = "LinuxX86-64Env.Set";
+        $OUTPATH        = "unxlngx6";
+      }
+      else {
+        $outfile        = "Linux" . $ABI . "Env.Set";
+        $OUTPATH        = "unxlng" . $ABI;
+        $EPM_FLAGS      = "-a $ABI";
+      }
+
       # Blackdown.org JDK porting project uses `amd64' and `server' in JDK 1.4.2 RC1
       $JRELIBDIR      = '$JAVA_HOME'.$ds."jre".$ds."lib".$ds."amd64";
       # has both server and client
@@ -533,15 +561,24 @@
    }
    elsif ($platform =~ m/^arm.*?l-/)
    {  print "Setting Linux ARM specific values... ";
-      $outfile        = "LinuxARMEnv.Set"; 
+
+      if ($ABI =~ "") {
+        $ABI="arm";
+        $outfile = "LinuxARMEnv.Set"; 
+        $OUTPATH = "unxlngr";
+      }
+      else {
+        $outfile = "Linux" . $ABI . "Env.Set"; 
+        $OUTPATH = "unxlng" . $ABI;
+      }
+
       $CPU            = "R";
       $CPUNAME        = "ARM";
-      $OUTPATH        = "unxlngr";
       $JRELIBDIR      = '$JAVA_HOME'.$ds."jre".$ds."lib".$ds."arm";
       $JRETOOLKITDIR  = '$JAVA_HOME'.$ds."jre".$ds."lib".$ds."arm".$ds."server";
       $JRETHREADDIR   = '$JAVA_HOME'.$ds."jre".$ds."lib".$ds."arm".$ds."native_threads";
       $JREEXTRALIBDIR = '$JAVA_HOME'.$ds."jre".$ds."lib".$ds."arm".$ds."xawt";
-      $EPM_FLAGS      = "-a arm";
+      $EPM_FLAGS      = "-a $ABI";
    }
    elsif ($platform =~ m/^mips/)
    {  print "Setting Linux MIPS specific values... ";

After some clean ups I am going to put that into CWS as soon as I find the time ... that's it for now :-)

tags: arm build building compile compiling cross openoffice openoffice.org qemu quilt

tags: openoffice openoffice.org performance

Hi OOo Folks,

some of you may have noticed that ARM is en vogue, though analysts deeply disagree over it's chances e.g. in the netbook market.

Nevertheless I thought it would be a good idea to see how OOo behaves and actually builds on ARM. Unfortunately, with the exception of my iPod Touch, which I received as price from Sun Learning eXchange for this little video about ODF@WWW, and by the way Dmitri Popov just reminded me to continue on that, I didn't have any hardware to build or run it on.

So suddenly I entered the wonderful world of QEMU. QEMU is an emulator, which can, amongst many other architectures, emulate various ARM CPUs and systems. While looking at it I stumbled over a really nice and unique feature, namely the QEMU user space emulator. In user space emulation, QEMU does not emulate a whole system, but allows to execute an alien binary (such as a binary build for ARM) under the host (e.g. x64) directly, dynamically mapping system calls etc. to the hosts architecture, avoiding a bigger part of emulation, which is typically done by whole system emulation.

I already did compile OOo on various flavours of Linux, Mac OS X and on Windows, using different CPUs etc., such as x86, x64 and PPC, but I never did cross compile it (though the term emu(-lator) compile may be more appropriate :-). Still I felt well equipped.

Days (or weeks?) later I actually can report success, just an hour ago I finished my first cross compilation of OOo for ARM CPUs successfully, and surprisingly the installed application seems to work as expected :-)

Initially I planned to utilize the QEMU system mode, were building should just have been like any native build ... unfortunately it was unbearable slow, so I switched to user space mode, which had it's own pitfalls, but which was by factors faster.

Building OOo with QEMU in system mode seemed to take weeks, while utilizing user space mode in principle allowed to finish in about 24 hours, I think there is potential to even push that down to may be 8 hours. By the way, the machine I used had 4 cores.

If there is interest, I may post more details regarding the QEMU user space build, and how I think it can be accelerated further.

Have a nice weekend

Kay

tags: arm build building compile compiling cross openoffice openoffice.org qemu

We are short before code freeze for OpenOffice.org 3.1 and I want to give you an overview about the major community contributions for the framework project. First of all I want to thank all contributors for their dedications and commitment. It's a pleasure to see how hard and target-oriented community developer work to reach their goal. This part wants to present the work from Ariel Constenla-Haile, a very active community member from Argentina. If you are involved in OpenOffice.org development you definitely stumbled over his name. Some months ago Ariel detected an issue which complained about the limited abilities of UNO AWT API regarding menus, especially setting an image for a menu item. UNO AWT is a project which provides an UNO based abstraction layer above VCL (OpenOffice.org user interface toolkit). This API is often used by extension developers to create their own user interface. The framework team received a patch for this issue from Shi Zhoubo, a developer from the RedFlag 2000 team, regarding menus and images. Ariel noticed that the patch fixes the issue but other VCL features for menus were not part. He decided to overcome the limitations of the first patch and worked on a complete solution. He analyzed the present situation and listed what features were missing. He thought about possible solutions and contacted me for discussion. We agreed to a solution which provides three additional interfaces for the missing features. He created patches for the projects involved and contributed them. Summing up Ariels contribution is a good example how people can help to fix/enhance OpenOffice.org.

The following image give you a impression what's is possible with this enhancement:

Ariel created a wiki page to explain what problems extension developers face, how to solve the problem and what's possible with his enhancement. I strongly recommend people that are interested using menus in OpenOffice.org to read his explanations.

If you want to help us to make OpenOffice.org better please contact me via e-mail or join the framework development mailing list. Everybody is welcome to join us. The framework team will help you to have an easy start. Just give it a try to become a development member of the leading open source Office suite.

In my next blog I want to continue the overview with an enhancement regarding keyboard shortcuts.

tags: community contribution framework openoffice openoffice.org

Have you ever been CONFUSED by creating make files? Have you ever WONDERED which files had to be rebuild because of changes in make files or header files? Have you ever thought about how CONCURRENT your make files actually are? Have you ever been disturbed by DISTRIBUTING IMPLEMENTATION DETAILS about a particular .c- or .h-file over multiple makefiles? Have you ever thought how concurrency could really work while invoking make MULTIPLE (or many) times and how that puts a lot of OVERHEAD into a build process? Have you ever found it hard to MODULARIZE your make files? Have you ever QUESTIONED the sense of make files in general?

THAN THIS BLOG POSTING IS FOR you  :-)

Warning: When coming down to creating, maintaining  or compiling stuff, I know I am different, so be warned  :*) I did briefly explain to others how I felt things should work ... and did hear from some that I am odd, while others seemed to like it ... anyway, I do believe in and successfully tried the below approach in my home projects and am currently playing with it while experimenting with packaging ... but judge yourself :-)

A Start: Let's start with simple things, in my little world I mostly have some .c- and .h-files, while I typically want to create some .o-, .bin- and .so- respectively lib*.so-files. Some files need special flags, e.g. for optimization or debugging. To understand how we can automate this - thus abandoning the makefile burden ideally completely - we may want to take a look at a typical C include.

A simple include looks like this:

foo.c:
    #include "a.h"

Compiling: For the corresponding .o-file (foo.o) this means, that it needs to be remade in case foo.c, a.h or any of the transitive includes of a.h changes.

Linking: To be able to link foo.o into an executable or shared library, we also need to link-in any objects or shared libraries implementing a.h (or its includes respectively).

Sample Code: Let's say, a.h only declares what's implemented in a.c, such that a.h does not include anything else. Unfortunately a.c includes (despite a.h) b.h as well as x.h .

a.c:
    #include "a.h"
    #include "b.h"
    #include "x.h"

Luckily, b.h does not include anything itself, but b.c does, additionally to b.h it also includes c.h, which is implemented in c.c:

b.c:
    #include "b.h"
    #include "c.h"

c.c:
    #include "c.h"

And if this had not been enough, a.c respectively a.o does also need the implementation of x.h to be properly linkable, which is not directly implemented in a .o-file but a linkable shared library named libx.so.

Looking into x.c

x.c:
    #include "x.h"
    #include "y.h"
    #include "d.h"

we see, that itself needs another library (liby.so) and at least one other .o-file (d.o). And foo.bin is not the only executable we would like to link, but bar.bin as well. bar.bin is partly based on the same objects as foo.bin, but links differently, I leave the details out here, please have a look at the bottom for all sample files.

And it's Makefile: Expressing this in GNU make looks like this:

============= handcraft.mk =============:
LDFLAGS+=-Wl,-rpath='$$ORIGIN/' -L.

a.o: a.c a.h b.h c.h x.h
	$(COMPILE.c) $(OUTPUT_OPTION) $<

b.o: b.c b.h c.h 
	$(COMPILE.c) $(OUTPUT_OPTION) $<

c.o: c.c c.h
	$(COMPILE.c) $(OUTPUT_OPTION) $<

x.o: x.c x.h y.h d.h
	$(COMPILE.c) $(OUTPUT_OPTION) $<

y.o: y.c y.h
	$(COMPILE.c) $(OUTPUT_OPTION) $<

libx.so: x.o liby.so
	$(LINK.o) $(LOADLIBES) $(LDLIBS) -shared $^ -L. -ly -o $@

liby.so: y.o
	$(LINK.o) $(LOADLIBES) $(LDLIBS) -shared $^ -o $@

foo.o: foo.c a.h
	$(COMPILE.c) $(OUTPUT_OPTION) $<

foo.bin: foo.o a.o b.o c.o libx.so
	$(LINK.o) $(LOADLIBES) $(LDLIBS) $^ -o $@

bar.o: bar.c b.h c.h d.h
	$(COMPILE.c) $(OUTPUT_OPTION) $<

bar.bin: bar.o b.o c.o d.o
	$(LINK.o) $(LOADLIBES) $(LDLIBS) $^ -o $@

Now we can build the concrete targets by invoking make as:

  > make -f handcraft.mk foo.bin
  > make -f handcraft.mk bar.o
  > make -f handcraft.mk libx.so
  > make -f handcraft.mk -j foo.bin bar.bin
  > make -f handcraft.mk ...

Automation: By increasing the number of targets and prerequisites such a make file can become pretty fast pretty complicated - after all building software properly is not an easy task.

Automatic dependency generation for include files is well known and has been done for long, what's missing to become fully automatic are compilation flags in particular and linking support in general. The point is, that only the moment building becomes fully automatic, make can be modularized reasonably and we can build a whole project invoking make once only (which is desirable :-).

So, let's see what information regarding linking has been implicitly expressed in the above makefile:

• If you include a.h, than you need to link-in a.o as well.
• If you link-in a.o, than you need to link-in b.o and c.o as well, and you have to link against libx.so .
• If you ...
  

The Rule: In principle, the rule seems to be: If you include a header, ensure that you link-in the implementing .o-files and shared libraries, as well as the required .o-files and shared libraries of these .o-files etc.

Cohesion: The hints regarding required .o-files, shared libraries and c-flags should be placed in the appropriate files and need to be available for make during building, otherwise this information gets repeated in multiple targets and makefiles again and again - which to my experience is typical ;*)

Concrete: If we have a header file which requires a particular object file to be linked-in, we can (initially hackily :-) express this directly in-line, prefixing it with a "magic" word, e.g. as:

//§ req_objs:=a.o

We may want to do the same with libraries:

//§ req_libs:=libx.so

And even compilation flags may be expressed this way:

//§ cflags:=-g

Applying the schema to the above files, we get the following additional lines:

foo.c:
  //§ cflags=-g

a.h:
  //§ req_objs:=a.o

b.h:
  //§ req_objs:=b.o

x.h:
  //§ req_libs=libx.so

...

Now, the only think left is to write some generic make "modules" to utilize this now explicit available information ... this is so obvious, that I leave that as a homework for the patient reader ... just joking :-) please have a look below.

Conclusion: By now we have developed a simple schema for linking binaries and shared libraries based on C files, which allows us to build any final or intermediate target fully automatically, expressing linking relevant information in a cohesive way, while eliminating the need to hassle with make-files ever again.

Scalability:  I have to admit that I did not yet test how scalable this approach is, in principle it should scale quite well ... and after all we are only talking about some thousand files for typical projects. At least for my home projects with some hundred files it works quite nicely.

Reasoning: The reason I brought this up is, that I believe that anything can be build fully automatically this way - even installation packages I am currently experimenting with.

In Principle: The overall rule set to build any kind of derived file fully automatically (and not only compiling C sources :-), seems to be something along the following lines:

1. We have prerequisite files and derived files.
2. Every derivation has exactly one primary prerequisite and n secondary prerequisites.
3. A derivation explicitly depends on its primary prerequisite.
4. A derivation implicitly depends on its secondary prerequisites, which need to be determined during runtime out of the primary prerequisite.
5. A prerequisite may specify transformation flags.
6. For every type of prerequisite-to-derivation transformation exactly one explicit rule is defined.
7. For any type of derivation there may be n+ explicit rules.
   

Note on -7-: As a derivation may not depend on its primary prerequisite only, in practice some functions need to be executed at runtime to determine all secondary prerequisites. Actually, the concrete function to be calculated depends on the actual explicit rule selected for the creation of the derivation. As ordinary makes (e.g. GNU make), do not offer a way to add prerequisites to a derivation after an explicit rule has been selected, statement -7- has to be adapted for these to work:

-7- For any type of derivation there is exactly one explicit rule.

Therefore, to still enable deriving one type from different types of prerequisites, we need to mangle in the prerequisites type.

Finally: If there is interest and people think that this general approach seems to be reasonable, I am going to explain in more detail how it actually works and what else could be done to make a builders life easier ... and I may even try it out on a concrete OOo module, may be SAL :-)

Last but not least I have to admit, that I am a descent GNU make fan :-)

Please find sample files etc.  below.

Best regards

Kay

Samples and make files for GNU make, tried on Debian Linux, just copy&paste the stuff into the named files ...

============= foo.c =============:
//§ cflags:=-g

#include "a.h"

int main(void) {
  return 0;
}

============= bar.c =============:
//§ cflags:=-O

#include "b.h"
#include "c.h"
#include "d.h"

int main(void) {
  return 0;
}

============= a.h =============:
//§ req_objs=a.o

============= a.c =============:
#include "a.h"
#include "b.h"
#include "x.h"

============= b.h =============:
//§ req_objs=b.o

============= b.c =============:
#include "b.h"
#include "c.h"

============= c.h =============:
//§ req_objs:=c.o

============= c.c =============:
#include "c.h"

============= d.h =============:
//§ req_objs=d.o

============= d.c =============:
#include "d.h"

============= x.h =============:
//§ req_libs=libx.so

============= x.c =============:
#include "x.h"
#include "y.h"
#include "d.h"

============= y.h =============:
//§ req_libs:=liby.so

============= y.c =============:
#include "y.h"

============= generic.mk =============:
# The master makefile.

# Preserve intermediate files.
.PRECIOUS: %.o

# To find linked libraries relatively.
LDFLAGS+=-Wl,-rpath='$$ORIGIN/' -L.

include functions.mk   # Some helpers.

include info_c.mk      # Create C infos.
include o_c.mk         # Create C objects.
include bin_o.mk       # Create object binaries.
include libso_o.mk     # Create object shared libraries.

$(foreach target,$(MAKECMDGOALS),$(call $(suffix $(target))_spreqs,$(target)))

============= functions.mk =============:
# Guard includes.
define ginc_t
  ifndef $(1)_inc_def
    $(1)_inc_def=:1 

    -include $(1) 
  endif
endef

ginc=$(eval $(ginc_t))

# Calculate the referential transitive closure.
define closure_t
  ifndef $(1)$(2)_cls_def
    $(1)$(2)_cls_def:=1

    ifdef $(suffix $(1))_info
      $$(call $(suffix $(1))_info,$(1))
    else
      $$(call ginc,$(1).info) 
    endif

    $(1)$(2)_cls:=$$(strip $$($(1)$(2)) $$(foreach file,$$($(1)$(2)),$$(call closure,$$(file),$(2))))
  endif
endef

# $(1) = file name
# $(2) = variable postfix
closure=$(eval $(closure_t)) $($(1)$(2)_cls)

============= info_c.mk =============:
# Rules and prerequisits for .info files .

define info_script
  echo Making $@
  echo "`grep ^//§ $<|sed s0//§\ 0$<_0g)`" > $@
  echo "$<_includes:=`echo \`grep \#include $<|sed s0\#include00g|sed s0\\\"00g\``" >> $@
endef

%.info: %
	@$(info_script)

============= o_c.mk =============:
# Rules and prerequisits for .o files, as well as .o.info .

define t_o_info
  ifndef $(1)_info_def
    $(1)_info_def:=1

    t_o_info_ppreq_$(1):=$(1:.o=.c)

    $$(call ginc,$$(t_o_info_ppreq_$(1)).info)

    t_o_info_all_includes:=$$(call closure,$$(t_o_info_ppreq_$(1)),_includes)
    $(1)_req_objs:=$$(subst $(1),,$$(foreach inc,$$(t_o_info_all_includes),$$($$(inc)_req_objs)))
    $(1)_req_libs:=$$(foreach inc,$$(t_o_info_all_includes),$$($$(inc)_req_libs))
  endif
endef

.o_info= $(eval $(t_o_info)) 


define t_o_spreqs
  ifndef t_o_spreqs_$(1)_def
    t_o_spreqs_$(1)_def:=1

    t_o_spreqs_ppreq_$(1):=$(1:.o=.c)

    $(1): $$(call closure,$$(t_o_spreqs_ppreq_$(1)),_includes)
    $(1): cflags_:=$$($$(t_o_spreqs_ppreq_$(1))_cflags)
  endif
endef

.o_spreqs=$(eval $(t_o_spreqs))



%.o: %.c
	$(COMPILE.c) $(OUTPUT_OPTION) $(cflags_) $<

============= bin_o.mk =============:
# Rules and prerequisits for .bin files .

define t_bin_spreqs
  ifndef t_bin_spreqs_$(1)_def
    t_bin_spreqs_$(1)_def:=1
    t_bin_spreqs_$(1)_ppreq:=$(1:.bin=.o)

    $$(call .o_spreqs,$$(t_bin_spreqs_$(1)_ppreq))

    t_bin_spreqs_$(1)_objs:=$$(call closure,$$(t_bin_spreqs_$(1)_ppreq),_req_objs)
    $$(foreach obj,$$(t_bin_spreqs_$(1)_objs),$$(call .o_spreqs,$$(obj)))

    t_bin_spreqs_$(1)_libs:=$$(subst $(1),,$$(foreach obj,$$(t_bin_spreqs_$(1)_ppreq) $$(t_bin_spreqs_$(1)_objs),$$($$(obj)_req_libs)))
    $$(foreach lib,$$(t_bin_spreqs_$(1)_libs),$$(call .so_spreqs,$$(lib)))

    $(1): objs_:=$$(t_bin_spreqs_$(1)_objs)
    $(1): libs_:=$$(t_bin_spreqs_$(1)_libs)
    $(1): $$(t_bin_spreqs_$(1)_objs)
    $(1): $$(t_bin_spreqs_$(1)_libs)
  endif
endef

.bin_spreqs=$(eval $(t_bin_spreqs))

%.bin: %.o
	$(LINK.o) $(LOADLIBES) $(LDLIBS) $< $(objs_) $(patsubst lib%.so,-l%,$(libs_)) -o $@

============= libso_o.mk =============:
# Rules and prerequisits for lib*.so files .

define t_so_spreqs
  ifndef t_bin_spreqs_$(1)_def
    t_so_spreqs_$(1)_def:=1
    t_so_spreqs_$(1)_ppreq:=$(patsubst lib%.so,%.o,$(1))

    $$(call .o_spreqs,$$(t_so_spreqs_$(1)_ppreq))

    t_so_spreqs_$(1)_objs:=$$(call closure,$$(t_so_spreqs_$(1)_ppreq),_req_objs)
    $$(foreach obj,$$(t_so_spreqs_$(1)_objs),$$(call .o_spreqs,$$(obj)))

    t_so_spreqs_$(1)_libs:=$$(subst $(1),,$$(foreach obj,$$(t_so_spreqs_$(1)_ppreq) $$(t_so_spreqs_$(1)_objs),$$($$(obj)_req_libs)))
    $$(foreach lib,$$(t_so_spreqs_$(1)_libs),$$(call .so_spreqs,$$(lib)))

    $(1): objs_:=$$(t_so_spreqs_$(1)_objs)
    $(1): libs_:=$$(t_so_spreqs_$(1)_libs)
    $(1): $$(t_so_spreqs_$(1)_objs)
    $(1): $$(t_so_spreqs_$(1)_libs)
  endif
endef

.so_spreqs=$(eval $(t_so_spreqs))

lib%.so: %.o
	$(LINK.o) $(LOADLIBES) $(LDLIBS) -shared $< $(objs_) $(patsubst lib%.so,-l%,$(libs_)) -o $@

> make -f generic.mk foo.bin
> make -f generic.mk bar.o
> make -f generic.mk libx.so
> make -f generic.mk -j foo.bin bar.bin
> make -f generic.mk ...

tags: automatic build building gnumake make openoffice openoffice.org packaging

If you are interested in streaming versions of all recorded sessions
from OOoCon2008, you can find them here:

https://library.network.com/CatalogQueryServer/app.jsp#tags/ooocon2008

The conference program including most of the slides can still be found here:

http://marketing.openoffice.org/ooocon2008/programme.html

tags: odf ooo ooocon ooocon2008 openoffice openoffice.org sun

For those who haven't been already informed by the plugin update mechanism of NetBeans. We have  uploaded a new version 2.0.3 of our plugin in the update center of NetBeans 6.5. It's a micro release with minor bugfixes and better or final support for NetBeans 6.5.

 The most interesting features are:

• OXT properties have been moved in the normal project properties dialog and support of the latest oxt specific properties. That makes it really easy for you to provide a complete oxt extension with localized display names for the extension manager, icons, localized licence files etc.
• better project structure by providing a virtual oxt folder to separate clearly between oxt and office specific files and Java or NetBeans project files.
• some code templates. For example urqi for an useful expansion to queryInterface or  oohd to easy insert an OpenOffice specific header in your files.

tags: api extension netbeans openoffice

Beside this event 5 other releases were done for OOo. So there were a lot work for the QA and release teams last year. 6 releases are more than in the past years!

The 6 product releases for OOo in 2008

• OOo 2.4
  • released end of March 2008 (with a delay of round about one month)
  • was the last but biggest feature release on OOo 2.x code line
• OOo 3.0 Beta 1
  • released mid of April 2008
  • nearly all features for OOo 3.0 were integrated
• OOo 2.4.1
  • released mid of June 2008
  • was the last bigger regular bug fix release on OOo 2.x code line (the next updates on OOo 2.x code line will fix security and very critical issues only)
• OOo 3.0 Beta 2
  • released begin of July 2008
  • final feature set for OOo 3.0 were integrated and stabilization of the code base were done
• OOo 3.0 Final
  • released mid of October 2008 (at 8 ^th anniversary of OOo - 13 ^th of October)
  • numbers about that release
• OOo 2.4.2
  • released on 28 ^th of October 2008
  • currently the last bug fix release on OOo 2.x code line, were ~25 critical issues were fixed

In January 2009 (the very next days) OOo 3.0.1 will be released. So stay tuned for the first bug fix release on OOo 3.x code line. Also Feature Freeze for OOo 3.1 was done end of December.

What other numbers can be interested for the last year?

• nearly 900 child work spaces (CWS) were integrated in the code lines (duplicate integrations in different code lines aren't counted)
• more than 4300 issues (features, enhancements, bug fixes etc.) were integrated in the code lines (duplicate integrations in different code line aren't counted)
• more than 12750 issues were reported in IssueTracker

What does this mean in detail and according to the past years?

Child Work Spaces (CWS) and integrated issues

894 CWS with 4360 issues were integrated in 2008. 83 CWSs with 741 issues of them include features and enhancements. The full allocation over the months can be seen in the next graphs.

How many CWSe and Issues were integrated in the years before is shown in the next graphs.

I asked myself, why we had so much integrations of CWS and issues in 2005 and also in 2006. The answer I found is, OOo worked in that time frame on bug fixing only. In October 2005 OOo 2.0 was released. Before this release features and bug fixes were done in parallel for a long time. And after the betas a mass of bug fixes had to be done. With OOo 2.0.1 in December 2005 and OOo 2.0.2 in February 2006 more than 3500 fixed issues – mostly bug fixes - were integrated. This rate OOo never reached again. Beside this the release model changed in 2007 with the toggle of Feature release with Bug-Fix releases. So since mid of 2006 the teams are working more and more on integration of new features than in 2005.

Incoming Issues in IssueTracker increased in 2008

The high interest on OOo 3.0 was noticeable also in IssueTracker (the issue tracking system for OOo). The number of issues increased before first Beta in March and was on a high level in Beta phases of OOo 3.0 until the final release in October 08.

In 2007 the duration of incoming issues per month were under 1000 issues. But in 2008 the number increased on nearly the same level as in 2006.

I know it isn't a good signal that OOo get more issues reported. More issues could mean more bug reports. I analyzed the incoming issues in more details.
12752 issues were reported in sum, Defects are 9388 (74%) issues, Feature and Enhancements 1981 (15%) issues and the rest are patches and tasks with 1383 (11%) issues. (More about integrated patches you can find in this blog).

Defects : issues 9388 (74%)				Feature and Enhancements : 1981 issues (15%)
which are fixed :	2734	(29%)		which are fixed :	374	(19%)
which are duplicate or invalid :	2984	(32%)		which are duplicate or invalid :	349	(18%)
open but targeted for OOo 3.1 :	258	(3%)		open but targeted for OOo 3.x :	226	(11%)
open but targeted for OOo 3.x :	1036	(11%)		open without an target :	1031	(52%)
open without an target :	2371	(25%)

These numbers say that 25% of all reported defects and 52% of all reported features and enhancements aren't addressed until now. In sum more than 3000 issues (~27%) aren't addressed. If this is a good or a bad number, I cannot say. I do not have the comparison with other bigger products until now. But I know that with OOo 3.0.1 and OOo 3.1 the quality will and should be increased. So stay tuned for these releases. As I wrote above OOo 3.0.1 will be out in the very soon.

On the other side more than 50% of all issues are done and 12% are addressed for the very next releases. So the general rate for issue handling isn't bad, I think. But I always know, there are still issues which hinder people on using OOo in their environment. It isn't possible to work on all that issues.

My resume

Because of the mass of releases – more than in the past years – and the big pressure to release OOo 3.0 it was a lot of work for all community teams. In my opinion we, the whole OOo community, did more than in the past years. Perhaps the code contribution in lines of code wasn't as high as in the past years (I haven't checked this by myself), but there were many other things to do, that these releases could be realized in time – nearly in time.

Last week Eric Bachard wrote about the number of contributors to OOo and Jim Parkinson, VP at Sun Microsystems, wrote that Sun will continue to support OOo. Also the contribution of Extensions for OOo started and increased dramatically in 2008. Everybody who want to contribute his/her new feature without build environment of OOo could use this method. And a lot of users/contributors did this.

But we need a lot of more people in all projects on OOo to increase the success of OOo in future. Some people talk about code contribution only. I talk about we need more people in all areas and projects – User Experience, QA, L10N, Development or any other project on OOo – to get dreams fulfilled.

So let 2009 be also a successful year and come to OOo – as community member or as user only.

tags: ooo openoffice openoffice.org quality release statistics

tags: ooo ooocon ooocon2008 openoffice openoffice.org qa quality video

As promised, just a quick heads up regarding the video of my talk at the OOoConf 2008. I uploaded it to Sun Learning eXchange, to make it more convenient to view ... and if I had known how, I would have placed it inline ;-)

Best regards

        Kay

Mailing lists

• announce@odf-at-www.openoffice.org
• users@odf-at-www.openoffice.org
• dev@odf-at-www.openoffice.org

Blog Postings

• 23 Dec 2008 - ODF@WWW - Video of my Talk at the OOoConf 2008
• 12 Dec 2008 - ODF@WWW - Heading for 0.1
• 14 Nov 2008 - ODF@WWW - OOo Conf 2008
• 26 Sep 2008 - ODF@WWW - Becoming an (Incubator) Project
• 01 Sep 2008 - ODF@WWW - Some State
• 22 Jul 2008 - ODF@WWW - Going forward ...
• 04 Jul 2008 - ODF@WWW - Simply Install
• 27 Jun 2008 - ODF@WWW -How it works
• 19 Jun 2008 - ODF@WWW (An ODF Wiki)

tags: document odf ooocon open opendocument openoffice openoffice.org video wiki