Weblog

Wednesday March 16, 2005

Pitfals of the Perl XS or what to do when things do not work as advertised

Introduction

I was working on providing Perl interface to the liblgrp(3LIB) library so that simple scripts can be written to understand NUMA topology of a given machine. I started by reading pretty good and extensive documentation explaining how to write Perl extensions: perlxstut(1), perlxs(1), perlguts(1) and h2xs(1).

While these documents explain things pretty well I stumbled across various things that didn't work as advertised, so I decided to do a walk through a simple example and show what can go wrong and how to fix it.

Initial stubs module

The Perl XS documentation suggests using h2xs as a starting point. Let's follow its recommendation. The h2xs documentation provides a simple example for extension based on .h and .c files. We will assume that the header file is called xs_test.h and we want the perl module be named as XS::Test. Let's follow the h2xs(1) man page and perform Start with a dummy run of h2xs:

  $ h2xs h2xs -Afn XS::Test

  Writing ext/XS-Test/ppport.h
  Writing ext/XS-Test/lib/XS/Test.pm
  Writing ext/XS-Test/Test.xs
  Writing ext/XS-Test/Makefile.PL
  Writing ext/XS-Test/README
  Writing ext/XS-Test/t/XS-Test.t
  Writing ext/XS-Test/Changes
  Writing ext/XS-Test/MANIFEST

Let's remove ppport.h and use long option names for readability option names.

  $ h2xs --skip-ppport  --omit-autoload --force --name=XS::Test

  Writing ext/XS-Test/lib/XS/Test.pm
  Writing ext/XS-Test/Test.xs
  Writing ext/XS-Test/Makefile.PL
  Writing ext/XS-Test/README
  Writing ext/XS-Test/t/XS-Test.t
  Writing ext/XS-Test/Changes
  Writing ext/XS-Test/MANIFEST

So far so good.

Adding header file

Now let's populate the directory with an empty header file

  $ touch ext/XS-Test/xs_test.h

and create the actual extension. The man pages suggests to run

    h2xs -Oxan Ext::Ension interface_simple.h

Note that -x requires the C::Scan(1) package. Let us assume that it is installed and continue by replacing short options with long ones and changing the module name:

  $ h2xs --skip-ppport --overwrite-ok --autogen-xsubs \
         --name=XS::Test xs_test.h

  Can't find --overwrite-ok.h in . /usr/include /usr/sfw/include
        ...
        /usr/include ext/XS/Test

Hmm... Looking at the h2xs we see that the actual option name is --overwrite_ok instead of --overwrite-ok as advertised in the man page. So let's retry using the right option name:

  $ h2xs --skip-ppport --overwrite_ok --autogen-xsubs  \
         --name=XS::Test xs_test.h

  Can't find xs_test.h in . /usr/include /usr/sfw/include /opt/sfw/include
        ...
        /opt/GNU/include /usr/include ext/XS/Test

Obviously it is looking in the wrong place, the directory should be ext/XS-Test instead of ext/XS/Test. This is another h2xs bug.

Let's fix both bugs and create our own copy of h2xs in ~/bin/h2xs.fixed:

  $diff /usr/perl5/bin/h2xs ~/bin/h2xs.fixed
  573c573
  <                 'overwrite_ok|O'     => \$opt_O,
  ---
  >                 'overwrite-ok|O'     => \$opt_O,
  787c787
  <     (my $epath = $module) =~ s,::,/,g;
  ---
  >     (my $epath = $module) =~ s,::,-,g;

And now try again:

  $ ~/bin/h2xs.fixed --skip-ppport --overwrite-ok --autogen-xsubs \
                     --name=XS::Test xs_test.h

  Overwriting existing ext/XS-Test!!!
  Scanning typemaps...
  Scanning /usr/perl5/5.8.4/lib/ExtUtils/typemap
  Scanning xs_test.h for functions...
  Scanning xs_test.h for typedefs...
  Writing ext/XS-Test/lib/XS/Test.pm
  Writing ext/XS-Test/Test.xs
  Writing ext/XS-Test/fallback/const-c.inc
  Writing ext/XS-Test/fallback/const-xs.inc
  Writing ext/XS-Test/Makefile.PL
  Writing ext/XS-Test/README
  Writing ext/XS-Test/t/XS-Test.t
  Writing ext/XS-Test/Changes
  Writing ext/XS-Test/MANIFEST

Wow, now we should have a working extension that does nothing useful!

Adding constants

Let's move on and populate the header file with few simple constants:

  $ cat >ext/XS-Test/xs_test.h
  #define XST_DEFINE 1

  typedef enum xst_enum {
          XST_ENUM_1,
          XST_ENUM_2,
  } xst_enum_t;

  typedef enum xst_enum_val {
          XST_ENUM_VAL_1 = 1,
          XST_ENUM_VAL_2 = 2,
  } xst_enum_val_t;
  ^D

Repeating the exercise we get

  $ ~/bin/h2xs.fixed --skip-ppport --overwrite-ok --autogen-xsubs \
                     --name=XS::Test xs_test.h

  Overwriting existing ext/XS-Test!!!
  Scanning typemaps...
  Scanning /usr/perl5/5.8.4/lib/ExtUtils/typemap
  Scanning xs_test.h for functions...
  Scanning xs_test.h for typedefs...
  Use of uninitialized value in exists at 
      /home/akolb/bin/h2xs.fixed line 1025.
  Writing ext/XS-Test/lib/XS/Test.pm
  Writing ext/XS-Test/Test.xs
  Writing ext/XS-Test/fallback/const-c.inc
  Writing ext/XS-Test/fallback/const-xs.inc
  Writing ext/XS-Test/Makefile.PL
  Files "ext/XS-Test/fallback/const-c.inc" and 
        "ext/XS-Test/const-c.inc" differ.
  It appears that the code in ext/XS-Test/Makefile.PL 
             does not autogenerate the files 
             ext/XS-Test/const-c.inc and 
             ext/XS-Test/const-xs.inc correctly.

  Please report the circumstances of this bug in h2xs version 1.23 using the
  perlbug script. 
  Writing ext/XS-Test/README
  Writing ext/XS-Test/t/XS-Test.t
  Writing ext/XS-Test/Changes
  Writing ext/XS-Test/MANIFEST

Something went wrong and we need to start digging around. Running

  $ grep XST ext/XS-Test/* ext/XS-Test/*/* ext/XS-Test/*/*/*

we note that XST_ENUM_1 and XST_ENUM_2 are not present in the generated code. It turns out that there is another bug in h2xs that incorrectly parses enums. Fixing it gives

  870c868
  < my ($key, $declared_val) = $item =~ /(\w*)\s*=\s*(.*)/;
  ---
  > my ($key, $declared_val) = $item =~ /(\w+)\s*[=,]*\s*(.*)/;
 
After we fix it, retry and grep again both missing constants appear together
with other constants.

Hurra! Now our simple extension module it understands all our constants. Or does it? Let's make sure that everything works:

  $ cd ext/XS-Test
  $ perl Makefile.PL
  Checking if your kit is complete...
  Looks good
  Writing Makefile for XS::Test
  $ make test
  ...
  Running Mkbootstrap for XS::Test ()
  chmod 644 Test.bs
  rm -f blib/arch/auto/XS/Test/Test.so
  LD_RUN_PATH="" cc  -G Test.o  xs_test.o  -o blib/arch/auto/XS/Test/Test.so      
  chmod 755 blib/arch/auto/XS/Test/Test.so
  cp Test.bs blib/arch/auto/XS/Test/Test.bs
  chmod 644 blib/arch/auto/XS/Test/Test.bs
  PERL_DL_NONLAZY=1 perl "-MExtUtils::Command::MM" "-e" \
      "test_harness(0, 'blib/lib', 'blib/arch')" t/*.t
  t/XS-Test....ok
  All tests successful.
  Files=1, Tests=2,  0 wallclock secs ( 0.03 cusr +  0.00 csys =  0.03 CPU)

That's promising! Now let's add simple tests to verify that we can actually access all constants from Perl. Going to t/XS_Test.t let's add tests that verify that constants have expected values:

  $ diff XS-Test.t.~1~ XS-Test.t
  8c8
  < use Test::More tests => 2;
---
  > use Test::More tests => 7;
  30a31,35
  > is(XST_DEFINE, 1);
  > is(XST_ENUM_1, 0);
  > is(XST_ENUM_2, 1);
  > is(XST_ENUM_VAL_1, 1);
  > is(XST_ENUM_VAL_2, 2);

repeating the mantra:

  $ make test
  PERL_DL_NONLAZY=1 perl "-MExtUtils::Command::MM" "-e" \
      "test_harness(0, 'blib/lib', 'blib/arch')" t/*.t
  t/XS-Test....ok 2/7
  Your vendor has not defined XS::Test macro XST_ENUM_1, used at t/XS-Test.t line 32
  # Looks like you planned 7 tests but only ran 3.
  # Looks like your test died just after 3.
  t/XS-Test....dubious
        Test returned status 255 (wstat 65280, 0xff00)
  DIED. FAILED tests 4-7
        Failed 4/7 tests, 42.86% okay
  Failed Test Stat Wstat Total Fail  Failed  List of Failed
  ----------------------------------------------------------
  t/XS-Test.t  255 65280     7    8 114.29%  4-7
  Failed 1/1 test scripts, 0.00% okay. 4/7 subtests failed, 42.86% okay.
  *** Error code 2
  make: Fatal error: Command failed for target `test_dynamic'

Not so good! To figure out what's going on let's take a look at const-c.inc file:

    ...
    switch (name[9]) {
    case '1':
      if (memEQ(name, "XST_ENUM_", 9)) {
      /*                        1     */
  #ifdef XST_ENUM_1
        *iv_return = XST_ENUM_1;
        return PERL_constant_ISIV;
  #else
        return PERL_constant_NOTDEF;
  #endif

That explains the problem. Of course, there are no defines for enum constants! We found another bug in h2xs. Instead of fixing it let's take a look at the ExtUtils::Constant module. Looking at the ``macro'' key in the C_constant section we see:

macro

The C pre-processor macro to use in the ``#ifdef''. This defaults to the name, and is mainly used if value is an ``enum''. If a reference an array is passed then the first element is used in place of the ``#ifdef'' line, and the second element in place of the ``#endif''. This allows pre-processor constructions such as

  #if defined (foo)
  #if !defined (bar)
  ...
  #endif
  #endif

to be used to determine if a constant is to be defined.

A ``macro'' 1 signals that the constant is always defined, so the ``#if''/``#endif'' test is omitted.

So let's tweak Makefile.PL a bit:

  $ diff Makefile.PL.~1~ Makefile.PL
  22,23c22,28
  <   my @names = (qw( XST_DEFINE XST_ENUM_1 XST_ENUM_2
  <                    XST_ENUM_VAL_1 XST_ENUM_VAL_2));
  ---
  >   my @names = (qw(XST_DEFINE),
  >            # Enums should be specified as references with macro set to 1.
  >            {name=>"XST_ENUM_1", macro=>"1"},
  >            {name=>"XST_ENUM_2", macro=>"1"},
  >            {name=>"XST_ENUM_VAL_1",macro=>"1"},
  >            {name=>"XST_ENUM_VAL_2",macro=>"1"});

And retry our test:

  $ perl Makefile.PL
  Writing Makefile for XS::Test
  $ make
  ...
  Running Mkbootstrap for XS::Test ()
  chmod 644 Test.bs
  rm -f blib/arch/auto/XS/Test/Test.so
  LD_RUN_PATH="" cc  -G Test.o  xs_test.o  -o blib/arch/auto/XS/Test/Test.so      
  chmod 755 blib/arch/auto/XS/Test/Test.so
  cp Test.bs blib/arch/auto/XS/Test/Test.bs
  chmod 644 blib/arch/auto/XS/Test/Test.bs
  Manifying blib/man3/XS::Test.3

  $ make test
  PERL_DL_NONLAZY=1 perl "-MExtUtils::Command::MM" \
                    "-e" "test_harness(0, 'blib/lib', 'blib/arch')" t/*.t
  t/XS-Test....ok
  All tests successful.
  Files=1, Tests=7,  0 wallclock secs ( 0.03 cusr +  0.00 csys =  0.03 CPU)

Finally it works correctly and we successfully created an extension which exports constants! Uphh... Seems like h2xs should be smart enough to treat enums as advertised, but we found a way around.

Alternative: names directly in the module name space

There is an alternative approach for our module that doesn't require autoloaded functions and auto-generated constants and instead stashes constants directly in the module name space using the XS BOOT keyword.

Our Test.xs file is very simple at this point:

  #include "EXTERN.h"
  #include "perl.h"
  #include "XSUB.h"

  #include <xs_test.h>

  #include "const-c.inc"

  MODULE = XS::Test             PACKAGE = XS::Test

  INCLUDE: const-xs.inc

Here is the trick:

  $ diff Test.xs.~1~ Test.xs

  7,8d6
  < #include "const-c.inc"
  < 
  11c9,27
 < INCLUDE: const-xs.inc
  ---
  > PROTOTYPES: ENABLE
  > 
  >  #
  >  # Define any constants that need to be exported.  By doing it this way we can
  >  # avoid the overhead of using the DynaLoader package, and in addition constants
  >  # defined using this mechanism are eligible for inlining by the perl
  >  # interpreter at compile time.
  >  #
  > BOOT:
  >     {
  >     HV *stash;
  > 
  >     stash = gv_stashpv("XS::Test", TRUE);
  >     newCONSTSUB(stash, "XST_DEFINE", newSViv(XST_DEFINE));
  >     newCONSTSUB(stash, "XST_ENUM_1", newSViv(XST_ENUM_1));
  >     newCONSTSUB(stash, "XST_ENUM_2", newSViv(XST_ENUM_2));
  >     newCONSTSUB(stash, "XST_ENUM_VAL_1", newSViv(XST_ENUM_VAL_1));
  >     newCONSTSUB(stash, "XST_ENUM_VAL_2", newSViv(XST_ENUM_VAL_2));
  >     }

This allows us to simplify Makefile.PL as well removing all uses of ExtUtils::Constant:

  $ diff Makefile.PL.~1~ Makefile.PL
  17,30d16
  < if  (eval {require ExtUtils::Constant; 1}) {
  <   my @names = (qw( XST_DEFINE XST_ENUM_1 XST_ENUM_2
  <                    XST_ENUM_VAL_1 XST_ENUM_VAL_2));
  <   ExtUtils::Constant::WriteConstants(
  <            NAME         => 'XS::Test',
  <            NAMES        => \@names,
  <            DEFAULT_TYPE => 'IV',
  <            C_FILE       => 'const-c.inc',
  <            XS_FILE      => 'const-xs.inc',
  <            );
  32,40d17
  < }
  < else {
  <   use File::Copy;
  <   use File::Spec;
  <   foreach my $file ('const-c.inc', 'const-xs.inc') {
  <     my $fallback = File::Spec->catfile('fallback', $file);
  <     copy ($fallback, $file) or die "Can't copy $fallback to $file: $!";
  <   }
  < }

Also there is no need for the autoload code in Test.pm file:

  $ diff lib/XS/Test.pm.~1~ lib/XS/Test.pm
  9d8
  < use AutoLoader;
  42,64d40
  < sub AUTOLOAD {
  <     # This AUTOLOAD is used to 'autoload' constants from the constant()
  <     # XS function.
  < 
  <     my $constname;
  <     our $AUTOLOAD;
  <     ($constname = $AUTOLOAD) =~ s/.*:://;
  <     croak "&XS::Test::constant not defined" if $constname eq 'constant';
  <     my ($error, $val) = constant($constname);
  <     if ($error) { croak $error; }
  <     {
  <     no strict 'refs';
  <     # Fixed between 5.005_53 and 5.005_61
  < #XXX        if ($] >= 5.00561) {
  < #XXX            *$AUTOLOAD = sub () { $val };
  < #XXX        }
  < #XXX        else {
  <         *$AUTOLOAD = sub { $val };
  < #XXX        }
  <     }
  <     goto &$AUTOLOAD;
  < }
  < 
  68,71d43
  < # Preloaded methods go here.
  < 
  < # Autoload methods go after =cut, and are processed by the autosplit program.
  <

Let's verify that everything works:

  $ make test
  Running Mkbootstrap for XS::Test ()
  chmod 644 Test.bs
  rm -f blib/arch/auto/XS/Test/Test.so
  LD_RUN_PATH="" cc  -G Test.o  xs_test.o  -o blib/arch/auto/XS/Test/Test.so      
  chmod 755 blib/arch/auto/XS/Test/Test.so
  cp Test.bs blib/arch/auto/XS/Test/Test.bs
  chmod 644 blib/arch/auto/XS/Test/Test.bs
  PERL_DL_NONLAZY=1 perl "-MExtUtils::Command::MM" \
             "-e" "test_harness(0, 'blib/lib', 'blib/arch')" t/*.t
  t/XS-Test....ok
  All tests successful.
  Files=1, Tests=7,  0 wallclock secs ( 0.03 cusr +  0.00 csys =  0.03 CPU)

It works! Final Test.pm looks pretty simple:

  package XS::Test;
  
  use strict;
  use warnings;
  
  require Exporter;
  our @ISA = qw(Exporter);
  
  our %EXPORT_TAGS = ( 'all' => [ qw(
        XST_DEFINE
        XST_ENUM_1
        XST_ENUM_2
        XST_ENUM_VAL_1
        XST_ENUM_VAL_2
  ) ] );
  
  our @EXPORT_OK = ( @{ $EXPORT_TAGS{'all'} } );
  our @EXPORT = qw(
        XST_DEFINE
        XST_ENUM_1
        XST_ENUM_2
        XST_ENUM_VAL_1
        XST_ENUM_VAL_2
  );
  
  our $VERSION = '0.01';
  
  require XSLoader;
  XSLoader::load('XS::Test', $VERSION);
  
  1;
  __END__

And the final Test.xs file:

  #include "EXTERN.h"
  #include "perl.h"
  #include "XSUB.h"
  
  #include <xs_test.h>
  
  MODULE = XS::Test             PACKAGE = XS::Test              
  
  PROTOTYPES: ENABLE
  
  BOOT:
        {
        HV *stash;
  
        stash = gv_stashpv("XS::Test", TRUE);
        newCONSTSUB(stash, "XST_DEFINE", newSViv(XST_DEFINE));
        newCONSTSUB(stash, "XST_ENUM_1", newSViv(XST_ENUM_1));
        newCONSTSUB(stash, "XST_ENUM_2", newSViv(XST_ENUM_2));
        newCONSTSUB(stash, "XST_ENUM_VAL_1", newSViv(XST_ENUM_VAL_1));
        newCONSTSUB(stash, "XST_ENUM_VAL_2", newSViv(XST_ENUM_VAL_2));
        }
    
    
This concludes our exercise. Now you should be able to write simple extensions
that just export constants from header files and avoid dangerous pitfalls!

Technorati Tag: Perl