Thursday Mar 12, 2009

A problem I encountered a few weeks ago was the following:

$ svcs -l sendmail
fmri         svc:/network/smtp:sendmail
svcs: svcs.c:335: Unexpected libscf error: invalid argument. Exiting. 

"svcs -a" produced no error and the correct output whereas "svcs -xv" also produced the above error.  It didn't matter which SMF service we chose, the same problem was seen when using the '-l' or '-xv' flags.  The question we needed to answer was what function within libscf was returning this error?  The source code showed several places where SCF_ERROR_INVALID_ARGUMENT could be set.

We can see that /usr/bin/svcs links with libscf.so.1:

$ ldd /usr/bin/svcs
    libcontract.so.1 =>     /usr/lib/libcontract.so.1
    libscf.so.1 =>     /usr/lib/libscf.so.1
    libuutil.so.1 =>     /usr/lib/libuutil.so.1
    libumem.so.1 =>     /usr/lib/libumem.so.1
    libc.so.1 =>     /usr/lib/libc.so.1
    libnvpair.so.1 =>     /usr/lib/libnvpair.so.1
    libgen.so.1 =>     /usr/lib/libgen.so.1
    libnsl.so.1 =>     /usr/lib/libnsl.so.1
    libmp.so.2 =>     /usr/lib/libmp.so.2
    libmd.so.1 =>     /usr/lib/libmd.so.1
    libm.so.2 =>     /usr/lib/libm.so.2 

As a first attempt at troubleshooting the problem we'd like to use the PID provider to look at what functions within libscf are called during the execution and see which are returning SCF_ERROR_INVALID_ARGUMENT.  However the first attempt produced an error:

$ dtrace -n 'pid$target:libscf.so.1::return /pid == $target/ {printf("= %d\n", arg1);}' -c "svcs -l sendmail" -o /var/tmp/dtrace.svcs.out
dtrace: invalid probe specifier pid$target:libscf.so.1::entry /pid == $target/ {printf("= %d\n", arg1);}: probe description pid1153:libscf.so.1::entry does not match any probes 

The issue here is that DTrace isn't loading the libscf.so.1 even though /usr/lib is in the LD_LIBRARY_PATH.  The trick/workaround is to use the LD_PRELOAD variable to pre-load the library before DTrace attaches.

$ export LD_PRELOAD_32=/usr/lib/libscf.so.1

Note we use LD_PRELOAD_32 for 32bit libraries and LD_PRELOAD_64 for 64bit libraries.  In theory setting LD_PRELOAD should allow the compiler/linker to select the correct ELF library but in this case it didn't which is why we use LD_PRELOAD_32 to force 32bit.

Running the DTruss on a working host we now get:

$  dtrace -n 'pid$target:libscf.so.1::return /pid == $target/ {printf("= %d\n", arg1);}' -c "svcs -l sendmail" -o /var/tmp/dtrace.svcs.out
dtrace: description 'pid$target:libscf.so.1::return ' matched 429 probes
fmri         svc:/network/smtp:sendmail
name         sendmail SMTP mail transfer agent
enabled      true
state        online
next_state   none
state_time   15 February 2009 01:49:26 GMT
logfile      /var/svc/log/network-smtp:sendmail.log
restarter    svc:/system/svc/restarter:default
contract_id  84 
dependency   require_all/refresh file://localhost/etc/mail/sendmail.cf (online)
dependency   require_all/refresh file://localhost/etc/nsswitch.conf (online)
dependency   optional_all/none svc:/system/filesystem/autofs (online)
dependency   require_all/none svc:/system/filesystem/local (online)
dependency   require_all/none svc:/network/service (online)
dependency   require_all/refresh svc:/milestone/name-services (online)
dependency   optional_all/refresh svc:/system/identity:domain (online)
dependency   optional_all/none svc:/system/system-log (online)
dtrace: pid 1406 has exited


This produced a 651k output file.  A better way might be to look for any functions that return -1, or SCF_ERROR_INVALID_ARGUMENT.

$ dtrace -n 'pid$target:libscf.so.1::return /(pid == $target) && (arg1 == -1)/ {printf("= %d\n", arg1);}' -c "svcs -l sendmail" -o /var/tmp/dtrace.svcs.out

Unfortunately the original problem was never fully investigated as the host was rebuilt before root cause could be established.

I was working on an issue that wasn't reproducible at will but had a failure frequency low enough to dtruss.  The problem with this approach is that it can leave you with a large dataset.  Unless you're actively watching for errors in /var/adm/messages or application logs and can quickly cross-reference or mark the dtruss output the only option is to post-process the ELAPSD column and calculate where in the dtruss dataset you need to look.  Here lies the first issue.  By default dtruss doesn't provide the date & time when the dtruss was started.  Easily fixed by running "date; dtruss <options>" but not ideal.  The first enhancement I've added to Brendan's dtruss is to print a walltimestamp in the dtrace:::BEGIN to make it similar to the truss(1M) output.  The second enhancement I added is to embed the walltimestamp in the truss output as a new column, the first column, which makes finding the relevant bit of dtruss trivial. I've added the '-w' flag to dtruss, also active via the '-a' argument.

# USAGE: dtruss [-acdeflhoLsw] [-t syscall] { -p PID | -n name | command }
#
#               -p PID          # examine this PID
#               -n name         # examine this process name
#               -t syscall      # examine this syscall only
#               -a              # print all details
#               -c              # print system call counts
#               -d              # print relative timestamps (us)
#               -e              # print elapsed times (us)
#               -f              # follow children as they are forked
#               -l              # force printing of pid/lwpid per line
#               -o              # print on cpu times (us)
#               -s              # print stack backtraces
#               -L              # don't print pid/lwpid per line
#               -w              # print walltimestamp (system date/time) 
#               -b bufsize      # dynamic variable buf size (default is "4m") 

Here's a quick example of my dtruss running showing both enhancements:

$ pfexec  ./dtruss -a -p `pgrep nwamd`
dtruss started tracing at 2009 Mar 12 22:57:44 
TIMESTAMP              PID/LWP    RELATIVE  ELAPSD    CPU SYSCALL(args)          = return
2009 Mar 12 22:57:55    4310/11:        65 10010277     11 lwp_park(0x0, 0xFDFCEF3C, 0x0)         = -1 Err#62
2009 Mar 12 22:58:05    4310/11:       118 10009670     10 lwp_park(0x0, 0xFDFCEF3C, 0x0)         = -1 Err#62
^C

CALL                                        COUNT
lwp_park                                        3

You can download my updated version of dtruss here.  Enjoy.

Monday Apr 28, 2008

In the previous blog entry we looked at how to obtain the video controls using the VIDIOC_QUERYCTRL & VIDIOC_QUERYMENU ioctls.  Now we'll take a look at how to use the VIDIOC_ENUM_FMT ioctl to enumerate image formats and codecs supported by the camera/capture device.

The man page for usbvc only gives a one-liner:

     VIDIOC_ENUM_FMT

         Enumerate the video formats supported by the device.

With no specific detail, we need to read the documentation which states:

"To enumerate image formats applications initialize the type and index field of struct v4l2_fmtdesc and call the VIDIOC_ENUM_FMT ioctl with a pointer to this structure. Drivers fill the rest of the structure or return an EINVAL error code. All formats are enumerable by beginning at index zero and incrementing by one until EINVAL is returned."

This is the v4l2_fmtdesc structure that we'll be using:

    336 /*
    337  *    F O R M A T   E N U M E R A T I O N
    338  */
    339 struct v4l2_fmtdesc
    340 {
    341     uint32_t        index;             /* Format number */
    342     enum            v4l2_buf_type  type;    /* buffer type */
    343     uint32_t        flags;
    344     char            description[32];        /* Description string */
    345     uint32_t        pixelformat;            /* Format fourcc     */
    346     uint32_t        reserved[4];
    347 };
    
The documentation tells us to use increasing integer values for 'index' starting at zero and incrementing by one until the ioctl call returns EINVAL.  That's just going to be a FOR loop so, nothing complicated there.  We need to do this for each of the buffer types (v4l2_buf_type).  These are the all the defined buffer types:

    120 enum v4l2_buf_type {
    121     V4L2_BUF_TYPE_VIDEO_CAPTURE     = 1,
    122     V4L2_BUF_TYPE_VIDEO_OUTPUT     = 2,
    123     V4L2_BUF_TYPE_VIDEO_OVERLAY     = 3,
    124     V4L2_BUF_TYPE_VBI_CAPTURE     = 4,
    125     V4L2_BUF_TYPE_VBI_OUTPUT     = 5,
    126 #if 1
    127     /* Experimental Sliced VBI */
    128     V4L2_BUF_TYPE_SLICED_VBI_CAPTURE = 6,
    129     V4L2_BUF_TYPE_SLICED_VBI_OUTPUT  = 7,
    130 #endif
    131     V4L2_BUF_TYPE_PRIVATE         = 0x80
    132 };

However the documentation also states that only the following is actually supported by the VIDIOC_ENUM_FMT ioctl:

    V4L2_BUF_TYPE_VIDEO_CAPTURE
    V4L2_BUF_TYPE_VIDEO_OUTPUT
    V4L2_BUF_TYPE_VIDEO_OVERLAY
    and custom (driver defined) types with code V4L2_BUF_TYPE_PRIVATE and higher.

The flags parameter within the v4l2_fmtdesc struct only has one option at this time, namely V4L2_FMT_FLAG_COMPRESSED.  If set then the current format is compressed else it's uncompressed.

The description paramter is just an ASCII name for the format and is based on the FourCC code.

The pixelformat parameter is a four character code that is calculated by the v4l2_fourcc macro:

     73 /*
     74  *    M I S C E L L A N E O U S
     75  */
     76
     77 /*  Four-character-code (FOURCC) */
     78 #define    v4l2_fourcc(a, b, c, d) \
     79     (((uint32_t)(a)<<0) | ((uint32_t)(b)<<8) | \
     80     ((uint32_t)(c)<<16)|((uint32_t)(d)<<24))
    
The unique FOURCC value assigned to every compression format and pixel layout allows video frames to be passed between file and codec by ensuring the FOURCC of the source frame matches a FOURCC supported by the codec.  To understand Fourcc further here's some reference material:
With the understanding of what needs to be done, we can now implement the following function to grab all supported formats/codecs.  get_supported_video_formats() is implemented within webcam.c to grab the video formats and codecs supported by the video camera/capture device.

/*
* get_supported_video_formats
*
*   @cam - Pointer to a camera object
*
*   Enumerate the video formats supported by the device using the
 *     VIDIOC_ENUM_FMT ioctl. 
 */
void get_supported_video_formats(camera_t * cam){
    /*
     * To enumerate image formats we have to initialise the type and index field
     *   of struct v4l2_fmtdesc and call the VIDIOC_ENUM_FMT ioctl with a pointer
     *   to this structure. Drivers fill the rest of the structure or return an
     *   EINVAL error code. All formats are enumerable by beginning at index zero
     *   and incrementing by one until EINVAL is returned.
     *
     * Only these buffer types are valid within the format field:
     *      V4L2_BUF_TYPE_VIDEO_CAPTURE
     *      V4L2_BUF_TYPE_VIDEO_OUTPUT
     *      V4L2_BUF_TYPE_VIDEO_OVERLAY
     *      and custom (driver defined) types with code V4L2_BUF_TYPE_PRIVATE and higher.
     *
     * This function only implements the first buffer types, i.e. no customer/private types
     */   

     struct v4l2_fmtdesc vid_fmtdesc;    /* Enumerated video formats supported by the device */
     memset(&vid_fmtdesc, 0, sizeof(vid_fmtdesc));
     vid_fmtdesc.index = 0;
     char *buf_types[] = {"VIDEO_CAPTURE","VIDEO_OUTPUT", "VIDEO_OVERLAY"}; /* Conversion between enumerated type & english */
     char *flags[] = {"uncompressed", "compressed"};  
     fprintf(stdout, "\nDiscovering supported video formats:\n");

     /* For each of the supported v4l2_buf_type buffer types */
     for (vid_fmtdesc.type = V4L2_BUF_TYPE_VIDEO_CAPTURE; vid_fmtdesc.type < V4L2_BUF_TYPE_VIDEO_OVERLAY; vid_fmtdesc.type++)
     {
         /* Send the VIDIOC_ENUM_FM ioctl and print the results */
         while( ioctl( cam->dev_fd, VIDIOC_ENUM_FMT, &vid_fmtdesc ) == 0 )
         {

             /* We got a video format/codec back */
             fprintf(stdout,"VIDIOC_ENUM_FMT(%d, %s)\n", vid_fmtdesc.index, buf_types[vid_fmtdesc.type-1]);
             fprintf(stdout, "  index        :%d\n", vid_fmtdesc.ind
             fprintf(stdout, "  type         :%s\n", buf_types[vid_fmtdesc.type-1]);
             fprintf(stdout, "  flags        :%s\n", flags[vid_fmtdesc.flags]);
             fprintf(stdout, "  description  :%s\n", vid_fmtdesc.description);

             /* Convert the pixelformat attributes from FourCC into 'human readable' format */
             fprintf(stdout, "  pixelformat  :%c%c%c%c\n",
                                vid_fmtdesc.pixelformat & 0xFF, (vid_fmtdesc.pixelformat >> 8) & 0xFF,
                                (vid_fmtdesc.pixelformat >> 16) & 0xFF, (vid_fmtdesc.pixelformat >> 24) & 0xFF);           

             /* Increment the index */
             vid_fmtdesc.index++;

         }
     }
}/* End of get_supported_video_formats() */   


The output of the above using a Toshiba Tecra M5 with a Logitech QuickCam Pro 5000 yields the following output:

Discovering supported video formats:
VIDIOC_ENUM_FMT(0, VIDEO_CAPTURE)
  index        :0
  type         :VIDEO_CAPTURE
  flags        :compressed
  description  :MJPEG
  pixelformat  :MJPG
VIDIOC_ENUM_FMT(1, VIDEO_CAPTURE)
  index        :1
  type         :VIDEO_CAPTURE
  flags        :uncompressed
  description  :YUYV
  pixelformat  :YUYV

The Sun Studio 12 Project (includes all source code) can be downloaded from here.

Sunday Apr 27, 2008

Following on from the previous blog entry where we got started with v4l2 & usbvc we'll now expand the functionality so we can query the device & driver to obtain the video controls (brightness, contrast, hue, etc).

The man page for usbvc(7D) states the following for the ioctl we're interested in

     VIDIOC_QUERYCTRL

         Query the device and driver  for  supported  video  con-
         trols.  Currently, the usbvc driver supports the bright-
         ness, contrast, saturation, hue, and  gamma  video  con-
         trols.

Unfortunately it doesn't elaborate as to what we expect happen when we send this ioctl.  For that we must go and read the documentation:

"To query the attributes of a control applications set the id field of a struct v4l2_queryctrl and call the VIDIOC_QUERYCTRL ioctl with a pointer to this structure. The driver fills the rest of the structure or returns an EINVAL error code when the id is invalid.

It is possible to enumerate controls by calling VIDIOC_QUERYCTRL with successive id values starting from V4L2_CID_BASE up to and exclusive V4L2_CID_BASE_LASTP1. Drivers may return EINVAL if a control in this range is not supported. Further applications can enumerate private controls, which are not defined in this specification, by starting at V4L2_CID_PRIVATE_BASE and incrementing id until the driver returns EINVAL.

In both cases, when the driver sets the V4L2_CTRL_FLAG_DISABLED flag in the flags field this control is permanently disabled and should be ignored by the application.[1]

When the application ORs id with V4L2_CTRL_FLAG_NEXT_CTRL the driver returns the next supported control, or EINVAL if there is none. Drivers which do not support this flag yet always return EINVAL."

Note that in Solaris, we do not implement the V4L2_CID_BASE_LASTP1, instead we have V4L2_CID_LASTP1.  Another subtle difference between the Linux & Solaris implementations.
 

The documentation asks us to iterate over the following id's which will auto-populate the v4l2_queryctrl structure.  I will only implement the first method outlined in the documentation.  For completeness however here are all the options defined within <sys/videodev2.h>:

    871 /*  Query flag, to be ORed with the control ID */
    872 #define    V4L2_CTRL_FLAG_NEXT_CTRL    0x80000000
    873
    874 /*  User-class control IDs defined by V4L2 */
    875 #define    V4L2_CID_BASE            (V4L2_CTRL_CLASS_USER | 0x900)
    876 #define    V4L2_CID_USER_BASE        V4L2_CID_BASE
    877 /*  IDs reserved for driver specific controls */
    878 #define    V4L2_CID_PRIVATE_BASE        0x08000000
    879
    880 #define    V4L2_CID_USER_CLASS        (V4L2_CTRL_CLASS_USER | 1)
    881 #define    V4L2_CID_BRIGHTNESS        (V4L2_CID_BASE+0)
    882 #define    V4L2_CID_CONTRAST        (V4L2_CID_BASE+1)
    883 #define    V4L2_CID_SATURATION        (V4L2_CID_BASE+2)
    884 #define    V4L2_CID_HUE            (V4L2_CID_BASE+3)
    885 #define    V4L2_CID_AUDIO_VOLUME        (V4L2_CID_BASE+5)
    886 #define    V4L2_CID_AUDIO_BALANCE        (V4L2_CID_BASE+6)
    887 #define    V4L2_CID_AUDIO_BASS        (V4L2_CID_BASE+7)
    888 #define    V4L2_CID_AUDIO_TREBLE        (V4L2_CID_BASE+8)
    889 #define    V4L2_CID_AUDIO_MUTE        (V4L2_CID_BASE+9)
    890 #define    V4L2_CID_AUDIO_LOUDNESS        (V4L2_CID_BASE+10)
    891 #define    V4L2_CID_BLACK_LEVEL        (V4L2_CID_BASE+11)
    892 #define    V4L2_CID_AUTO_WHITE_BALANCE    (V4L2_CID_BASE+12)
    893 #define    V4L2_CID_DO_WHITE_BALANCE    (V4L2_CID_BASE+13)
    894 #define    V4L2_CID_RED_BALANCE        (V4L2_CID_BASE+14)
    895 #define    V4L2_CID_BLUE_BALANCE        (V4L2_CID_BASE+15)
    896 #define    V4L2_CID_GAMMA            (V4L2_CID_BASE+16)
    897 #define    V4L2_CID_WHITENESS        (V4L2_CID_GAMMA) /* ? Not sure */
    898 #define    V4L2_CID_EXPOSURE        (V4L2_CID_BASE+17)
    899 #define    V4L2_CID_AUTOGAIN        (V4L2_CID_BASE+18)
    900 #define    V4L2_CID_GAIN            (V4L2_CID_BASE+19)
    901 #define    V4L2_CID_HFLIP            (V4L2_CID_BASE+20)
    902 #define    V4L2_CID_VFLIP            (V4L2_CID_BASE+21)
    903 #define    V4L2_CID_HCENTER        (V4L2_CID_BASE+22)
    904 #define    V4L2_CID_VCENTER        (V4L2_CID_BASE+23)
    905 #define    V4L2_CID_LASTP1            (V4L2_CID_BASE+24) /* last CID + 1 */

Here's the v4l2_queryctrl structure which will get populated.  Again, it is defined in <sys/videodev2.h>:

    840 /*  Used in the VIDIOC_QUERYCTRL ioctl for querying controls */
    841 struct v4l2_queryctrl
    842 {
    843     uint32_t        id;
    844     enum            v4l2_ctrl_type type;
    845     char            name[32];     /* Whatever */
    846     int32_t         minimum; /* Note signedness */
    847     int32_t         maximum;
    848     int32_t         step;
    849     int32_t         default_value;
    850     uint32_t        flags;
    851     uint32_t        reserved[2];
    852 };

The enumerated v4l2_ctrl_type structure is defined as:

    134 enum v4l2_ctrl_type {
    135     V4L2_CTRL_TYPE_INTEGER        = 1,
    136     V4L2_CTRL_TYPE_BOOLEAN        = 2,
    137     V4L2_CTRL_TYPE_MENU         = 3,
    138     V4L2_CTRL_TYPE_BUTTON         = 4,
    139     V4L2_CTRL_TYPE_INTEGER64     = 5,
    140     V4L2_CTRL_TYPE_CTRL_CLASS     = 6
    141 };
    
The documentation goes on to say:   

"Additional information is required for menu controls, the name of menu items. To query them applications set the id and index fields of struct v4l2_querymenu and call the VIDIOC_QUERYMENU ioctl with a pointer to this structure. The driver fills the rest of the structure or returns an EINVAL error code when the id or index is invalid. Menu items are enumerated by calling VIDIOC_QUERYMENU with successive index values from struct v4l2_queryctrl minimum (0) to maximum, inclusive. " 
   
In other words if the v4l2_queryctrl.type is of type V4L2_CTRL_TYPE_MENU then we need to do some more processing by issuing VIDIOC_QUERYMENU ioctl which will populate the v4l2_querymenu structure, defined as:

    854 /*  Used in the VIDIOC_QUERYMENU ioctl for querying menu items */
    855 struct v4l2_querymenu
    856 {
    857     uint32_t        id;
    858     uint32_t        index;
    859     uint8_t        name[32];    /* Whatever */
    860     uint32_t        reserved;
    861 };
   
With a better understanding of what we need to do, we can implement the function to enumerate the controls.  Within webcam.c the enum_video_ctrls_and_menus(camera_t * cam) function has been implemented.  Note that all variables are local, ie. there's nothing added to our camera_t structure.  If we decide later to keep these values then we could update the structure, but for now this suits the purpose.

/*
* enum_video_ctrls_and_menus
*
*   @cam - Pointer to a camera object
*
*   Enumerate controls and menu control items
 *     VIDIOC_QUERYCTRL & VIDIOC_QUERYMENU ioctls. 
 */
void enum_video_ctrls_and_menus(camera_t * cam){
/*
  * To query the attributes of a control applications set the id field of a struct
  * v4l2_queryctrl and call the VIDIOC_QUERYCTRL ioctl with a pointer to this
  * structure. The driver fills the rest of the structure or returns an EINVAL
  * error code when the id is invalid.
  *
  * To enumerate controls call VIDIOC_QUERYCTRL with successive
  * id values starting from V4L2_CID_BASE up to and exclusive V4L2_CID_LASTP1,
  * or starting from V4L2_CID_PRIVATE_BASE until the driver returns EINVAL.
  */ 
  struct v4l2_queryctrl queryctrl;    /* Query Control structure as defined in <sys/videodev2.h> */

struct v4l2_querymenu querymenu;    /* Query Menu Structure as defined in <sys/videodev2.h> */

  fprintf(stdout,"Discovering controls:\n");
 

  memset (&queryctrl, 0, sizeof (queryctrl));  
  for (queryctrl.id = V4L2_CID_BASE; queryctrl.id < V4L2_CID_LASTP1; queryctrl.id++) {
      if ( ioctl (cam->dev_fd, VIDIOC_QUERYCTRL, &queryctrl) == 0 ) {
          /* Check to see if this control is permanently disabled and should be ignored by the application */
          if (queryctrl.flags & V4L2_CTRL_FLAG_DISABLED)
              continue;
          /* We got a video control back */
          fprintf(stdout,"VIDIOC_QUERYCTRL(V4L2_CID_BASE+%d)\n", queryctrl.id-V4L2_CID_BASE);
          fprintf(stdout,"   id: %d\n", queryctrl.id);
          switch (queryctrl.type){
              case V4L2_CTRL_TYPE_INTEGER:
                  fprintf(stdout, "   type: INTEGER\n");
                  break;
              case V4L2_CTRL_TYPE_BOOLEAN:
                  fprintf(stdout, "   type: BOOLEAN\n");
                  break;
              case V4L2_CTRL_TYPE_MENU:
                  fprintf(stdout, "   type: MENU\n");
                  /* Additional information is required for menu controls, the name of menu items.
                   * To query them applications set the id and index fields of struct v4l2_querymenu
                   *   and call the VIDIOC_QUERYMENU ioctl with a pointer to this structure. The driver
                   *   fills the rest of the structure or returns an EINVAL error code when the id or
                   *   index is invalid. Menu items are enumerated by calling VIDIOC_QUERYMENU with
                   *   successive index values from struct v4l2_queryctrl minimum (0) to maximum, inclusive.
                   */
                  querymenu.id = queryctrl.id;
                  for (querymenu.index = queryctrl.minimum; querymenu.index < queryctrl.maximum; querymenu.index++){
                      fprintf(stdout, "      menu id: %d\n", querymenu.index);
                      fprintf(stdout, "      menu name: %s\n", querymenu.name);
                  }
                  break;
              case V4L2_CTRL_TYPE_BUTTON:
                  fprintf(stdout, "   type: BUTTON\n");
                  break;
          }
          fprintf(stdout,"   name: %s\n", queryctrl.name);
          fprintf(stdout,"   minimum: %d\n", queryctrl.minimum);
          fprintf(stdout,"   maximum: %d\n", queryctrl.maximum);
          fprintf(stdout,"   step: %d\n", queryctrl.step);
          fprintf(stdout,"   default_value: %d\n", queryctrl.default_value);
          fprintf(stdout,"   flags: %d\n", queryctrl.flags);

      } else {

             if (errno == EINVAL)
                 continue;

             perror("VIDIOC_QUERYCTRL");
             break;
      }

   }     

} /* End of enum_video_ctrls_and_menus() */

Using my Toshiba Tecra M5 & Logitech Quickcam Pro 5000 camera I get the following output when running the enum_video_ctrls_and_menus() function:

Discovering controls:
VIDIOC_QUERYCTRL(V4L2_CID_BASE+0)
   id: 9963776
   type: INTEGER
   name: Brightness
   minimum: 0
   maximum: 255
   step: 1
   default_value: 127
   flags: 0
VIDIOC_QUERYCTRL(V4L2_CID_BASE+1)
   id: 9963777
   type: INTEGER
   name: Contrast
   minimum: 0
   maximum: 255
   step: 1
   default_value: 32
   flags: 0
VIDIOC_QUERYCTRL(V4L2_CID_BASE+2)
   id: 9963778
   type: INTEGER
   name: Saturation
   minimum: 0
   maximum: 255
   step: 1
   default_value: 32
   flags: 0
If you want the Sun Studio 12 Project, which includes all the source code, you can download it here.


 

Saturday Apr 05, 2008

Introduction 

The Video4Linux2 and usbvc are kernel drivers to allow the interaction of applications with webcams.  Before diving in I strongly recommend you read the man page for usbvc(7D) and study the associated header files:

  • /usr/include/sys/usb/clients/video/usbvc/usbvc.h
  • /usr/include/sys/videodev2.h

If you're unfamiliar with this type of low-level API one of the first things that strikes you is a distinct lack of user functions.  That's because we're extremely close to kernel land and the way to manage userland <-> kernel interaction is via ioctl(2m) or read/write commands if the device/driver supports it.  It's up to you to write the necessary wrapper functions for your application.  There are some very good wrappers if you're on Linux or Windows including:

  • Intel Open Computer Vision (OpenCV)
  • Logitech's libwebcam

We could port these to Solaris but that's no fun, although I'll be porting them at a later date when I get time.  You learn much more by writing your own which is why I'm going about it this way.  I'll be using the aforementioned libraries as a guide because there's no need to reinvent the wheel.

Getting Started 

You can use whatever editor & compiler you feel most comfortable with but I'll be using Sun Studio 12 only because I've never used it before and thought this would be a good time to learn plus it's got some great features and I expect this app/API/Library to grow across multiple files so what better way to manage the project.

If you've read the man page for usbvc(7D) you'll note that one of the first ioctl commands it talks about is VIDIOC_QUERYCAP. 
 

Devices                                                 usbvc(7D)

     VIDIOC_QUERYCAP

         Query the device capabilities. Besides device  capabili-
         ties, the usbvc driver returns structure v4l2_capability
         which includes information on the driver, data  bus  and
         OS  kernel.  Please  note  that  the "Version" structure
         member has no meaning in Solaris and is always set to 1.

 

Sounds like a good place to start.

If you want my Sun Studio12 project directory you can grab it from here.  Download and extract it to your Sun Studio12 project directory then import it using File -> Open Project.  You'll need to do a clean build (Shift+F11) to ensure it all works before doing any editing.  If you just want the webcam.c & webcam.h files you can download webcam_cap_filesonly.tar.gz but you'll have to build your own Makefile or use the correct compiler options.

Defining the camera_t structure 

If you read the videodev2.h and usbvc.h files you'll see that there's lots of features available so the best way to manage all those features is to declare a structure which we can then use throughout our code.  This way we only need to declare a single pointer to the structure rather than having multiple variables.  It also makes things easier when we add support for multiple capture devices.  Within webcam.h I've declared the following structure:

typedef struct camera {
    int dev;                            /* Camera Device Number 0..N */
    char *video_dev;                    /* Video device path, i.e. /dev/videoN - where N is the device number */
    int dev_fd;                         /* File descriptor for /dev/videoN */
    int contrast;                       /* Contrast value */
    int brightness;                     /* Brightness value */
    int colour;                         /* Colour value */
    int hue;                            /* Hue value */
    int wb;                             /* White Balance value */
    int frame_number;                   /* Frame Number 0..N */
    struct v4l2_capability vid_cap;     /* Video capability bit field as defined in <sys/videodev2.h>*/
    struct v4l2_buffer vid_buf;         /* Video buffer structure as defined in <sys/videodev2.h> */
} camera_t;

Not all of the above is required for this simple example but eventually this structure will grow as we add more features so I've stuck a few things in here while I remember.  The purpose here is to populate the "v4l2_capability vid_cap" structure so we know what we can and can't do with our webcam.  It's also a good test to ensure it's plugged in and that the usbvc can read/write to it.

webcam.c/main()

If you look at the main() function within webcam.c it's a fairly straight forward piece of code.  All we're doing here is declaring the object camera_t object as a usable pointer (*cam), set a few defaults for our object, verify we can open the  webcam device read/write, and if successful we then call our get_camera_cap() function to find out what capabilities our device has.

   141  int main(int argc, char** argv) {
142 camera_t cam_object, *cam; /* Declare a pointer to a camera object. See webcam.h */
143 cam = &cam_object;
144
145 /* Set some defaults for the camera device */
146 cam->frame_number = 0;
147 cam->video_dev = "/dev/video0"; /* We're only going to support one device for now.
148 * In future we'll get the device or device number
149 * from the command line arguments. */
150 fprintf(stdout, "Video Device: %s\n", cam->video_dev);
151
152 /* Attempt to open the camera device read/write */
153 if ((cam->dev_fd = open(cam->video_dev, O_RDWR)) == -1)
154 {
155 perror("Cannot open /dev/video0");
156 exit(1);
157 } else {
158 fprintf(stdout, "Succesfully opened /dev/video0 and got file descriptor: %d \n", cam->dev_fd);
159 /* Get and display the camera capabilities */
160 fprintf(stdout, "Ascertaining camera capabilities...\n");
161 get_camera_cap(cam);
162 };
163
164 /* Clean up before exiting */
165 close(cam->dev_fd);
166
167 return (EXIT_SUCCESS);
168 }

Once we've obtained the webcam's capabilities we then close the device and exit.

webcam.c/get_camera_cap()

The man page states that is we send the VIDIOC_QUERYCAP ioctl, and it's successful, we should expect to get a v4l2_capability structure returned.  This v4l2_capability structure has been declared within our camera_t structure as vid_cap so we'll access it via cam->vid_cap.

If we look at /usr/include/sys/videodev2.h that structure is defined as:

   207  struct v4l2_capability
   208  {
   209          uint8_t         driver[16];     /* i.e. "bttv" */
   210          uint8_t         card[32];       /* i.e. "Hauppauge WinTV" */
   211          uint8_t         bus_info[32];   /* "PCI:" + pci_name(pci_dev) */
   212          uint32_t        version;        /* should use KERNEL_VERSION() */
   213          uint32_t        capabilities;   /* Device capabilities */
   214          uint32_t        reserved[4];
   215  };
   216
   217  /* Values for 'capabilities' field */
   218
   219  /* Is a video capture device */
   220  #define V4L2_CAP_VIDEO_CAPTURE          0x00000001
   221
   222  /* Is a video output device */
   223  #define V4L2_CAP_VIDEO_OUTPUT           0x00000002
   224  #define V4L2_CAP_VIDEO_OVERLAY          0x00000004  /* Can do video overlay */
   225
   226  /* Is a raw VBI capture device */
   227  #define V4L2_CAP_VBI_CAPTURE            0x00000010
   228
   229  /* Is a raw VBI output device */
   230  #define V4L2_CAP_VBI_OUTPUT             0x00000020
   231    #if 1
   232  
   233    /* Is a sliced VBI capture device */
   234    #define V4L2_CAP_SLICED_VBI_CAPTURE    0x00000040
   235  
   236    /* Is a sliced VBI output device */
   237    #define V4L2_CAP_SLICED_VBI_OUTPUT    0x00000080
   238    #endif
   239    #define V4L2_CAP_RDS_CAPTURE        0x00000100  /* RDS data capture */
   240  
   241    #define V4L2_CAP_TUNER            0x00010000  /* has a tuner */
   242    #define V4L2_CAP_AUDIO            0x00020000  /* has audio support */
   243    #define V4L2_CAP_RADIO            0x00040000  /* is a radio device */
   244  
   245    #define V4L2_CAP_READWRITE        0x01000000  /* read/write systemcalls */
   246    #define V4L2_CAP_ASYNCIO        0x02000000  /* async I/O */
   247    #define V4L2_CAP_STREAMING        0x04000000  /* streaming I/O ioctls */

The v4l2_capability structure is simple but we do have a bit field to contend with for the device capabilities.  Remember, on Solaris 'version' isn't supported so we only expect to get "1" returned.  Essentially all the get_camera_cap() function does at this stage is issue the ioctl and if successful print the resulting v4l2_capability structure to stdout.  Most of the code within this function handles the capabilities bitfield to determine what flags are set and which are not.  Line 38 is where we send the ioctl:

    26  /*
27 * get_camera_cap()
28 *
29 * @cam - Pointer to a camera object
30 */
31 void get_camera_cap(camera_t * cam){
32 char *msg;
33
34 /* Send the VIDIOCGCAP ioctl to the camera and see what it returns.
35 * If successful, the ioctl will return a populated v4l2_capability
36 * structure which we deposit within our cam object for later reference.
37 */
38 if(ioctl(cam->dev_fd, VIDIOC_QUERYCAP, &cam->vid_cap) == -1) {
39 fprintf(stdout, "Could not connect to video device (%s).\nPlease check connection.", cam->video_dev);
40 exit(0);
41 } else {
42 fprintf(stdout, " Driver: %s\n", cam->vid_cap.driver);
43 fprintf(stdout, " Card: %s\n", cam->vid_cap.card);
44 fprintf(stdout, " Bus Info: %s\n", cam->vid_cap.bus_info);
45 fprintf(stdout, " Version: %2d\n", cam->vid_cap.version);
46 fprintf(stdout, " Capabilities: \n");
47
48 /*
49 * Test each flag to see what is and is not set
50 * Refer to "D R I V E R C A P A B I L I T I E S" section of videodev2.h (lines 204-247)
51 */
52
53 /* Is this a video capture device? (Yes/No) */
54 if ( cam->vid_cap.capabilities & V4L2_CAP_VIDEO_CAPTURE ){
55 fprintf(stdout, " Video capture support: Yes\n");
56 } else {
57 fprintf(stdout, " Video capture support: No\n");
58 }
59 /* Does the device support video output? (Yes/No) */
60 if ( cam->vid_cap.capabilities & V4L2_CAP_VIDEO_OUTPUT ){
61 fprintf(stdout, " Video output support: Yes\n");
62 } else {
63 fprintf(stdout, " Video output support: No\n");
64 }
65 /* Does the device support video overlay? (Yes/No) */
66 if ( cam->vid_cap.capabilities & V4L2_CAP_VIDEO_OVERLAY ){
67 fprintf(stdout, " Video overlay support: Yes\n");
68 } else {
69 fprintf(stdout, " Video overlay support: No\n");
70 }
71 /* Supports raw VBI capture (Yes/No) */
72 if ( cam->vid_cap.capabilities & V4L2_CAP_VBI_CAPTURE ){
73 fprintf(stdout, " RAW VBI capture support: Yes\n");
74 } else {
75 fprintf(stdout, " RAW VBI capture support: No\n");
76 }
77 /* Supports raw VBI output (Yes/No) */
78 if ( cam->vid_cap.capabilities & V4L2_CAP_VBI_OUTPUT ){
79 fprintf(stdout, " RAW VBI output support: Yes\n");
80 } else {
81 fprintf(stdout, " RAW VBI output support: No\n");
82 }
83 /* Supports sliced VBI capture (Yes/No) */
84 if ( cam->vid_cap.capabilities & V4L2_CAP_SLICED_VBI_CAPTURE ){
85 fprintf(stdout, " Sliced VBI capture support: Yes\n");
86 } else {
87 fprintf(stdout, " Sliced VBI capture support: No\n");
88 }
89 /* Supports slices VBI output (Yes/No) */
90 if ( cam->vid_cap.capabilities & V4L2_CAP_SLICED_VBI_OUTPUT ){
91 fprintf(stdout, " Sliced VBI capture support: Yes\n");
92 } else {
93 fprintf(stdout, " Sliced VBI capture support: No\n");
94 }
95 /* Supports RDS Capture (Yes/No) */
96 if ( cam->vid_cap.capabilities & V4L2_CAP_RDS_CAPTURE ){
97 fprintf(stdout, " RDS capture support: Yes\n");
98 } else {
99 fprintf(stdout, " RDS capture support: No\n");
100 }
101 /* Is this a tuner device? (Yes/No) */
102 if ( cam->vid_cap.capabilities & V4L2_CAP_TUNER ){
103 fprintf(stdout, " Device has a tuner: Yes\n");
104 } else {
105 fprintf(stdout, " Device has a tuner: No\n");
106 }
107 /* Device has audio support (Yes/No) */
108 if ( cam->vid_cap.capabilities & V4L2_CAP_AUDIO ){
109 fprintf(stdout, " Audio support: Yes\n");
110 } else {
111 fprintf(stdout, " Audio support: No\n");
112 }
113 /* Is a radio device (Yes/No) */
114 if ( cam->vid_cap.capabilities & V4L2_CAP_RADIO ){
115 fprintf(stdout, " Device has radio support: Yes\n");
116 } else {
117 fprintf(stdout, " Device has radio support: No\n");
118 }
119 /* Supports read/write system calls (Yes/No) */
120 if ( cam->vid_cap.capabilities & V4L2_CAP_READWRITE ){
121 fprintf(stdout, " Supports read/write system calls: Yes\n");
122 } else {
123 fprintf(stdout, " Supports read/write system calls: No\n");
124 }
125 /* Supports asynchronous I/O (Yes/No) */
126 if ( cam->vid_cap.capabilities & V4L2_CAP_ASYNCIO ){
127 fprintf(stdout, " Supports async i/o: Yes\n");
128 } else {
129 fprintf(stdout, " Supports async i/o: No\n");
130 }
131 /* Supports streaming */
132 if ( cam->vid_cap.capabilities & V4L2_CAP_STREAMING ){
133 fprintf(stdout, " Supports streaming: Yes\n");
134 } else {
135 fprintf(stdout, " Supports streaming: No\n");
136 }
137
138 }
139 } /* End of get_camera_cap() function */

When I run this on my Toshiba Tecra M5 with a USB Logitech Quickcam Pro 5000 I get:

 

Video Device: /dev/video0
Succesfully opened /dev/video0 and got file descriptor: 3
Ascertaining camera capabilities...
  Driver: usbvc
  Card: Generic USB videoclass device
  Bus Info: usb
  Version:  1
  Capabilities:
    Video capture support: Yes
    Video output support: No
    Video overlay support: No
    RAW VBI capture support: No
    RAW VBI output support: No
    Sliced VBI capture support: No
    Sliced VBI capture support: No
    RDS capture support: No
    Device has a tuner: No
    Audio support: No
    Device has radio support: No
    Supports read/write system calls: Yes
    Supports async i/o: No
    Supports streaming: Yes

This is what you'd expect to see from just a webcam.  Now that we know what the driver will let us do with the device we can extend this code to do something useful, like grab frames.  I'll cover this in the next installment.

 


Tuesday Mar 18, 2008

There's a couple of multitouch system methods, FTIR and Diffuse Illumination.  I'll be building the FTIR system.
 
FTIR (frustrated total internal reflection) is a very simple principle that basically says that if you light up a certain material with some light, the light will stay inside the material itself, boucing on it's external surfaces until the external surfaces is "frustrated", for example, with a finger touch. In this case the light is bounced back from the finger and exit the material. So, with FTIR multitouch display, this material is plexiglas and we use the usual IR-pass modified camera to track the finger's position. This principle works also with infrared light so that the tracking light will be not disturbed by visible light.

FTIR.gif

I wanted to start off small given that this is only a "dip your toe in the water" type of project.  After a bit of research I decided on the following shopping list:

  • 500mm x 500mm x 6mm Acrylic Sheet
  • 500mm x 500mm x 2mm Acrylic Sheet
  • Safety mirror (A3 size)
  • A0 Tracing Paper
  • Two 2m x 10mm x 10mm C/U channel extruded aluminum (8mm internal dimension)
  • InfraRed LEDs
  • Resistors
  • 350W ATX PSU (I didn't have any old ones lying around)
  • Wet & Dry paper (300, 600, 800, & 1000 grade)
  • Brasso Metal Polish
  • 2m x 50cm strip of cord material
  • Webcam

Most of this stuff can be found in local shops or purchased off the internet.
 

InfraRed LEDs 

The first decision was regarding the IR LEDs and the circuit design.  Near infrared leds (the type we want) are in the 810nm-1400nm range.  IR leds around 840/850nm are used in remote controls.  They provide lots of IR light however, because they are close to the red end of the spectrum the naked eye can see them glowing red.  I didn't want this so I moved up the scale towards the 940/950nm range which is outside what the eye can detect.  The final decision was made purely on a cost basis.  Farnell (www.farnell.co.uk) had some wide angle 940nm 5mm IR Emitting LEDs for £0.10 each if you bought 100 or more.  I opted for Avago Technologies HSDL-4271.  The data sheet can be found on this page.  The datasheet is very important when designing the circuit and calculating the resistor values.

There's no hard and fast rules about how many LEDs you'll need so I decided I would try and experiment a little.  Some people have leds along one side, two sides, or along all four.  I chose the latter to give maximum experimental potential.  The maths suggested that if I spaced my LEDs at 2cm intervals I would need 24 (3 groups of 8) for each side making 96 LEDs in total.  The experimentation would come later as I proposed to build the tablet such that I could connect & disconnect each bank of 8 LEDs to see what impact it had.

Using this great online LED Resistor Calculator I had my circuit and the required resistors (27R 1/2W).  Here's what the wizard suggested:

Solution 2: 8 x 12 array uses 96 LEDs exactly

+12V
+-|>|--|>|--|>|--|>|--|>|--|>|--|>|--|>|--/\/\/\-+R = 27 ohms
+-|>|--|>|--|>|--|>|--|>|--|>|--|>|--|>|--/\/\/\-+R = 27 ohms
+-|>|--|>|--|>|--|>|--|>|--|>|--|>|--|>|--/\/\/\-+R = 27 ohms
+-|>|--|>|--|>|--|>|--|>|--|>|--|>|--|>|--/\/\/\-+R = 27 ohms
+-|>|--|>|--|>|--|>|--|>|--|>|--|>|--|>|--/\/\/\-+R = 27 ohms
+-|>|--|>|--|>|--|>|--|>|--|>|--|>|--|>|--/\/\/\-+R = 27 ohms
-|>|--|>|--|>|--|>|--|>|--|>|--|>|--|>|--/\/\/\-R = 27 ohms
+-|>|--|>|--|>|--|>|--|>|--|>|--|>|--|>|--/\/\/\-+R = 27 ohms
+-|>|--|>|--|>|--|>|--|>|--|>|--|>|--|>|--/\/\/\-+R = 27 ohms
+-|>|--|>|--|>|--|>|--|>|--|>|--|>|--|>|--/\/\/\-+R = 27 ohms
+-|>|--|>|--|>|--|>|--|>|--|>|--|>|--|>|--/\/\/\-+R = 27 ohms
+-|>|--|>|--|>|--|>|--|>|--|>|--|>|--|>|--/\/\/\-+R = 27 ohms
The wizard says: In solution 2:
  • each 27 ohm resistor dissipates 270 mW
  • the wizard thinks 1/2W resistors are needed for your application Help
  • together, all resistors dissipate 3240 mW
  • together, the diodes dissipate 11520 mW
  • total power dissipated by the array is 14760 mW
  • the array draws current of 1200 mA from the source.

 

That's quite a high current draw so the cheapest power supply is a computer AT/ATX PSU.  These can be purchased very cheaply now and even a 350W PSU will be more than enough.
 

 Webcam

 OpenSolaris hasn't had webcam support for very long but any UVC supported webcam should work.  These are the ones known to work with OpenSolaris:

    Device Name                          Vendor ID, Product ID

Logitech Quickcam Pro 5000                046d,08c5
Logitech Quickcam Pro 5000                046d,08ce
Logitech Quickcam Fusion                  046d,08ca
Logitech Quickcam Orbit MP                046d,08cc
Logitech Quickcam Pro for Notebooks       046d,08cb
Logitech Quickcam Pro for Notebooks       046d,0991
Logitech Quickcam Ultra Vision            046d,08c9
Logitech Quickcam Deluxe for notebooks    046d,09c1
Logitech Quickcam Communicate Deluxe      046d,0992
Logitech Quickcam Pro 9000                046d,0990

I managed to find the 'Logitech Quickcam Pro 5000' on eBay for £25 so I ordered it as it looked to be one of the easier models to modify and convert it to an InfraRed camera.  Normally cameras have an Infrared filter built in such that they only detect visible light.  We want the opposite, we need to block visible light and allow IR light in.  I'll cover how to accomplish this in a later blog but essentially I'll be taking the camera apart to remove the IR filter and replacing it with some 35mm film which is an excellent and cheap filter for my needs.
 


Monday Mar 17, 2008

There seems to be quite a plethora of multitouch projects springing up all over the web so I'm jumping on the band wagon and having a go myself. The only difference being that the host OS will of course be OpenSolaris. If you're not sure what a multitouch interface is punch "multitouch" into YouTube and enjoy.

With the advent of usbvc(1m) being introduced into OpenSolaris b57 (iirc) I'm a little late to the table (current release is b86) but better late than never. In fact I've known about these types of interfaces for years but never got around to actually building one...until now. As yet I haven't found anyone using OpenSolaris so is this going to be a first?

The project has many personal goals including:

  • Learning C (again)
  • Learning the usbvc & Video4Linux2 API's
  • Implementing some Computer Vision/Real Time image processing
  • Building a hardware based project (all projects so far have been software)

The ultimate aim of course is to produce a platform which colleagues can use for their own projects and applications.

There's two branches to this project; the hardware and the software. Both can be done in parallel to some degree depending on what I'm doing on any given day.

Let the quest begin....

This blog copyright 2009 by Steve Scargall