RAW digital photos on OpenSolaris and gcc vs Sun Studio

Tuesday Apr 10, 2007

Ireland was blessed with clear skies during the lunar eclipse of 3 March 2007. My combination of a Celestron 500mm F5.6 Maksutov mirror lens telescope and a Pentax *ist DL worked well. I set the white balance to daylight so the camera wouldn't try to compensate for the unearthly red tint earth's atmosphere caste on the moon. I also set the camera to RAW mode, to avoid JPEG artifacts and allow me to make use of the full dynamic range of the sensor chip. Unfortunately because I had previously used the camera to photograph documents in support of my Irish residency (important documents which have not yet been returned), the 6 Megapixel camera was set to 1 Megapixel. So Simon Phipps and Tim Foster both have better photos of this event.

I did learn something from this experiment; gimp, eog and other GNOME applications which are available on recent OpenSolaris desktops don't know how to handle RAW images. There's a very good reason for this. RAW isn't a digital image standard. RAW is a word which is used to refer to one of a number of relatively unprocessed proprietary digital camera image formats. These formats range from Sony's encrypted "RAW" format to formats which resemble tiff or Adobe's Digital Negative (DNG.) Ideally the manufacturers would agree on a standard such as Adobe's DNG the lossless JPEG used in medical imaging, or even Microsoft's "JPEG Killer", as long as patents, closed standards and other all too familiar tactics aren't used to "embrace and extinguish" digital cameras and internet imaging. Can you imagine if the open standard (JPEG and PNG) hadn't replaced GIF on the web and IBM and Unisys had decided to collect royalties?

Until camera manufacturers standardize on an open format, we will need conversion utilities. Fortunately these utilities exist. Dave Coffin wrote one such utility called "dcraw". I like the program, it's a small command line utility which only depends on standard C libraries. A portable command line utility probably makes more sense than a library because next week a camera manufacturer might come up with yet another "RAW" format and you'll have to change the source code and rebuild. If you're building this on Solaris or OpenSolaris, you'll probably want to apply a tiny patch:

bash-3.00$ diff -u dcraw.c.orig dcraw.c
--- dcraw.c.orig        Tue Mar  6 11:18:08 2007
+++ dcraw.c     Sat Mar 24 00:04:15 2007
@@ -317,7 +317,7 @@
 {
   fread (pixel, 2, count, ifp);
   if ((order == 0x4949) == (ntohs(0x1234) == 0x1234))
-    swab (pixel, pixel, count*2);
+    swab ((const char *) pixel, (char *) pixel, count*2);
 }
 
 void CLASS canon_600_fixed_wb (int temp)
@@ -1923,7 +1923,7 @@
   size_t nbytes;
 
   nbytes = fread (jpeg_buffer, 1, 4096, ifp);
-  swab (jpeg_buffer, jpeg_buffer, nbytes);
+  swab ((const char *) jpeg_buffer, jpeg_buffer, nbytes);
   cinfo->src->next_input_byte = jpeg_buffer;
   cinfo->src->bytes_in_buffer = nbytes;
   return TRUE;
@@ -7162,7 +7162,7 @@
           FORCC ppm [col*colors+c] = lut[image[soff][c]];
       else FORCC ppm2[col*colors+c] =     image[soff][c];
     if (output_bps == 16 && !output_tiff && htons(0x55aa) != 0x55aa)
-      swab (ppm2, ppm2, width*colors*2);
+      swab ((const char *) ppm2, (char *) ppm2, width*colors*2);
     fwrite (ppm, colors*output_bps/8, width, ofp);
   }
   free (ppm);

Slow

Building with Sun Studio requires a couple of options and still gives warnings about some of the unsigned char data exceeding 127, but gcc ignores it so we'll ignore it for now also:

bash-3.00$ cc -o sdcraw dcraw.c -lm -DNO_JPEG -DNO_LCMS
"dcraw.c", line 309: warning: implicit function declaration: getc_unlocked
"dcraw.c", line 3555: warning: initializer does not fit or is out of range: 128
"dcraw.c", line 3556: warning: initializer does not fit or is out of range: 136
"dcraw.c", line 3563: warning: initializer does not fit or is out of range: 128
"dcraw.c", line 3563: warning: initializer does not fit or is out of range: 136
"dcraw.c", line 3569: warning: initializer does not fit or is out of range: 128
"dcraw.c", line 3570: warning: initializer does not fit or is out of range: 136
bash-3.00$ time ./sdcraw ../Documents/Photos/20070401/IMGP3420.PEF

real    0m21.769s
user    0m19.671s
sys     0m0.222s

Guess why I named the executable sdcraw? Because it's slow, 21 seconds for a 6 megapixel Pentax file. That's because the Sun Studio in Solaris Express Developer edition still defaults to no optimization! Let's see how gcc does with no optimization flags:

Faster

bash-3.00$ /usr/sfw/bin/gcc -o sgdcraw dcraw.c -lm -DNO_JPEG -DNO_LCMS
bash-3.00$ time ./sgdcraw ../Documents/Photos/20070401/IMGP3420.PEF

real    0m15.308s
user    0m13.339s
sys     0m0.211s

That's a bit faster.  By default gcc doesn't seem to care if you're initializing
unsigned chars to values greater than 127, but it's default optimization seems 
somewhat better than Sun Studio's.  Let me try enabling some gcc optimization:

bash-3.00$ /usr/sfw/bin/gcc -O4 -o gdcraw dcraw.c -lm -DNO_JPEG -DNO_LCMS
bash-3.00$ time ./gdcraw ../Documents/Photos/20070401/IMGP3420.PEF

real    0m6.767s
user    0m5.220s
sys     0m0.189s

That's getting respectable!  Now let me try Sun Studio with the -fast, -xpentium and
-xspace flags.  You have to be careful with -fast, because by itself it assumes that
the target architecture is the same as the build architecture.  Since I'll be using it
on the same laptop I'm building it on, that's O.K.

Fastest!

bash-3.00$ cc -xc99=none -i -xspace -xstrconst -xpentium -mr -o fdcraw -fast dcraw.c -lm -DNO_JPEG -DNO_LCMS
"dcraw.c", line 3555: warning: initializer does not fit or is out of range: 128
"dcraw.c", line 3556: warning: initializer does not fit or is out of range: 136
"dcraw.c", line 3563: warning: initializer does not fit or is out of range: 128
"dcraw.c", line 3563: warning: initializer does not fit or is out of range: 136
"dcraw.c", line 3569: warning: initializer does not fit or is out of range: 128
"dcraw.c", line 3570: warning: initializer does not fit or is out of range: 136
bash-3.00$ time ./fdcraw ../Documents/Photos/20070401/IMGP3420.PEF
real    0m5.659s
user    0m4.160s
sys     0m0.208s

Sun Studio's default of no optimization is probably a good thing. I'm sure someone will complain if (when?) this default is changed, but until it is changed, newcomers to OpenSolaris might take a quick glance at Sun Studio and say "this sucks". If you're willing to read some man pages and explore some optimizations and tools (e.g. Sun Studio's analyzer profiler), you might grow to prefer Sun Studio.

If you're looking for a Solaris application and find it doesn't exist, try to find the source of a GNU/Linux or OSX application which does what you need and rebuild it for Solaris. And thank the maintainer of the application, Dave Coffin's dcraw is pretty useful. Once I've converted my Pentax PEF "RAW" original into IMGP3420.tiff, I can pull it into GIMP where I crop, resize and upload it to this blog entry. The above photo was taken at the Howth dolman a couple of weekends ago. The rhododendrons were already blooming.