Things on my mind. George Drapeau's Weblog

Sometimes my TiVo gets confused after a power outage, and it takes forever to reboot (I think it's trying to recover its files which takes a while when you have over 1TB of storage connected to the TiVo).  Sometimes, I lose some of my shows, and that makes me sad.

So I decided to find some software I could get running on OpenSolaris so I could use my home media server as a backup device for my TiVo shows.

I did it, using the Galleon open source Java application; here's how.

Now, if somebody smarter than me would just port the TiVo software to OpenSolaris, we'd have the benefits of ZFS on a TiVo box.  That would be fantastic.

Powered by ScribeFire.

Sun's ISV Engineering group has been working hard all spring and summer to get fistfuls of popular and important open source applications into the OpenSolaris "contrib" repository, a repo for third-party applications that can be assumed to have some level of sanity checking done on them.

Eric Reid in ISV Engineering has submitted three different releases of Drupal; today, the final release (Acquia Drupal) was approved and placed into /contrib.  These three applications are

1. Drupal 5 (release 5.20)
   
2. Drupal 6 (release 6.14)
   
3. Acquia Drupal, created by the commercial enterprise backing Drupal: Acquia.com
   

It's nice and easy to try out any of these Drupal packages in OpenSolaris.  Want help?  Read this entry on drupal.org for complete yet succinct instructions.

(update)
I should also point out that when you install Drupal in OpenSolaris, the package management system will pull in the Sun WebStack components automatically, which is all free as you would expect.  It's the AMP stack components you know and love, but optimized by Sun for out-of-the-box performance improvement, plus a management console that lets you see what's going on with the components while you're running your Drupal web site.

(Okay, that may have sounded a bit like an advertisement, but we really did put a lot of engineering effort into optimizing the AMP components on Sun's systems and the new analytics thingy is pretty cool looking.)
(/update)

Meanwhile, this adds one of the premier open source content management systems to the stable of apps freely and easily available to you when you install OpenSolaris on your computer.  You can even choose from amongst the leaders in blog engines / content management systems / web site building tools: Drupal, Joomla!, WordPress are all available now.  Here is a list of packages published to the contrib repository.

Check 'em out!

Powered by ScribeFire.

The OpenSolaris "/contrib" repository is a place where anybody can contribute open source software packages, which are then available for you to install on OpenSolaris.  Before a packages gets promoted to the contrib repo, you submit your package / application to a repository called "pending".  The idea is that people can test packages and get others to try them out before they get promoted so that the packages in "contrib" have been at least smoke-tested.

Recently I've noticed the number of packages and apps available growing pretty nicely, and I've been testing some of the apps that our own group (Sun's ISV Engineering organization) has been contributing.  Most of the packages I've tested have gone smoothly.  One gave me a little hiccup so I thought I'd describe what I did to get it working.

That package is the Joomla! content management system.

Installation was actually a snap; once you install the package with the OpenSolaris Package Manager, just fire up the web browser and go to http://localhost/joomla and you get the Joomla! installer.

Well, it's a little more complicated than that; you need to make sure you create a database for Joomla! to use, like this:

mysqladmin -u joomla -p create joomla

mysql -u joomla -p
mysql> grant all privileges on joomla.* to 'joomla'@'localhost' identified by [password in single quotes here];

(assuming I created a MySQL database user called "joomla", I'm also creating a database called "joomla")

After a few web screens, I can login as the Joomla! admin just fine, but trying to go to the Joomla! front page as a regular non-admin person causes a long series of errors to spit out at me (and rather rudely, I might add); see the end of this post for the details.

Turns out it's a MySQL problem; you can see the discussion here.  The current workaround is to run MySQL in 64-bit mode, but in order to do that, you need to make sure you're running OpenSolaris in 64-bit mode.  If you type "isainfo -k" and it returns "x86", that's not good enough.

I am doing my testing in VirtualBox; to make sure I'm running OpenSolaris in 64-bit mode, I go to the VirtualBox main application panel where I can see the settings for my virtual machine.  Near the top, the section called "General", The "OS Type" field sais "OpenSolaris".  I change that to "OpenSolaris (64-bit)" and the next time I boot OpenSolaris, it's in 64-bit mode.

One more step; making sure to launch the 64-bit version of the MySQL service.  Do this:

svccfg -s mysql:version_51 setprop mysql/enable_64bit=true
svcadm refresh mysql:version_51
svcadm restart mysql:version_51

I did just this, and went back to http://localhost/joomla and everything worked perfectly well.  Joomla! works just fine in OpenSolaris, as you would expect.  Nice looking app, too.

P.S.:  (Joomla / MySQL error message below)



* Error loading Modules:MySQL server has gone away SQL=SELECT
id, title, module, position, content, showtitle, control, params FROM
jos_modules AS m LEFT JOIN jos_modules_menu AS mm ON mm.moduleid =
m.id WHERE m.published = 1 AND m.access <= 0 AND m.client_id = 0 AND (
mm.menuid = 1 OR mm.menuid = 0 ) ORDER BY position, ordering
* Error loading Modules:MySQL server has gone away SQL=SELECT
id, title, module, position, content, showtitle, control, params FROM
jos_modules AS m LEFT JOIN jos_modules_menu AS mm ON mm.moduleid =
m.id WHERE m.published = 1 AND m.access <= 0 AND m.client_id = 0 AND (
mm.menuid = 1 OR mm.menuid = 0 ) ORDER BY position, ordering
* Error loading Modules:MySQL server has gone away SQL=SELECT
id, title, module, position, content, showtitle, control, params FROM
jos_modules AS m LEFT JOIN jos_modules_menu AS mm ON mm.moduleid =
m.id WHERE m.published = 1 AND m.access <= 0 AND m.client_id = 0 AND (
mm.menuid = 1 OR mm.menuid = 0 ) ORDER BY position, ordering
* Error loading Modules:MySQL server has gone away SQL=SELECT
id, title, module, position, content, showtitle, control, params FROM
jos_modules AS m LEFT JOIN jos_modules_menu AS mm ON mm.moduleid =
m.id WHERE m.published = 1 AND m.access <= 0 AND m.client_id = 0 AND (
mm.menuid = 1 OR mm.menuid = 0 ) ORDER BY position, ordering
* Error loading Modules:MySQL server has gone away SQL=SELECT
id, title, module, position, content, showtitle, control, params FROM
jos_modules AS m LEFT JOIN jos_modules_menu AS mm ON mm.moduleid =
m.id WHERE m.published = 1 AND m.access <= 0 AND m.client_id = 0 AND (
mm.menuid = 1 OR mm.menuid = 0 ) ORDER BY position, ordering
* Error loading Modules:MySQL server has gone away SQL=SELECT
id, title, module, position, content, showtitle, control, params FROM
jos_modules AS m LEFT JOIN jos_modules_menu AS mm ON mm.moduleid =
m.id WHERE m.published = 1 AND m.access <= 0 AND m.client_id = 0 AND (
mm.menuid = 1 OR mm.menuid = 0 ) ORDER BY position, ordering
* Error loading Modules:MySQL server has gone away SQL=SELECT
id, title, module, position, content, showtitle, control, params FROM
jos_modules AS m LEFT JOIN jos_modules_menu AS mm ON mm.moduleid =
m.id WHERE m.published = 1 AND m.access <= 0 AND m.client_id = 0 AND (
mm.menuid = 1 OR mm.menuid = 0 ) ORDER BY position, ordering
* Error loading Modules:MySQL server has gone away SQL=SELECT
id, title, module, position, content, showtitle, control, params FROM
jos_modules AS m LEFT JOIN jos_modules_menu AS mm ON mm.moduleid =
m.id WHERE m.published = 1 AND m.access <= 0 AND m.client_id = 0 AND (
mm.menuid = 1 OR mm.menuid = 0 ) ORDER BY position, orderingTechnorati Tags: opensource, opensolaris, solaris, joomla, virtualbox, mysql, install, installation, content management

Powered by ScribeFire.

In our ISV Engineering organization, we do some pretty cool work with a variety of software companies built around open source business models; here are just a few of our more strategic open source partners.  This week, we published some work we've been doing with Terracotta for the last few months to help them optimize their technology on Sun's products.  The 4-page document provides an overview of the business benefits of Terracotta for Java developers, plus some results of testing we did with Terracotta on both x64 and CMT servers.  We also ran their same tests on RedHat Enterprise Linux to see how we did.  We did great.

I really like what Terracotta's done; my overly-simplistic explanation of what they do is to hook into a Java Virtual Machine (JVM) and link it with other JVMs working together, so that a cluster of JVMs look like a single, big JVM to the Java developer.  The significance is this: if you're a Java developer and you want to easily scale up your application so it can take on more load, Terracotta makes it really easy for you to do that.



In the document that we published, we showed the results of tests we did with sample workloads that Terracotta created to demonstrate what it can do for some common Java application scenarios.  (one scenario models an online test-taking application where many people login concurrently to take their tests, maybe leave the test midway through, come back where they left off, etc.)  If you look at the results table, you'll see a couple of results that I find interesting:

1. Performance of Terracotta on Solaris vs. RedHat.  Everything else was the same: same JVM, same physical hardware.  But Terracotta on Solaris performed much better, making more efficient use of the compute resources.  You leave less of your computing budget on the table with Terracotta on Solaris, is what this says to me.
2. Terracotta performance on CMT.  On the T5240 CoolThreads server, we didn't get the top result, but we had plenty of headroom to go (using 9% of the CPU resource available), which means we could launch more copies of Terracotta, or the Java application itself.  Our tests with Terracotta show us we can use CMT to get massive scaling; the results table clearly reflects that.

Once we started scaling up with Terracotta on CMT, we started to notice that their persistence mechanism was becoming a bottleneck (if you read more about Terracotta, you find that they make your cluster of JVMs reliable because Terracotta keeps track of Java objects that change, and persists those changes to its local disk).  So we introduced Terracotta to our solid state disk (SSD) products and configured the Terracotta server to persist its data to the SSDs instead of spinning disk.  That essentially gave us reliability at in-memory speeds which means that you don't have to make the tradeoff of performance vs. reliability.  It's very cool.

We've had a blast working with Terracotta; they're sharp people, and they create a product that I think is hugely valuable to Java developers, especially those trying to write apps that work at large scales on the web.  If you're such a developer, you should check them out.  Their software is available as open source and it works.

Powered by ScribeFire.

This week's announcement that VMWare is acquiring SpringSource sure caught me by surprise.  I'm not surprised that VMWare is making an acquisition; it clearly needs to get away from the "one trick pony" problem of being known only as a hypervisor vendor, because there are many entrants in that field now, and it will become commoditized soon.

What surprised me was that VMWare picked a company known for its support of software development frameworks, namely, the Spring framework.  Also what surprised me was that VMWare pretty much explicitly commits to Java as the programming language of choice for their cloud offering.

Don't get me wrong: I love Java as a choice of programming language.  But I have two basic questions about this choice:

1. Really?  Java is the language you're choosing for rapid-beyond-belief development and deployment of applications on the web?
2. Why is VMWare locking themselves into a single language?  That's essentially what I get out of the announcement and press.

Maybe there weren't any other credible candidate frameworks in PHP, for example.

Here is a nicely-written opinion piece by an IBM guy about what the deal means to VMWare and the industry.  I like his take that this purchase is less help to VMWare's cloud vision than what you might think, but it does help VMWare get into the addressable market for application servers

But what do you think is going on?  Why did VMWare choose to buy SpringSource?  Does this really make VMWare more credible as a cloud vendor, or do you think VMWare is off track with this decision?

Powered by ScribeFire.

I listen to a lot of podcasts, usually when during my runs or while I'm driving.  One of the podcasts I've been listening to over the past few years is This Week in Tech (TWiT), hosted by Leo Laporte.

Well, he's got a network of podcasts on the TWiT Network and I recently found a new one that I think is pretty good: FLOSS (Free Libre Open Source Software).  It's a weekly show in an interview format; each episode is about a different open source topic or person.

Recently, FLOSS interviewed Sun's own Glynn Foster to discuss OpenSolaris.  Not a bad introduction to OpenSolaris if you haven't checked it out yet.

Another Sun-related FLOSS Weekly is the ZFS podcast.

I also enjoyed the interview with Jono Bacon of Canonical; it looks like there are plenty of other FLOSS episodes worth a listen.

Powered by ScribeFire.

I recently learned about a company that's been around for a couple of years called spigit, and a fascinating piece of software they produce.  If you're familiar with the term "decision market" or with the book "The Wisdom of Crowds" then you have a great headstart understanding what they do.  If you haven't encountered the term or read the book, I'd explain it like this: the problem they solve is how to harness the brainpower any large company or community has in solving problems.  I'll use an example to try to explain what they do.

Suppose you are running a record company and you want to pick next year's musical artists to produce.  There are thousands of musical acts you could go with but you want to make the best few choices you can, because you can only fund a relatively small number out of those thousands.  In your record company, you've got all kinds of people who know a lot about their piece of the music industry.  So what you do is to use spigit to create a sort of a game: anybody can suggest an artist for the company to produce.  The person who suggests an artist posts to the spigit collaboration site with whatever info she wants to post that will get people to vote yes on that artist.  Other people can vote yes or no on that artist; they can post additional information about that artist (maybe a reason why to support or not support that artist).  Anybody can participate; you end up getting a wide variety of opinions from all around the company, ultimately ending up in a ranked list of artists that the company can produce.

In the meantime, people are voting on artists but also on the opinions and suggestions of other employees, so that employees build up a reputation within spigit.  The higher your reputation, the more your votes tend to count.  Reputation can go up or down; you can build up your reputation but you can also ruin it.

There's a lot more to spigit, but this is the basic idea.  The application does a nice job of combining current web 2.0 kinds of technologies and adding the concept of prediction / decision markets.  It's worth checking out if you want to make the best use of the collective intelligence of a community of people.

It looks like there are some open source prediction market packages as well; I'll have to check those out and see what they can do compared with spigit.

Powered by ScribeFire.

Here's something I don't understand: I'm trying to compile the open source program "ffmpeg", useful for transcoding from one kind of movie file and video encoding format to another, plus a whole big bag-o-tricks.

So I download the source code:

svn checkout svn://svn.ffmpeg.org/ffmpeg/trunk ffmpeg

Then I go into the ffmpeg source directory and type "./configure".  It fails, reporting:

check_ld
check_cc
BEGIN /tmp/ffconf.XXRJaOkG.c
    1   #include <signal.h>
    2   static void sighandler(int sig){
    3       raise(SIGTERM);
    4   }
    5   int main(void){
    6       signal(SIGILL, sighandler);
    7       signal(SIGFPE, sighandler);
    8       signal(SIGSEGV, sighandler);
    9   #ifdef SIGBUS
   10       signal(SIGBUS, sighandler);
   11   #endif
   12       {     volatile int i=0;
   13       __asm__ volatile (
   14           "xorl %%ebp, %%ebp"
   15       ::: "%ebp");
   16       return i; }
   17   }
END /tmp/ffconf.XXRJaOkG.c
gcc -D_ISOC99_SOURCE -D_POSIX_C_SOURCE=200112 -std=c99
-D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D__EXTENSIONS__
-fomit-frame-pointer -c -o /tmp/ffconf.XXYJaalG.o
/tmp/ffconf.XXRJaOkG.c
gcc -o /tmp/ffconf.XXTJaWkG /tmp/ffconf.XXYJaalG.o
./configure: line 663: 16530: Terminated

I can't tell why it fails even making a Makefile, but after scanning the web for a while, I find a suggestion that says "use bash as your shell instead of whatever shell you were using."  I had been using tcsh, then tried ksh, then tried sh, all with the same error result.

So then I try "bash configure".  What do you know?  It created a Makefile just fine.

Now why is that?  Can somebody help me out here?  Because I don't know why the choice of a shell would make the configure script succeed or fail.  And that seems bad for the other shells available on OpenSolaris.

The app seems to compile, mostly.  Next: see how much of the app got compiled and how much of it runs.

Why am I doing this?  Because it's part of "pyTiVo", which I'd like to get running on my OpenSolaris home media server, so that I can back up the TiVo's content onto this nice, ZFS-enabled server.  I'll update my progress here.

Powered by ScribeFire.

As I'm learning how to use Drupal for creating and deploying web sites, I'm keeping track of it in blog entries.

The way I work: I do my development on a virtualized environment: I run OpenSolaris as a guest OS under VirtualBox; that way, I can easily blow away my development/test environment or send it to other machines running VirtualBox.  Then, I install the WebStack (PHP, Apache web server, MySQL), then I install Drupal using the instructions on the drupal.org web site.

Once I get my test Drupal site working, I follow these instructions to deploy to an actual production server.  It makes the develop-test-deploy cycle pretty easy, and I can develop and test pretty much anywhere.

Powered by ScribeFire.

I'm in Washington DC for the DrupalCon 2009 conference.  Drupal, for those not familiar with the technology, is an open source content management system (CMS) built using the PHP programming language.  If that explanation doesn't help, think of it this way: lots of people use Drupal to build web sites.  I mean, lots of people.  For example, the Obama administration's recovery.gov web site is using Drupal.  And during the conference, I've talked to online newspaper sites and people who built web sites for the major music labels (e.g., Warner Music, Sony, others) and they use Drupal to build their web sites.  The popularity and growth of Drupal is astounding.  You can feel it at the convention center here.

I'm here to learn more about the Drupal community: I intend to learn more about the big problems the community wants to address, their main customer pain points, the technical and non-technical problems they face, how the community works...and ultimately, how Sun can play an important role in the Drupal community.

Sun has a table at DrupalCon, and we're getting a lot more people stopping by the table to talk to us than in previous conferences I've been at.  It's a great opportunity for us to learn what people want, and how (or whether) we can help.  Lots of people use MySQL, and they pretty much all seem to be aware that Sun purchased MySQL.  I was a bit surprised at how many people use VirtualBox, but not everybody knew that It's a Sun product.  (so we're telling them, and hearing how they use VBox).  General feedback I hear from visitors is that they are glad to see Sun showing support for the Drupal community and for open source in general.  I wouldn't say people are rushing to our table, but generally when we ask them questions, they happily stay and talk with us about what they're doing.

Here is what I'm learning so far about why Drupal is so successful: it is simple to extend, by design.  (built in PHP, you can get up-to-speed on how Drupal is built pretty quickly and the barriers to you contributing something new are reasonably low)  You don't have to contribute code to the core part of Drupal; you can create your own "module" to add new features, or you can contribute themes to help others' web sites look cool.  These are lower-risk ways of including the maximum number of people to add to Drupal, and they do.  Drupal really wants a lot of people to contribute, and they make it pretty easy to do it.

Also, Drupal is simple to get running; you can have a simple web site running in less than an hour, and you don't have to be a database administrator or a developer to do it.

There is a commercial entity formed by Drupal's founder, Dries Buytaert.  That commercial entity, called Acquia, contributes a lot to Drupal but they take steps not to try to assert too much control over Drupal.  In fact, of all the committers to the Drupal core, only one works at Acquia (Dries).

Time and time again I have heard from Drupal fans that they see Dries spending a lot of time and energy to make the community open and welcoming; getting people to contribute appears to be a cornerstone of Drupal's direction.  I've talked to a lot of people building web sites and asked them why they're at DrupalCon; many of them say they're looking for people to hire, and they use Drupal because it's easy to find developers who already know Drupal.  It's your classic developer play: Drupal succeeds in part because they have a larger developer base than competing open source CMSs like Joomla! or Wordpress.  I like this aspect because it shows the payoff of spending enormous energy in opening your community; the payoff is a huge ecosystem of features and add-ons, and an army of Drupal-trained workforce that makes it easy for customers to get somebody to build their websites in Drupal for a good price.

I installed Drupal on my OpenSolaris distro, running inside VBox on my MacBook Pro.  It took about 20 minutes from start to finish, including the time to download the software, unpack it, read the instructions, activate the WebStack components (MySQL, Apache), and configure the database for Drupal to get going.  If I were to do it again, it would take me about 10 minutes to get Drupal going, maybe 5.

Two main themes I'm hearing about what's important to work on for Drupal: simplicity, and performance.  Simplicity seems to be the much bigger issue, because Drupal is trying to expand its community by an order of magnitude.  They made it easy for PHP developers to contribute and build web sites, but they want to attract people who don't want to learn PHP.  Maybe these people know a little HTML, maybe a little CSS, but not PHP.  Drupal has to make things even simpler to create web sites with interesting content.  Anything to make it easier to install, configure, and run Drupal with a usable set of modules is a huge benefit to the Drupal community.

Performance seems to be not quiet as important, but still plenty worth talking about.  For smaller sites that don't get a lot of concurrent users hitting their web sites, performance of Drupal and the underlying software stack on a single server is plenty fine.  But when the web site starts to grow its audience, performance issues start to come into play, and I think there's a lot that Sun can do if we can show that we understand how to optimize performance on the whole software and hardware stack, from Drupal on down to the server.  I believe we can do that, and more: Sun has some interesting technology to make snapshots and test environments quick, easy, and reliable to do (ZFS comes into play here).  And we have a great tool for observing performance of Drupal sites live, in deployment (DTrace).  These are not for everybody -- I hear a lot of people saying they don't want to spend their time learning how to do detailed performance tuning, they'd rather spend their time getting web sites live -- but I think that people who see these tools at our BOF on Friday will see how these tools can be a real help.

One nice thing they're doing at the conference is the use of Twitter as a public address system: follow the twitter user "drupalcon" and you get updates about where lunch is being served, parties happening after hours, reporting on status of wi-fi downtime (mostly wireless access has been great, but when it goes down, the DrupalCon people tweet about it right away and ask people to tweet back saying where they find dead spots).  When slides get posted to web sites, it's announced on twitter.  Same with videos of the talks.  Somebody lost a phone; drupalcon tweeted a lost-and-found message.  It's a great use of Twitter.

I'll leave this entry with a quote from the opening keynote.  It was attributed to Clay Shirky, to represent the approach of the Drupal community: "Replace planning with coordination".  Simple example: going to lunch as a group.  Before cell phones, if you want to go to lunch as a group you have to talk about it before-hand and plan where you're going to go, because communication is complicated in a group.  With web-enabled cell phones, you can call, text, or tweet your mates in real time, coordinating what you're doing right then.  Interesting approach; I'd say it's a good summary of the open source way of doing things that I've seen in my own career working with open source.

More soon; the conference is a firehose, and I'm having a blast drinking from it.

Powered by ScribeFire.