What happens downstream?: Scott Mattoon's blog

No sooner had we put the wrap on an April 9 Commonwealth Club panel interview on Collaborating for Change, than PBS announced a really cool collaborative project on Nova to design the "Car of the Future".   Both of these recent productions focus on the application of open source design to social and economic needs beyond software.  The promise of open source economics is popping up everywhere.  It must be something in the water, (or the atmosphere).   Network based open source design efforts have been written about before, and there's more than a few established non-software open source design projects, but they were hardly regarded as mainstream.  And open source as a business model has been a fringe enterprise.  But all that is changing.

The upcoming Nova special, and the Commonwealth Club interview (with Amy Novogratz, Kate Stohr, Maria Giudice, and myself (video courtesy fora.tv)) serve as proof points that this phenomena has exceeded meme status and is spilling over into the broader socioeconomic graph.   But we knew this was inevitable, right?  We just needed the right conditions for humanity's collaborative tendency to come out of the proprietary deep freeze.

The substrate upon which this new culture is rising pairs flexible licensing models a'la Creative Commons with accessible technology for building collaborative online communities a'la Drupal and Wordpress,  Yahoo!groups and PBWiki.  Among the catalysts for this reaction are frustration over obscene economic inequities around the world, abuses of people and planet for profit, and utter neglect by federal governments.  As was discussed here in the video interview about the Open Architecture Network, these frustrations can be overcome by collaborating for change on the net.

Need more proof of the trend toward an open source economy?  Just check with the folks at Open Everything.  They're tracking numerous open collaboratives, which are exogenous to  the software world, but infused with many of the same principles, practices and tools as open source software projects. 

One of the most prominent tools applied to these new collaboratives is Drupal, and we discuss it's role in the Open Architecture Network in the video (at :37:30, :46:30, and :51:00).

Ten years ago who would have imagined that:

• IBM would begin opening their patent portfolio?
• Major news bureaus would sponsor citizen journalism?
• Educational curriculum would be opened and shared for free?
• The Pirate party could get on a national election ballot?
• Sun Microsystems would convert its software portfolio to open source, and it would be profitable?  (Well, that one you might have imagined.)
  

Yet these, and plenty of other examples show that collaborative culture is on the rise.  Does this signal the next generation economy in which businesses profit less from market lockout and legal protection and more from direct value delivered in open markets?  Or does it lead to a more fundamental shift wherein socioeconomic prosperity derives less through commerce than through collaborations for which the primary incentive to contribute is sociocentric good? 

Sitting in the Is Beauty Truth? session at TED2008 today, I am reminded of a phone conference with TED Curator Chris Anderson last fall to bring him up to date on the status of one of the previous year's TED wishes.

When Sun wrapped up its formal role in developing the Open Architecture Network (OAN) it handed over a sustaining challenge to the site's owner and community leader, Architecture for Humanity (AFH).   When TED Curator Chris Anderson asked Sun why the TED Prize winner was left in a lurch I gave a short answer, "It was primarily due to reasons of expediency".  In actual fact, Sun never walked away from AFH.  Sun was, and continues to be, committed to their success and continues to be involved.  As of today, we now we see a clear path to a sustaining model that leverages the Drupal community and frees AFH from the dependence cycle it was caught in with Sun.  I look forward to bringing that good news to Chris before the conference wraps up on Saturday.

The first step on this path is to refactor the site such that it runs on an unadulterated Drupal core.  To do that AFH and Sun have contracted with CivicActions to migrate the OAN from a hacked Drupal 4.7 to a clean Drupal 5.X.  (It was the hacking aspect that I explained away to Chris Anderson as "expediency".  Corners were cut, compromises were made, but AFH's and TED's primary goal, to launch the site at TED2007, was achieved.  Incidentally, of the three TED2006 prize winner, only AFH's wish was realized by TED2007.)  CivicActions won the bid to perform the migration by doing a professional and efficient assessment of the OAN's current state and the effort required to bring it up to the high standards of a showcase Drupal site.

My next few posts will describe the process of setting up this development environment as we open Chapter 2 in the OAN's odyssey.  I'll describe how we use OpenSolaris to enable efficient development, testing, and deployment for multiple contributors working on multiple tasks and timelines.

For more on why OpenSolaris was chosen as the development and deployment platform for the OAN, see this article on the Sun Developer Network, and this brief interview.

A nice 10 minute interview with Cameron Sinclair aired on PBS tonight.  I learn something new every time I hear Cameron speak.

"It's a little known fact," says Cameron "most plastic tents are made in Pakistan ... when the Kashmir earthquake happened, because of the series of natural disasters we faced that year ... they ran out of tents."  Watch the video to find out what was conceived by Pakistanis caught in this desperate situation.

Nancy and I caught up with Cameron two weeks ago at a Pecha Kucha event in SOMA.  He told us that Autodesk is now working with AFH to add Project Freewheel models to the OAN.  Freewheel is the "web ready" export format from Revit and other Autodesk tools.  Revit Architecture is the Building Information Management system that Nancy has been training architects to use recently.  Her experience with a Freewheel 3D model (of my house) seemed to indicate that data rich models are too dense to smoothly manipulate (zoom, rotate).  Given the size and complexity of some of the AFH projects, this could pose a problem when adding DWF files to the OAN.

As Cameron said in the interview, the simple solutions are the important ones.  Maybe 3D orbiting of structural designs in a web page is not so important.

The People Formerly Known As The Audience landed a piece on Wired.com today about the Open Architecture Network (OAN).  

The story was part of a pro-am journalistic experiment orchestrated through Assignment Zero that produced 80 distinct articles dealing with crowdsourcing, of which 12 earned a spot on Wired's home page.  As one of the OAN story contributors, I bagged my first byline on Wired.com, one in which I was decidedly on the am side of pro-am.  While my contribution was relatively small, it was enough to give me a taste of the post-edit blues - most of my copy was either red-lined, or reduced to the point of inaccuracy.   My original reporting expanded on the technology choices and process of developing a collaboration site using the open source CMS Drupal.   The final piece gives a nod to Drupal and Sun Microsystems, but leaves the wrong impression:

"Even the software powering the site -- designed by Sun Microsystems -- is open source: the Drupal content management system chosen by thousands of nonprofits for its ease of use."

Sun did not design Drupal, and while ease of use is one of the virtues of Drupal it's an oversimplified view of why so many nonprofits use it.  Granted, the focus of the story was not so much on technology as the potential for open source design, but one of the points red-lined from my copy was perhaps one of the most relevant given the context of this crowdsourcing experiment:

"Organizations that use Drupal for their online communities include... Assignment Zero."

Drupal is everywhere.   Drupal's integral role in the explosion of collaboratives born of open sharing and grass roots participation is worthy of an entire Wired issue, if not more than just a mention buried deep in one story.  Even among citizen journalism sites, it's a dominant software platform - witness The Witness Project, for example.

Still, good to see the momentum for Architecture for Humanity continuing in the media - in addition to the Wired piece, another story about the OAN appeared this week in Business Week, and AFH's founder, Cameron Sinclair, was added to the distinguished list of "Thinkers of Tech" for Fortune's iMeme conference next week in San Francisco.

We're still the audience, we just have something to say now. 

Planetizen, the online commmunity for urban planners and designers, ranked the Open Architecture Network as one of the Top Ten websites for 2007.  Like the OAN, Planetizen is built on the Drupal CMS. 

While browsing through Planetizen I came across a link to AssignmentZero a crowdsourcing journalistic experiment.  The site is also built on Drupal, and they've got an assignment posted to write about Architecture for Humanity, who they describe as "one of numerous organizations practicing open design".   Along with three other writers, I took the assignment, natch.

The Open Architecture Network has posted a "project" chronicling in photos the process of bringing this community online.  Except for one particularly unflattering photo of me, there's a good sequence of photos of the SunFire X2200 M2 servers and Storagetek 3511 storage array racked in AMD's data center. 

With all the available space in that rack why did we stick the shiny new gear at the bottom of the rack?  In densely populated racks, servers mounted in the top half of the rack have as much as a 50% lower MTBF than servers in the bottom half.  The working rule of thumb is for every 10^o F above 68^o the failure rate doubles.  The gear that is typically most sensitive to high temperature is the power supply, hard drive, and fan.  Good thing we've got two of each in these boxes, but I wouldn't expect a heat problem anyway - the cold air blowing on my head whilst working on these machines reminded me of winter in Duluth.

Resources for managing data center cooling:

• Site Planning Guide for Sun Servers
• 42U.com rack cooling solutions
  
• Computational Fluid Dynamics FAQ for the Data Center 
  

The Coolstack 1.1 AMP package installs the 32-bit version of MySQL by default.  We want to let the horses out of the corral on this SunFire X2200 M2, so we also install the 64-bit version, which is provided as a separate package.  Since we need the 32-bit version in order to compile php5, we keep it in its original  /opt/coolstack/mysql_32bit  location.

After running  mysql_install_db and the other steps in  /opt/coolstack/mysqlREADME we then to prep MySQL to be a first class citizen on Solaris 10.

Convert MySQL to SMF

Like the the CoolStack 1.1 Apache, CoolStack 1.1 MySQL is not integrated with SMF.    Here are the resulting manifest and method files to get MySQL working cleanly as a service:

/var/svc/manifest/network/mysql.xml : 

<?xml version='1.0'?>
 <!DOCTYPE service_bundle SYSTEM '/usr/share/lib/xml/dtd/service_bundle.dtd.1'>
<!--
        Manifest for MySQL
 -->

 <service_bundle type='manifest' name='CSKmysql:mysql'>

 <service
         name='network/mysql'
         type='service'
         version='1'>
         <create_default_instance enabled='false' />
         <single_instance />

       <!--
                  Wait for network interfaces to be initialized.
        -->
                <dependency name='network'
                    grouping='require_all'
                    restart_on='error'
                    type='service'>
                    <service_fmri value='svc:/milestone/network:default'/>
                </dependency>

                <!--
                  Wait for all local filesystems to be mounted.
                -->
                <dependency name='filesystem-local'
                    grouping='require_all'
                    restart_on='none'
                    type='service'>
                    <service_fmri
                        value='svc:/system/filesystem/local:default'/>
                </dependency>

         <exec_method
                 type='method'
                 name='start'
                 exec='/lib/svc/method/CSKmysql start'
                 timeout_seconds='60'>
         </exec_method>

         <exec_method
                 type='method'
                 name='stop'
                 exec='/lib/svc/method/CSKmysql stop'
                 timeout_seconds='60'>
         </exec_method>

         <exec_method
                 type='method'
                 name='restart'
                 exec='/lib/svc/method/CSKmysql restart'
                 timeout_seconds='60'>
         </exec_method>

 </service>
 </service_bundle> 

/lib/svc/method/CSKmysql :

#!/usr/bin/sh
#
#        Method file for MySQL
#
# This uses the MySQL packages from CoolStack 1.1
# CSKmysql
#
# Modify accordingly!
#
# NOTE: Make sure DB_DIR is owned BY the mysql user and group and chmod
# 700.
#

. /lib/svc/share/smf_include.sh

DB_DIR=/site-data0/data
PIDFILE=${DB_DIR}/`/usr/bin/uname -n`.pid

case "$1" in
        start)
        /opt/coolstack/mysql/bin/mysqld_safe --user=mysql --datadir=${DB_DIR} --pid-file=${PIDFILE} > /dev/null &
                ;;
        stop)
                if [ -f ${PIDFILE} ]; then
                /usr/bin/pkill mysqld_safe >/dev/null 2>&1
                /usr/bin/kill `cat ${PIDFILE}` > /dev/null 2>&1 && echo -n ' mysqld'
                fi
                ;;
'restart')
        stop
    while pgrep mysqld > /dev/null
      do
      sleep 1
    done
        start
        ;;
        *)
                echo ""
                echo "Usage: `basename $0` { start | stop | restart }"
                echo ""
                exit 64
                ;;
esac

Then import the service:

# svccfg import /var/svc/manifest/network/mysql.xml

Before starting MySQL we need to put a config file in  /etc.  The example small config provided with MySQL is just right for now.

# cp /opt/coolstack/mysql/share/mysql/my-small.cnf /etc/my.cnf

Then change the datadir setting in /etc/my.cnf to point to the 1.3TB zfs pool on the external StorageTek 3511:

39  datadir=/site-data0/data

Finally, set the data dir with proper ownership:>

# chown myqsql:mysql /site-data0/data

And make sure it starts:

# svcadm -v enable mysql

At this point we're ready to set up the content management system, Drupal.

Additional tips for MySQL on Solaris:

• Configuring MySQL to use SMF
  
  

The new Solaris AMP stack, a.k.a. CoolStack 1.1 is here. And not a moment too soon, as I sit down to build another server for the Open Architecture Network. This is server #2, which will provide the n+1 scaling and redundancy necessary to keep the the OAN up and functional in the face of any one component failure and through a good slash-dotting. 

Of all the goodies in this new release, it was the GD library that we needed in particular. It is also nice to see suhosin from the hardened-php project included in this release. Here's a quick breakdown of version differences between CoolStack 1.0 and 1.1:

Convert CoolStack Apache to SMF 

First, I notice that the services in coolstack are not integrated with SMF. We need apache to run under SMF so its privileges can be easily limited.  I convert it to SMF, and prepare it for limited privileges by creating a service manifest and service method based on the original apache service shipped with Solaris 10.

# cp /lib/svc/method/http-apache2 \
/lib/svc/method/http-CSKapache2
# cp /var/svc/manifest/network/http-apache2.xml \
/var/svc/manifest/network/http-CSKapache2.xml

In /lib/svc/method/http-CSKapache2 change

 
    11  APACHE_HOME=/usr/apache2
    12  CONF_FILE=/etc/apache2/httpd.conf
    13  PIDFILE=/var/run/apache2/httpd.pid


    20          /bin/mkdir -p /var/run/apache2 

to

    11  APACHE_HOME=/opt/coolstack/apache2
    12  CONF_FILE=/opt/coolstack/apache2/conf/httpd.conf
    13  PIDFILE=/var/apache2/run/httpd.pid


    20          /bin/mkdir -p /var/apache2/run
 

In /var/svc/manifest/network/http-CSKapache2.xml change

    10  <service_bundle type='manifest' name='SUNWapch2r:apache'>

    23          <instance name='apache2' enabled='false'>

   100                                  manpath='/usr/apache2/man' />

to

    10  <service_bundle type='manifest' name='CSKapch2r:apache'>

    23          <instance name='CSKapache2' enabled='false'>

   100                                  manpath='/opt/coolstack/apache2/man' />

Then import the service:

# svccfg -v import /var/svc/manifest/network/http-CSKapache2.xml

Minimize Apache's Service Privileges

Next, we configure the new service to run with minimal privileges following the example in 
Glenn's Limiting Service Privileges BluePrint.  After the procedure the CSKapache2 privileges
should look like this:

# svcprop -v -p start CSKapache2
start/timeout_seconds count 60
start/type astring method
start/exec astring /lib/svc/method/http-CSKapache2\ start
start/user astring webservd
start/group astring webservd
start/privileges astring basic,!proc_session,!proc_info,!file_link_any,net_privaddr
start/limit_privileges astring :default
start/use_profile boolean false
start/supp_groups astring :default
start/working_directory astring :default
start/project astring :default
start/resource_pool astring :default

Note that the changes to the PidFile and LockFile directives specified in this minimization procedure will be overridden  by the Server-pool management configuration that is loaded by

474  Include conf/extra/httpd-mpm.conf

Unless the corresponding directives are commented out of  /opt/coolstack/apache2/conf/extra/httpd-mpm.conf

Increase Semaphores for PHP 

By default the php5_module is loaded in the CoolStack 1.1 apache.  I observed that PHP was causing the maximum number of semaphores to be exceeded, so I created a project  httpd.php  to bump the max from 128 up to 256:

# projadd -c "Apache-PHP" -U webservd httpd.php
# projmod -sK "project.max-sem-ids=(privileged,256,deny)" httpd.php

then added the project to the service configuration:

svccfg -s http:CSKapache2 setprop start/project = astring: httpd.php

Enable suhosin

Because the site is expected to receive lots of publicity, and it will not have a 24x7 SWAT team ready to jump in and thwart the bad guys, we want it to be as hardened to attacks as possible.  Suhosin gets us a long way toward that goal.  Since it's already built for us in CoolStack, we just need to enable it by  uncommenting  extension="suhosin.so"  in  /opt/coolstack/php5/lib/php.ini

Now we're ready to setup the CoolStack 1.1 MySQL ...

Additional SMF resources:

• BigAdmin: developing services to run within SMF
• OpenSolaris SMF Community
  
• SMF BluePrint
• Converted Services: Manifests and Methods on OpenSolaris.org
  

This was the reply I received from Kate after sharing a post on tips for fostering a community online. The post couldn't be more relevant to her and the rest of Architecture for Humanity as they approach the launch of the Open Architecture Network (OAN), which will coincide with this year's TED conference. Lessons from the social networking front line are pouring in, and I expect the OAN will begin it's contribution of do's and dont's, how-to's, and testimonies to the power of the network soon.

Anyone who has launched an online social network will tell you that the experience is, indeed, a very interesting social experiment. Some of the most compelling communities have taken shape with minimal intervention and control from it's founders, yet their vision and values manage to persist through its evolution by virtue of their participation. The social networking movement puts Metcalfe's law squarely back in good standing as a reasonable measure of the value of the network, which the law says is proportional to the square of the number of users. In fact, it may be time to revise the formula as we see true benefit reach well beyond the network itself, to those who have never touched a computer. And the Open Architecture Network is poised to be one of the those networks whose beneficiaries may have never before been touched by the network effect.

If you give a damn, listen to the Treehugger interview with Architecture for Humanity's other co-founder, Cameron Sinclair. And watch this space for breaking news about the new Open Architecture Network.