Bonnie Corwin's Weblog: Riding the Roller Coaster

Last time I wrote, changes were coming to the opensolaris.org website this summer.

Yesterday, Phase 1 completed.

As a reminder: Phase 1 was the move to using the new Auth application and database for membership management.

Now that this change has been made, everyone with an existing account on opensolaris.org needs to validate their account the next time they log in.

If you have forgotten your password, please send email to website-admin AT opensolaris DOT org and we will get you started.

If you have problems using auth.opensolaris.org  or think something about your data is not quite right, also send email to website-admin AT opensolaris DOT org, and we'll figure it out.  FAQS are also available: an opensolaris.org Website Transition FAQ and a User Action FAQ.  As you use the FAQs, we'd love input about questions to add and how to make answers better.

Next up will be Phase 2: the move to an XWiki-based site.  Email with information about that will be coming later this week.

I do want to thank the members of my team who have been working long hours for a long time to make this transition go well.  It's a big transition; it covered lots of ground within the site infrastructure; and it will enable more change to happen.  Thanks doesn't begin to cover it.

Changes are coming to the opensolaris.org website this summer.

These changes will enable support for a more complete open development infrastructure and provide wiki support on the site.  The changes will occur in phases to hopefully minimize confusion and manage the chaos (I can always hope :-).

Phase 1 is about new membership management, and the target date is late July.

Currently, some of the applications on the site manage their own data for historical reasons.  The original portal application that runs the site was written before anyone knew exactly what roles and rights would be needed for working efficiently on the site and before there was a community constitution that defined governance roles.  As a result, the database associated with the portal application isn't sufficient for all current needs and requirements.  Some site applications use less-than-optimal methods for checking that people are registered on the site and some use their own databases of user information.

Moving forward, there will be one database.  Its information can then be used across the site to perform tasks such as confirming commits to source repositories, managing governance and site rights, and identifying people as being registered on the site for access to certain functionality like defect tracking, the test farm or the source juicer.  Over time, the various applications running on the site will transition to use the new single database.

You can find pointers to information about the changes in the Website Community Group on the site.  Scroll down to "Phase 1" under "Website Transition: Implementation Phases".  There are pointers there to pages that explain the roles and collectives in the new system and the plan for migrating data from existing data sources to the new single database.

You can try out a BETA TEST SITE VERSION of the new membership management application at: http://auth.opensolaris.org.  NOTE THAT THIS IS A BETA TEST SITE VERSION. Changes made here exist ONLY in this beta version.  Changes made here have NO EFFECT on opensolaris.org.  If you need to make a real change to your account on opensolaris.org or if you wish to register on opensolaris.org, you must do that at http://opensolaris.org.

A minimal amount of email will be sent to all current site registrants to communicate about the coming changes (the first went out late last week).  But ongoing announcements, upates and other information is being sent to the website-discuss mailing list (website dash discuss at opensolaris dot org), so subscribe to really follow along.

Who knew that my fair city (Colorado Springs) and my county (Colorado's El Paso Country) each have departments dedicated to removing dead animal carcasses?   Turns out, they remove such carcasses only from public streets and rights-of-way, not from private property.  And who knew that there are four (four??) companies in my city that remove dead animal carcasses from private property?

I do know I live in the 'wilderness' (that from the man who came to remove the carcass :-).  We are in the hills at the base of Pikes Peak, where developers built houses into animal habitat.  Our 'yard' is about an acre of hillside that is mostly wild - grasses, yucca, prickly pear cactus, scrub oak, etc.  Not a place to go barefoot in the summertime.  And on a regular basis we have deer, fox, bears and even mountain lions in our yard.

But yesterday morning around 6:30am, I opened the garage door to leave for work and noticed about half a dozen magpies on the rocks in a bed by our front door.  Magpies, for those unaware, are scavengers, one of nature's garbage collectors.  I noticed two red strips on which they were feeding and walked closer to see if it was a snake.

Apparently there's a reason hunting takes skill because animals really do blend in with their surroundings.  Lying on the rocks was a very young buck, with nubs showing on his head as the beginnings of antlers.  Eyes wide open, ribs exposed, stomach hanging out, food for the magpies, quite dead.  And then I noticed the blood - spread across the driveway - and a bloody paw print on the step to the front door.

The theory is coyotes.  One animal I actually haven't seen in my yard.  I've heard them.  And heard about them.  But not seen them.  But the print was too small and not the right shape to be a mountain lion.

I mess with my teenage kids at dinner, asking them the stereotypical question, "what did you learn today?".  On Sunday night, the older one started to chuckle after telling me that was a stupid question.  I asked what was so funny, and he said very sheepishly, "I actually did learn something today".  :-)

Yesterday I learned a lot.  Some of which I could have lived without.  I get the whole living-with-nature thing, I get the food chain, and I have come to believe that a scientist I heard last year might be right, that deer in general are simply "rats with antlers".  But the dead young buck about five feet from my front door, and still-fresh, bright red blood all over the driveway indicating that the fight was quite recent, was all a bit eery.

And I understand in more than an intellectual way now why the Native Americans thanked the animals they had to kill for food.

Congratulations to Rafael Vanoni and Chad Mynhier.

Rafael did a putback to the O/N Consolidation today of two bug fixes from Chad: 6730130 and 6733214.  In addition to being useful changes to DTrace code, these changes mark the 399th and 400th integrations done via the Sponsor Program.

Chad has been a contributor for almost two years, has had 16 code changes integrated, and currently has six other code contributions in progress via the Sponsor Program. 

If you're curious about contributions to date via the Sponsor Program, check out the request-sponsor table on opensolaris.org.  That page also contains a pointer to the Improving OpenSolaris page which has information to get you started on your first contribution.

Contributions to date via the Sponsor Program have been made to the O/N, Install, Cluster, Desktop, G11N, SFW, X and Pubs consolidations.  There was a dip in the total number of code integrations in Year 2, but the numbers have been growing since then.  And when we hit our fourth anniversary next month, Year 4 will show the most code integrations in a year so far.

Many thanks to all the contributors and Sun engineers who enable this participation while we work to make it all easier.

I seem to continually collect bits of information.  Because of questions I've been asked recently, I was interested in companies that are working on http://opensolaris.org.  And I found a quick sampling:

Code contributions to the ON consolidation that have come in through the Sponsor Program:

• AMD/11 putbacks/2 contributors
• Intel/20 putbacks/6 contributors
• Ohio State University/2 putbacks/3 contributors
  

Projects on the site:

• AMD              Enhancements for AMD-based Platforms
• Intel               Enable/Enhance Solaris support for Intel Platform
• Intel               Intel Graphics
• Intel               LatencyTOP
• NSA              Flexible Mandatory Access Control
• Sine Nomine  Systemz
• Songbird       Nightingale: Port Songbird to OpenSolaris
  

I didn't do an exhaustive investigation, so I'm sure I'm missing information.  But I think it's exciting to see the collaboration that's happening.

And we're continuing to work on improving the developer experience.  Information about areas of work and current plans are in the Open Development Infrastructure project.

In December, I updated a list I keep.  Most of these additions were announced when they happened via email of some kind and were logged in the old roadmap.  But I think seeing the list as a list is impressive, and I love that it hasn't stopped growing.  It's four years since we started publishing source code on http://opensolaris.org, and we're still adding:

2005

---

01/25/05: OS/Networking (ON) Consolidation: DTrace Source Code

06/14/05: ON Consolidation Source Code

10/28/05: JDS Consolidation Source Code

11/10/05: DevPro Consolidation: SCCS/make Binaries

11/15/05: OpenGrok Source Browser Source Code

11/16/05: ZFS Project integrated into ON build 27

2006

---

01/27/06: Network Storage Consolidation Soure Code

02/22/06: DevPro Consolidation: libm/libmvec Source Code

02/28/06: DevPro Consolidation: libmtsk Binaries

03/06/06: Install Consolidation: Packaging Tools Source Code

03/29/06: SFW Consolidation Source Code

03/31/06: Documentation Consolidation (2 books total): Source for

• ZFS Administration Guide

• Device Driver Tutorial

03/31/06: X Window System Consolidation Source Code

05/10/06: Globalization Consolidation: Source for OS/Net Consolidation Message Files

05/31/06: Documentation Consolidation (4 books total): Source for

• Solaris Dynamic Tracing Guide

• System Administration Guide: Solaris Containers--Resource Management and Solaris Zones

06/12/06: DevPro Consolidation: medalib Source Code

06/26/06: Companion CD Source Code

06/30/06: Documentation Consolidation (6 books total): Source for

• OpenSolaris Developer's Reference

• Solaris Containers: Resource Management and Solaris Zones Developer's Guide

07/28/06: Documentation Consolidation (8 books total): Source for

• Solaris Volume Manager System Administration Guide

• Solaris Express Installation Guide: Basic Installations

08/31/06: Documentation Consolidation (16 books total): Source for

• Solaris Trusted Extensions Installation and Configuration Guide

• Solaris Trusted Extensions Label Administration

• Solaris Trusted Extensions User's Guide

• Solaris Trusted Extensions Transition Guide

• Solaris Trusted Extensions Developer's Guide

• Solaris Express Installation Guide: Solaris Flash Archives (Creation and Installation)

• System Administration Guide: Basic Administration

• System Administration Guide: Advanced Administration

09/11/06: BrandZ Project integrated into ON build 49

09/26/06: DevPro Consolidation: SUNWlibC (C++ runtime libraries) Binaries

09/29/06: Documentation Consolidation (19 books total): Source for

• Application Packaging Developer's Guide

• DTrace User Guide

• Solaris Trusted Extensions Administrator's Procedures

10/06/06: Solaris PowerPC source release.

10/20/06: SPARC Graphics Consolidation: Device Driver Binaries

11/30/06: Documentation Consolidation (23 books total): Source for

• Solaris Express Installation Guide: Custom JumpStart and Advanced Installations

• Solaris Express Installation Guide: Planning for Installation and Upgrade

• Solaris Tunable Parameters Reference Manual

• System Administration Guide: Security Services

12/08/06: Man Page Consolidation: Source for an initial set of 356 man pages.

12/18/06: Test: Source for the NFSv4 Test Suite

12/19/06: DevPro Consolidation: Source for SCCS and make

12/22/06: Globalization Consolidation Source Code

2007

---

01/18/07: Globalization Consolidation: Source for multiple translations

01/26/07: Man Page Consolidation: Source for a second set of 2790 man pages (total: 3146)

02/20/07: Documentation Consolidation (24 books total): Source for

• System Administration Guide: IP Services

04/12/07: Documentation Consolidation (26 books total): Source for

• System Administration Guide: Naming and Directory Services (DNS, NIS, and LDAP)

• System Administration Guide: Network Services

05/24/07: Man Page Consolidation: Source for third set of 622 man pages (total: 3768)

06/21/07: Man Page Consolidation: Source for fourth set of 306 man pages (total: 4074)

06/26/07: Test: Source for the ZFS test suite

06/27/07: Open HA Cluster: Sun Cluster Agents (data services) Source Code

06/27/07: Test: Source for:

• Common Test Infrastructure (CTI) for the Test Environment Tookit (TET)

• iSCSI test suite

06/29/07: Documentation Consolidation (28 books total): Source for

• Memory Thread Placement and Optimization Developer's Guide

• Solaris Modular Debugger Guide

07/05/07: Open HA Cluster: Globalization Source Code for the Solaris Cluster Agents

07/20/07: Test: Source for the Sharemgr Test Suite

08/03/07: Documentation Consolidation (31 books total): Source for

• Solaris Express Developer Edition Release Notes

• Solaris Express Developer Edition What's New

• Solaris Express Package List

09/04/07: Documentation Consolidation (34 books total): Source for

• Solaris Express Developer Edition Installation Guide: Laptop Installations

• Solaris Express Installation Guide: Network-Based Installations

• Solaris Express Installation Guide: Solaris Live Upgrade and Upgrade Planning

09/21/07: Test: Source for

• Sharemnt Test Suite

• Kerberos Test Tools (krb5tools)

09/24/07: Man Page Consolidation: Source for fifth set of 87 man pages (total: 4161)

09/28/07: Website: Source for Portal Application Code

10/18/07: Self-Service Testing: OpenSolaris Self-Service Testing application available and ready for use by OpenSolaris contributors.

12/04/07: Open HA Cluster: Sun Cluster Geographic Edition product Source Code

12/21/07: Man Page Consolidation: Source for 17 additional man pages (total: 4278)

2008

---

01/25/08: Man Page Consolidation: Source for 21 additional man pages (total: 4299)

02/12/08: Documentation Consolidation (36 books total): Source for

• Solaris CIFS Administration Guide

• System Administration Guide: Solaris Printing

03/24/08: Test: Source for

• Secure by Default (SBD) Test Suite

• Key Management Framework (KMF) Test Suite

03/27/08: Man Page Consolidation: Source for 9 additional man pages (total: 4308)

04/24/08: Test: Source for the NIC Driver Test Suite

05/01/08: Documentation Consolidation (37 books total): Source for

• System Administration Guide: Devices and File Systems

05/29/08: Open HA Cluster: Source for Solaris Cluster Core (final source for the Solaris Cluster product)

05/30/08: Test: Source for

• FNFS (NFSv4) Test Suite

• SMF Test Suite

05/30/08: Man Page Consolidation: Source for 8 additional man pages (total: 4316)

06/09/08: Documentation Consolidation (39 books total): Source for

• Writing Device Drivers

• OpenSolaris Development Guide

06/11/08: Test: Source for the Storage Driver Test Suite (SDTS)

06/26/08: Man Page Consolidation: Source for 401 additional man pages (total: 4717)

10/31/08: Test: Source for the JIST (Java Technology-based Interoperability System/Software/Storage) Test Suite

11/13/08: Man Page Consolidation: Accumulated source for additional man pages over the last five months (total: 4717)

11/20/08: Test: Source for the Megacli and Virt-Install test suites

11/20/08: Test: Source for the NetCat test suite

12/11/08: Man Page Consolidation: Source for 852 new man pages (total: 5569 which is 95% of all the man pages in the SunOS Reference Manual)

I'm late on this, but it's been a busy week.  Last Friday email was sent announcing the availability of a preview version of the new opensolaris.org portal and user management application.

http://hub.opensolaris.org is a test site.  It's a chance to see how things will work with wiki functionality and the new user management system, and it's a test of the underlying infrastructure.  It's not skinned.  It's not ready for data migration and everyday use. 

But this is a milestone.  The community has wanted wiki functionality, and XWiki was chosen for use (you can see the requirements and evaluations in the Website Community).  Much has been done that isn't visible: work on XWiki, creation of a new database schema, new membership rules worked out with the OGB, technical issues resolved for the new authentication mechanism, integration of XWiki with the new user management application.  But now we have a visible step that people can see and try.  So take a look.  Log in, and wander around.  If you have comments or questions, please send them to website-discuss at opensolaris dot org.

Someone wondered this week on a mailing list what's up with developer tools work.  Coincidentally, my program managers and I are working on a roadmap to post on opensolaris.org.

The original roadmap posted when we launched was focused on milestones related to getting the site and the community started, and for quite a while, progress on the website itself was slow at best because my team had cycles to pretty much only try to keep things running.  Now that staffing has shifted, forward movement is happening, and we need to outline changes coming to the site and provide associated information useful to the developer community.

So we're creating a roadmap that outlines the work going on in my group: Open Development Infrastructure, opensolaris.org Platform Development, Content plans, etc.  The roadmap will likely point to associated projects on the site that will contain more detail and information about how people can get involved.

In the meantime, a snapshot:

• A new version of the SCM Console was made available last month after iterating on versions of it with people on the SCM Migration Project and in the Website community for a couple of months.  Many thanks to all of you who tried it out and provided input.  Stay tuned for the next phase.
  
• We're trying to get a test version of the new membership management application and XWiki support out this month so people can start helping us test the first set of that functionality.
• Michelle Olson continues to evolve the front page and general content pages on the site to improve the user experience.  Again, thanks to those of you in the website community who iterate about changes.
• We are regrouping on the topic of Open Development Infrastructure because this arena is multi-faceted and very complicated.
• We are, in fact, taking ownership of the bugs.opensolaris.org application so we can make enhancements to it.  Unfortunately, like many other areas of work we own, the invisible infrastructure work needed to take ownership and get it running on our machines will have to happen before we can start making most of the enhancements.

With a roadmap in place in January, we're hoping to enable people to better watch the areas in which they are interested and get involved in those areas as well.

On the subject of improving developer collaboration, some of you will be glad to know we are investigating changes to bugs.opensolaris.org.

This bugs-by-mail mechanism was written by someone no longer around and has not been owned by my team for reasons that don't matter.  It was originally meant to be only a temporary stopgap until open defect tracking was available (we were so blissfully naive when all this started :-).  That plan clearly hasn't worked out so well as three and a half years later we are still using it and will be for the foreseeable future.  Yes, even with defect.opensolaris.org, some bugs will need to remain in and be worked via Sun's database.

Because issues with bugs.opensolaris.org and how it works are clearly dissatisfiers for many community members, my team is investigating the possibility of taking ownership of this service so that we would be able to make enhancements to improve its usefulness and people's experience with it.

More information will be available on the SCM Migration Project page in the new year once we have completed the investigation.  So watch the project page or join the discussion to see how this progresses.  And I'll also be back with an update as soon as one is available.

My team's, not mine.

The name of our team has always been difficult because we are not typical: we are not a typical software development team nor are we a typical IT team or a general hosting or sysadmin support team.

We were originally referred to as the "OpenSolaris Engineering team" which did not work well once there was an OpenSolaris product that has development engineering teams working on it.  So we tried "OpenSolaris Infrastructure Engineering team" to highlight the website infrastructure work we do.  But it turns out the word "Infrastructure" has become quite overloaded and sometimes confusing, at least within Sun.

The focus of the opensolaris.org website and most of what we do is the developer community, and we are trying to improve developer collaboration in multiple ways through the site and the services it offers.  Which is why my team includes a mixture of software engineers, sysadmins and program managers.

So "OpenSolaris Developer Collaboration" team seems a better description.

So we'll try that for a while until it seems it should change again :-)

My team has been growing lately, and we were fortunate enough to hire a senior system administrator named Elaine Ashton.  Elaine joined the team in September, works out of Sun's Burlington campus and clearly has hidden talents.

For anyone wishing to have a similar pumpkin for Halloween, "instructions of a sort are in pictures" (according to Elaine) at her Flickr site.

Happy Halloween.

A new project opened on opensolaris.org on Friday, October 10, that represents another milestone for OpenSolaris. 

Systemz is also known as "Sirius" and is a port of OpenSolaris to the IBM System Z mainfraim operating in z/VM mode.  We've been working to enable this project creation, so it's exciting to see it open. 

The project page include implementation notes, release notes, and an invitation to get involved.  You can download a source tarball or browse the source hierarchy.  So check out this new project.

The ON organization talks a lot about "FCS Quality All the Time".  That focus is the reason for much of its software development process and is mentioned often in discussions, advice to new developers and ON documentation such as "The Quality Death Spiral".

This focus is also a reason nightly ON builds are run on at least one internal engineering server.  This sometimes causes internal pain when something goes wrong, but the production testing exposes problems that will not be encountered at customer sites.  And it's a terrific learning experience if your putback causes a problem :-).

So what does this have to do with my team?  We run Nevada builds on the machines that support the opensolaris.org website.

Yes, development builds on external-facing production machines.  I get interesting responses when people hear about this or figure it out for the first time.  Some responses are simply dropped jaws and looks of incredulity.  Some are comments such as 'are you crazy?' or 'why in the world would you do that?'.

When we investigated self-hosting before the initial launch, we knew we wanted to run Solaris on the machines that supported the OpenSolaris program.  Doing anything else just didn't make sense.

But then we decided it would be better to run Nevada builds because we wanted to make use of new functionality not in Solaris releases at the time (e.g., ZFS) and new features related to existing technologies like Zones and SMF.

And it's worked.  From August 2006 until around March of this year, we ran Build 45 on the infrastructure machines (note that the current build in progress is Build 100).  We finally hit some system problems this Spring that were fixed in later ON builds, so we started upgrading.  And in general, we want to set up a regular rolling upgrade plan to keep more recent builds on these machines.

But a development build worked for more than a year and a half.  Without system problems.  On external-facing production machines that support a public website.  Perhaps I'm easily amused, but that's amazing to me.

And, I think,  a tribute to the focus on "FCS quality all the time".  So thanks to the ON organization for its diligence about this focus.  It clearly makes a difference.


P.S. Yes, we plan to move to OpenSolaris builds at some point in the future.  But at the moment, our list is long and the product is new, so planning for that transition is not in the works yet.

More about contributions.

We hit another milestone this week when Mark Johnson sponsored a putback to the ON consolidation from AMD.  AMD engineer Hans Rosenfeld provided code that allows the live loading of microcode updates for AMD CPUs (see #290 on the request-sponsor table).

Expanding the functionality of the operating system is always a good thing, but it's also very exciting to have another large company contributing to OpenSolaris.

Find out more about work related to AMD-based platforms on the team's project page: Enhancements for AMD-based Platforms.

I recently pulled together data about OpenSolaris contributions that have come through the Sponsor Program.  And I was excited by what I learned.

OpenSolaris is 3.25 years old.   The Sponsor Program was set up to enable community members to contribute code to consolidation gates before they are all outside on opensolaris.org.

Contributions

As of September 12, 2008, 657 contributions have been offered via the Sponsor Program to multiple consolidations: ON primarily, but also to G11N, Open HA Cluster, Install, SFW, Documentation and X.

Of the 657 contributions offered:

• 288 (44%) have been integrated into a consolidation code base.
• 118 (18%) are currently in progress.
• 89 (14%) have been 'closed', meaning that the bug was closed for some reason (will not fix, not a bug, bug is a duplicate of another bug, etc.) or that the bug was fixed by work already in progress at Sun.
• 161 (24%) have been 'suspended', usually meaning that the original contributor no longer wants or has time to work on the problem.  These bugs are available for other community members to pick up and work on.
  

Contributors

While the contribution numbers are interesting, information about the actual contributors is very cool.

168 unique contributors have submitted code contributions through the Sponsor Program:

• 42 in Year 1
• 39 new in Year 2
• 82 new in Year 3
•  5 new so far in Year 4
  

The contributors span 22 countries across five continents:

• Asia: India, Israel, Japan, Singapore, Thailand
  
• Australia
  
• Europe: Belgium, Czech Republic, Denmark, England, France, Germany, Netherlands, Poland, Portugal, Russia, Switzerland, Wales
• North America: Canada, Mexico, USA
• South America: Colombia

The pool of Sun Contributor Agreements on file for OpenSolaris is larger than 168 - almost double that at 325.  This pool is larger because people file SCAs to work on technologies like translations and documentation and to work directly on projects hosted on opensolaris.org, in addition to making contributions to consolidations through the Sponsor Program.  This pool spans 36 countries across six continents:

• Africa                    1%
• Asia                     42%
• Australia                2%
• Europe                28%
  
• North America      25%
• South America       2%

I am excited by the numbers and by the diversity represented around the world.  If you are interested in finding a project to join or want to start one of your own, start on the Projects page.  Information about participating via the Sponsor Program can be found on the Participation page.