Bob Porras' Blog

The community showed up in full force last week in San Francisco, CA for JavaOne.  To kick things off the CommunityOne day had the OpenSolaris 2008.05 release on center stage.  Lot's of Live CDs were distributed and even more downloads occurred last week.  For those of you who could not attend you can get the image to download and burn a CD by clicking the big grey arrow inside the circle below.

For those of you who do not want to download you can have a CD mailed to you free of charge by clicking here.  The Live CD can be booted on your x86 hardware without requiring a full install on your hard drive unless you wish to do so.  If your not comfortable doing this then you can download and install VirtualBox on your OS running on metal and boot OpenSolaris 2008.05 as a virtual machine.

Lot's of features to experiment with including the new package repository, zfs booting, snapshots, kernel CIFS server, network configurator, data services and a new look and feel.  Keep in mind this OpenSolaris CD comes from the same pedigree as the Solaris that many customers are running their business in the enterprise for scalability (100s of CPUs, 100s of threads), performance and stability. 

Help us make it better with new enhancements, more packages in the repository and your time.  Join OpenSolaris.org and SDN today for free.

A year ago Sun announced its OpenStorage initiative.  OpenSolaris is enabling the open storage revolution with the industry's first open storage software community and it is thriving and growing.  Companies are actively contributing source code as well as building appliances and solutions with this OpenStorage software stack.  This is not a head on battle with proprietary storage vendors.  Rather it is a flank.  OpenStorage  provides customers the advantage of a global community, with all the building blocks they need to accelerate business and market response at 1/10th the cost, with freedom to change vendors. Unlike the competition, Sun remains active in the community, offering the full range of service and support to help you at any point along the path to OpenStorage.  The community is enabled to provide OpenStorage software pre-installed on selected servers and contributed to the community for download.

All community members love to share to the degree that they choose.  That is the beauty... participate actively or maybe just watch for the moment from your vantage point.  It is rewarding to observe the participation through the efforts of others.  From podcasts of enthused individuals destroying disk drives to community members touting the value of this open sourced software-- one point is consistent.  ZFS is a file system that keeps appearing in the news more and more.  For example, end to end data integrity WITHOUT  intelligent hardware RAID controllers using free software on commodity hardware is news. Simon blogging about an open sourced home file server is news.  Fear of the impact of this free technology to some proprietary business models is news.  Seeing what others are doing with this technology is news.  Interest from other companies both large and small on using this file system is news.  Tim Thomas talking about configuring native CIFS in WorkGroup mode on OpenSolaris is news.  When Tim discusses Domain mode that is news as well. Seeing Jim Hughes and his YouTube postings helps makes the news as well. 

OpenStorage is no longer coming.  OpenStorage is here and customers are containing and retiring their proprietary storage. 

Set your storage free...  Get connected.

Randy Pausch is a professor at Carnegie Mellon University who has been diagnosed with terminal cancer.  Here is a clip from his last lecture.  Gut wrenching but at the same time some worthy advice.  Make it count...  I certainly do and can relate to him.  ABC News will be showing a program this Wed April 9th, 2008 @ 10pm/9 CST.  Here's to Randy.

When I first viewed this video I was amazed more than my kids.  I'm still in awe with what can be created when you pull various engineering disciplines together.  The mechanics, digital electronics and software enable this machine to perform as an almost living organism.  The gas engine sound is a good clue that you are seeing a machine.  However if you turn the volume down and view at a distance you will think otherwise.  A machine that can be kicked, slip on ice, climb in snow and leap tall bounds and recover from these events certainly speaks artificial intelligence, finite state automaton and advanced research.  The fact that this robot is carrying a heavy payload of over ~100 lbs. (~45 kilos) is impressive.  In college I took courses in AI and state machine theory, but this video has been the best demonstration for me of that theory in practice. This robot was created by Boston Dynamics, a company that specializes in human simulation and robotics.  The machine below was funded in part by DARPA.  Cool stuff indeed.  Makes you think will Sarah Connor and Terminator remain fiction?

Another storage code base has been posted at opensolaris.  This most recent contribution focuses on the area of hierarchical storage management (HSM).  The technology is much more than standard backup.  It addresses automated data management via policies driven by data and metadata.  HSM drives some of the largest data repositories out there in the industry today.  With our open sourcing of SAM-Q we have completed an extremely large complex effort of open sourcing our *entire* storage stack!

This milestone is only the end of the beginning since we have many new open storage projects in process at opensolaris.org. These new storage projects are all being developed out in the open with the community.  Everything from data services, protocols, file systems, compression, encryption, replication, snapshots, drivers and archive software is available to the community.  There is no other comprehensive open sourced storage stack out there in the industry.

However, there are other comprehensive proprietary storage stacks out there that are quite good but you pay a hefty price (premium) for each part of the storage stack.  If you have the time but not the money the opensolaris community may be the place where you can contribute.  The community may also be the place for you if you are trying to establish your business or solution at a revolutionary price point.  In either case pure economics is a driving force. 

You may want to check out my most current read. "Alan Greenspan - The Age of Turbulence"  If you enjoy economics, history and want to ponder the power of the open source movement (aggregate demand )-- this book is a must read.  Alan also gives you some insight into the current market meltdown.

Read what some of the many team members Margaret Hamburger, Ted Pogue and Lynn Rohrer have to say about our latest opensource efforts. The entire team's pace and execution responded to a very aggressive goal set by me ~1 year ago... "Open Source the entire storage stack."  I'm also excited by the code contributions made by partners and vendors to the community.  It is also equally exciting to see customers using the open sourced storage technologies to build their storage products for their businesses. 

We at Sun also have the opportunity to build hybrid storage solutions with the opensolaris storage stack as well.  After all, open source software is in our DNA and we are the largest commercial contributors of open source software in the world.  A big thank you to the entire team.

I've just returned from beautiful Johannesburg, South Africa and Tech Days.  It was a very busy week of interacting with developers and many customers.  Africa is another continent going through rapid change.  The growth opportunities are tremendous and South Africa is another example of the new global economy.  The people and climate are great.  The Pilanesburg National Park is spectacular and beautiful.  At Tech Days in Johannesburg we demonstrated an enterprise open sourced operating system in action on a whitebox.  Matt Ahrens, Jim Hughes, Wyllis Ingersol and Renier Sevenster all got together and installed OpenSolaris Developer Preview 2 on the whitebox.  A RAID-Z stripe was created across 8 80GB disk drives that were connected via USB ports.  A MD5 checksum was run on a file that was layed out across all of the 8 drives.  Then Jim took a sledge hammer to 2 disk drives without any data loss.  The data still remained preserved and accessible.   Try doing that with another operating system on a laptop or PC!  In fact view the video below and leave the demolition of hardware for another day.  Checkout for yourself the capability of opensolaris.  Download the Developer Preview 2 here.  If you need a MD5 checksum tool get it here.  If you need a CD burner tool get it here.  Then install your live CD  on your laptop or PC and compare it to what you are running today.  No need to partition or wipe out your hard drive as the system runs off of the CD (Live CD).  I heard strongly in South Africa that open source is a requirement for customers.  Even better if your open source solution is built from a proven enterprise operating system.

I am XAM.  XAM I am.  Sun recently made another open source contribution-- this time in the space of fixed content storage software.  Fixed content storage is growing at an exponential rate.  As an example, stop and imagine how much fixed content data will be generated by the Summer Olympics 2008. Pictures, video, tickets, security data, statistics, invoices, hotel bills, airline reservations, etc, etc. etc.  A massive amount of data that will be stored and preserved digitally for a long time.  Customers depend on Archive Products, even better if they are open.

There is now a fully open source code base contributed to opensolaris.org, java.net and the SNIA XAM TWG.  Read Scott Tracy's perspective. This code base eliminates the need to roll your own digital archive using piece parts (server, RAID HW, database, etc.)  Yet another storage solution built on the equation:

• Open Standards + Open Source = Open Storage

Powerful considering this implementation is not proprietary and uses industry standard APIs. Our contribution to XAM further emphasizes the commitment to eliminate the barriers in the fixed content storage arena. No more closed APIs to a specific vendors hardware or software stack, but rather as an industry standard such as Ethernet. Some vendors are being forced to open their APIs as opensource is having a positive effect for customers.

Code contrbutions can be found at java.net, opensolaris.org and for more info on joing SNIA please go here.

A great start for building your own digital archive appliance with proven enterprise software that is available as an open source code base.  A common theme for some vendors that are leading the way. 

Mårten Mickos the CEO of MySQL hit upon a key trend occurring in the web core which is also known as SAAS, web hosting services,  the internet cloud, etc. . "Some people spend time to save $$$ and some people spend $$$ to save time."  In the open source arena the majority of folks (actually it's a pretty vast majority) are spending the time to save the $$$.  50,000 downloads a day of the most popular open sourced database is an eye opener.  That certainly sounds like a vibrant and active community to me!  People who help develop, enhance, promote, utilize, advocate, lead, govern, market, plan, discuss, etc. There is power in numbers  Are individuals using this database?  Yes in the droves.  Are companies using the database?  Wikipedia, Yahoo, Google, Facebook, SecondLife are using MySQL.  Would banks be interested in this technology alternative?  That is a good question.  Are some enterprise companies experimenting with CentOS today?  The answer is "yes."  Can you purchase commercial grade support on CentOS?  It is probably a barrier for certain application deployments in the enterprise today.  Seems like the potential is there... Commercial grade support from a Fortune 500 company can broaden the reach to new customers. A company that  generates new technology rather than simply gather and glue technology together in a distro can be an advantage as well as an attraction.

Does an open source business model want to shift enterprise customers from spending the time to spending the $$$.  Of course. When those customers are ready to do so on their terms because it is a business decision.  Think of "free" but drop the letter 'r' (for me it's easy since I'm from Boston) and you have "fee" for commercial grade deployment-- which typically means support, various service offerings and SLAs.  If you become "the" largest open source company in the world you drive for new and repeat customers via opportunities not by mouse traps.  Opportunities are generated by downloads, partners, OEMs and direct sales.  While the MySQL acquisition is subject to closing and regulation approvals it is clearly a move that complements nicely an existing software business that is growing.

How much data is currently being stored via this relational database?  I imagine it is many tables of stored data and many more tables of the relations between those tables of stored data.  Can MySQL help drive synergies with storage products and other offerings? 

A big yes.

This is making too much sense.

Few can say that storage is not one very hot topic in the IT sector today.  The internet has created quite a medium for content delivery of podcasts, blogs, webcasts, webinars, etc.  This blog itself is an example.  Storage solutions are abundant, but don't forget that some storage vendors charge you for every neat feature.  Yes you pay for that special hardware that does compression to save storage space.  You also pay for every one of those protocols that you need (iSCSI, CIFS, NFS, etc.) and let's not forget about those important data services for protection (replication, clustering, anti-virus engine, etc.)  Some vendors even charge big money when you grow out of your storage pool and want expansion.  Can you say fork lift upgrade?  Well not all storage vendors want to charge you twice.  As the storage market approaches commodity some of us are getting ahead of the curve.  It is true that all storage companies want to make money... but the difference is leading the way versus fighting against something that will happen anyway.  Even new fast storage hardware will become a commodity as others join with similar offerings. It is important to understand that storage today is heavily reliant on low level software such as device drivers, frameworks and protocols that enable the higher level software in the storage stack to simply work.  If a company can expose and open up the storage stack it has a good possibility to attract not only customers but developers as well... which is what I'll call a community.

At OpenSolaris.org there is a project being done by the community called COMSTAR (Common Multiprotocol SCSI Target).  It is a clever framework which enables protocol plug-ins which speak differnent flavors of storage like Fibre Channel, iSCSI, etc.  For me it is analogous to  the old port and class driver model of my youth.  For more info on COMSTAR click here. 

Also see what this community member has to say. Some other notables about the storage plumbing at OpenSolaris.  The COMSTAR effort puts code into the kernel for optimization.  It will improve upon the current iSCSI Target already available and in good use today.

Open sourcing the entire storage stack implies the storage plumbing too.  For example it enables this storage stack to work for you rather than you working to pay for that expensive storage solution.

Remember Sun's OS, middleware, database and infrastructure products have the following in common-- they are models of open sourced software that more and more customers are demanding from all vendors and providers.

In anticipation of some Winter Holiday downtime and unplugging from the internet cloud-- I envisioned some history reading and some home projects that required tools from a Craftsman Toolbox.  On Dec 24th my home computer hardware decided to get real sick.  The system uses fast RDRAM memory but all 2GB became usless when a single memory address line failure corrupted my system disk.  I was able to invoke extended memory tests then managed to isolate the faulty DIMM module and once again had good memory.  My next step was to try and recover the system disk.  I ran chkdsk (its just like fsck), mucked with the boot partion, etc. but could not recover the disk.  Previously I was fortunate to recover the system disk two times prior on this machine but this time my luck ran out.  Since all my data is on a separate data disk that is incrementally backed up daily to a network drive, I still had the most important items intact... The DATA.  So I reformatted the system disk and reinstalled the OS, then all the numerous updates, patches and ALL the applications (including configuring them).  It took about a day but all is normal on the home computer once again.  My wife and I realized how dependent we are on this device for running our domestic endeavours (paying bills, kids activities, purchases, getting information, etc.)  While we have other computers running various OSes in our house, this specific one was the nerve center.  It really resonated with me that I had taken precautions to preserve the data and they payed off.  It felt similar to the car accident question.  "Are you all right?"  We can replace the car but not you (the data). Personal digital data that an individual generates is just as important as the digital data of corporations.  We live in a sea of digital data today.  With free email, picture sharing and archives that store massive amounts of data that increases every second, protecting the data grows in importance as well.

My son received a new gaming console as a holiday present and the initial experience was amazing.  The ability of the graphic processor (GPU) to generate and manipulate polygons is incredible.  The effects generated to create waterfalls, reflection, smoke, 3D, etc. is excellent.  The visual effect is most stunning when the game's HD visual output is displayed in 1080i mode on a HD display.  It is safe to say the gaming console is a powerful personal computer.  The ultimate gaming experience is when you plug into the network cloud and play online with others.  Forget for a moment the complexities of configuring your network router, wireless LAN and the correct settings of open not restricted NAT and UPnP.  To play online and communicate in real time via VoIP, while you visualize in HD, hear in surround sound and control the events of the game is an experience you could only previously get at Walt's Kingdom.  The online response was great... for a few days when suddenly response started to get slow.  I told my son that good old latency had arrived.  He said "What?"  I told him that if 10 kids in each city of the world just received a new game console for the holidays that would be a lot of new consoles.  By the time most of them connected online would be about now...  As a result there are servers somewhere that are hosting these online games that are probably getting strained/hot and can't keep up with all the requests.  In other words the computer infrastructure (lots of servers, storage, etc.) that is keeping all game consoles in harmony can't do it as effortlessly as before because there are now too many of you playing while on school vacation. Then it happened...  His online id on his console got corrupted.  He could not recover his id.  All his scores, points, you name it data and a lot of it was unreachable on his console.  I told him I think his hard drive on his console is corrupted.  So I reformatted his hard drive (deja vu) and went through a recovery procedure.  Fortunately we recovered his online id and all his data since those busy servers with loads of storage kept a copy of his data too.

These technology woes are opportunities for those who provide the infrastructure of the internet... which is one large data cloud growing and growing and growing. 

FOSS is a check box item for new startup companies as well as enterprise corporations who are consolidating, upgrading or issuing new application deployments.  The high tech industry will continue to have companies acquire other's technology as part of alignment and pure business economics.  Some companies acquire open source software and their intent is to continue to FOSter the community with this software, while being able to monetize the asset.   Counter to this strategy some proprietary companies may be inclined to purchase an open source software stack simply to eliminate its growing popularity by customers.  The software industry should embrace, as have universities, that more and more new deployments require solutions based on open source software code bases.  The following table shows very large deployments of storage assets based on proprietary and open source models. Open source software does create a low exit barrier for unhappy customers, but it does enable a low hurdle for a company that wants to take advantage of the opportunity to engage.  If you have built your business model around open source software you have probably listened to your customers and have realized strategically where the software industry is headed.  On the other side of the coin if your business model is to stay proprietary you may be inclined to believe that open source software is a trend and you will be able to continue to differentiate in a commodity market.  The debate continues but customers vote with their purchases.  It is my opinion that os virtualization solutions both proprietay and open sourced will shed some light on the momentum or trend of open source software.  A robust, stable, enterprise OS that can virtualize other OSes as guests has an opportunity.  The market will embrace multiple choices for OS virtualization rather than have a single choice.  With the amount of vendors who have announced OS virtualization solutions that are both proprietary and open sourced the end results are still open for debate.  Who has the momentum?  I remember the VHS and Betamax debate and who tried to dictate rather than listen to customers.

Why talk about an industry standard protocol called NDMP (Network Data Management Protocol).  I agree by itself it is a low level medium that is a check box item in the world of storage.  However it is an important piece of plumbing that enables your "house" to function.  Numerous software storage applications, particularly backup software and tape products, utilize this protocol for providing services.  While it is analogous to other storage protocols such as iSCSI and FCoE it is "the" protocol when it comes to basic backup.  Pretty sophisticated backup software choices are out there all driven by this basic protocol.  This includes multiple configuration choices such as local backup, 3-way and DAR. 

• SUN is a working member of the SNIA community to advance the adoption of industry standards. We will be contributing source code to the SNIA effort to help update the SNIA software from NDMP v3 to NDMP v4.

• SUN is fostering its own OpenSolaris Storage Platform community in cooperation with SNIA and other industry standards bodies to implement and enhance storage industry standards.

• SUN plans to incorporate the SNIA software being created by the NDMP software TWG into OpenSolaris when it is approved by the SNIA membership.

The storage stack of OpenSolaris keeps getting stronger.  While NDMP is a robust service for backup, when it comes to management of massive amounts of data your mileage may vary.

Today any OS needs to provide support for the 2 dominant file sharing protocols in the industry (NFS and CIFS).  While there are good  implementations such as SAMBA which run  "On" the OS in user space, ideally you want both file sharing protocols to run portions of the implementation "In" the kernel.  Now that OpenSolaris has a kernel based CIFS server along side NFS, I want to give kudos to a few people who made it a reality. Keep in mind implementing any file protocol in any kernel is difficult.  When you introduce both Windows and OpenSolaris together at the kernel level one can appreciate the complexities introduced to coexist.  Coexist has the caveat that code gets added in a seamless fashion and that services are not negatively impacted by this new foreign object.  Fundamental items such as the file system, security, permissions, marshaling,  etc., play an important part in making the seamless coexistence a technical mountain to climb.  That said the OpenSolaris team solved some pretty significant technical issues to make the CIFS Server in OpenSolaris available to anyone.  Despite some self proclaimed experts insisting the feat would never be accomplished, I'll point you to a few folks who know how to collaborate, innovate and in some instances dictate how to "get it done."  You can see for yourself the numerous ARC (~35) cases sponsored and approved as well as some intricate details of how problems were solved.  You can hear from the developers directly and since OpenSolaris is open sourced-- peruse ~370 thousands of lines of code that are now part of OpenSolaris.  Let's start with Mike Shapiro and Alan M. Wright.  One is a Patriots and Red Sox fan (like me) and the other is fond of rugby (unlike me... I like the Patriots).  Both are top notch engineers who can architect with ease and crank out massive amounts of code... that just works.  Mike and Alan cracked a heck of a technical problem (yes these are the hard problems that motivate them both) with respect to "Unified POSIX and Window Credentials for Solaris."  Mike describes the problem in detail on a recent post at his blog.  Alan takes us through, via his blog, the evolution of how a fully integrated CIFS service was integrated into OpenSolaris.  He is very literate on topics such as SMB autohome shares and why they evolved.  Let's not forget about Afshin, Doug and Nico.  All quite literate as well on the details of the hows and whys of a fully integrated CIFS service into OpenSolaris that was not just munged to sort of work.  From an architectural perspective numerous design requirements were diligently reviewed.  Take for example endianness.  The CIFS protocol is sensitive to the x86 processor endian order of significance.  However careful care was taken to enable the CIFS protocol in OpenSolaris to work on both big-endian and little-endian architectures by putting intelligence into marshaling.  So that means that the OpenSolaris CIFS Server will run on a SPARC-based platform if a community member were to build an appliance or simply run the service as part of a general purpose server.  Yes Niagara-II and Rock microprocessors work as it is part of OpenSolaris architecture conformance.  What other open sourced OS supports a kernel based CIFS service that runs on both big and little endian machines?

There I was standing at the railway station in the city of Odawara, Japan waiting for the express train back to Tokyo.  Out of nowhere a N700 bullet train sped by without stopping.  I'm estimating it was clocking ~90mph (~145kph) right through the station!!!  The N700 bullet train is an engineering marvel.  In fact the bullet train system in Japan is a wonderful experience.  The service is superb and in my opinion the most efficient means for transportation in and out of Tokyo.  Instead of dealing with stop and go traffic in any congested major city of the world, you ride on what feels like a cushion of air.  You also see the electric poles on the side of the track flash by you quickly...  You know you are going fast.  The N700 will top out at ~320kph (~200mph) which is equivilent to the French TGV  bullet trains.  Previously the N700 was bidding to top out at ~360kph (~224mph) but the East Japan Railway Company decided to scale back due to vibration and environmental concerns.  I wanted to take a picture of the train in motion.  The pure speed not only startled me but had me in awe of this beautiful machine.  The nose of the train is very aerodynamic in appearance.  The number of passenger cars attached, fully occupied with passengers, must of numbered ~20.  Despite the number of fully loaded cars the N700 appeared to effortlessly move  due to momentum and kinetic energy.

The N700 reminds me of the OpenSolaris train that has gathered a lot of speed and momentum during the past 18 months.  This train  is carrying more passengers (community members) and the acceleration is noticeable.  With the OpenSolaris release of the Developer Preview, MMS, Kernel based CIFS Server and yes infamous ephemeral UIDs the validation of open source software keeps building momentum.  Furthermore even more technology is yet to come.  While a picture may be worth a 1000 words-- I encourage you to look, touch and participate in the millions of lines of source code available to everyone.  So you can create your own picture.