What happens downstream?: Scott Mattoon's blog

Today's Green:Net event in San Francisco's Presidio District brought together some of the most informed and influential people working at the intersection of sustainable business and high tech. It was good to see this cross section so enthusiastically pursuing opportunities for improving the environment and their bottom lines while most of the rest of the economy is struggling to stay afloat. I was proud to represent Sun, which was one of the three event sponsors.

No Consensus Here

Kudos to the Green:Net organizers for hosting a wide spectrum of perspectives in the keynotes and panels of this one day conference. The poles of the spectrum were aptly represented by Saul Griffith and Bob Metcalfe. Both of these engaging speakers are big heros of mine.

Both have expansive knowledge relevant to dealing with society's energy dependence. But they differ on the approach to meeting the challenge. I think Griffith would agree with Metcalfe when he said, "Our goal is not to darken the Earth," but he would differ with the Internet tycoon on priorities. Metcalfe, true to his venture capitalist roots, believes the real challenge is to feed the every increasing demand for energy, and he cites the evolution of the Internet as a great reference case for meeting that challenge. Griffith's perspective, on the other hand, places the priority squarely on conservation. I have to side with Griffith, with all due deference to the man behind the Law of network effects, if for no other reason that it represents a bigger shift for society and implicitly acknowledges that there is much that is broken in our current consumption orientation. In an era where some mainstream scientists are projecting the decimation of human populations, knocking our species down from six billion to one billion by the end of the century due the effects of climate change, it's time to consider an emergency exit strategy. Griffith gave his emergency exit strategy in a condensed version of the one hour talk he gave to the Long Now in January, and still managed, in 20 minutes, to convince many of the 200+ attendees that the situation is dire and, crucially, that we (that's the royal We,) have a shot (about a 1 in 3 shot,) of keeping CO2 below the 450ppm threshold where scientist agree we might still avoid irreversible feedback effects. Griffith and Metcalfe effectively sanctioned the rules for the Green:Net conversation: consensus is not required to participate; differing opinions are valued.

Open_Source is a Verb

One of the highlights of the event was the Sun Workshop titled, Open Sourcing The Sustainability Challenge: Technology for Social Good. The panel discussion was Moderated by Josie Garthwaite with Sun's own Lori Duval, Natural Logic's Gil Friend, EQ2's Steve Burt, and AMEE's Gavin Starks providing the subject matter expertise. Data was the focal point of this discussion with lots of emphasis on accuracy and integrity being critical success factors for measuring GHG inventories and environmental effects. It was clear that these leaders appreciate the magnitude of the challenge, yet they do not shrink from it. It was also clear from this workshop that the companies and NGO's and governments building these data collection and analysis systems are going to need a lot of computing infrastructure and specialized information systems knowledge to manage the Big Data sets required for accurate and meaningful analysis. Openeco.org was cited in the discussion as a good example of an open model for collecting, protecting and comparing data. However, Openeco.org, and the range of the carbon calculators and GHG inventory tools on the web do not fill the need for a comprehensive analytics system needed by decision makers who deal with GHG emissions policy and standards. Bigger, more sophisticated and integrated systems are needed here.

Can the Internet Help?

Another highlight for me was the Panel titled The Green Web Effect, which dealt with the use of web technologies to create a successful call to action in the green business movement. Much of the highly evolved thinking about the network effects of the Internet that is typified by Clay Shirky's work was abundant in this panel consisting of Moderator Alexis Madrigal, Erin Carlson, Director of Yahoo! for Good, Ron Dembo, Founder and CEO of Zerofootprint, Jason Karas, Founder and President of Carbonrally.com, Kevin Marks, Developer Advocate for OpenSocial at Google, and Dara O’Rourke, CEO of GoodGuide. Alexis opened the discussion by saying, "We didn't necessarily start the fire, but we're trying to help put it out." Framed in that context, he then asked the panelists, "What do you want to have happen out in the world?" I liked O'Rourke's answer the best, because it aligns with a vision I've had for many years. He said, "We want to help people to make better decisions by providing the right information at the right moment." He was speaking of the GoodGuide service, which helps consumers understand the environmental, social, and health impacts of their purchases at the decision point in a retail environment. I've just started using their new iPhone app and am pretty impressed with what they've accomplished in less than a year as a business. I am very excited about the potential for this service when they add barcode scanning capability. I'd also really like to see them add some reputation enabled crowdsourcing capabilities. I'd also really like to help them build that.

At the end of the day, I think it was pretty clear to everyone at the event that open networked systems for dealing with global warming, and energy efficiency are not only possible, but imperative if we are going to have any chance of avoiding catastrophic consequences of climate change. The event was another reminder of why I'm proud to work at Sun.

The Demand Response Research Center (DRRC) at LBNL provides a system that enables electric utility customers to automate energy load shedding during peak demand periods.  It's called Demand Response Automation Server (DRAS).  Basically, it takes a feed from the utility whose payload includes data for: Event Pending, Price levels, and Price Schedule.  The system can interface with environmental management systems to turn off lights and raise set points when curtailment events and price jumps occur.  It could also be used in combination with systems management software to automate data center load shedding, but has yet to be adopted for this purpose.   With advanced virtualization and automation technologies, there is ultimately no reason that workloads could not be migrated according to a demand / cost equation.

The barrier to DRAS adoption in the data center has been a mix of scant awareness, legacy perceptions, and a shortage of creative thinking.  But all that is changing.  DRRC is on a campaign now to build awareness among all eligible facilities and IT managers about the potential for DRAS.   And the old saw that says power cycling a computer reduces it's MTBF is just that: old.  Today's systems, for the most part, are engineered to withstand daily power cycles well beyond the typical useful life of a computer.  With the benefit of this knowledge and the findings of a Harris Interactive Poll commissioned by Sun, IT and facilities managers are inducing employees to turn off their computers when not in use.  But we still need more creative thinking.

While the traditional facilities folks appear to be abundantly creative with power saving measures (in a DRAS webinar last month, Aimee McKane from LBNL, cited one example where a bakery participating in DRAS bought more bread pans to avoid running the dishwasher during peak demand periods,) creative applications of DRAS in the data center are in short supply.  

Still, the conversations are happening, as Walter Bays demonstrated on his blog today.  If a courageous (and creative) company were to combine Dynamic Infrastructure technology from Sun and DRAS from DRRC they could begin to realize the savings possible in Walter's energy utopia.  Services from Sun and others specializing in Demand Response systems like EnerNoc, can help these first mover companies develop strategies for capitalizing on these huge energy saving opportunities.

Dave Douglas kicked off the EnergyCamp "Unconference" and incited a heated debate by asking the opening session panel, "Do we need five big innovations or billions of little ones?"

Hunter Lovins responded first, saying we have the technology to get to sustainability: efficiency first, to buy time, then follow the model of nature, strive to produce products locally, at ambient temp, using waste from one production as inputs to the next.  The economics are beginning to make sense, as evidenced by the emergence, in China, of the world's first green billionaire.

Adam Werbach answered facetiously, "Kill the experts."  What he means is that the "experts'" legacy thinking is getting in the way.  Grass roots non-experts, in the shape of WalMart employees, have turned off the lights in coke machines in WalMart's employee break rooms (and Pepsi machines too).  Once Coke saw sales dip, they responded by putting up a  "My light is out but it's cold inside"sign, then saw their sales rise while Pepsi's fell.

Ted Nordhaus said our priority is to grapple with inexpensive energy, which is a root cause of climate change.  This will be a process of unleashing human power to change the political reality, he predicts.

Michael Shellenberger claimed that international policy alone is not a solution.  He cites that, since Kyoto, GHG emissions have risen in Canada and the EU faster than in U.S.

The panelists' positions having been posed, debate ensued over many topics, especially the potential role of nuclear power in solving climate change.  Nice to see some substantive disagreement - Sun did a great job picking a panel that would not mimic the too oft aligned community of experts.

More from the Unconference coverage here.

The San Francisco Bicycle Coalition will be doing community outreach at tomorrow's OpenEco EnergyCamp.  SFBC is one of San Francisco's most influential grass roots organizations.  They've been a model for many other urban bicycle coalitions, and have helped change the character of San Francisco's streets by advocating for bike lanes, and helping promote bicyling to events like EnergyCamp.

Biking to the event and using the web to promote biking to the event are two great examples of how Technology can be applied to mitigate the adverse effects of Population and Affluence.

Check out the EnergyCamp wiki to see what sessions are being planned for this "unconference" sponsored by Sun.  It looks like it's shaping up to be a great event.

EnergyCamp will take place tomorrow Thursday, January 10 at the Mission Bay Conference Center in San Francisco.  Register here.

This question was the underlying theme of many sessions at the 2007 Uptime Institute's Design Charette, which I attended this week.  But it's the wrong question.  As I wrote in a follow up to the EPA Energy Star report on data center efficiency the bigger question is:  How can IT create value, in the broader economy, that replaces other less efficient modes of commerce and interaction?  In that context, any goal to reduce data center energy use is probably unattainable.

Data center energy consumption is projected to be 2.5% of total U.S. electricity demand by 2011, and it's tracking to double every 5 years.   Should IT managers be focused on driving that ratio and rate down?   That's another wrong question.

My design charette team was focused on Data Center Management & Metrics.  Green Grid contributor Ken Uhlman from Eaton was on the team.  He  posits:  "Managers get things done right.  Leaders get the right things done."  Accepting that axiom, it becomes clear that efficiency potential can only be maximized if we have both managers and leaders focus on the challenge.  Data Center managers need to reduce the marginal energy use per unit of work executed in the data center.  Leaders need to find ways to deliver economic value over the network that are more efficient than current business and social practices.

For instance, how much energy can be saved by services like those offered by MillionsOfUs.com?  Every "test drive" of an automobile in these virtual worlds uses some amount of electricity and causes a puff of CO2 to be emitted from a power plant, but the watts/joules/calories and GHG emissions involved are infinitesimal compared to that of a trip in the combustion powered vehicle down to dealer row to try out cars.

Clearly, IT driven efficiency has been at work for a long time.  Over the last 40 years, global economic productivity gains have been driven largely by IT, and much of this gain has arguably resulted in a net reduction in energy use (modulo the indirect demand for energy driven by IT).  But how much?  And what is the size of the opportunity ahead to do even more?   Studying these effects was a clear call to action in the EPA Energy Star report, but no such action appears to be underway.

While it is critical for managers to get a handle on efficiency within the data center envelope - and the potential here is huge - real leadership in energy efficiency will come in the form of value creation over the network that displaces less efficient value creation.

Sun hosted a well attended Silicon Valley Leadership Group (SVLG) event focused on Energy Efficient Datacenters last Thursday at Sun's Santa Clara campus.  The agenda consisted of case studies, an energy demand management program and a facilities tour.  I'll share some of the highlights here.

Small Retrofit Investment Yields Big Savings 

NetApp Facilities Director, Dan Hoffman presented the results of some very targeted energy efficiency measures in a mid-sized (6764 sq ft.) data center.  The net results of a $146k investment were impressive:

• Estimated Energy Savings: 1,042,000 kWh/year
• Cost Savings:   $125,000 -$145,000 year
  
• GHG emission reduction:  3.6 million lbs./year (equal to removing 150 cars from highways annually)

And under a PG&E rebate program they were able to recoup their entire initial investment, so the project had a zero year payback.

The key measures, developed with the assistance of Lawrence Berkeley National Labs (LBNL), that made these savings possible included:

• Supply right amount of air at correct temperature where it is needed, i.e., to the inlet of IT equipment
• Install curtains between hot and cold aisles to minimize Source Air (SA) and Return Air (RA) mixing
  
• Replace SA registers that direct air to top of racks with grills that direct air downward to reduce stratification & hot spots
• Install array of wireless temperature sensors strategically placed on cold aisle side of racks to enable raising SA temperature (i.e., raise the set point on the thermostat by 2-4 degrees)
  
• Allow RA to rise to 80-90°F to enable a substantial reduction in RA/SA fan speed
• Optimize “free-cooling”effect of economizer by raising SA temperature and allowing dampers to modulate longer to reduce the load on the cooling system
  

I always hear the question, "Why don't we build data centers in northern climates where we can use the outside air for cooling?"  So it was of key interest to me to hear Dan's experience with "free cooling" and the economizers' effect on airborne particle concentrations.  When they increased "free cooling" to use 85% outside air the particle concentrations jumped to 11 micrograms/m³, nearly double their normal level.  This is well below ASHRAE standards, but still of potential concern to data center managers worried about contamination.   The environment used 40% filters, which is typical of modern data centers.  It was estimated that the particle count and size could be reduced significantly (over 50%) by replacing the 40% filters with 85% filters.  The energy required to force air through these finer filters, of course, is higher, so may not be rationalized in terms of net energy use.

Demand Response Coming to a Data Center Near You

LBNL Computer Systems Engineer, Girish Ghatikar shared details of the Demand Response (DR) programs being developed in conjunction with utility companies, and gave some compelling data on why voluntary participation in PG&E's AutoDR plan could be beneficial to medium and large IT equipment users. 

The current PG&E DR Program provides technical assistance and requisite equipment to help facilities managers shed electrical load during critical peak usage periods.  Savings through discounts on electricity can be significant (up to $50 per kw saved) under the program if the facility can reduce usage during "events" (project to be between 12 and 15) during the summer (May 1 - October 31).  Participants are notified day before or day of an event, and participants control the level of curtailment during the period.

For most data center managers, the idea of switching off IT equipment during business hours and on short notice, is akin to fingernails scratching a chalkboard.  The mere mention of cycling the power on a server gets a visible shiver from the average system administrator.  But many of the concerns are becoming manageable, and the notion that cycling the power reduces MTBF is dated - most modern IT equipment does not suffer any significant reduction in reliability due to moderate frequency of power cycles. 

In response to the management problem - how to efficiently and gracefully respond to peak event notification - the LBNL folks have developed a systems architecture that can be integrated into existing environments.  The infrastructure includes a Demand Response Automation Server and Energy Management Control System (EMCS), which transmit event notifications and trigger load shifting actions.  The architecture leverages a new SOA standard, OpenADR based on SOAP, that enables a publish and subscribe model.  The payload of the OpenADR messages contain Event Pending, Price levels, and Price Schedule.  This architecture, while promoted through the AutoDR program can be used for finer grained control that can be integrated with other event notifications besides PG&E's critical peak days.  For example, if price exceeds $0.20/kWH then initiate a load shed activity.

State of The Art - Sun's New Santa Clara Labs

When I first met the distinguished LBNL Engineer Bill Tschudi at this meeting, he said, "You guys are doing some great stuff, but why aren't you using free cooling in your new rooms?"  Good question, Bill.  I don't know, but I suspect it has something to do with the somewhat unique mixed use environments that are characteristic of the new labs being built in Building 12 on the Santa Clara campus. 

As Dean Nelson, Sun's Shared Lab Services Director, pointed out on our tour of these nearly completed rooms, engineers will be working in these environments as part of their normal day to day routine.  For the most part, these are not lights out environments.  Systems engineers and Services engineers will be using these large labs to do everything from prototype component testing to troubleshooting customer problems.  They're designed with ergonomics and frequent physical movement of equipment in mind.  A lot of the space in these labs is dedicated to benches where gear can be disassembled and tested easily.

Two huge cooling towers are the centerpiece of a mechanical plant (pictured here) that sits behind an office building facade just outside the labs.  These 1000 ton chillers use adaptive frequency drives that change the frequency and voltage to optimize the load to what is sensed.  Dynamic cooling within the data centers takes the cooling intelligence down to the rack level, and spot cooling minimizes unwanted air mixture.

One of the highlights of the tour was a look at the Hot Aisle Containment solutions by Liebert and APC.  There were several installations of each, and Dean expected huge savings in cooling costs due to their efficient circulation of chilled air.  The Liebert Solution has a higher total CFM, so was chosen for the more dense lab needs, while the APC Solution with some sensing and adjusting capability and lower maximum CFM were chosen for the less dense, or not always on environments.  Dean's team will be generating benchmark data that compares both solutions once they get populated and running. 

Watch this space for more on the magnitude of savings and reduction in greenhouse gas emissions as a result of the new Santa Clara labs project.

---

Related reading:

• High Performance Building for High-Tech Industries
  
• Demand Response Research Center
• PG&E's Auto Demand Response Program 
• What Sun is doing to green its business
  

My alma mater, U of Minnesota, is collaborating with UCLA and other universities under a DOD award to study biologically assembled electronics.   The study builds upon work by professor Richard A. Kiehl at the home of the Gophers in which he demonstrated metallic nanoparticle assembly by DNA scaffolding. While the researchers have not touted the energy saving potential of such precise biological assembly of nanoparticle arrays, there is every reason to believe that such a process would be more energy efficient than the 1,400^oF process used in the fabrication of most silicon based semiconductors today.

 

News of LBNL's server energy use report, released last week refers to the study as recent, yet the most recent actual data used in the study is from 02005.

The purpose of the report, "to accurately characterize electricity used by servers in the U.S. and the world so that public debate can proceed based on accurate data," was fulfilled. The report also established a process for aggregating relevant data and attribution of power use to specific server models. Furthermore, Koomey and other LBNL scientists helped to dispel the myth that office use of IT equipment approaches 13-15% of total U.S. energy consumption, and they reinforce some of the widely held assumptions related to energy use in IT: 60-80% of energy supplied to IT equipment in data centers is consumed by servers, and 50% of total data center energy use goes to cooling. Most importantly, the study established that the growth in energy demand generated by servers doubled over the period 02000 - 02005, and, at current growth rates we can expect to see a 40% increase by 02010.

However, IT shops operating in today's business climate should be careful to not assign too much value to the report in their efforts to tackle infrastructure efficiency and carbon neutrality. Important innovation and policy change has occurred since Koomey's report was compiled, which are of much greater relevance to these eco and efficiency initiatives. For example,

• Sun released the UltraSPARC T1 processor, which powers our Cool Threads servers - the only servers eligible for an efficiency incentive rebate from energy utility Pacific Gas & Electric.
• Water cooled solutions to the chronic challenge of cooling data center are now becoming economically attractive.
• Congress has enacted HR 5646 specifically promoting the use of energy efficient servers, which has already accelerated the pace of innovation in server efficiency.

I knew I could count on Aaron to put it in the proper perspective relative to Sun, but other recent coverage of the study (Green Wombat, ars technica and Treehugger) omitted the relevance of recent innovations and the leadership role that Sun is playing in the advancement of data center efficiency.

Hats off to Koomey and his colleagues for establishing a comprehensive, peer reviewed baseline and methodology for aggregating the data and putting server energy use into a proper context. I am hopeful that Sun and other industry leaders will be able to apply the gains in understanding toward the challenge of solving contemporary business challenges.