What happens downstream?: Scott Mattoon's blog

Today's Green:Net event in San Francisco's Presidio District brought together some of the most informed and influential people working at the intersection of sustainable business and high tech. It was good to see this cross section so enthusiastically pursuing opportunities for improving the environment and their bottom lines while most of the rest of the economy is struggling to stay afloat. I was proud to represent Sun, which was one of the three event sponsors.

No Consensus Here

Kudos to the Green:Net organizers for hosting a wide spectrum of perspectives in the keynotes and panels of this one day conference. The poles of the spectrum were aptly represented by Saul Griffith and Bob Metcalfe. Both of these engaging speakers are big heros of mine.

Both have expansive knowledge relevant to dealing with society's energy dependence. But they differ on the approach to meeting the challenge. I think Griffith would agree with Metcalfe when he said, "Our goal is not to darken the Earth," but he would differ with the Internet tycoon on priorities. Metcalfe, true to his venture capitalist roots, believes the real challenge is to feed the every increasing demand for energy, and he cites the evolution of the Internet as a great reference case for meeting that challenge. Griffith's perspective, on the other hand, places the priority squarely on conservation. I have to side with Griffith, with all due deference to the man behind the Law of network effects, if for no other reason that it represents a bigger shift for society and implicitly acknowledges that there is much that is broken in our current consumption orientation. In an era where some mainstream scientists are projecting the decimation of human populations, knocking our species down from six billion to one billion by the end of the century due the effects of climate change, it's time to consider an emergency exit strategy. Griffith gave his emergency exit strategy in a condensed version of the one hour talk he gave to the Long Now in January, and still managed, in 20 minutes, to convince many of the 200+ attendees that the situation is dire and, crucially, that we (that's the royal We,) have a shot (about a 1 in 3 shot,) of keeping CO2 below the 450ppm threshold where scientist agree we might still avoid irreversible feedback effects. Griffith and Metcalfe effectively sanctioned the rules for the Green:Net conversation: consensus is not required to participate; differing opinions are valued.

Open_Source is a Verb

One of the highlights of the event was the Sun Workshop titled, Open Sourcing The Sustainability Challenge: Technology for Social Good. The panel discussion was Moderated by Josie Garthwaite with Sun's own Lori Duval, Natural Logic's Gil Friend, EQ2's Steve Burt, and AMEE's Gavin Starks providing the subject matter expertise. Data was the focal point of this discussion with lots of emphasis on accuracy and integrity being critical success factors for measuring GHG inventories and environmental effects. It was clear that these leaders appreciate the magnitude of the challenge, yet they do not shrink from it. It was also clear from this workshop that the companies and NGO's and governments building these data collection and analysis systems are going to need a lot of computing infrastructure and specialized information systems knowledge to manage the Big Data sets required for accurate and meaningful analysis. Openeco.org was cited in the discussion as a good example of an open model for collecting, protecting and comparing data. However, Openeco.org, and the range of the carbon calculators and GHG inventory tools on the web do not fill the need for a comprehensive analytics system needed by decision makers who deal with GHG emissions policy and standards. Bigger, more sophisticated and integrated systems are needed here.

Can the Internet Help?

Another highlight for me was the Panel titled The Green Web Effect, which dealt with the use of web technologies to create a successful call to action in the green business movement. Much of the highly evolved thinking about the network effects of the Internet that is typified by Clay Shirky's work was abundant in this panel consisting of Moderator Alexis Madrigal, Erin Carlson, Director of Yahoo! for Good, Ron Dembo, Founder and CEO of Zerofootprint, Jason Karas, Founder and President of Carbonrally.com, Kevin Marks, Developer Advocate for OpenSocial at Google, and Dara O’Rourke, CEO of GoodGuide. Alexis opened the discussion by saying, "We didn't necessarily start the fire, but we're trying to help put it out." Framed in that context, he then asked the panelists, "What do you want to have happen out in the world?" I liked O'Rourke's answer the best, because it aligns with a vision I've had for many years. He said, "We want to help people to make better decisions by providing the right information at the right moment." He was speaking of the GoodGuide service, which helps consumers understand the environmental, social, and health impacts of their purchases at the decision point in a retail environment. I've just started using their new iPhone app and am pretty impressed with what they've accomplished in less than a year as a business. I am very excited about the potential for this service when they add barcode scanning capability. I'd also really like to see them add some reputation enabled crowdsourcing capabilities. I'd also really like to help them build that.

At the end of the day, I think it was pretty clear to everyone at the event that open networked systems for dealing with global warming, and energy efficiency are not only possible, but imperative if we are going to have any chance of avoiding catastrophic consequences of climate change. The event was another reminder of why I'm proud to work at Sun.

The U.S. EPA Energy Star Program released a report on server and data center efficiency(PDF) this month.  The study was in response to Public Law 109–431(PDF), which required EPA to analyze "the rapid growth and energy consumption of computer data centers by the Federal Government and private enterprise".   The Law goes on to say, "It is the sense of Congress that it is in the best interest of the U.S. for purchasers of computer servers to give high priority to energy efficiency as a factor in determining best value and performance for purchases of computer servers."

The increased federal attention to server efficiency is good news for Sun, particularly in light of the recently announced UltraSPARC T2 processor, a.k.a. Niagra 2, which, at two watts per thread, will clobber the competition in most commercial SWaP comparisons.

Where it comes to energy demand, the EPA report keeps the big picture in focus, citing that IT is not only part of the problem, but also part of the solution:

"The data processing and communication services provided by data centers can also lead to indirect reductions in energy use in the broader economy, which can exceed the incremental data center energy expenditures in some cases. For instance, e-commerce and telecommuting can reduce both freight and passenger transportation energy use."

The authors recommend quantifying this indirect reduction through IT services in future research.  This is a largely untapped source of energy conservation for which a range of alternatives exist.  For example, using technology such as the 4 watt SunRay thin client, businesses could shift employee desktop computing tasks to run on optimally efficient servers in the data center, rather than the mostly idle 200+ watt computer at every desk scenario that dominates corporate work environments.  Companies could also employ work at home programs like Sun's Open Work, and make much greater use of video conferencing and web meeting software.  These conservation efforts inevitably increase energy demand in data centers, but clearly offset much larger energy demand by providing reasonable alternatives to some very energy intensive practices that dominate business culture today.

Seperate from any empirical consideration of such indirect energy reductions, the report estimates that by 2011 U.S. businesses could shave off $4.1B in data center electricity costs annually just by following best practice outlined in the report.   Considering that total U.S. data center electricity costs in 2006 were $4.5B, that's a lot of efficiency gain by 2011.

Interestingly, the $4.1B potential data center savings is mirrored by the potential savings determined by a Harris Interactive poll commissioned by Sun, for conservation in the office by workers.  The results of the poll, released August 1, indicate that energy-conscious behaviors of U.S. office workers can save $4.3B in energy costs per year.  With a flip of two switches (lights off, computer off,) workers can make a huge collective difference, equivalent to taking 6.1M cars' CO2 emissions out of the atmosphere.

So we're looking at potential savings of $8.4B, just by doing what we already know how to do, with no compromise to services or productivity.  Add in whatever additional energy can be saved by replacing energy intensive business practices with services over the network and you've got a really good economic case for aggressively pursuing energy efficiency in the data center and the workplace.

---

 Further reading:

• EPA’s Enterprise Server and Data Center Energy Efficiency Initiatives Web site

Sun hosted a well attended Silicon Valley Leadership Group (SVLG) event focused on Energy Efficient Datacenters last Thursday at Sun's Santa Clara campus.  The agenda consisted of case studies, an energy demand management program and a facilities tour.  I'll share some of the highlights here.

Small Retrofit Investment Yields Big Savings 

NetApp Facilities Director, Dan Hoffman presented the results of some very targeted energy efficiency measures in a mid-sized (6764 sq ft.) data center.  The net results of a $146k investment were impressive:

• Estimated Energy Savings: 1,042,000 kWh/year
• Cost Savings:   $125,000 -$145,000 year
  
• GHG emission reduction:  3.6 million lbs./year (equal to removing 150 cars from highways annually)

And under a PG&E rebate program they were able to recoup their entire initial investment, so the project had a zero year payback.

The key measures, developed with the assistance of Lawrence Berkeley National Labs (LBNL), that made these savings possible included:

• Supply right amount of air at correct temperature where it is needed, i.e., to the inlet of IT equipment
• Install curtains between hot and cold aisles to minimize Source Air (SA) and Return Air (RA) mixing
  
• Replace SA registers that direct air to top of racks with grills that direct air downward to reduce stratification & hot spots
• Install array of wireless temperature sensors strategically placed on cold aisle side of racks to enable raising SA temperature (i.e., raise the set point on the thermostat by 2-4 degrees)
  
• Allow RA to rise to 80-90°F to enable a substantial reduction in RA/SA fan speed
• Optimize “free-cooling”effect of economizer by raising SA temperature and allowing dampers to modulate longer to reduce the load on the cooling system
  

I always hear the question, "Why don't we build data centers in northern climates where we can use the outside air for cooling?"  So it was of key interest to me to hear Dan's experience with "free cooling" and the economizers' effect on airborne particle concentrations.  When they increased "free cooling" to use 85% outside air the particle concentrations jumped to 11 micrograms/m³, nearly double their normal level.  This is well below ASHRAE standards, but still of potential concern to data center managers worried about contamination.   The environment used 40% filters, which is typical of modern data centers.  It was estimated that the particle count and size could be reduced significantly (over 50%) by replacing the 40% filters with 85% filters.  The energy required to force air through these finer filters, of course, is higher, so may not be rationalized in terms of net energy use.

Demand Response Coming to a Data Center Near You

LBNL Computer Systems Engineer, Girish Ghatikar shared details of the Demand Response (DR) programs being developed in conjunction with utility companies, and gave some compelling data on why voluntary participation in PG&E's AutoDR plan could be beneficial to medium and large IT equipment users. 

The current PG&E DR Program provides technical assistance and requisite equipment to help facilities managers shed electrical load during critical peak usage periods.  Savings through discounts on electricity can be significant (up to $50 per kw saved) under the program if the facility can reduce usage during "events" (project to be between 12 and 15) during the summer (May 1 - October 31).  Participants are notified day before or day of an event, and participants control the level of curtailment during the period.

For most data center managers, the idea of switching off IT equipment during business hours and on short notice, is akin to fingernails scratching a chalkboard.  The mere mention of cycling the power on a server gets a visible shiver from the average system administrator.  But many of the concerns are becoming manageable, and the notion that cycling the power reduces MTBF is dated - most modern IT equipment does not suffer any significant reduction in reliability due to moderate frequency of power cycles. 

In response to the management problem - how to efficiently and gracefully respond to peak event notification - the LBNL folks have developed a systems architecture that can be integrated into existing environments.  The infrastructure includes a Demand Response Automation Server and Energy Management Control System (EMCS), which transmit event notifications and trigger load shifting actions.  The architecture leverages a new SOA standard, OpenADR based on SOAP, that enables a publish and subscribe model.  The payload of the OpenADR messages contain Event Pending, Price levels, and Price Schedule.  This architecture, while promoted through the AutoDR program can be used for finer grained control that can be integrated with other event notifications besides PG&E's critical peak days.  For example, if price exceeds $0.20/kWH then initiate a load shed activity.

State of The Art - Sun's New Santa Clara Labs

When I first met the distinguished LBNL Engineer Bill Tschudi at this meeting, he said, "You guys are doing some great stuff, but why aren't you using free cooling in your new rooms?"  Good question, Bill.  I don't know, but I suspect it has something to do with the somewhat unique mixed use environments that are characteristic of the new labs being built in Building 12 on the Santa Clara campus. 

As Dean Nelson, Sun's Shared Lab Services Director, pointed out on our tour of these nearly completed rooms, engineers will be working in these environments as part of their normal day to day routine.  For the most part, these are not lights out environments.  Systems engineers and Services engineers will be using these large labs to do everything from prototype component testing to troubleshooting customer problems.  They're designed with ergonomics and frequent physical movement of equipment in mind.  A lot of the space in these labs is dedicated to benches where gear can be disassembled and tested easily.

Two huge cooling towers are the centerpiece of a mechanical plant (pictured here) that sits behind an office building facade just outside the labs.  These 1000 ton chillers use adaptive frequency drives that change the frequency and voltage to optimize the load to what is sensed.  Dynamic cooling within the data centers takes the cooling intelligence down to the rack level, and spot cooling minimizes unwanted air mixture.

One of the highlights of the tour was a look at the Hot Aisle Containment solutions by Liebert and APC.  There were several installations of each, and Dean expected huge savings in cooling costs due to their efficient circulation of chilled air.  The Liebert Solution has a higher total CFM, so was chosen for the more dense lab needs, while the APC Solution with some sensing and adjusting capability and lower maximum CFM were chosen for the less dense, or not always on environments.  Dean's team will be generating benchmark data that compares both solutions once they get populated and running. 

Watch this space for more on the magnitude of savings and reduction in greenhouse gas emissions as a result of the new Santa Clara labs project.

---

Related reading:

• High Performance Building for High-Tech Industries
  
• Demand Response Research Center
• PG&E's Auto Demand Response Program 
• What Sun is doing to green its business
  

The Ziff Davis eSeminar today on Managing the Exploding Energy Costs in the Data Center was billed as an opportunity to learn "How to build the energy-smart datacenter". If you run an overheated, power hungry Dell and Intel datacenter this was an hour for you. The tools, technologies and services presented were specific to these platforms. The seminar was really a commercial for these brands.

It's great to see other industry leaders focused on this problem. It's not great to see Sun's truly game-changing technologies for energy efficiency so often absent from the discussion.

Innovation coming out of Dell for policy based shut down of systems at night was featured in the presentation. This is a valuable tool for the enterprise who has predictable workloads involving periods of non-use. Unfortunately, the businesses with the highest demand for compute resource are global operations that deliver their value online, so don't have the option of shutting down systems at night. (Desktops are generally good candidates for this efficiency measure, but the period of non-use for a 200 watt desktop would have to be more than 23 hours every day in order to use the same amount of energy as a 4 watt SunRay 2. Sign me up for the 5 hour work week.)

Dell made the requisite claim that their servers are optimized for power and deliver 25% greater performance per watt over similarly configured servers. They went on to compare with the HP DL 380 and the IBM x3650.

Intel discussed a lot of the long range promise for improving processor efficiency through advances like 45nm process and, and the potential for 10X reduction in power consumption for equivalent or better performance through the use of Indium Antimonide transistors. Their material included an intersting axiom that smaller silicon means lower power consumption. They didn't answer my question about how to reconcile that with the historical evidence to the contrary - as processors have gotten smaller their power consumption has risen.

Intel also took the opportunity to promote their Ultra-dense processor line which is the Xeon L53x0, their 40W Dual-Core, and 50W Quad-Core low voltage processors. The Quad-Core due is due to ship in the next few weeks. The quad-core is basically a dual die design that combines two of their dual-core wafer on one chip. The wattage spec. for these is interesting. Like other vendor specs, it does not include the power draw of the supporting chip set, but in the L5300 series, where they've moved the memory controller off of the processor, this means there's at least 30 watts not accounted for in the spec if you were looking for an apples to apples comparison.

The most valuable insight offered in the talk was a cost comparison of liquid vs. air cooling. David Moss from Dell said that in high density environments where air cooling has the worst efficienty, liquid is cheaper, at least in theory, than air.

Someone asked a great question about the state of the art in using waste heat from the datacenter to heat office space, and using free cooling from the ambient air outside to cool a datacenter. The answer, basically, was that people are talking about it, and it's not very practical in a place like Austin. It was disappointing that these experts really aren't thinking about the potential efficiency gains from such an industrial ecology approach. There are some practical applications of this already in use in Scandinavia. Other than Project Blackbox I'm not sure what Sun is doing in this area, but it seems like there is a big market opportunity here.

You can get the audio and slides from this and other eSeminars at www.eseminarslive.com