Thursday Feb 21, 2008
More SPECtacular awards given at SPEC's 2008 annual meeting in San Francisco, these to members of the Web Committee. As before, I won't post anyone's name without permission, but you know who you are and SPEC is grateful for your contributions.
It's a bit more complicated to tell you what these engineers did to earn this recognition. In 2007 SPEC released a maintenance version 1.20 to SPECweb2005 which included a lot of improvements. Most notable to me are new code that extends the level playing field of comparison, and tightened standards compliance rules that ensure real world applicability.
In order to focus on performance of the web server, to simplify the configurations tested and so reduce the cost of benchmarking, SPECweb2005 uses a simulated back-end database, BESIM, instead of the database server typically found with real world web applications. The great majority of the computing load is, by design, on the web server. Therefore the database component being synthetic code has not been an issue with fair comparison across platforms. However we discovered that when using an ISAPI implementation, two Ethernet packets were exchanged per HTTP response instead of the single packet with other implementations. No such results had been published, so there were no unfair comparisons - yet. But the problem needed to be fixed in order that ISAPI solutions could be measured on a level playing field. The solution was to adapt BESIM to use ISAPI v2.0 interface based on the Windows IIS implementation (with permission from Microsoft), so now different solutions behave alike, and they behave like real world web servers.
Another potential risk to fair comparison was in the Java Server Pages (JSP) which is typically used in benchmark results in preference to other slower scripting languages like PHP. We added a requirement to the run rules that software products prove their standards compliance by passing the relevant test suite. In the case of JSP that means that web servers which say they implement JSP must really implement JSP, not a subset needed to run the benchmark but omiting some slower features needed to implement real world web applications. Again, we don't know of any published results which did gain a performance advantage by cutting corners on standards conformance. Now that the rules are tightened we know there won't be any.
Thursday Oct 04, 2007
I measured electricity usage of our
home computers using a cheap clamp-on ammeter from Radio Shack. I
concluded that you need to make your own measurements because many of
my preconceptions did not hold up to actual measurement. Since then
I bought a more accurate and convenient Watts
Up meter for $21 from amazon.com, and re-measured everything.
To my
colleagues at HP: I apologize for saying your PC used 300 watts.
100 watts is more like it. I likewise
overstated the power usage of other PC's. I correct the measurements
here, along with some discoveries about Windows and Ubuntu power
management. You may be asking, why should I believe you now? Good
question. You should not believe some random person you find on the
net. You should get your own meter and measure your own equipment.
Why might you halfway believe me,
enough to read a bit more? Well, Watts Up does seem to be a more
reliable instrument, and unlike my earlier measurements it corrects
for the power factor. It displays the power factor, which can vary
greatly from one device to another. As a check on the new meter I
measured a 3-way light bulb: the 60w setting measured 44w, the 100w
setting measured 89w, and the 150w setting measured 135w. At least
I'm assured it's not measuring high.
A colleague in the SPEC power committee
laughed at my new meter, “You mean you trust a Watts Up meter?”
For the upcoming SPEC power
benchmarks that team of engineers from many companies, universities,
and agencies has written elaborate requirements for the power meter
and measurement protocol to be sure that results tested by one lab
will be comparable to results tested by another lab. But for home, if you can get any information at all about the power your electronic gear uses, that's good.
First I looked at how many watts they
used in normal home, light load operation. I switched the Linux
distro on PC2002 to Ubuntu 7.04 Feisty Fawn, and its power management
software seems to be the equal of Windows.
|
OS |
Computer |
Monitor |
Speakers |
Total |
| PC1999 |
Linux |
39 |
26 |
1 |
66 |
| PC1999 |
Windows 98 |
37 |
26 |
1 |
64 |
| PC2002 |
Linux JDS |
95 |
34 |
- |
129 |
| PC2002 |
Windows XP |
61 |
34 |
- |
95 |
| PC2002 |
Linux Ubuntu |
57 |
34 |
- |
91 |
| PC2006 |
Solaris |
77 |
22 |
- |
99 |
| PC2006 |
Windows XP |
67 |
22 |
- |
89 |
Next I looked at how many watts they
used under a heavy load. These weren't identical controlled loads,
and workload really matters to how much power is used.
|
OS |
Computer |
Monitor |
Speakers |
Total |
| PC1999 |
Linux |
46 |
26 |
1 |
73 |
| PC1999 |
Windows 98 |
55 |
26 |
1 |
82 |
| PC2002 |
Linux JDS |
95 |
34 |
- |
129 |
| PC2002 |
Windows XP |
100 |
34 |
- |
134 |
| PC2002 |
Linux Ubuntu |
118 |
34 |
- |
152 |
| PC2006 |
Solaris |
? |
22 |
- |
? |
| PC2006 |
Windows |
115 |
22 |
- |
137 |
How much of difference does workload
make? Here is the power consumption of PC2002 running Ubuntu:
|
Operation
|
Watts (PC only)
|
|
download OS updates
|
70
|
|
system quiescent
|
57
|
|
100-Base-T file copies
|
80 - 100
|
|
Play Supertux arcade game
|
116
|
|
Scanning document with USB scanner and xsane
|
118
|
|
Afterwards [Note 1]
|
115
|
|
Killed mysqld
|
60
|
[Note 1] I don't know what set off the
MySQL daemon, whether it was doing useful work, or had just run wild.
But without the power meter, had I noticed anything amiss at all, I
would have just been puzzled that the fan was running. When I stopped
the daemon, power usage fell back down to normal levels.
Next I suspended each PC by pressing
the sleep button as the system is configured out of the box.
|
OS |
Computer |
| PC1999 |
Linux |
27 |
| PC1999 |
Windows |
27 |
| PC2002 |
Windows |
2 |
| PC2002 |
Linux Ubuntu |
2 |
| PC2006 |
Windows |
44 |
But wait, what's going on with that new
HP PC taking so much more power suspended than the old HP PC? I
looked at the Windows power management control panel and started
reading the documentation. It turns out that the old PC was set by
default for the sleep button to put it into what Windows calls
“standby” state, while the new PC was set by default for
the sleep button to put it into “away” mode. Away mode
doesn't do much, and is intended for systems that might still be
doing some processing without a person sitting there. So it's kind of
like forcing it immediately into the quiescent state it would
eventually go into if you just left it alone, and the monitor blanks.
I changed it to go into suspend state, and then it used just 6 watts.
Hibernate state used 4 watts. Michael Chu wrote a great
description of how Windows XP power management works.
Here are some details of my
configurations.
|
OS |
Computer |
Monitor |
Speakers |
| PC1999 |
Primary: Sun JDS3 Linux; Secondary Windows 98SE |
PC, Sony Vaio, Pentium III, 500MHz |
NEC, 14” LCD, 1024x768 |
Sony |
| M2000 |
Primary: Linspire Linux; Secondary: Windows 98SE |
Laptop, Sharp Actius, Mobile Pentium II, 300MHz |
integrated |
integrated |
| PC2002 |
Primary: Ubuntu 7.04 Linux; Secondary Windows XP Home |
PC, Compaq S6000Z, Athlon 2600XP, 1.9GHz |
Samsung 170MP, 17” LCD, 1280x1024 |
integrated |
| PC2006 |
Primary: Solaris Nevada; Secondary: Windows XP Media |
PC, HP a1540n, Athlon 64 x2 4200 dual core, 2.2GHz |
Sharp LL193A, 19” LCD, 1280x1024 |
integrated |
| M2007 |
Debian Linux |
Tablet, Nokia N800, TI OMAP 2420, 330MHz |
integrated |
integrated |
