« links for 2007-06-18 | Main | links for 2007-06-19 »
20070618 Monday June 18, 2007

Hardware Archaeology

Cactus Pillow

One of the running themes of Free/open source software has been how easily drivers for chipsets in old (and often not-so-old) hardware can be created. Some semiconductor vendors keep a tight hold on the details of their chips, and discourage F/OSS developers strongly. The recent news that Open Sound is available as Free software under both GPLv2 and CDDL is a great step forward, and the availability of drivers for R500-family AMD video cards is also fantastic.

As a long-term manufacturer of fantastic hardware, Sun is frequently approached by groups of developers looking for documentation for the chips used in its products. These days, Sun's newest chips are open source already, and there's a good web-site for documentation for current chips. This has already resulted in great things.

More of an issue, though, are the older systems. In plenty of cases, rather than using I/O chips sold on the open market by companies like Broadcom, nVidia, and ATI, Sun's engineers have actually designed chips specifically for the work at hand. Unlike those companies which sell I/O chips for a living, and thus have a moral duty to provide complete, externally presentable documentation for their chips, Sun's engineers often had no business need to document the chip design for public consumption (or, in quite a few older cases, at all!) Even worse, in some cases the chips Sun has made use designs and ideas acquired from third parties as a 'trade secret'. In those cases, public documentation just can't be made available.

So what happens when people ask for documentation? Well, there are at least five cases:

  1. The product in question may be past it's end-of-life date and there's actually no team anywhere in Sun that can answer the question.
  2. The documentation may not exist. Writing it would require costly reverse engineering (even if people with the skills to do so are still at Sun).
  3. It may exist for internal use, but releasing it outside Sun would need legal review to check for 'trade secrets' belonging to others. That legal review is time consuming and costs real money. And the answer may turn out to be "can't release, can't name the company that's obstructing us".
  4. It may exist and it may be possible to release it. Doing that, though, is actually a non-zero cost since usually the documentation is in an old format of some kind.
  5. It may actually exist in an online-ready form.

The question has once again been raised (over on Jonathan's blog) of whether Sun is acting in bad faith over interface documentation for older systems. I've done a cursory check and I believe answers of "no, we can't help" fall almost exclusively in the first four classes above and mostly in the first two. But I do understand why passions are raised and scepticism is rife.

Jonathan asked me to look into this, to ensure we're pursuing an open path across all of Sun, not simply the software group. We take all input seriously, and we can't solve all problems for all parties, but we're committed to doing our best to faithfully engage with all the communities we serve, in the same spirit as the existing Open Source Ombudsman Scheme. With the support of my team and others in the community I'll try to build a new scheme that is fair and transparent. Watch this space!

Update Aug 27: The new FOSS Open Hardware Documentation wiki is now open for business.

Posted by webmink ( Jun 18 2007, 06:30:52 PM PDT ) Permalink Comments [32] Trackback
technorati del.icio.us digg slashdot
Comments:

And what about north bridges for modern Sparc systems ?

Also, you probably kidding about ATI, Broadcom and Nvidia. Try to get documentation about Broadcom and Nvidia ethernet.

Posted by kk on June 19, 2007 at 04:43 AM PDT #

kk: I'm not saying they do provide it, no, just that it's reasonable to expect them to do so.

Posted by Simon Phipps on June 19, 2007 at 04:47 AM PDT #

So which case applies to the Sun V215/V245 servers that were mentioned in Theo de Raadt's comment?

Posted by chithanh on June 19, 2007 at 05:03 AM PDT #

chithanh: No idea yet, sorry. I do note that David Miller seems to have written drivers for them though.

Posted by Simon Phipps on June 19, 2007 at 05:13 AM PDT #

The freeing of OpenSound is great news! I think you linked to the wrong press release though; try this.

Posted by Luke on June 19, 2007 at 05:14 AM PDT #

Luke: Fixed, thank-you.

Posted by Simon Phipps on June 19, 2007 at 05:15 AM PDT #

It seems to me the first step is openness in communication. I suggest you set up a page where people can ask for classification of their requests. Then you put each request under one of the five categories above and encourage internal and external review of the list. This way, knowledge might surface that wouldn't have done so otherwise.

Posted by My Name is Nobody on June 19, 2007 at 05:33 AM PDT #

If vendors are locking Sun into NDA's that prevent Sun from being open, then going forward Sun should reject partnering with vendors that lock Sun into NDA's that hurt Sun's customers. Do I think Theo is a jerk? Yes. But his open source credentials are beyond challenge. His operating system is superlative, and the Solaris team should be paying attention to what he's teaching you through his own work. Sun should really be bending over backwards to accommodate OpenBSD. When Theo says "this vendor has been very cooperative", a lot of us tend to flock to that vendor because we know they are going to be well supported by the OS. But Theo has been having issues with Sun saying one thing and doing another (more like doing nothing) for a long time. So please, use whatever leverage you must to keep your lawyers in line (they do tend to run the company into the ground if you let them) and instruct them to find a way to cooperate more fully with projects like OpenBSD and the Linux kernel team. Linux support on Sun (SPARC) hardware is at best experimental right now. I'd really love to see that change.

Posted by Magnus on June 19, 2007 at 06:34 AM PDT #

Your 5 points about whether documentation can or cannot be released seem fair. May I suggest that (a) for the first four that you also check whether or not Sun could enter into an NDA with developers so that developers could get access to the info and do a cleanroom style build? (I.e. the developers themselves would have to do much of the work, and would generate docs that others could use.) Certainly this could be done with a group like the Linux Foundation, and perhaps Sun could even get the costs covered. And (b) even if 'a' is not an option, please let the community know why - even if it just saying "sorry we can't; a vendor is blocking us and we cannot say anything more than that" or naming something else from your list. I think that in itself would solve a lot of the issues. Who knows - Sun may even find groups willing to pay for the work.

Posted by TemporalBeing on June 19, 2007 at 07:22 AM PDT #

TemporalBeing: My understanding of the OpenBSD position is that information under NDA is never acceptable. Certainly I will try to get as much information out as possible, including offering NDA access where that's the only option. I've already discovered that I can likely get access to Psycho, Cassini, Schizo, GEM (all out of production), Tomatillo, and Fire documentation under NDA at pretty short notice, for example.

As to the transparency you describe: Yes, that's the plan :-)

Posted by Simon Phipps on June 19, 2007 at 07:37 AM PDT #

There's another issue with FreeBSD trying to import dtrace. They need some *headers* under a BSD-like license so that dtrace won't have to be patched in every time some wants to try it. The problem was serious enough that the dtrace port, which was going to be committed to FreeBSD 7.0, was suspended ... in fact it smells dead already.
Yes, this is not a hardware issue, but the fact that such a simple issue has not had any response, makes us think that SUN is not willing to work with the community and not being all that opensource at all.

Posted by Pedro on June 19, 2007 at 09:38 AM PDT #

There's another issue with FreeBSD trying to import dtrace. They need some *headers* under a BSD-like license so that dtrace won't have to be patched in every time some wants to try it. The problem was serious enough that the dtrace port, which was going to be committed to FreeBSD 7.0, was suspended ... in fact it smells dead already. Yes, this is not a hardware issue, but the fact that such a simple issue has not had any response, makes us think that SUN is not willing to work with the community and not being all that opensource at all.
The license is not a Sun issue. The OpenSolaris community is now in control of that. If you want to see it changed, petition the community, not Sun. At this point, I still haven't seen a full, legitimate reasoning as to why the BSD project can't integrate the CDDL code, header files or not, when Apple is doing so without any issues.

Posted by Shawn on June 19, 2007 at 09:42 AM PDT #

Shawn: It's not the BSD project, it's the BSD projects, they are not a singular entity, they are several distinct groups with their own goals and rules and code bases. OpenBSD is the hardline one, the one that is extremely BSD, going so far as to replace everything it can that does not use a BSD-like licence with code that does. A BSD cannot integrate code that is more restrictive than the BSD licence, else it is not a BSD anymore. The simple requirement of credit is the only one which BSD developers hold dear, anything else is intrusive and offensive in their code. FreeBSD doesn't mind near so much as OpenBSD, but if there is a choice between two codebases, one BSD and one CDDL, FreeBSD will use the BSD, because in spite of it's desire for an increased feature list, it still wants to be a BSD. Apple is no BSD, they use a licence that is more restrictive for their Darwin sources. Integrating the GPL, MPL, APSL, or CDDL is counter to everything that is BSD, because the BSDs are about being transparently easy to use for whatever someone may want, they need only give due credit to the original creators of the code. All the nonsense about viral effects, patents and such is entirely more hassel than the BSDs want to deal with, let alone make their users deal with.

Posted by John P. Dykstra on June 19, 2007 at 11:08 AM PDT #

John, Pedro: My view is that *BSD can freely use the header files needed for DTrace since, being CDDL-licensed, they do not have a "viral" effect that extends outside the actual header files. If they were being merged pre-compile-time into a larger file there might be an issue, but they are not and I do not believe any relicensing is needed to allow you to proceed.

Posted by Simon Phipps on June 19, 2007 at 11:32 AM PDT #

Simon:

While header file usage is all well and good for people who don't care about the entire codebase, OpenBSD specifically does not allow for such additions to it's codebase.

While your view may make sense to you, it does not sync with the attitudes of the BSD folk. For them it is either MIT/BSD/ISC liberal or not good enough for them. Even the four clause BSD licence is not particularly desirable anymore.

It's a line in the dirt that they refuse to cross any further, for them they've got more non-liberal code than they'd like already. I am sure there are several OpenBSD developers that would love to use DTrace, and indeed there are many FreeBSD developers who are actively working on making DTrace function with FreeBSD. The same can be said for ZFS, but even FreeBSD would not allow the filesystem in base, it will have to be a module.

It's unfortunate, but relicensing is the only means by which the code would ever be integrated into the system.

Posted by John P. Dykstra on June 19, 2007 at 12:03 PM PDT #

What about the Elite3D firmware (afb.ucode) which is needed for afbinit on SPARC/Linux? It can't be distributed easily by Linux distros because of legal stuff. It is not free software which, so bugs can't be fixed easily.

Posted by Daniel van Eeden on June 19, 2007 at 12:46 PM PDT #

I guess OpenBSD should start writing OpenCC to replace gcc then, huh? </troll>

Posted by James on June 19, 2007 at 06:03 PM PDT #

Very promising first step - while you are looking at north bridges - can you also consider your system management hardware - and open up what Sun uses for IPMI/sensors-type management. Or better yet get the hw side to switch over to IPMI ;-). You have arrived when ubuntu can turn on the funky LEDs on my SunFire V210 and read temperatures/fans.

Posted by ragneyi on June 19, 2007 at 08:37 PM PDT #

James:

Wait for it, it is an eventuality. First expect OpenCVS to be finished off, some time around release 4.3. They've done sort, pf, grep, gzip, ntp and many other bits in the past. They'll probably finish off replacing the random smaller bits first, but eventually the gcc will be the only thing left to remove.

Had Bell Labs released the ken thompson c compiler set under the MIT licence like they did some other bits of plan 9 code, kencc would probably already be in base in order to start hammering out the missing bits from it - but it's not, so the compiler is going to have to come from something else.

Posted by John P. Dykstra on June 19, 2007 at 09:15 PM PDT #

FWIW, in relation to OpenBSD, Theo de Raadt and finding a BSD-licensed gcc replacement, I did suggest a few years ago, in an email to Theo himself, that the OpenBSD team start with TenDRA C compiler, since it was available under the BSD license. He _is_ an irascible character. :-) I have the greatest respect for his ability to keep his team together. Most people that irascible are perpetual loners. :-) So, a Bugzilla for required info on hardware. Thumbs up! Not a bad idea.

Posted by Wesley Parish on June 20, 2007 at 01:12 AM PDT #

a BSD-licensed c compiler has been around for ages (TenDRA) but it does not seem the BSD folks have embraced it so much. Anyway this is OT, isn't it? About the real topic: even if the first 4 cases do apply, would it be possible to actually _know_ which one does? Saying "we don't know what is happening in that chip, sorry about that" is quite different from not giving an answer at all. But thanks for taking a talkative approach to this issue.

Posted by riffraff on June 20, 2007 at 01:21 AM PDT #

I'm not sure the world cares that much about OpenBSD. Theo got kicked out of NetBSD for a reason (several, actually). Getting the FreeBSD community to adopt the header files for dtrace would be good though.

Posted by Jim Thompson on June 20, 2007 at 03:42 AM PDT #

kencc is part of both Plan 9 and Inferno. While the version in Inferno lacks ELF support and merely outputs ECOFF, they're both quite similar, and adding ELF support wouldn't be that trivial (now shared libraries, .8 files, the cursed assembler syntax, etc. are different beasts). (TenDRA/Ten15 is schizo and probably discontinued. Compiling in Plan 9 is I/O bound, speed-wise…) And all of Inferno is actually MIT licenced. When I told Theo about it, he basically told me to fuck off (quite in contrary to their usual behaviour of just ignoring us). There's more than the licence of a reason to abandon gcc though - the crappy code and their politics (dubbed Pinski-isms by us). Actually, while Sun's C compiler is probably going to be GPL'd as well, we're following that closely, as we could already build native (MirBSD) code with the (Linux) compiler. PS: I want the soundcard in my SPARCstation 20 to be supported ☺

Posted by 213.23.202.22 on June 20, 2007 at 05:29 AM PDT #

I think it is a great thing that folks at Sun are responding and explaining the challenges. However I hope that Sun can understand that it does make SPARC a "red headed stepchild" when it comes to USIII and USIV (assuming that USIV also has the same issues that Theo raises on USIII). It literally makes your hardware have less value. Personally, I think you ought to ship a T1000 to Theo with a three foot stack of docs and help him get OpenBSD up as either a bare-metal port (unlikely to scale well though) or get it running in an LDOM.

Posted by Patrick Giagnocavo on June 20, 2007 at 06:15 AM PDT #

> I'm not sure the world cares that much about > OpenBSD. Theo got kicked out of NetBSD for a > reason (several, actually). Getting the FreeBSD > community to adopt the header files for dtrace > would be good though. > > Posted by Jim Thompson on June 20, 2007 at 03:42 AM PDT # Jim, I' pretty sure that the world doesn't care about you or your stupid ideas. OpenBSD DOES make sense. Everything makes sense. Period. OpenBSD guys are asking for releasing FREE documentation. Not only for them. For everyone. Everyone will benefit of it. Sun could ship MORE USIII systems (they ARE in stock RIGHT NOW!) to not only Solaris users. Sun hardware is great but applications differ. One wants to run OpenBSD on it -- don't mind about the other's reasons. Simon, NDA doesn't make sense. Increase sales? Release docs! :)

Posted by 192.18.43.225 on June 20, 2007 at 07:45 AM PDT #

Thorsten Glaser:

Considering you are constantly spamming everywhere about your unstable system everywhere OpenBSD is mentioned, pretty much anything you say is responded to with hostility or ignored, you should be used to it. Quit spamming about your system and it will be less of a problem for you, and the people who must deal with the spam.

riffraff and Wesley Parish:

As Thorsten mentioned, Tendra is pretty much dead in the water - and it was never really much better than that. It's support for hardware outside of the i386 world was never all that good. The policy of, "wait until it supports other systems better," was in place, and Trendra never has. Jim Thompson:

Considering how big a role OpenBSD plays in the non-BSD world, I doubt, "noone cares that much about OpenBSD." In fact, that the Mozilla Foundation gave them ten thousand dollars illustrates that it's mostly trolls and those who've no idea what OpenBSD does that talk smack about it.

Posted by John P. Dystra on June 20, 2007 at 08:19 AM PDT #

Simon: Back to the dtrace headers issue ... I agree that it could be done, but there is a sound policy behind it: it's a matter of freedom (like in copycenter not copyleft). ZFS is part of FreeBSD because no one is forced to use it: You can take all the CDDL'd code and remove it from the FreeBSD and you still have an OS. You can take all the GPL'd code and remove it from FreeBSD and assuming you have a decent compiler (SUN Studio for FreeBSD anyone?) you still have a working OS, but if we include those headers you will get to a point where the BSD code wouldn't add up to a working OS.
Shawn: don't be naive, even if a poll on opensolaris-discuss agrees that OpenSolaris should be relicensed under the BSDL and merged into OpenBSD, that won't happen. There goes your community.

Posted by Pedro on June 20, 2007 at 02:12 PM PDT #

There is nothing in OpenBSD that you can't get elsewhere. If you want to run on non-x86 hw, you can run NetBSD (which is where OpenBSD forked from anyway.) sshd? OK, nice work. the OpenBSD ntp daemon? Its a nice effort, but not a replacement for a working ntp daemon except on the simplest of leaf-nodes. Mozilla gave OpenBSD $10k because they would have evaporated without it. Especially given that DARPA pulled funding for OpenBSD after Theo opened his yap. Ever wonder why, say, FreeBSD doesn't have the same financial issues? Sounds like most people don't care to me. And I love, love love folks who respond anonymously. Show 'em you care folks, hide behind your IP address!

Posted by Jim Thompson on June 20, 2007 at 07:24 PM PDT #

Simon, couldn't Sun just give the FreeBSD project a nice lawyer-letter clearing the issue?

Posted by Jim Thompson on June 20, 2007 at 07:26 PM PDT #

Jim:

As if responding with a name means anything at all, names can easily be pulled out of the air, I could have called myself Simone Taransky and you'd not know that I am actually John, you'd be none the wiser because there is no way to tell, so don't go whining about people using their IPs, it gives a lot more information than a name does.

There are actually quite a few things you don't get in other systems that come in OpenBSD, perhaps you should try to read up on the subject rather than spouting nonsense, I won't bother to educate you, since you've proven yourself to have prejudged the situation and obviously have no interest in learning about the topic you so freely discuss. Besides, this has gone quite off topic.

Posted by John P. Dykstra on June 20, 2007 at 09:10 PM PDT #

I have never used OpenBSD, but it was the first OS to support the NX bit and it's been admitedly the one OS that has been obsessed with cleaning the codebase and doing the right thing always. Their approach and Theo's particular character has many drawbacks but they deserve due respect. You can get most of those features on other OSs b/c they of them showing the way to do it and thanks to their code being under a BSD license. Hmm.. there was some talk about making OpenSolaris more like linux.. how about making it more BSD first.. there are plenty of features the BSDs have that Solaris needs: UFS2, geom, netgraph, NetBSD's ext2fs, and you can change the license to CDDL if you like.
Hmm.. about the dtrace thing... I think this could be talked to the FreeBSDFoundation.org , but I honestly see no interest in SUN to ease things up.

Posted by Pedro on June 21, 2007 at 11:30 AM PDT #

there are *lots* of things you get in openbsd that you don't get anywhere else, but the most important one is unwavering commitment to freedom. mr. de raadt may be a flawed human being like all the rest of us, but his integrity and laser focus on freedom cannot be faulted.

Posted by pete gilman on June 22, 2007 at 08:49 PM PDT #

Post a Comment:

Comments are closed for this entry.