Normally, I'm hesitant to engage people on the internet in an argument. Typically, you get about one or two useful responses before it descends into a complete flame war. So I thought I'd take my one remaining useful response and comment on this blog which is a rebuttal to my previous post, and accuses me of being a "single Sun misinformed developer".
To state that the Linux kernel developers don't believe in those good, sound engineering values, is pretty disingenuous ... These [sic] is pretty much a baseless argument just wanting to happen, and as he doesn't point out anything specific, I'll ignore it for now.
Sigh. It's one thing to believe in sound engineering values, and quite another to develop them as an integral part of your OS . I'm not saying the Linux doesn't care about these things at all, just that they're just not a high priority. The original goal of my post was not "our technology is better than yours," only that we have different priorities. But if you want a technology comparison, here are some Solaris examples:
- Reliability - Reliability is more than just "we're more stable than Windows." We need to be reliable in the face of hardware failure and service failure. If I get an uncorrectable error on a user process page, predictive self healing can re-start the service without rebooting the machine and without risking memory corruption. Fault Management Architecture can offline CPUs in reponse to hardware errors and retire pages based on the frequency of correctable errors. ZFS provides complete end-to-end checksums, capable of detecting phantom writes and firmware bugs, and automatically repair bad data without affecting the application. The service management facility can ensure that transient application failures do not result in a loss of availability.
- Serviceability - When things go wrong (and trust me, they will go wrong), we need to be able to solve the problem in as little time as possible with the lowest cost to the customer and Sun. If the kernel crashes, we get a concise file that customers can send to support without having to reproduce the problem on an instrumented kernel or instruct support how to recreate my production environment. With the fault management architecture, an administrator can walk up to any Solaris machine, type a single command, and see a history of all faulty components in the system, when and how they were repaired, and the severity of the problems. All hardware failures are linked to an online knowledge base with recommended repair procedures and best practices. With ZFS, disks exhibiting questionable data integrity can automatically be removed from storage pools without interruption of normal service to prevent outright failure. Dynamic reconfiguration allows entire CPU boards can be removed from the system without rebooting.
- Observability - DTrace allows real-world administrators (not kernel developers) to see exactly what is happening on their system, tracing arbitrary data from user applications and the kernel, aggregating it and coordinating with disjoint events. With kmdb, developers can examine the static state of the kernel, step through kernel functions, and modify kernel memory. Commands like trapstat provide hardware trap statistics, and CPU event counters can be used to gather hardware-assisted profiling data via libcpc.
- Resource management - With Solaris resource management, users can control memory and CPU shares, IPC tunables, and a variety of other constraints on a per-process basis. Processes can be grouped into tasks to allow easy management of a class of applications. Zones allow a system to be partitioned and administrated from a central location, dividing the same physical resources amongst OS-like instances. With process rights management, users can be given individual privileges to manage privileged resources without having to have full root access.
That's just a few features of Solaris off the top of my head. There are Linux projects out there approaching some semblance of these features, but I'd contend that none of them is as polished and comprehensive as those found in Solaris, and most probably live as patches that have not made their way into few mainstream distributions (RedHat), despite years of development. This is simply because these ideas are not given the highest priority, which is a judgement call by the community and perfectly reasonable.
Crash dumps. The main reason this option has been rejected is the lack of a real, working implementation. But this is being fixed. Look at the kexec based crashdump patch that is now in the latest -mm kernel tree. That is the way to go with regards to crash dumps, and is showing real promise. Eventually that feature will make it into the main kernel tree.
I blamed crash dumps on Linus. You blame their failure on poor implementation. Whatever your explanation, the fact remains that this project was started back in 1997-99. Forget kernel development - this is our absolute #1 priority when it comes to serviceability. It has taken the Linux community seven years to get something that is "showing real promise" and still not in the main kernel tree. Not to mention the post-mortem tools are extremely basic (30 different commands, compared with the 700+ available in mdb).
Kernel debuggers. Ah, a fun one. I'll leave this one alone only to state that I have never needed to use one, in all the years of my kernel development. But I know other developers who swear by them. So, to each their own. For hardware bringup, they are essential. But for the rest of the kernel community, they would be extra baggage.
Yes, kernel debuggers are needed in a relatively small number of situations. But in these situations, they're absolutely essential. Also, just because you haven't used one yet doesn't mean it isn't necessary. All bugs can be solved simply by looking at the source code long enough. But that doesn't mean it's practical. The claim that it's "extra baggage" is bizarre. Are you worried about additional source code? Binary bloat? How can a kernel be extra baggage if I don't use it?
Tracing frameworks. Hm, then what do you call the kprobes code that is in the mainline kernel tree right now? :) This suffered the same issue that the crash dump code suffered, it wasn't in a good enough state to merge, so it took a long time to get there. But it's there now, so no one can argue it again.
Yes, the kprobes code that was just accepted into the mainline branch only a week and a half ago (that must be why I'm so misinformed). KProbes seems like a good first step, but it needs to be tied into a framework that administrators can actually use. LTT is a good beginning, but it's been under development for five years and still isn't integrated into the main tree. It's quite obvious that the linux kernel maintainers don't perceive tracing as anything more than a semi-useful debugging tool. There's more to a tracing framework than just KProbes - any of our avid DTrace customers (administrators) are living proof of this falsehood.
We (kernel developers) do not have to accept any feature that we deem is not properly implemented, just because some customer or manager tells us we have to have it. In order to get your stuff into the kernel, you must first tell the community why it is necessary, and so many people often forget this. Tell us why we really need to add this new feature to the kernel, and ensure us that you will stick around to maintain it over time.
First of all, every feature in Solaris 10 was conceived by Solaris kernel developers based on a decade of interactions with real customers solving real problems. We're not just a bunch of monkeys out to do management's bidding. Second of all, you don't implement a feature "just because some customer" wants it? What better reason could you possibly have? Perhaps you're thinking that because some customer really wants something, we just integrate whatever junk we can come up with in a months time. If this were true, don't you think you'd see Kprobes in Solaris instead of DTrace?
First off, this [binary compatibility] is an argument that no user cares about.
We have customers paying tens of millions of dollars precisely because we claim backwards compatibility. This is an example of where Linux is just competing in a different space than Solaris, hence the different priorities. If your customer is a 25 year old linux advocate managing 10 servers for the University, then you're probably right. But if your customer is a 200,000 employee company with tens of thousands of servers and hundreds of millions of dollars riding on their applications, then you're dead wrong..
The arguments he makes for binary incompatibility are all ones I've heard before. Yes, not having binary compatibility makes it easier to change a kernel interface. But it's not exactly rocket science to maintain an evolving interface with backwards compatibility. Yes, you can rewrite interfaces. But you can do so in a well defined and well documented manner. How hard is it to have a stable DDI for all of 2.4, without worrying that 2.4.21 is incompatible with 2.4.22-ac? As far as I'm concerned, changing compiler options that break binary compatiblity is a bug. Fix your interfaces so they don't depend on structures that change definition at the drop of a hat. Binary compatibility can be a pain, but it's very important to a lot of our customers.
And when Sun realizes the error of their ways, we'll still be here making Linux into one of the most stable, feature rich, and widely used operating systems in the world.
For some reason, all Linux advocates have an "us or them" philosophy. In the end, we have exactly what I said at the beginning of my first post. Solaris and Linux have different goals and different philosophies. Solaris is better at many things. Linux is better at many things. For our customers, for our business, Linux simply isn't moving in the right direction for us. There's only so many "we're getting better" comments about Linux that I can take: the proof is in the pudding. The Linux community at large just isn't motivated to accomplish the same goals we have in Solaris, which is perfectly fine. I like Linux; it has many admirable qualities (great hardware support, for example). But it just doesn't align with what we're trying to accomplish in Solaris.
No offense, but I think you, your colleagues, and your readers would be better served by more technical discussions. I've been a sysadmin for a good number of years, and I continually find great nuggets in the tech posts your teams have written. I'll admit, be it age, lack of focus, or life in general, a lot of the more technical aspects ellude me, but there's usually an item or three I can take away with me. And please give the other Sun bloggers a cattle-prod jolt to post more (though I know you're all busy, esp. with 10 coming out soon). The technical items in the blogs are the best thing Sun's communications policies have done in ages. This will be even more the case as the fallout of the Blueprints authors' cutbacks manifest. <ps> Keep on blogging!
Posted by Rainer Heilke on September 24, 2004 at 02:07 PM PDT #
Rainer -
Thanks for the comment. I agree that most arguments with zealots are futile - after one or two rebuttals you end up stuck down a rabbit hole. But OpenSolaris is part of our Solaris strategy, so I've got to throw a punch every now and then :-) This will be last you hear of this argument for a while, at least from me.
I promise to get back to some more technical posts real-soon-now. Stephen has been going crazy with SMF posts; I'm wondering how he finds the time. As you mention, we're all pretty busy trying to get the thing out the door. But I'll try to give everyone a good whack with the blogging stick...
Posted by Eric Schrock on September 24, 2004 at 02:20 PM PDT #
I do have a strong server OS preference: Solaris. What Eric says here is very much what I seen from the actual products. I know a lot of people who feel the same way. It's very much appreciated, and nice to have the option.
Sometimes I think of "Solaris vs Linux" as "gurus vs geeks" or "gurus vs hackers". Certainly a different mindset between the two groups.
PS With regards to what Rainer says, though I agree, since I get into OS/kernel arguments from time to time, it's nice to see some discussion by experts on the subject.
Posted by Chris Rijk on September 24, 2004 at 03:51 PM PDT #
Posted by Sahil on September 24, 2004 at 06:38 PM PDT #
As for Linux vs Solaris, I think it doesn't make sense. It should be Solaris 10 vs RedHat Advance Server 3 U3 vs Win03 vs AIX 5.3. These are products that I can buy. I cannot buy Linux, just like I cannot buy UNIX.
Posted by Iwan Ang on September 25, 2004 at 09:25 AM PDT #
Posted by 42 on September 25, 2004 at 10:44 AM PDT #
Posted by David Akhond on September 25, 2004 at 06:36 PM PDT #
Posted by Greg k-h on September 26, 2004 at 09:46 AM PDT #
Mostly in response to Rainer, but also to you,
Don't call people zealots all the time. Just because one has a strong opinion does not make one a zealot. I find this "And what's so "pseudo" about some of them?" very insulting. GPL "non-believers" are just as fanatic about their point of view as GPL "zealots".
Besides, it has nothing to do with the technical merits of Solaris on the one hand and Linux on the other. Personally I'm quite happy with Linux and, apparently, so are the customers of the company I work for, and I suppose the same is true for Sun and Solaris. But there is room for improvement in both.
Instead of a "mine is longer than yours" discussion, it would be much more interesting to see what both kernels can learn from each other, once OpenSolaris is out there.
Posted by Anonymous on September 27, 2004 at 01:35 AM PDT #
Posted by Sunburnt on September 27, 2004 at 08:46 AM PDT #
Linux was not, and is not, designed for the "high-end" enterprise market. If it were, Linus would never have survived the first "brown paper bag" release. That would have been it for Linux.
On the other hand, Solaris is too heavy to run well on ix86 platforms, and was even slated for cancellation on that platform at one point. It's OK for number-crunching scientific work, but there are far more Linux systems in the Supercomputer top 500 list than Solaris systems. That's not just a matter of price, it's also a consequence of the extremely high level of Linux R&D.
The current conflict between Sun and Linux is stupid and strategically naive on both sides. There are very few markets in which they both compete on equal footing. For the most part, the differing demands of different users require that BOTH Operating Systems continue to be developed in much the way they have been.
There is only one serious threat to either. A threat from a known monopolist with a penchant for "knifing the baby" (to quote their own memos). Until a strategic alliance can be developed to confront the real menace in our midst, both operating systems (and all the companies involved in either) risk near-certain destruction.
I like Sun. I like Sun a lot. I like Sun enough that I'm taking this time to remind it that the Great White Shark over there in the corner is a more substantial threat than any other fish in the sea, no matter how it may appear at the time. Keep focussed on the real problems.
Posted by Jonathan Day on September 27, 2004 at 08:54 AM PDT #
Posted by Don Williams on September 27, 2004 at 09:20 AM PDT #
Posted by Petros Diveris on September 27, 2004 at 09:41 AM PDT #
Posted by Michael Chaney on September 27, 2004 at 10:03 AM PDT #
Posted by S.R. Prozak on September 27, 2004 at 10:44 AM PDT #
Two things are clear: Solaris/X86 can give you a serious advantage over Linux, driver support on Linux is all that is superior right now, and secondly if your going to run X86, Sun provides the best X86 systems to run on. Either way, running Solaris or Linux, Sun can handle your needs. Come one, come all.
Posted by benr on September 27, 2004 at 10:46 AM PDT #
Tell it like it is! I hope that people who disagree with you realize how huge you are. Folks, he could kick your ass! Lookout!
- Ben
Posted by bhsigelm on September 27, 2004 at 11:45 AM PDT #
Posted by Jeff on September 27, 2004 at 11:55 AM PDT #
Posted by Anonymous on September 27, 2004 at 12:30 PM PDT #
Posted by Don Williams on September 27, 2004 at 12:30 PM PDT #
Posted by kenneth topp on September 27, 2004 at 12:43 PM PDT #
Posted by Don Williams on September 27, 2004 at 01:03 PM PDT #
Every identifiable position on any issue, from technical to metaphysical, has proponents who express themselves well, and proponents who express themselves badly.
I don't think there's anything inherently "religious" in the GPL - to me it's quite practical. It's fair to say neither the FSF or anyone else wants to force or trick you into writing free software.
When you strip away the personalities of individuals, the idea itself is utterly benign. After that, there is a disproportionately negative reaction the GPL gets from a minority of people who may be threatened by it, and by what it represents.
But there are very simple, concise (you could even say "technical") reasons for free software. These are just ordinary prosaic things that are of daily importance to a software user. The more they (individually and collectively) have access to and control over their systems, the safer and better off they are. This used to be dismissable as theory, but we are starting to see the evidence stacking up.
The part of the GPL that you and your customers don't like, concerning derivative works and "restrictions" that the GPL puts on users sounds counterintuitive to me, and I put restrictions in quotes, since the goal, and the result so far, is to guarantee freedoms to the user at the "end of the chain." To follow the license, you give up your freedom to restrict them. :D
Non-free software, even when open source, has many pitfalls that, if you close your ears and say "zealot," you may not learn to appreciate properly. Of course, no system is perfect.
Respectfully but non-zealousy,
David
Posted by David Wood on September 27, 2004 at 01:18 PM PDT #
Posted by Don Williams on September 27, 2004 at 01:43 PM PDT #
Regarding binary compatibility of drivers: this is not merely about performance. FreeBSD takes generally the same approach as Solaris regarding a stable interface and binary compatibility. Is this approach useful?
Let's look at the ammount of available drivers: the amount of free drivers for linux versus the total (free + propriatary) amount of drivers for any of Solaris or FreeBSD. If the approach of Linux is so bad for driver vendors, why are there so few drivers for those other systems?
Now think abain about servicability: when a system has two separate propriatary drivers, there is no single entity in the world that has the full sources for it and thus can fully service it.
Reliebility: All the points you mention require hardware support. Once the hardware support is there, patching linux to support them is not a big issue. The infrastracture is basically there, as I mentioned.
Servicability: This is mostly userspacfe. If the linux kernel is flexible enough to save a crash dump on the disk and to send the panic message in morse code, it can surely do whatever else is necessary. If /var/log/kernel is not good enough for you, you can fix the logging in userspace.
Posted by Tzafrir Cohen on September 27, 2004 at 02:34 PM PDT #
Incomplete Quote:"First off, this [binary compatibility] is an argument that no user cares about."
Your Answer: We have customers paying tens of millions of dollars precisely because we claim backwards compatibility. This is an example of where Linux is just competing in a different space than Solaris, hence the different priorities. . If your customer is a 25 year old linux advocate managing 10 servers for the University, then you're probably right. But if your customer is a 200,000 employee company with tens of thousands of servers and hundreds of millions of dollars riding on their applications, then you're dead wrong..
The Full Quote:
"First off, this is an argument that no user cares about. Applications written for Linux 1.0 work just fine on the 2.6 kernel. Applications written for older versions of Linux probably work just fine too, I don't know if anyone tried it in a long time. The Linux kernel developers care a lot about userspace binary compatibility. If that breaks, let us know and we will fix it. So to say that old apps written for old versions of Solaris will work on the latest Solaris, is not a valid argument at all.
Posted by George Kamp on September 27, 2004 at 02:47 PM PDT #
<em>"30 different commands, compared with the 700+ available in mdb"</em>
Is that an argument for or against mdb? :)
<em>"Solaris is better at many things. Linux is better at many things."</em>
Never a truer word spoken. It's a joy to see a developer such as yourself engaging in a constructive debate with another tallented developer "from the other side" (so to speak) such as Greg. :)
Thanks.
Posted by Charlie on September 27, 2004 at 03:07 PM PDT #
Posted by justin on September 27, 2004 at 03:28 PM PDT #
Posted by Seongbae Park on September 27, 2004 at 03:52 PM PDT #
Price out a Sun V440 4-way with 4GB Ram. Now go to the Dell website and price out their Intel 4-way with 4GB of Ram. The Sun box is cheaper.
Things are not always as they seem :)
Posted by Kerry Schwab on September 27, 2004 at 06:47 PM PDT #
Posted by chizang on September 28, 2004 at 01:29 AM PDT #
Some good comments have been made in the last two blog entries; don't dismiss them because you prefer Linux (or some other OS). Use them to improve your platform of choice (if you can--re-read Mr. Schrock's comments on Linux kernel development). But Eric's right. With OpenSolaris being bounced around (and the continued debate over Java), a couple posts on the matter were in order, and it's better getting some solid ones from someone in the trenches rather than marketroid spin. :-)
Thanks again, Eric. Keep on truckin'. I'm looking forward to the next technical post.
Rainer
Posted by Rainer Heilke on September 28, 2004 at 08:06 AM PDT #
Posted by Ean Price on September 28, 2004 at 06:24 PM PDT #
Ean -
No worries. I have never looked at any GPLed Linux kernel code, AFAIK. All the information I have about Linux is from reading linux mailing lists (avoiding the patches themselves), changelogs, newsgroups, presentations, general discussion, and use of the software.
I don't want to be the subject of a lawsuit 5 years down the road ;-)
Posted by Eric Schrock on September 28, 2004 at 06:30 PM PDT #
Having read the discussion, I'm a tad dumbfounded by the arguments people use.
I admin sun servers. I admin linux servers. I admin FreeBSD servers. I admin NetBSD servers. I even admin OpenBSD servers.
Each system has its own advantages and disadvantages.
Solaris. Horrible on the desktop. It's often hell to compile the applications you need due to them not being written cross-platform enough. You often have to patch them yourself to get them up and working. On the other hand - it's an excellent system on servers - except that you can't sniff the lo0 interface with tcpdump (which has caused me quite a bit of grief). On the other hand, software written for solaris is very, very, nice.
Also, it is - in my experience - a tad tiresome to update solaris. While it's a breeze with for example YaST under SuSE, it can be quite annoying to patch solaris systems.
Linux. Excellent on the desktop. Wide variety of applications. Lots of excellent distributions with nice install and patch tools. Huge amounts of programs to choose from by default. Quite nice on some servers too, but it's a tad more difficult to get good support for commercial software on commercial hardware.
*BSD - excellent on "special purpose" devices or boxes. Wonderfull to have as firewalls, VPN-servers, and so forth. Quite nice as desktops too, but not as "polished" as various linux distributions there. In addition, most BSD's are very nice to update (just remake the world! :-) . The ports collections has caused me quite a lot of greif, though.
Posted by Rune Kristian Viken on September 29, 2004 at 02:52 AM PDT #