CIFS ... in Solaris
CIFS ... in Solaris. That's an interesting concept; one that has evoked
a variety of emotions within Sun.
I was first asked about adding an in-kernel CIFS service to Solaris about
4 years ago, "Yes, it's possible but it's probably a lot more intrusive
than you think. Are you sure you want to do that?"
We had been working on an independent CIFS implementation for several years,
and we would use this as the basis for the Solaris CIFS project, but it
would take time for everything necessary to fall into place: there is a
big difference between what management at Sun would like to happen and
what the engineers at Sun will endorse. It took a while to get the
necessary support and it would be another two years before the Solaris CIFS
server project became a reality.
Although I didn't know it at the time, this all started about 16 years ago
when I was working on multi-OS, distributed, transaction processing
systems (I seemed to be forever porting ONC RPC to yet another OS) and I
came across something called OSF DCE, which was a distributed computing
environment that used DCE RPC for client-server communication, X.500 name
services and Kerberos security. We built a prototype using DCE
across various different UNIX operating systems but decided to use a
different technology to create products. People were talking about using
DCE for CORBA (object request broker) servers, and we'd used a DCE based
transaction processing monitor, but I didn't think DCE would catch on and
I thought I'd never hear about it again - I had no idea.
About 10 years ago, I was asked to help design and implement a new
transaction oriented, journalling file system (not ZFS) for a small NAS
company. We started working on it but a large OEM deal came along and
the file system was put aside to productize what would eventually become
the StorageTek 5320 NAS. There were many things to take care of but one
of the main problems was the lack of a comprehensive CIFS service.
No-one on the team had much experience with Windows or CIFS so I took it on,
and it didn't take long to realize that I'd run straight back into DCE.
CIFS had evolved: MSRPC is essentially DCE RPC and Active Directory is
based on LDAP and Kerberos. Oh Joy!
And 10 years on ...
Many people assumed or desired that the Solaris CIFS server project would
be like Samba but what would that achieve? Sure, it would avoid breaking
any eggs, i.e. avoid making substantial changes to Solaris, but Samba is
available on Solaris today. There is no point in creating another Samba.
If you truly want an integrated CIFS implementation, that can really
inter-operate with Windows at a fundamental level, the operating system
has to support certain core features. Eggs will have to be broken.
Not surprisingly, the most contentious topic seemed to be the possibility
that someone might introduce case-insensitivity into a Solaris file system.
This was anathema to many, which led to many repetitive discussions, and
was an interesting distraction because it wasn't the thing that really
concerned me. Case-insensitivity is important for transparent inter-
operability with Windows, as is ensuring that file names can be shared by
applications and protocols using disparate character encoding schemes,
but I could visualize solutions for those things - solutions that would
not compromise POSIX conformance. My real concern was how to support
integrated file access control within the file system given the mismatch
between UNIX and Windows credentials. The idea of non-unique UIDs and
GIDs is so pervasive in UNIX that I wasn't sure it would be possible to
effect the level of change necessary to achieve true CIFS integration.
CIFS access control relies on access tokens (Windows credentials) and
security descriptors, in which users and groups are represented by
globally unique, variable length security identifiers (SID). NFSv4 and
ZFS offer partial compatibility through NFSv4 access control lists (ACL)
but those ACLs and Solaris credentials are still founded on UIDs and GIDs,
which are fixed size and non-unique across NIS domains.
Afshin, a senior member of the CIFS project team, and I spent a lot of
time working on white boards and came to the conclusion that we couldn't
get round the credential issue. So, in December 2006 - the same month
that I submitted the CIFS Service project proposal (PSARC case 2006/715),
Afshin wrote and distributed a white paper titled "CIFS/NFSv4 Unified
Authorization", which proposed a unified access control model for CIFS,
NFSv4, local users and ZFS, and introduced the concept of the FUID that
is now used in ZFS. This generated some good discussion but the
mountain remained; how to make such a huge change a reality.
We continued working on other things, there was no shortage of other things,
but it was difficult to see a light at the end of the tunnel until February
2007, when a happenstance discussion over coffee with Jeff Bonwick and
Mike Shapiro, both Sun Distinguished Engineers who need no introduction
here, changed the future of the CIFS project. They asked how things were
going and I explained the magnitude of the change I wanted to make. A short
discussion and white board session later we had consensus and Mike said,
"Let me do some reading this weekend and I'll write something up." That
write-up became
PSARC case 2007/064
(Unified POSIX and Windows Credentials for Solaris) and we had the way
forward for the Solaris CIFS service.
We already had the basic CIFS service building on Solaris but it took
another 8 months, 22 more ARC cases, a lot of helping hands and many
late nights to deliver the project. On October 25th, 2007, the CIFS
service project putback over 800 files, approximately 370,000 lines of
code (including 180,000 lines of new code) to the Solaris operating
system.
It's a large, complex project (several years in the making, including a very
intense final year), which incorporates some fundamental changes to Solaris.
So what is CIFS, why add support to Solaris and what did we change?
The Common Internet File System (CIFS), also known as SMB, is the standard
for Windows file sharing services, and one of the primary goals for Solaris
is to continue to improve and enhance it as a storage operating system and
platform. Adding CIFS, to provide seamless, ubiquitous file sharing for
both CIFS (Windows, MacOS etc) clients and NFS clients is a major step
towards achieving this goal. Together, with the CIFS client, which is
also an OpenSolaris project, the CIFS server helps provide comprehensive,
integrated native Windows interoperability on Solaris.
What does this mean for Samba on Solaris? Not a lot really. Samba is
a good, multi-platform application service that provides file and print
service for Windows and CIFS clients. It is a portable, user space
application, and is actively supported on Solaris. The Solaris CIFS
service is a native kernel implementation; a first-class citizen of the
Solaris operating system that has been integrated with NFS, ZFS and many
OS feature enhancements to provide seamless, ubiquitous, cross-protocol
file sharing.
There is a common misconception that Windows interoperability is just a
case of implementing file transfer using the CIFS protocol. Unfortunately,
that doesn't get you very far. Windows interoperability also requires that
a server support various Windows services, typically MSRPC services, and
it is very sensitive to the way that those services behave: Windows inter-
operability requires that a CIFS server convince a Windows client or server
that it "is Windows". This is really only possible if the operating system
supports those services at a fundamental level.
In addition to the CIFS/SMB and MSRPC protocols and services:
Posted at 06:04PM Nov 02, 2007 by Alan Wright in Sun | Comments[37]
Great work!
Posted by Marc Hamilton on November 02, 2007 at 07:48 PM PDT #
Now I can understand why Solaris 10 didn't ship with kernel aware CIFS support. I had asked in Marc Hamilton's blog on Solaris and CIFS (http://blogs.sun.com/marchamilton/entry/solaris_and_cifs) about it but had know idea it was so involved!
It's been rumored for a long time that Sun wanted in on the filer market and that requires real CIFS support (ala NetApp) in addition to NFS support.
Amazing how many things are being worked on at Sun these days! Keep up the good work.
Posted by Bryan on November 02, 2007 at 07:51 PM PDT #
Thank you for adding CIFS - you just made my life a lot easier, and made it a lot easier to push Sun even higher on the approved vendor lists. Go SUN, Go!
Posted by Big Sun User on November 02, 2007 at 09:56 PM PDT #
In what build will this be available to the public? Great work!
Posted by KG on November 05, 2007 at 05:45 AM PST #
Any idea when PSARC/2007/064 will be available to the public on http://www.opensolaris.org/os/community/arc/caselog/2007/ ?
Cheers,
Ceri
Posted by Ceri Davies on November 05, 2007 at 07:02 AM PST #
Oh my goodness! This is the best explanation for all the trouble I've had with Samba on Linux/Solaris. Our issues have never been about providing "file transfer" to Windows/Mac clients via SMB. Rather, they're about "file SERVICES" complete with access control, file locking, etc.
I'm really hoping we can develop an open, inexpensive, DISRUPTIVE solution using ILM from SAMFS (we have a site license someone else paid for, so I have a motivation to make it work ;) but without a lot of layers (resharing, 3rd party code, etc.) to support Windows, Mac and Unix clients.
Now I just need to get Sun to lend me a couple of Thumpers, and some Solaris and storage engineers to architect and test the whole thing :)
Great work!
Posted by Charles Soto on November 05, 2007 at 08:01 AM PST #
Well done! I hope that more mountains can be moved in the future, too!
Posted by Richard Elling on November 05, 2007 at 10:05 AM PST #
Outstanding! Congrats to you and the team on this outstanding accomplishment.
Posted by Jeff Cheeney on November 05, 2007 at 10:22 AM PST #
I'm really looking forward to using the in-kernel CIFS support, just as soon as I can get my hands on a copy of build 77.
Is there a "samba to in-kernel CIFS" migration guide available somewhere?
Posted by James C. McPherson on November 05, 2007 at 12:31 PM PST #
CIFS is great, but will you be integrating it with snapshots in the same way Samba does?
At the moment for windows clients, Samba & ZFS are a better file server solution than any windows server because of this. You can use ZFS snapshots to back up the data, and Microsoft's Shadow Copy Client to give end users the ability to right-click any file and see previous versions to restore.
It's incredibly powerful, even NetApp haven't been able to confirm their devices can do this.
Posted by ross on November 06, 2007 at 06:01 AM PST #
PS. As a network admin for a 100% windows shop, I can't tell you how excited I am to see the work Sun are putting into this. We're looking for a SAN right now and we're seriously considering the x4500 Thumper running Solaris instead of a traditional SAN. The cost is far cheaper and it looks like it's soon going to have better functionality than any SAN. You've already got ZFS, CIFS and iSCSI targets. If you shoehorn VSS support in there somehow you've then got a better file server for us than anybody else (including Microsoft).
PPS. Clustering would be nice too, but I'll try not to be greedy :)
Posted by ross on November 06, 2007 at 08:37 AM PST #
Great blog Alan, I had no idea how involved this is and the visibility into CIFS integration being not just a file transport mechanism, but a much deeper integration challenge crystalized a lot of details for me.
BTW, everyone else... stay tuned on snapshot, VSS, iSCSI targets etc... there are more surprises in the making....c
Posted by Chris on November 07, 2007 at 11:24 AM PST #
Posted by Open Source on November 20, 2007 at 12:44 PM PST #
Hi Alan,
Of course, as you well know (because we've spoken to Sun and asked for this many, many times in the past) that all of the advantages of being "a first-class citizen of the Solaris operating system" could also trivially be available to Samba, and I hope that you'll expose the user-space API's in order to allow us to take advantage of them.
Linux has already done this - exposing the file lease API for oplocks, inotify for change notify etc. Solaris could expand on this by giving us access to atomic NT-ACL create, NTFS stream support, the ability to push SID credentials into the system from winbindd and attach to a process etc. We already support case-insensitive filesystems of course.
In fact you could have helped us put these things trivially into Samba, and thus shared the implementation burden, rather than re-inventing everything and pushing a rather large and dangerous protocol into your kernel - something I think you might come to regret later on.
I'm looking forward to seeing the docs for all the user-space API's that will allow Samba to take advantage of these cool new features in Solaris - in fact if you want to work with us on adding this code into Samba I'd be happy to help.
Cheers,
Jeremy Allison,
Samba Team.
Posted by Jeremy Allison on November 20, 2007 at 03:11 PM PST #
ross wrote :
> Clustering would be nice too, but I'll try not to be greedy :)
Check out ctdb.samba.org - this is the Samba clustering solution, built on top of the existing Samba code. It's currently being buit into product by IBM and others.
Again, if Sun would like to cooperate and share in the creation of this rather than trying to build everything themselves you'll get your clustered Solaris CIFS sooner rather than later :-).
Jeremy Allison,
Samba Team.
Posted by Jeremy Allison on November 20, 2007 at 04:50 PM PST #
And if Samba weren't under the GPL people wouldn't have to.
Posted by Nel on November 21, 2007 at 08:18 AM PST #
> And if Samba weren't under the GPL people wouldn't have to.
What's the problem?
Posted by Segedunum on November 21, 2007 at 09:52 AM PST #
Nel wrote :
> And if Samba weren't under the GPL people wouldn't have to.
Ah, the old "but I want to *own* it and do what *I* want with it" complaint :-). Sorry, we in the Samba Team don't own it either. It's communal property.
I've just been referred to this link :
http://opensolaris.org/os/community/arc/caselog/2007/064/final-materials/spec-txt/
I hadn't seen this comparison before :
> Samba has other drawbacks which have prevented NFS + Samba from
> seriously competing as an enterprise multi-protocol file
> service solution, most notably performance. These issues are
> outside the scope of this document, but are related to Sun's
> strategic decision to port Procom's CIFS stack to Solaris,
> and make this technology more broadly available to the
> OpenSolaris community.
which is rather interesting, as several times in the past I've worked with Sun engineers on rolling out very large, scalable Samba implementations on Solaris :-). Performance was one of the critieria they were using to chose Samba :-).
If I were to put one mark on Sun's report card it would be "does not play well with others" :-)
Jeremy.
Posted by Jeremy Allison on November 21, 2007 at 09:55 AM PST #
The other interesting thing about that link is the complete absence of information on winbindd. The identity mapping discussion in it is from the state of Samba id mapping in 1997 it seems.
That might be why Sun is reinventing that particular wheel. Again, if Sun is willing to work with us on improving winbindd (which there are at least 2 other commerical companies doing right now) we'll get portable common identity mapping solutions sooner.
Jeremy.
Posted by Jeremy Allison on November 21, 2007 at 10:06 AM PST #
Of course Jeremy, and spouting nonsense on someone's blog telling them what to do with their resources is playing nice? You're telling someone not to compete with you, why? Is it because you don't wnat to see the system you work on overshadowed by a superiour competing project?
You may as well go to the lsh developers and tell them to stop it, and just work on OpenSSH. And the FreeBSD, OpenBSD and NetBSD developers, you better tell them to all start working on Linux instead.
Posted by Nel on November 21, 2007 at 10:42 AM PST #
Nel wrote :
> You're telling someone not to compete with you, why? Is it because you don't wnat to see
> the system you work on overshadowed by a superiour competing project?
Touched a nerve, huh :-). I'm quite happy to see competing projects, although to be honest client implementations are more important than server ones to help break the Microsoft monopoly. I just hate to see the waste and duplication of effort. You've been down this path before with AS/U (aka. PC-Netlink). Did you learn nothing ? Doing this on your own is a pointless waste of time. If we work together we will achieve far more than working on multiple implementations under deliberately incompatible licenses.
And how about those userlevel API's we'll need on Solaris ? Are you going to expose them to us ? Or are you afraid of a little competition ? :-).
Sun needs to get over its "fear and loathing" of the GPL, especially for Solaris :-). Personally I think that's a legacy from Bill "the robots are coming, the robots are coming !" Joy, IMHO :-). I've said this many times to Simon Phipps. GPLing Java was a good start - now go the whole way. If you do that you'll be able to cut-and-paste entire chunks of Samba into your kernel and good luck to you !
> And the FreeBSD, OpenBSD and NetBSD
> developers, you better tell them to all
> start working on Linux instead.
That would be a really good idea actually :-).
Jeremy.
Posted by Jeremy Allison on November 21, 2007 at 11:03 AM PST #
I just wish Sun can be honest and tell yte world why they did this. As an expert in the matter I see that Sun had to break the same eggs they'd have to break to make Samba do what they needed. Juts instead of joining efforts they split them and burnt a pile of cash to reinvent the wheel.
Well I guess Sun has all the reasons to do so, now would Sun be willing to share the same interfaces so that samba can take advantage of the broken eggs now that the omlette is done?
Posted by Simo Sorce on November 21, 2007 at 11:32 AM PST #
They say there's a fine line between the policing mind and the criminal mind, and based on Jeremy's postings here I would say that there is a fine line between monopolists and those that would defeat them. So as it apparently needs to be restated: monopolies -- of any stripe -- tend to be unhealthy. Samba now has some competition in the open source CIFS server space, and this is a Very Good Thing -- for OpenSolaris, for Samba, and especially for those who operate CIFS servers.
Posted by Bryan Cantrill on November 21, 2007 at 01:15 PM PST #
Bryan Cantrill wrote :
> monopolies -- of any stripe -- tend to be unhealthy
I agree heartily. If the CIFS server market were in the position of web servers or other healthy markets then no one would be complaining about Sun expending resources on their own version of a server. The problem is we're not in a healthy market. We're competing with a super-dominant monopolist who defines the protocol in question.
In that position we should be pooling our efforts (which the GPL does very well) until the market is in a healthier state.
My philosphical view is that the GPL works best in markets dominated by one large (non OSS) player, as it allows small competitors to flourish, and Apache or BSD licensed code work best in open competitive markets.
It's more a moan about the wasted opportunity here than anything else. We have so few resources in this space that reinventing any of our wheels is counter-productive.
Jeremy.
Posted by Jeremy Allison on November 21, 2007 at 01:52 PM PST #
Jeremy, I am not a Sun worker, I just hate pompous fools wandering around telling other people what they should do, it's better to mind one's own.
And the fact that you think using Linux instead of continuing the development of OpenBSD is a good idea proves you're definately off your rocker.
Posted by Nel on November 21, 2007 at 02:00 PM PST #
So let's talk details here. Adding a CIFS server into your kernel is *insane*. This protocol is much more complex and dangerous than NFS - even NFSv4, and sticking the packet parsing for it into your kernel is a receipe for long-term insecurity.
Look at the Windows kernel for an excellent example of this.
IMHO what would really help is to add these useful features (case insensitivity, NT-style ACLs & stream support, file leasing, change notifty) into your kernel and make them available to userspace processes in a sane way.
That way you end up setting the standard for these features - Linux will probably add the same API's and you've moved the entire OSS/Unix market forward.
You don't need an in-kernel server to get speed. The only places you really need speed are read/write operations for bulk data - so make them zero-copy. Sendfile handles the read side or things - and I'd appreciate a "recvfile" call for the write side of things. Samba already has userspace support for any kernel that implements these calls and can flip bits to/from the filesystem to socket buffers without passing though userspace.
Add in async mechanisms for doing open/readdir etc. and you have everything you need to do this safely in userspace. Is the opensolaris community so replete with kernel programmers that you can burn them in stabilizing a CIFS server in kernel space ? Even the Linux people don't do that, and they have many more people than you do.
If you're going to do kernel work, concentrate on the CIFS client space, make a really good client implementation, and create the userspace API's to allow anyone to build a really good server on top.
Just my 2 cents. But you'd give up control of the server there. But that's not such a bad thing to do :-). We gave up control of Samba the day it went under the GPL and there are many many versions that are weird, wonderful (and some awful :-) out there that people experiment with.
You don't need to do this on your own unless you need to control it all. That won't help you long term.
Jeremy.
Posted by Jeremy Allison on November 21, 2007 at 02:14 PM PST #
Posted by HPCC 64 on November 21, 2007 at 03:48 PM PST #
Hi Jeremy,
I agree that it's a waste of resources to reinvent the wheel but I would like to ask you "What's the big deal?"
Of course its better for society (the common good-thing) to collaborate and use licenses but when that's not possible I belive we have to run our own race.
Samba is a great product. I think that Samba 4 is going to be a real AD-killer when it's ready. I wrote a review of it when you released the first TP-release and I liked what I saw.
I would say - just keep up the good work and "if you build it - they'll come"
Posted by Niklas Andersson, TechWorld Open Source on November 22, 2007 at 06:17 AM PST #
"If you truly want an integrated CIFS implementation, that can really inter-operate with Windows at a fundamental level, the operating system has to support certain core features. Eggs will have to be broken."
Does this have anything to do with Patent claim avoidance? or will these 'core feature' shared with others besides the OpenSolaris Community? I think what has been thought out here are the very important aspects of interoperability between the unix and the windows styles of file serving. I just hope it is going to be beneficial to more than an 'added few' to 'the few' of prior.
Posted by Shawn on November 22, 2007 at 10:43 AM PST #
Shawn,
The eggs comments were about the changes to the Solaris OS and had nothing to do with patents or anything else.
I'm not sure if I'm interpreting the other part of your comment correctly but all of the Solaris OS changes are described in the fast-tracks listed in the slides on http://www.opensolaris.org/os/project/cifs-server/, all of which should be available online.
Alan
Posted by 192.18.43.225 on November 27, 2007 at 04:09 PM PST #
`` Adding a CIFS server into your kernel is *insane*. This protocol is much more complex and dangerous than NFS - even NFSv4, and sticking the packet parsing for it into your kernel is a receipe for long-term insecurity.``
Curious, where does NetApp implement their CIFS protocol? And why?
It's no secret Sun wants in on the CIFS/NFS server business. To do that they need to compete performance wise.
Posted by Bryan on November 28, 2007 at 07:03 AM PST #
> Curious, where does NetApp implement their
> CIFS protocol? And why?
In their kernel of course, as I don't believe they have a generic userland POSIX system (if anyone knows better please correct me). So where else could they stick it :-).
And yes, that's a receipe for insecurity and crashes IMHO.
At to "compete performance wise" - tell me, do processors magically go faster when they're running in kernel space ? All you need to do to improve performance is avoid memory references and data copying, this can be done just as well in userspace as in kernel space (eg. sendfile/recvfile).
Jeremy.
Posted by Jeremy Allison on November 28, 2007 at 12:07 PM PST #
Sigh. I take no pleasure in this.....
My report card stands. "Does not play well with others"
http://directorymanager.wordpress.com/2007/11/28/an-open-letter-to-the-opends-community-and-to-sun-microsystems/
Jeremy.
Posted by Jeremy Allison on November 28, 2007 at 12:12 PM PST #
Jeremy, you seem have a penchant for the dramatic but the reality is that implementing an in-kernel CIFS protocol parser is just a design choice. Whether or not it makes sense is based on the project requirements and objectives, and complexity doesn't necessarily make it dangerous, which I assume is a euphemism for risk. Risk is mitigated by experience and having good quality assurance, design, implementation and review processes; it's not inherent from the protocol specification that the project or implementation is high risk.
If the objective had been to provide a portable, non-OS specific implementation, perhaps we would have made a different choice but that's not what we were trying to do, and I made that point in my blog. One of our main goals was to leverage and deliver the features of the Solaris OS, and ZFS in particular, to CIFS clients in an integrated fashion.
We went to great lengths, not least of which were all the ARC reviews, to try to ensure that the interfaces being added or changed would be generic and extensible - I encourage you to look at the fast-tracks and at the CIFS architecture. I cannot go to the ARC and request interface changes for a hypothetical Samba project that may or may not ever happen but, if you really want to do this, you can propose an OpenSolaris Samba project that includes building on or extending interfaces.
Posted by Alan on November 28, 2007 at 02:19 PM PST #
I was confused as to the relevance of the open letter, which states,
"Please note that I don’t feel that this action was
representative of Sun’s true open source strategy,
but was a relatively isolated incident ..."
I don't know if this letter provides a balanced view of this particular incident or not but, regardless of that, it seems unfair to extrapolate from this incident, involving a few individuals on a completely different project, particularly given the quote above, to the CIFS project.
Alan
Posted by Alan on November 28, 2007 at 03:27 PM PST #
Also cutting down the risk was buying Procom Technology over two years ago. I think it's great that with that acquisition Sun set the stage to allow the code to be open sourced in OpenSolaris!
June 13, 2005 - Sun Microsystems, Inc. (NASDAQ: SUNW) today announced it has completed the acquisition of all intellectual property rights of Procom Technology, Inc. relating to its Network Attached Storage (NAS) offerings. In addition, the company has welcomed the majority of Procom's employees related to NAS into the Sun organization.
Through a previous software licensing agreement with Sun, Procom's technology is already embedded and currently shipping within the Sun StorEdge 5000 family of NAS Appliances. With the close of this acquisition, Sun now owns the intellectual property rights relating to Procom's NAS product offerings and therefore gains additional engineering expertise, enabling it to build future NAS and next-generation file-based storage systems much faster and more cost-effectively.
http://www.sun.com/smi/Press/sunflash/2005-06/sunflash.20050613.2.xml
Posted by Bryan Althaus on November 28, 2007 at 07:37 PM PST #
Posted by HPCC 64 on December 24, 2007 at 03:13 AM PST #