Split PCI bus on T-Class servers
Split PCI bus capability has become pretty common among the T-Class systems. The ability to split the PCI bus provides multiple I/O Domain support. With extra I/O Domains you can do all sorts of cool stuff like NIC fail-over and even direct card control.
If you've ever had trouble understanding (or in my case even just remembering) how the split bus configurations work, here's the site to check out: LDoms Community Cookbook, Section 1 - Hardware & Split PCI.
This page (and the whole site) is fantastic. Check out this image -- it makes understanding the bus allocations trivial:
I've definitely got to spend some more time on wikis.sun.com
This feature is really rather useful if you want to cut down on the number of single points of failure when doing ldoms. Also it makes maintenance easier, because it's actually possible to boot (and upgrade) the control domain while the ldoms keep going along. There's just one tiny problem - since the controller is stuck on one of the pci buses, you have to have external storage for your ldoms. Not too bad in a T2000, but take a T5240 with room for 8 internal disks and it suddenly does become rather annoying.
Posted by Mads on September 26, 2009 at 03:15 AM EDT #
@Mads: It's funny that you mention that. Just yesterday I was longing for an extra SAS controller on a T5240 that I was using. The T5440 is the same -- just one controller. I understand the rationale. If you're doing legacy system consolidation or even just massively multi-threaded application processing, internal storage probably won't cut it. Nevertheless, it would make life easier on the admins.
Posted by Jason Callaway on September 26, 2009 at 03:07 PM EDT #
I got a couple of T55240s, and someone had decided to spec it with 8 internal drives. In reality I'm not even using a third of the first two drives.
I have been thinking about using some of the internal disks for a couple of low priority ldoms. Just adding an smf service to shut them down when the control domain is going down. I would much rather have had the option to add in a controller for the other half of the disks. I hope we get that option in a future version - even if it is just to avoid having to put swap on SAN.
Posted by Mads on September 27, 2009 at 03:01 AM EDT #