Monday April 25, 2005

What's a SNIC?

A SNIC is a Storage Network Interface Card. It's how you connect a compute server to the storage network.

Direct-attached storage is connected to compute servers through Host Bus Adapters (HBAs), so called, because they adapt one bus to another. Servers have an internal bus, such as PCI, designed with one set of characteristics - short distance, word addressible, short quick data bursts, etc. On the other hand, external storage busses such as SCSI are designed for longer cable lengths, longer bursts of streamed data, etc. The HBA adapts the characteristics of the external storage bus to the internal one.

Now, go to many of today's large data centers that have over a hundred individual storage devices connected by a complex network with over a thousand ports, dozens of switches and miles of FC or ether network cabling. These are true networks running network protocols. How do you connect compute servers/data clients to these networks? With a SNIC. What else?

( Apr 25 2005, 10:07:13 AM PDT ) Permalink

Friday April 22, 2005

Great technical insights on storage network technology

For great technical insights on storage networking and related technology trends, see Richard McDougall's weblog

Richard is co-author of the Solaris Internals book and has recently been working on storage network technology. Richard is much smarter than I am about how the technology really works.

( Apr 22 2005, 02:07:45 PM PDT ) Permalink

This is only a test This is a test of MarsEdit, a weblog editor for Macs. It is available HERE. MarsEdit simplifies creation of HTML for weblog postings from OS X.

We now return to our regularly scheduled programming....

( Apr 22 2005, 08:59:36 AM PDT ) Permalink

Wednesday April 20, 2005

Are we really open?

Yes,we are very serious about open standards and we are actively driving creation of new open storage management standards. The forum for this is the Storage Networking Industry Association (SNIA) at http://www.snia.org

When we created native multipathing for Solaris, we recognized that there was no standard way to monitor and manage multipathing so we initiated creation of a SNIA MP-API as part of the OS-Attach Technical Workgroup in SNIA. Sun engineer Paul von Behren has served as editor for this API which is now nearing final draft. You can download a copy from the SNIA website here:

http://www.snia.org/tech_activities/publicreview/

In addition to putting an iSCSI driver into Solaris, we have also been participating in SNIA's IP Storage Technical Workgroup since its inception. This workgroup is chartered to define standard solutions for management of IP based storage networks. Sun storage engineer John Forte is helping edit the iSCSI Management API (IMA), a set of C programmatic interfaces for host based management of iSCSI initiators and discovery of iSCSI storage devices. Sun will be providing the Solaris version of IMA later this year. This same SNIA workgroup has also played a key role in defining the SMI-S iSCSI profiles that enable CIM based management of iSCSI initiators and storage. A draft of this spec is also available at:

http://www.snia.org/tech_activities/publicreview/

In addition to creating the specs for the open APIs, we have gone the next step to make our API implementation available as open source projects. See the Source Forge project for the MP-API here:

http://sourceforge.net/projects/mp-mgmt-api/

See the sourceforge project for IMA here:

http://sourceforge.net/projects/iscsi-mgmt-api/ ( Apr 20 2005, 09:47:59 AM PDT ) Permalink Comments [0]

Sunday April 17, 2005

More on Multipathing

A reader asked if Solaris native multipathing can do multiple types of multipath/failover simultaneously, allowing connection to multiple types of storage arrays at the same time. The answer is definitely YES.

This highlights another reason why operating systems need to include multipath for block storage. Compute servers connect into networks with a variety of storage, from a variety of vendors. One driver stack, and one set of HBAs has to talk to all of them. The old model of one type of HBA, with one type of driver, for each storage array doesn't work any more than you can have a different NIC for each type of Web server you connect to on the Internet. ( Apr 17 2005, 03:58:42 PM PDT ) Permalink Comments [1]

Monday April 11, 2005

Multipathing. How did we get here?

I saw statistics from one of the computer marketing groups (Gartner, I think) showing that 90% of mid-range servers go into data-centers that use networked storage. Although not mentioned in the report, I bet the vast majority of those servers have high-availability requirements that demand that all storage components be redundant including array controllers, switches, cabling, and host adapters, and that failover happens automatically.

Given this fact, selling a server and OS without multipathing in the storage stack seems like selling cars without drive shafts. Administrators have been buying new servers, THEN they have to buy the missing multipath part from a third party, install it themselves, and since these drivers weren't originally designed as part of the operating system's I/O framework, they don't integrate as well with the OS as they should.

The industry got here because of the way the technology evolved. It was the array vendors who originally invented dual controllers with failover. Some early redundant controllers used ID aliasing for failover. Say controller 1 appeared on SCSI target ID 1, controller 2 at ID 2. If controller 1 died, controller 2 detected that its alternate was no longer there, and began responding to requests for both SCSI IDs 1 and 2. The problem with this approach was that the interconnect and the HBA were still single points of failure so they started developing add-on drivers for each OS they supported. So, administrators got used to buying and installing these drivers and that's the way the industry worked for several years.

Since then, the OS vendors and the standards bodies have finally caught up. The ANSI T10 SCSI committee now has a standard for asymmetric controller failover. Solaris 10 has the concept of virtualized paths built natively into the device tree and it knows how to do path virtualization, as well as controller failover for most common storage arrays. This is available as a patch for S8 and 9 as well.

On the storage side at Sun, we are moving to use the native multipathing in every OS as well. Most Sun arrays work with the native 'md' driver in Linux. See a paper on how to configure it HERE

Windows is adding a native multipathing framework called MPIO and we are working with Microsoft to support Sun storage. (in the meantime, we have a traditional add-on multipath driver available – yes, we know how to write Windows drivers). HP is now bundling the Veritas VM with DMP into HP-UX as their multipath solution so we are negotiating with Veritas to make sure that 'native' solution supports our arrays.

So, stop buying third-party drive shafts and use an OS that has everything needed to start working with your storage.

( Apr 11 2005, 04:30:30 PM PDT ) Permalink Comments [1]

Friday April 08, 2005

More on iSCSI Two additional comments on iSCSI:

A reader asked if we recommend IPMP (IP Multipathing) or Trunking. We tested and recommend IP Multipathing. We have a Sun Blueprints doc in the works that explains this better, including how it works with embedded storage multipathing (Traffic Manager) in the Solaris OS. Stay tuned for the Blueprints doc. I will post a pointer to it when it's published

Another alert reader pointed out that there ARE iSCSI HBAs (Host Bus Adapters) available that offload iSCSI and TCP/IP protocols with the ability to DMA data directly to/from the user's buffer. These offload the CPU just like Fibre Channel and SCSI HBAs, as well as providing an embedded BIOS for boot support. In my original posting when I said iSCSI uses the main processor, I was referring to the software driver integrated in Solaris 10 Update 1. ( Apr 08 2005, 12:58:09 PM PDT ) Permalink Comments [0]

Wednesday April 06, 2005

Scaling, Fault Management, and Open Standards I'm having a scaling problem at my house that has taught me the value of Fault Management and Open Standards. My wife and I have two teenagers who drive so we now have four cars that need to be maintained.

Cars have gotten more complex over the years, but I've been happy to learn that they have also developed a great fault management architecture built into every new car. They have sensors all over the engine, transmission, and other critical components that send a stream of telemetry through an event bus to embedded service processors running fault management tasks. These tasks continuously analyze streams of events looking for a failure, or sensor readings consistently out of range, turn on the check engine light and provide a specific problem code without requiring you, or the mechanic to search through logs of raw events.

Now, what's really made this nice for me is combining it with open standards. Our garage is heterogeneous. All four of our cars come from different auto companies but, it turns out that every car sold in the U.S. since 1996 provides a standard interface to the fault management system - a connector usually located under the left side of the dashboard. For about $80 I bought an interface adapter that connects this to a standard serial port on my notebook and I downloaded an open-source program that retrieves and interprets the diagnostic codes. So, I can plug my notebook PC into any of our cars and communicate with the service processor.

I've used it a few times. Once it basically told me that we had to take our daughter's Honda in for a new catalytic converter. In other cases though, it saved trips to the shop. Once, a couple days after bringing my car home from the dealer after routine maintenance, it started running rough. The diagnostic code showed that one spark plug wasn't working and after a quick check under the hood, I found that a spark plug wire wasn't seated properly and reconnected it.

We've built some of these same concepts into Sun's Fault Management Architecture. Through our participation in the SNIA storage management standards group we are helping create similar open standards. Even better, since the vast majority of our servers are connected to the Internet, we have the ability to send codes back to a Sun support center. In many cases, we can send a support engineer to the customer's site before they even know they have a problem. ( Apr 06 2005, 09:49:13 PM PDT ) Permalink Comments [0]

Monday April 04, 2005

Another choice for building storage networks Although Fibre Channel is the most common, and highest performance interconnect for storage networks, they can also be built using traditional ethernet, or a combination of ether and Fibre Channel using bridging devices, or switches that bridge both interconnects.

Solaris 10 Update 1 will have an integrated iSCSI driver that sends SCSI commands and data over standard ethernet NICs. To applications, file systems, and volume managers, the iSCSI driver looks like the existing SCSI drivers so these layers will run the same on iSCSI/Ethernet as on parallel SCSI or Fibre Channel.

The disadvantage of the iSCSI driver is that more protocol execution is done by the processor, and more significantly, data is copied between user memory and ethernet buffers. Fibre, on the other hand, can DMA directly in and out of application memory as well as offload protocol processing to the HBA.

iSCSI offers at least two benefits though. One is lower cost. It runs on low cost NICs and ethernet switches. Second, the storage network can be managed, in part, using familiar ethernet protocols and tools.

We have customers who are planning to use Sun's rack-mount x64 servers in their data center and who plan to use iSCSI in the Solaris OS on these servers. They have evaluated the performance of these x64 servers and decided they have enough processing power to spare. They like that Solaris iSCSI is integrated (no adding extra patches) and have found it to perform very reliably when faced with a variety of hardware failures and storage network error scenarios. Most plan to bridge iSCSI to FC to access their existing consolidated SAN storage, and some are looking to use native iSCSI arrays.

So, data center administrators have another choice for building storage networks. Either way, the drivers are built-into the Solaris OS, they are highly available, extensively tested and, as always, open based on standards. ( Apr 04 2005, 07:23:03 PM PDT ) Permalink Comments [2]

Friday April 01, 2005

What is Storage Network Engineering? Trace the evolution of storage. In the old days, storage was directly connected as a peripheral to the server. The server booted up, discovered it had storage attached and took control of it for its exclusive use. This approach had several limitations. The amount of storage you could attach was limited by the number of I/O slots and storage bus connections (e.g. SCSI IDs). Storage couldn't be easily shared so if two server needed to use the same data, multiple copies were required, and it was hard to pool storage for consolidated management.

In the 90's new interconnects like Fibre Channel (FC) were developed that solved some of these problems. Early FC allowed much higher expandability and performance. It also allowed some amount of pooling and central management of physical storage assets. The term Storage Area Networks (SANs) was used to describe these limited storage networks. A problem though was operating systems still operated under the direct-attached, storage-as-peripheral model. They still tried to scan every storage device that was visible to them, then assumed exclusive control of that storage. OS's also assumed that the configuration couldn't change unless the server was powered-down, or that the number of devices was small enough that the system administrator could list them all in a configuration file. This forced administrators to partition up their SAN through zoning to give each OS only a small view of the SAN that appeared like a small direct-attached configuration, prevented multiple servers from seeing the same storage, and still allowed server to claim exclusive ownership of storage across the FC interconnect.

Today we have entered the age of true storage networks. Data centers now have storage networks with well over a thousand nodes. These are true networks on which the disk arrays and tape libraries have become Servers, providing block storage or tape archiving services. The Compute Servers are now Data Clients that discover and use these shared storage services across a network. The old direct-attach assumptions built into operating systems don't apply anymore. It would take hours to boot a compute server if the OS tried to scan the network for every device visible to it. You can't reboot the compute server every time the network is reconfigured or a new device is added. Also, you can't keep configuration files to describe the physical address of every storage device. Imagine if your web browser required you to maintain a file with the ethernet MAC addresses of every web server you visited.

In Storage Network Engineering we have designed a storage networking protocol stack into the Solaris OS making it the best Data Client on the storage network. Storage is discovered and exposed to applications as needed so boot times are fast. There is no editing config files with physical addresses. Multi-pathing for redundancy, routing through the network, and fail-over is built-in. New storage can be dynamically detected and path changes through the network are detected and adapted to without needing to reboot the Solaris Data Client.

For a more thorough description of the Solaris Storage Network stack see the Sun Blueprints document "Increasing Storage Area Network Productivity" located here:

http://www.sun.com/blueprints/0704/817-7544.pdf ( Apr 01 2005, 06:53:15 AM PST ) Permalink Comments [3]