More on Sun's Virtual NEM
The Virtual NEM can be connected externally to one or two 10 GbE network switch ports. These external 10 GbE links are then shared by up to 10 blades using virtual NICs. When using a single 10 GbE connection, the Inter ASIC Link (IAL) connects the two halves of the virtual NIC. When using two 10 GbE connections, the Virtual NEM can be configured so that each set of 5 blades shares one of the 10 GbE connections, providing a total of 20 Gb of bandwidth. The IAL also supports dynamic failover so that if one of your two 10 GbE connection fails, the Virtual NIC will automatically switch over the 5 blades on the failed connection to the other half of the Virtual NIC. This makes it quite simple to build highly available networks by connecting the Virtual NEM to two separate external switches.
The virtual NIC allows each blade to be configured with a 10 GbE driver and 10 GbE port and each blade can run at full 10 GbE speeds although the total bandwidth at any one time is limited to either 10 or 20 Gb. Almost no servers require dedicated, constant 10 GbE traffic, but many modern servers can drive short bursts of 10 GbE traffic.
In addition to the two 10 GbE ports, the Virtual NEM also includes (not shown):
The Virtual NEM is hot pluggable into the Sun Blade 6000 chassis which supports single or dual Virtual NEM configurations. The Virtual NEM is passively cooled for high availability (no fans to fail). A web based management interface along with command line interface make setup and monitoring simple.
Windows, Linux, Solaris, and VMware are all supported by the Virtual NEM hardware, although customers will have to wait approximately 60 days for the VMware driver to be completed. We have tested the Virtual NEM with a wide variety of 3rd party 10 GbE switches including Cisco, Foundry, Force10, Extreme, Fujitsu, HP ProCurve, and Arista Networks.
Since the NEM also has 10 1GigE ports, can each blade use both the VNIC and the dedicated NIC port at the same time? OR is it one or the other?
Posted by Damian on March 20, 2009 at 08:15 PM PDT #
Damian,
The Virtual NEM provides a dedicated 1 GigE port to each blade in addition to the 10 GbE ports. A blade can use both at the same time.
Posted by Marc Hamilton on March 20, 2009 at 09:08 PM PDT #
Just one follow-on note on the SAS connectivity. Currently the Virtual NEM only supports SAS external connectivity from compute blades to Sun Blade 6000 storage blade,
http://www.sun.com/servers/blades/6000storage/
Posted by Marc Hamilton on March 22, 2009 at 08:43 AM PDT #
Are the 10 GbE ports using Sun's multithreaded 10 GbE ASIC?
http://www.sun.com/products/networking/ethernet/10gigethernet/index.xml
Thanks for all the informative posts Mark.
Posted by Eric on March 22, 2009 at 01:07 PM PDT #
Eric,
The 10 GbE ports on the Sun Virtual NEM are an ASIC implementation which leverages the IP of the ASIC in our standalone 10 GbE network cards which is very similar to the 10 GbE logic built into our UltraSPARC T2 CPUs, although it is not the same exact ASIC. There are actually two ASICs in the Sun Virtual NEM each handling traffic for 5 blades and connected via the IAL. The "multithreading" works as follows:
RX unicast traffic is load balanced across 4 DMA channels per blade which allows the host OS to leverage OS threads for processing packet streams. Solaris drivers take further advantage of this via Solaris IP Instances and Solaris Crossbow technology. On the ASIC side, the load balancing is based on 4-Tuple matching using layer 4 information stored in a HW TCAM (ternary content addressable memory)
Blade TX is arbitrated with a Deficit Round Robin (DRR) algorithm. DRR transmits packets at the head of individual blade queues where the deficit counter (credits) > than the packet's size. If the deficit counter is < packet size then it is increased by a given value called quantum. After transmission the deficit counter is decreased by the size of packets being served. When credits are exhausted the next blade's queue is serviced
The marketing message for Sun Virtual NEM can still be summarized as "makes blade networking and i/o affordable and easy", but for technical folks, it really is an amazing collection of hardware and a great example of the value of a systems company to be able to integrate its software (OS), compute, storage, and networking assets to solve complex customer problems, all while being open and usable with 3rd party OSes, network switches, or storage.
Posted by Marc Hamilton on March 23, 2009 at 01:02 AM PDT #