Protoss

NPort ID Virtualization

What is NPort ID Virtulation
N_Port ID Virtualization (NPIV) is a technology that allows one physical HBA to login multiple times to a switch, with different WWN.
This is useful especially for virtualized server environments, such as xVM, LDom or VMWare.
By using virtual N Ports for each virtual server, the administrator can use traditional zoning and LUN masking techniques for SAN management.
However, it can also be useful for non xVM environment. In this way, one application could be given its own WWN.
A complimentary project is to do quality of service for the different NPIV ports.
This would allow one physical port to be fairly shared among devices.

Why use NPIV
1. Leverage Existing Fabric tools for storage management. For example: switch zoning, lun masking based on WWN
2. Broken HBA - no problem, replace HBA, and use the old WWN in the new HBA
3. Migrate virtual Severs from one physical server to another, zoning just works, because WWN follows virtual server. Currently, xVM must use open zoning, or at least zoning which includes all HBAs.
4. QOS. Use tools to segregate traffic based on WWN. Another implementation is to do per virtual port throttling - guarantee certain bandwidth to a certain port. This could be very useful in some environments, for example if a ISP sells a "virtual server"
5. Why do we need to access at SCSI layer This is important, because today xVM does not support SCSI access; in particular there is no easy way to use tape devices.

Hardware Support
Emulex : 4 Gb
Qlogic : 4 Gb, some 2 Gb
Brocade, Cisco, McData switches with recent firmware

How to use
You can use new user command fcadm to configure npiv port
# fcadm
Usage:  fcadm -?,-V,--help
Usage:  fcadm create-npiv-port [-?]
Usage:  fcadm delete-npiv-port [-?]
Usage:  fcadm hba-port [-?]
Usage:  fcadm remote-port [-?]
Usage:  fcadm create-port-list [-?]
For more information, please see fcadm(1M)

1. create a virtual port with random WWN
First we need check if the HBA card supports NPIV feature, we cann't create virtual port on a HBA card without supporting NPIV.
HBA port information shows "NPIV Not Supported" if it does not support NPIV,
otherwise it will show the maximum number of npiv ports it can support.

show physical ports on this system
bash-3.2# luxadm -e port            
/devices/pci@7c0/pci@0/pci@1/pci@0,2/SUNW,emlxs@1/fp@0,0:devctl    CONNECTED
/devices/pci@7c0/pci@0/pci@1/pci@0,2/SUNW,qlc@2/fp@0,0:devctl      CONNECTED

show physical ports info on this system
bash-3.2# fcadm hba-port
HBA Port WWN: 10000000c96dff32
        Port Mode: Initiator
        Port ID: 20a00
        OS Device Name: /dev/cfg/c4
        Manufacturer: Emulex
        Model: LP11002-S
        Firmware Version: 2.72a2 (B3F2.72A2)
        FCode/BIOS Version: 1.50a9
        Serial Number: 0999BG0-07460001HN
        Driver Name: emlxs
        Driver Version: 2.31j (2008.07.22.16.03)
        Type: N-port
        State: online
        Supported Speeds: 1Gb 2Gb 4Gb
        Current Speed: 2Gb
        Node WWN: 20000000c96dff32
        NPIV Not Supported
HBA Port WWN: 210000e08b84e3ea
        Port Mode: Initiator
        Port ID: 10300
        OS Device Name: /dev/cfg/c3
        Manufacturer: QLogic Corp.
        Model: 375-3294-01
        Firmware Version: 04.04.01
        FCode/BIOS Version:  BIOS: 1.4; fcode: 1.11; EFI: 1.0;
        Serial Number: 0402F00-0549112906
        Driver Name: qlc
        Driver Version: 20080617-2.30
        Type: N-port
        State: online
        Supported Speeds: 1Gb 2Gb 4Gb
        Current Speed: 4Gb
        Node WWN: 200000e08b84e3ea
        Max NPIV Ports: 63
        NPIV port list:

show physical ports attach point link to the devices path
bash-3.2# ls -l /dev/cfg/c3
lrwxrwxrwx   1 root     root          62 Sep  5 11:53 /dev/cfg/c3 -> ../../devices/pci@7c0/pci@0/pci@1/pci@0,2/SUNW,qlc@2/fp@0,0:fc
bash-3.2# ls -l /dev/cfg/c4
lrwxrwxrwx   1 root     root          64 Sep  5 11:53 /dev/cfg/c4 -> ../../devices/pci@7c0/pci@0/pci@1/pci@0,2/SUNW,emlxs@1/fp@0,0:fc

create a virtual port with random WWN (no WWN assigned)
bash-3.2# fcadm  create-npiv-port 210000e08b84e3ea
bash-3.2# fcadm hba-port
HBA Port WWN: 10000000c96dff32
        Port Mode: Initiator
        Port ID: 20a00
        OS Device Name: /dev/cfg/c4
        Manufacturer: Emulex
        Model: LP11002-S
        Firmware Version: 2.72a2 (B3F2.72A2)
        FCode/BIOS Version: 1.50a9
        Serial Number: 0999BG0-07460001HN
        Driver Name: emlxs
        Driver Version: 2.31j (2008.07.22.16.03)
        Type: N-port
        State: online
        Supported Speeds: 1Gb 2Gb 4Gb
        Current Speed: 2Gb
        Node WWN: 20000000c96dff32
        NPIV Not Supported
HBA Port WWN: 210000e08b84e3ea
        Port Mode: Initiator
        Port ID: 10300
        OS Device Name: /dev/cfg/c3
        Manufacturer: QLogic Corp.
        Model: 375-3294-01
        Firmware Version: 04.04.01
        FCode/BIOS Version:  BIOS: 1.4; fcode: 1.11; EFI: 1.0;
        Serial Number: 0402F00-0549112906
        Driver Name: qlc
        Driver Version: 20080617-2.30
        Type: N-port
        State: online
        Supported Speeds: 1Gb 2Gb 4Gb
        Current Speed: 4Gb
        Node WWN: 200000e08b84e3ea
        Max NPIV Ports: 63
        NPIV port list:
          Virtual Port1:
                Node WWN: c0007d5dc67c75f9
                Port WWN: c0007d5dc67c75f8

show new created virtual port info
bash-3.2# fcadm hba-port c0007d5dc67c75f8
HBA Port WWN: c0007d5dc67c75f8
        Port Mode: Initiator
        Port ID: 10301
        OS Device Name: /dev/cfg/c6
        Manufacturer: QLogic Corp.
        Model: 375-3294-01
        Firmware Version: 04.04.01
        FCode/BIOS Version:  BIOS: 1.4; fcode: 1.11; EFI: 1.0;
        Serial Number: 0402F00-0549112906
        Driver Name: qlc
        Driver Version: 20080617-2.30
        Type: N-port
        State: online
        Supported Speeds: 1Gb 2Gb 4Gb
        Current Speed: 4Gb
        Node WWN: c0007d5dc67c75f9
        NPIV Not Supported
bash-3.2# luxadm -e port
/devices/pci@7c0/pci@0/pci@1/pci@0,2/SUNW,emlxs@1/fp@0,0:devctl    CONNECTED
/devices/pci@7c0/pci@0/pci@1/pci@0,2/SUNW,qlc@2/fp@0,0:devctl      CONNECTED
/devices/pci@7c0/pci@0/pci@1/pci@0,2/SUNW,qlc@2/fp@1,0:devctl      CONNECTED
bash-3.2# ls -l /dev/cfg/c6
lrwxrwxrwx   1 root     root          62 Sep  5 14:18 /dev/cfg/c6 -> ../../devices/pci@7c0/pci@0/pci@1/pci@0,2/SUNW,qlc@2/fp@1,0:fc

2. create a virtual port with assigned WWN
bash-3.2# fcadm  create-npiv-port -p 1000000000000021 -n 1000000000000020 210000e08b84e3ea
bash-3.2# fcadm hba-port 210000e08b84e3ea
HBA Port WWN: 210000e08b84e3ea
        Port Mode: Initiator
        Port ID: 10300
        OS Device Name: /dev/cfg/c3
        Manufacturer: QLogic Corp.
        Model: 375-3294-01
        Firmware Version: 04.04.01
        FCode/BIOS Version:  BIOS: 1.4; fcode: 1.11; EFI: 1.0;
        Serial Number: 0402F00-0549112906
        Driver Name: qlc
        Driver Version: 20080617-2.30
        Type: N-port
        State: online
        Supported Speeds: 1Gb 2Gb 4Gb
        Current Speed: 4Gb
        Node WWN: 200000e08b84e3ea
        Max NPIV Ports: 63
        NPIV port list:
          Virtual Port1:
                Node WWN: 1000000000000020
                Port WWN: 1000000000000021

3. delete a virtual port
bash-3.2# fcadm hba-port 210000e08b84e3ea
HBA Port WWN: 210000e08b84e3ea
        Port Mode: Initiator
        Port ID: 10300
        OS Device Name: /dev/cfg/c3
        Manufacturer: QLogic Corp.
        Model: 375-3294-01
        Firmware Version: 04.04.01
        FCode/BIOS Version:  BIOS: 1.4; fcode: 1.11; EFI: 1.0;
        Serial Number: 0402F00-0549112906
        Driver Name: qlc
        Driver Version: 20080617-2.30
        Type: N-port
        State: online
        Supported Speeds: 1Gb 2Gb 4Gb
        Current Speed: 4Gb
        Node WWN: 200000e08b84e3ea
        Max NPIV Ports: 63
        NPIV port list:
          Virtual Port1:
                Node WWN: c0007d5dc67c75f9
                Port WWN: c0007d5dc67c75f8
bash-3.2# fcadm delete-npiv-port -p c0007d5dc67c75f8 210000e08b84e3ea
bash-3.2# fcadm hba-port 210000e08b84e3ea
HBA Port WWN: 210000e08b84e3ea
        Port Mode: Initiator
        Port ID: 10300
        OS Device Name: /dev/cfg/c3
        Manufacturer: QLogic Corp.
        Model: 375-3294-01
        Firmware Version: 04.04.01
        FCode/BIOS Version:  BIOS: 1.4; fcode: 1.11; EFI: 1.0;
        Serial Number: 0402F00-0549112906
        Driver Name: qlc
        Driver Version: 20080617-2.30
        Type: N-port
        State: online
        Supported Speeds: 1Gb 2Gb 4Gb
        Current Speed: 4Gb
        Node WWN: 200000e08b84e3ea
        Max NPIV Ports: 63
        NPIV port list:
bash-3.2# luxadm -e port
/devices/pci@7c0/pci@0/pci@1/pci@0,2/SUNW,emlxs@1/fp@0,0:devctl    CONNECTED
/devices/pci@7c0/pci@0/pci@1/pci@0,2/SUNW,qlc@2/fp@0,0:devctl      CONNECTED

NPIV and Sun xVM Server
Generally, and for most efficient operation, xVM requires device drivers to be modified to support xVM.
Device drivers operate either in the guest domain (called DOMU) or the device driver domain (usually DOM0).
Device drivers communicate between DOMU and DOM0 via descriptor rings.
These rings, or circular queues, are similar to what device drivers traditionally use to access hardware.
It's important to note that there is no way to directly call between DOM0 and DOMU.
The layer we're choosing is NPIV implemented entirely in DOM0.
The advantage is that it exposes a SCSI interface to DOMU, so devices such as tape can be used.
Issues (only cover NPIV in DOM0) DOMU does not have visibility into FC layer - hence no fcinfo, luxadm, cfgadm. However, DOM0 would have all this information.
1. Multipathing is simpler. All multipathing is done by Solaris DOM0. Could potentially be a negative, if customer wants to use a device for which Solaris does not support multipathing.
2. FCIP maybe is more complicated - have to interact with xVM networking?
3. Code should be simpler - no complicated interface. No need for agreement with other xVM operating systems on interface. Also, Leadville driver is identical between xVM and non xVM implementations. See

How to play it with xVM
For xVM supported NPIV, we use virsh to attach a LUN of FC target to the guest domain as a disk.

Attach a LUN of FC target to the guest domain.
virsh attach-disk <domian-name> <npiv:physical port wwn/virtual port wwn/array port wwn/LUN number> <target>
On Dom0 run
bash-3.2# virsh attach-disk 4632dom  npiv:210000e08b84e3ea/1000000000000020/20030003ba13f3fa/1 5
bash-3.2# fcadm hba-port 210000e08b84e3ea
HBA Port WWN: 210000e08b84e3ea
        Port Mode: Initiator
        Port ID: 10300
        OS Device Name: /dev/cfg/c3
        Manufacturer: QLogic Corp.
        Model: 375-3294-01
        Firmware Version: 04.04.01
        FCode/BIOS Version:  BIOS: 1.4; fcode: 1.11; EFI: 1.0;
        Serial Number: 0402F00-0549112906
        Driver Name: qlc
        Driver Version: 20080617-2.30
        Type: N-port
        State: online
        Supported Speeds: 1Gb 2Gb 4Gb
        Current Speed: 4Gb
        Node WWN: 200000e08b84e3ea
        Max NPIV Ports: 63
        NPIV port list:
          Virtual Port1:
                Node WWN: 1000000000000020
                Port WWN: 1000000000000021
On Domain 4632dom run format, you can find new LUNs attached to this domain
bash-3.2# format
Searching for disks...done


AVAILABLE DISK SELECTIONS:
       0. c0d5 <DEFAULT cyl 1024 alt 0 hd 128 sec 32>
          /xpvd/xdf@5
       1. c0t0d0 <DEFAULT cyl 2559 alt 0 hd 128 sec 32>
          /xpvd/xdf@51712
Specify disk (enter its number):

Detach a LUN of FC target to the guest domain.
bash-3.2# virsh detach-disk 4632dom 5

Enjoy it :)

Posted at 11:23上午 十一月 20, 2008 by Shu-Jun Allan Ou in Sun  |  评论[5]