Protoss
千金散尽还复来，莫使金樽空对月
NPort ID Virtualization
What is NPort ID Virtulation
N_Port ID Virtualization (NPIV) is a technology that allows one physical HBA to login multiple times to a switch, with different WWN.
This is useful especially for virtualized server environments, such as xVM, LDom or VMWare.
By using virtual N Ports for each virtual server, the administrator can use traditional zoning and LUN masking techniques for SAN management.
However, it can also be useful for non xVM environment. In this way, one application could be given its own WWN.
A complimentary project is to do quality of service for the different NPIV ports.
This would allow one physical port to be fairly shared among devices.
Why use NPIV
1. Leverage Existing Fabric tools for storage management. For example: switch zoning, lun masking based on WWN
2. Broken HBA - no problem, replace HBA, and use the old WWN in the new HBA
3. Migrate virtual Severs from one physical server to another, zoning just works, because WWN follows virtual server. Currently, xVM must use open zoning, or at least zoning which includes all HBAs.
4. QOS. Use tools to segregate traffic based on WWN. Another implementation is to do per virtual port throttling - guarantee certain bandwidth to a certain port. This could be very useful in some environments, for example if a ISP sells a "virtual server"
5. Why do we need to access at SCSI layer This is important, because today xVM does not support SCSI access; in particular there is no easy way to use tape devices.
Hardware Support
Emulex : 4 Gb
Qlogic : 4 Gb, some 2 Gb
Brocade, Cisco, McData switches with recent firmware
How to use
You can use new user command fcadm to configure npiv port
# fcadm
Usage: fcadm -?,-V,--help
Usage: fcadm create-npiv-port [-?]
Usage: fcadm delete-npiv-port [-?]
Usage: fcadm hba-port [-?]
Usage: fcadm remote-port [-?]
Usage: fcadm create-port-list [-?]
For more information, please see fcadm(1M)
1. create a virtual port with random WWN
First we need check if the HBA card supports NPIV feature, we cann't create virtual port on a HBA card without supporting NPIV.
HBA port information shows "NPIV Not Supported" if it does not support NPIV,
otherwise it will show the maximum number of npiv ports it can support.
show physical ports on this system
bash-3.2# luxadm -e port
/devices/pci@7c0/pci@0/pci@1/pci@0,2/SUNW,emlxs@1/fp@0,0:devctl CONNECTED
/devices/pci@7c0/pci@0/pci@1/pci@0,2/SUNW,qlc@2/fp@0,0:devctl CONNECTED
show physical ports info on this system
bash-3.2# fcadm hba-port
HBA Port WWN: 10000000c96dff32
Port Mode: Initiator
Port ID: 20a00
OS Device Name: /dev/cfg/c4
Manufacturer: Emulex
Model: LP11002-S
Firmware Version: 2.72a2 (B3F2.72A2)
FCode/BIOS Version: 1.50a9
Serial Number: 0999BG0-07460001HN
Driver Name: emlxs
Driver Version: 2.31j (2008.07.22.16.03)
Type: N-port
State: online
Supported Speeds: 1Gb 2Gb 4Gb
Current Speed: 2Gb
Node WWN: 20000000c96dff32
NPIV Not Supported
HBA Port WWN: 210000e08b84e3ea
Port Mode: Initiator
Port ID: 10300
OS Device Name: /dev/cfg/c3
Manufacturer: QLogic Corp.
Model: 375-3294-01
Firmware Version: 04.04.01
FCode/BIOS Version: BIOS: 1.4; fcode: 1.11; EFI: 1.0;
Serial Number: 0402F00-0549112906
Driver Name: qlc
Driver Version: 20080617-2.30
Type: N-port
State: online
Supported Speeds: 1Gb 2Gb 4Gb
Current Speed: 4Gb
Node WWN: 200000e08b84e3ea
Max NPIV Ports: 63
NPIV port list:
show physical ports attach point link to the devices path
bash-3.2# ls -l /dev/cfg/c3
lrwxrwxrwx 1 root root 62 Sep 5 11:53 /dev/cfg/c3 -> ../../devices/pci@7c0/pci@0/pci@1/pci@0,2/SUNW,qlc@2/fp@0,0:fc
bash-3.2# ls -l /dev/cfg/c4
lrwxrwxrwx 1 root root 64 Sep 5 11:53 /dev/cfg/c4 -> ../../devices/pci@7c0/pci@0/pci@1/pci@0,2/SUNW,emlxs@1/fp@0,0:fc
create a virtual port with random WWN (no WWN assigned)
bash-3.2# fcadm create-npiv-port 210000e08b84e3ea
bash-3.2# fcadm hba-port
HBA Port WWN: 10000000c96dff32
Port Mode: Initiator
Port ID: 20a00
OS Device Name: /dev/cfg/c4
Manufacturer: Emulex
Model: LP11002-S
Firmware Version: 2.72a2 (B3F2.72A2)
FCode/BIOS Version: 1.50a9
Serial Number: 0999BG0-07460001HN
Driver Name: emlxs
Driver Version: 2.31j (2008.07.22.16.03)
Type: N-port
State: online
Supported Speeds: 1Gb 2Gb 4Gb
Current Speed: 2Gb
Node WWN: 20000000c96dff32
NPIV Not Supported
HBA Port WWN: 210000e08b84e3ea
Port Mode: Initiator
Port ID: 10300
OS Device Name: /dev/cfg/c3
Manufacturer: QLogic Corp.
Model: 375-3294-01
Firmware Version: 04.04.01
FCode/BIOS Version: BIOS: 1.4; fcode: 1.11; EFI: 1.0;
Serial Number: 0402F00-0549112906
Driver Name: qlc
Driver Version: 20080617-2.30
Type: N-port
State: online
Supported Speeds: 1Gb 2Gb 4Gb
Current Speed: 4Gb
Node WWN: 200000e08b84e3ea
Max NPIV Ports: 63
NPIV port list:
Virtual Port1:
Node WWN: c0007d5dc67c75f9
Port WWN: c0007d5dc67c75f8
show new created virtual port info
bash-3.2# fcadm hba-port c0007d5dc67c75f8
HBA Port WWN: c0007d5dc67c75f8
Port Mode: Initiator
Port ID: 10301
OS Device Name: /dev/cfg/c6
Manufacturer: QLogic Corp.
Model: 375-3294-01
Firmware Version: 04.04.01
FCode/BIOS Version: BIOS: 1.4; fcode: 1.11; EFI: 1.0;
Serial Number: 0402F00-0549112906
Driver Name: qlc
Driver Version: 20080617-2.30
Type: N-port
State: online
Supported Speeds: 1Gb 2Gb 4Gb
Current Speed: 4Gb
Node WWN: c0007d5dc67c75f9
NPIV Not Supported
bash-3.2# luxadm -e port
/devices/pci@7c0/pci@0/pci@1/pci@0,2/SUNW,emlxs@1/fp@0,0:devctl CONNECTED
/devices/pci@7c0/pci@0/pci@1/pci@0,2/SUNW,qlc@2/fp@0,0:devctl CONNECTED
/devices/pci@7c0/pci@0/pci@1/pci@0,2/SUNW,qlc@2/fp@1,0:devctl CONNECTED
bash-3.2# ls -l /dev/cfg/c6
lrwxrwxrwx 1 root root 62 Sep 5 14:18 /dev/cfg/c6 -> ../../devices/pci@7c0/pci@0/pci@1/pci@0,2/SUNW,qlc@2/fp@1,0:fc
2. create a virtual port with assigned WWN
bash-3.2# fcadm create-npiv-port -p 1000000000000021 -n 1000000000000020 210000e08b84e3ea
bash-3.2# fcadm hba-port 210000e08b84e3ea
HBA Port WWN: 210000e08b84e3ea
Port Mode: Initiator
Port ID: 10300
OS Device Name: /dev/cfg/c3
Manufacturer: QLogic Corp.
Model: 375-3294-01
Firmware Version: 04.04.01
FCode/BIOS Version: BIOS: 1.4; fcode: 1.11; EFI: 1.0;
Serial Number: 0402F00-0549112906
Driver Name: qlc
Driver Version: 20080617-2.30
Type: N-port
State: online
Supported Speeds: 1Gb 2Gb 4Gb
Current Speed: 4Gb
Node WWN: 200000e08b84e3ea
Max NPIV Ports: 63
NPIV port list:
Virtual Port1:
Node WWN: 1000000000000020
Port WWN: 1000000000000021
3. delete a virtual port
bash-3.2# fcadm hba-port 210000e08b84e3ea
HBA Port WWN: 210000e08b84e3ea
Port Mode: Initiator
Port ID: 10300
OS Device Name: /dev/cfg/c3
Manufacturer: QLogic Corp.
Model: 375-3294-01
Firmware Version: 04.04.01
FCode/BIOS Version: BIOS: 1.4; fcode: 1.11; EFI: 1.0;
Serial Number: 0402F00-0549112906
Driver Name: qlc
Driver Version: 20080617-2.30
Type: N-port
State: online
Supported Speeds: 1Gb 2Gb 4Gb
Current Speed: 4Gb
Node WWN: 200000e08b84e3ea
Max NPIV Ports: 63
NPIV port list:
Virtual Port1:
Node WWN: c0007d5dc67c75f9
Port WWN: c0007d5dc67c75f8
bash-3.2# fcadm delete-npiv-port -p c0007d5dc67c75f8 210000e08b84e3ea
bash-3.2# fcadm hba-port 210000e08b84e3ea
HBA Port WWN: 210000e08b84e3ea
Port Mode: Initiator
Port ID: 10300
OS Device Name: /dev/cfg/c3
Manufacturer: QLogic Corp.
Model: 375-3294-01
Firmware Version: 04.04.01
FCode/BIOS Version: BIOS: 1.4; fcode: 1.11; EFI: 1.0;
Serial Number: 0402F00-0549112906
Driver Name: qlc
Driver Version: 20080617-2.30
Type: N-port
State: online
Supported Speeds: 1Gb 2Gb 4Gb
Current Speed: 4Gb
Node WWN: 200000e08b84e3ea
Max NPIV Ports: 63
NPIV port list:
bash-3.2# luxadm -e port
/devices/pci@7c0/pci@0/pci@1/pci@0,2/SUNW,emlxs@1/fp@0,0:devctl CONNECTED
/devices/pci@7c0/pci@0/pci@1/pci@0,2/SUNW,qlc@2/fp@0,0:devctl CONNECTED
NPIV and Sun xVM Server
Generally, and for most efficient operation, xVM requires device drivers to be modified to support xVM.
Device drivers operate either in the guest domain (called DOMU) or the device driver domain (usually DOM0).
Device drivers communicate between DOMU and DOM0 via descriptor rings.
These rings, or circular queues, are similar to what device drivers traditionally use to access hardware.
It's important to note that there is no way to directly call between DOM0 and DOMU.
The layer we're choosing is NPIV implemented entirely in DOM0.
The advantage is that it exposes a SCSI interface to DOMU, so devices such as tape can be used.
Issues (only cover NPIV in DOM0) DOMU does not have visibility into FC layer - hence no fcinfo, luxadm, cfgadm. However, DOM0 would have all this information.
1. Multipathing is simpler. All multipathing is done by Solaris DOM0. Could potentially be a negative, if customer wants to use a device for which Solaris does not support multipathing.
2. FCIP maybe is more complicated - have to interact with xVM networking?
3. Code should be simpler - no complicated interface. No need for agreement with other xVM operating systems on interface. Also, Leadville driver is identical between xVM and non xVM implementations. See
How to play it with xVM
For xVM supported NPIV, we use virsh to attach a LUN of FC target to the guest domain as a disk.
Attach a LUN of FC target to the guest domain.
virsh attach-disk <domian-name> <npiv:physical port wwn/virtual port wwn/array port wwn/LUN number> <target>
On Dom0 run
bash-3.2# virsh attach-disk 4632dom npiv:210000e08b84e3ea/1000000000000020/20030003ba13f3fa/1 5
bash-3.2# fcadm hba-port 210000e08b84e3ea
HBA Port WWN: 210000e08b84e3ea
Port Mode: Initiator
Port ID: 10300
OS Device Name: /dev/cfg/c3
Manufacturer: QLogic Corp.
Model: 375-3294-01
Firmware Version: 04.04.01
FCode/BIOS Version: BIOS: 1.4; fcode: 1.11; EFI: 1.0;
Serial Number: 0402F00-0549112906
Driver Name: qlc
Driver Version: 20080617-2.30
Type: N-port
State: online
Supported Speeds: 1Gb 2Gb 4Gb
Current Speed: 4Gb
Node WWN: 200000e08b84e3ea
Max NPIV Ports: 63
NPIV port list:
Virtual Port1:
Node WWN: 1000000000000020
Port WWN: 1000000000000021
On Domain 4632dom run format, you can find new LUNs attached to this domain
bash-3.2# format
Searching for disks...done
AVAILABLE DISK SELECTIONS:
0. c0d5 <DEFAULT cyl 1024 alt 0 hd 128 sec 32>
/xpvd/xdf@5
1. c0t0d0 <DEFAULT cyl 2559 alt 0 hd 128 sec 32>
/xpvd/xdf@51712
Specify disk (enter its number):
Detach a LUN of FC target to the guest domain.
bash-3.2# virsh detach-disk 4632dom 5
Enjoy it :)
Posted at 11:23上午 十一月 20, 2008 by Shu-Jun Allan Ou in Sun | 评论[5]
迷人的Solaris
Solaris是Sun公司开发和发布的操作系统，是UNIX系统的一个重要分支。
起源：首先从BSD Unix移植而来，随后逐渐转移到了System V标准。
支持平台：目前，Solaris除可以运行在SPARC CPU上外，还可以运行在X86 (IA32)的CPU上。最近又被移植到了最新的AMD64位CPU Opteron上.
市场地位：在服务器市场上，Sun的硬件平台具有高可用性和高可靠性，Solaris是当今市场上处于支配地位的Unix类操作系统。
下载：这些版本都可以从Sun的官方网站下载，也可以从国内外其他一些站点免费下载。对于难以接触到Sun SPARC架构计算机的用户可以通过使用Solaris x86体验世界知名大厂的商业Unix的风采。当然Solaris x86也可以用于实际生产应用的服务器。在遵守Sun的有关许可条款的情况下，Solaris x86可以免费用于学习研究或商业应用。
参考书目：国内关于Solaris的书籍还是比较多，从一些网上书店可以搜索到不少的关于Solaris的书籍,
强烈推荐《Solaris Internals: Solaris 10 and OpenSolaris Kernel Architecture, Second Edition》
Solaris的DTrace：
就像奥黛丽.赫本那清澈的明牟，
就像玛丽莲那性感的乌痣，
谈到Solaris10，我们不得不谈DTrace，华尔街时报2006年科技创新金奖得主。
再多华丽的辞藻，再多洋溢的赞美，再多羡慕的目光赠予这一项创新软件设计都不过分。
当然广告做的好没有用，要看它的疗效，
谁用谁知道呀～～～～
DTrace是一种故障诊断软件，
她是不同于我们以前用过的Debuger（调试工具），
什么单步跟踪，什么设置断点统统的没有，
使用DTrace检测，将对程序执行的影响降到最低，她就那样默默的注视着，注视着...
个人认为她更像是printf或printk，
她可以打印所有函数调用参数、所有内核全局变量、函数执行时的时间戳、函数调用的堆栈、调用函数的进程或线程，我靠太多东东啦（不好意思说了个dirty word，太激动了)
DTrace的设计理念是这样的：
各位程序员兄弟姐妹们，我呢为你们提供一些检测点（专业的说是 probe)，你们可以用这些检测点记录也好、打印也罢显示内核或用户进程运行的相关信息。当然这些检测点会被一些特定的行为激活，激活之后你就可以定义一些动作，比如打印函数的调用参数值等等。
当然设计理念之类的东东都是很抽象地，
就像马克思主义理论、毛泽东思想，都是总结之总结、概括之概括、集大成之集大成
不是你我这类凡夫俗子所能理解的，还是举个例子来的直接，
下面就是一个简单的DTrace脚本
#!/usr/sbin/dtrace -s
syscall::write:entry
{
trace(timestamp);
}
这里“syscall”是检测点提供者（provider），“write”就是检测点（probe），“entry”就是检测点被激活的条件，当write被调用的时候激活，定义的动作（action）就是打印函数被调用时的时间戳。
更多细节可以参考
《Dtrace User Guide》和《Solaris Dynamic Tracing Guide》（强烈推荐）
可以在 http://docs.sun.com下载到
Posted at 01:56下午 八月 13, 2007 by Shu-Jun Allan Ou in Sun | 评论[2]
今日点击： 30