Nitin's Weblog Nitin's Weblog

Shining, new OpenSolaris is here!. This is undoubtedly very exciting to share the efforts with everyone around. Here is some code internals and information about configuring IPoIB node over Infiniband.

IPoIB is a DLPI based OpenSolaris driver. It supports unconnected datagram mode over IB. The IPoIB mac address is 20 bytes long. The address is comprised of GID and QPN; GID is relatively stable while QPN can change across boot. During the init operation, IPoIB driver will try to join the broadcast group. It is expected of any Subnet Manager to create this group before the join operation takes place.

Here is an example of how to configure IPoIB node. IPoIB supports most of the common Subnet Managers. Make sure the driver is present and add_drv'ed.

bash-3.00# modinfo | grep ibd
 79 7bfd8000   8ae4   -   1  ibdm (InfiniBand Device Manager 1.18)
165 7bf66000   7070 232   1  ibd (InfiniBand DLPI Driver 1.17)
bash-3.00# cat /etc/path_to_inst | grep ibd
"/pci@8,600000/pci@1/pci15b3,5a44@0/ibport@1,7fff,ipib" 0 "ibd"
"/pci@8,600000/pci@1/pci15b3,5a44@0/ibport@1,8001,ipib" 1 "ibd"
"/pci@8,600000/pci@1/pci15b3,5a44@0/ibport@2,7fff,ipib" 2 "ibd"
"/pci@8,600000/pci@1/pci15b3,5a44@0/ibport@2,8001,ipib" 3 "ibd"

Partition key 8001 is a valid user level pkey which we can use to plumb the ibd interface. Codewise, the ibd driver init routine will do a query on the broadcast group. On success it records the MTU size and then joins this group. Code shown as follows (only the query part):

static ibt_status_t
ibd_find_bgroup(ibd_state_t *state)
{
	ibt_mcg_attr_t mcg_attr;
	uint_t numg;
	uchar_t scopes[] = { IB_MC_SCOPE_SUBNET_LOCAL,
	    IB_MC_SCOPE_SITE_LOCAL, IB_MC_SCOPE_ORG_LOCAL,
	    IB_MC_SCOPE_GLOBAL };
	int i, mcgmtu;
	boolean_t found = B_FALSE;

	bzero(&mcg_attr, sizeof (ibt_mcg_attr_t));
	mcg_attr.mc_pkey = state->id_pkey;
	state->id_mgid.gid_guid = IB_MCGID_IPV4_LOW_GROUP_MASK;

	for (i = 0; i < sizeof (scopes)/sizeof (scopes[0]); i++) {
		state->id_scope = mcg_attr.mc_scope = scopes[i];

		/*
		 * Look for the IPoIB broadcast group.
		 */
		state->id_mgid.gid_prefix =
		    (((uint64_t)IB_MCGID_IPV4_PREFIX << 32) |
		    ((uint64_t)state->id_scope << 48) |
		    ((uint32_t)(state->id_pkey << 16)));
		mcg_attr.mc_mgid = state->id_mgid;
		if (ibt_query_mcg(state->id_sgid, &mcg_attr, 1,
		    &state->id_mcinfo, &numg) == IBT_SUCCESS) {
			found = B_TRUE;
			break;
		}

	}

	if (!found) {
		ibd_print_warn(state, "IPoIB broadcast group absent");
		return (IBT_FAILURE);
	}

	/*
	 * Assert that the mcg mtu <= id_mtu. Fill in updated id_mtu.
	 */
	mcgmtu = (128 << state->id_mcinfo->mc_mtu);
	if (state->id_mtu < mcgmtu) {
		ibd_print_warn(state, "IPoIB broadcast group MTU %d "
		    "greater than port's maximum MTU %d", mcgmtu,
		    state->id_mtu);
		return (IBT_FAILURE);
	}
	state->id_mtu = mcgmtu;

	return (IBT_SUCCESS);
}

We now plumb the interface, assign an IP address to our interface and start using it.

bash-3.00# ifconfig ibd3 plumb
bash-3.00# ifconfig ibd3 192.168.1.1 up
bash-3.00# ifconfig ibd3
ibd3: flags=1000843 mtu 2044 index 3
        inet 192.168.1.1 netmask ffffff00 broadcast 192.168.1.255
        ipib 0:0:4:9:0:0:0:0:0:0:12:34:0:2:c9:1:8:fe:cc:62
bash-3.00# arp -a
Net to Media Table: IPv4
Device   IP Address               Mask      Flags   Phys Addr
------ -------------------- --------------- ----- ---------------
ibd3   224.0.0.2            255.255.255.255       00:ff:ff:ff:ff:12:40:1b:80:01:00:00:00:00:00:00:00:00:00:02 
bash-3.00# ping -i ibd3 sun.com
sun.com is alive