Nitin's Weblog Nitin's Weblog

As you may be aware, Crossbow provides features to allocate resources for network processing using dladm interface. Administrators can now assign a particular set of cpu's on the fly to restrict traffic over a certain interface stay constrained on only this set of cpu's.

I am going to use a T5220 sun4v server with 64 strands for this example. By default when the machine boots up, a default set of cpu's is allocated to the NIC. This allocation is done using a simple round robin scheme. However if the user wishes to change the allocation to a specific set of cpu's, one can do it using dladm interface as follows.

bash-3.2# dladm show-linkprop -p cpus e1000g1
LINK         PROPERTY        PERM VALUE          DEFAULT        POSSIBLE
e1000g1      cpus            rw   --             --             --
bash-3.2#

No user defined set of cpu's assigned as from seen above.

bash-3.2# echo ::interrupts | mdb -k

        Device   Shared  Type    MSG #   State   INO     Mondo    Pil    CPU
      e1000g#4      no   MSI        5    enbl    0x21    0x21       6      0
        nxge#0      no   MSI-X      9    disbl   0x20    0x20       6     63
        nxge#0      no   MSI-X      8    disbl   0x1f    0x1f       6     62
        nxge#4      no   MSI-X      7    disbl   0x1e    0x1e       6     61
        nxge#4      no   MSI-X      6    disbl   0x1d    0x1d       6     60
      e1000g#3      no   MSI        4    enbl    0x1c    0x1c       6      0
      e1000g#2      no   MSI        3    enbl    0x1b    0x1b       6      0
      e1000g#1      no   MSI        2    enbl    0x1a    0x1a       6     18

Interrupt for e1000g1 for the time being associated with CPU 18.

Lets assign a set of cpu's to this NIC.

bash-3.2# dladm set-linkprop -p cpus=14,15,16,17,18,19,20,21 e1000g1
bash-3.2# dladm show-linkprop -p cpus e1000g1
LINK         PROPERTY        PERM VALUE          DEFAULT        POSSIBLE
e1000g1      cpus            rw  14,15,16,17,18,19,20,21 -- --

The above command also retargets the interrupt assigned for the CPU to first
specified CPU in the list. Note, some platforms do not allow retargetting
a single interrupt, retargetting won't work for such platforms.

bash-3.2# echo ::interrupts | mdb -k

        Device   Shared  Type    MSG #   State   INO     Mondo    Pil    CPU
      e1000g#4      no   MSI        5    enbl    0x21    0x21       6      0
        nxge#0      no   MSI-X      9    disbl   0x20    0x20       6     63
        nxge#0      no   MSI-X      8    disbl   0x1f    0x1f       6     62
        nxge#4      no   MSI-X      7    disbl   0x1e    0x1e       6     61
        nxge#4      no   MSI-X      6    disbl   0x1d    0x1d       6     60
      e1000g#3      no   MSI        4    enbl    0x1c    0x1c       6      0
      e1000g#2      no   MSI        3    enbl    0x1b    0x1b       6      0
      e1000g#1      no   MSI        2    enbl    0x1a    0x1a       6     14

All the supporting threads including the interrupt are now confined to the set
of cpu's assigned by the user.  One can reset this static assignment using

bash-3.2# dladm reset-linkprop -p cpus e1000g1
bash-3.2# dladm show-linkprop -p cpus e1000g1
LINK         PROPERTY        PERM VALUE          DEFAULT        POSSIBLE
e1000g1      cpus            rw   --             --             --

Above was a simple example of using Crossbow feature to perform resource allocation for network processing.