Crossbow on CMT
Traditional uniprocessors are unable to drive high speed interfaces to link speed. Although there is significant improvement in uniprocessor performance, it is becoming increasingly unviable to push the limits on such uniprocessors. There is a large push towards Chip level multiprocesor architectures such as Niagara from Sun. Crossbow network stack is designed to take advantage of such CMT archictetures. It is based on assigning a group of connections to set of dedicated threads. These threads can then process messages in parallel taking advantage of the CMT architecture. Incoming packets are spread out to multiple rings and worker threads operate on these rings to process the packets. In order to demonstrate this point, I am going to show a simple example of incoming packet fanout as described above. After injecting packets on a Niagara platform, the fanout thread for UDP spreads traffic on 8 softrings. As you can see from the mpstat output below, there are 8 threads processing incoming traffic. In addition, there is a poll thread bringing in packets from the interface and a fanout thread spreading out packets to the softring.
CPU minf mjf xcal intr ithr
csw icsw migr smtx srw syscl usr sys wt idl
0
0 0 77 203 104
0 0 0 0
0 0 0 1
0 99
1 0 0
0 14246 14245 0 0
0 26447 0 0
0 63 0 37
2 0
0 0 1
0 0 0 0
0 0 0 0
0 0 100
3 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
4 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
5 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
6 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
7 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
8 0 0
0 3 0
3 0 0 0
0 3 0 0
0 100
9 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
10 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
11 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
12 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
13 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
14 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
15 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
16 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
17 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
18 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
19 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
20 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
21 0 0 4509 3474
0 6945 0 0 10262
0 0 0 43
0 57
22 0 0 3461
516 0 6922 0 0
32105 0 0
0 100 0 0
23 0
0 0 1630 0 3570
0 0 61742 0
0 0 92 0 8
24
0 0 0 14
0 34 0 0 45917
0 0 0 100 0
0
25 0 0 0
49 0 132 0
0 45610 0 0
0 100 0 0
26 0
0 0 82 0
221 0 0 45120
0 0 0 100 0
0
27 0 0 0
103 0 270 0
0 44182 0 0
0 99 0 1
28 0
0 0 813 0 1926
0 0 60564 0
0 0 96 0 4
29
0 0 0 691
0 1652 0 0 60248
0 0 0 97
0 3
30 0 0
0 611 0 1447 0
0 59956 0 0
0 98 0 2
31 0
0 0 1
0 0 0 0
0 0 0 0
0 0 100
32 1 0
0 2 0
1 0 0 0
0 203 1 0 0
99
33 0 0 0
1 0 0 0
0 0 0
0 0 0 0 100
34
0 0 0 2
1 0 0 0
0 0 0 0
0 0 100
35 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
36 0 0
0 2 0
2 0 0 0
0 0 0 1
0 99
37 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
38 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
39 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
40 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
41 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
42 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
43 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
44 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
45 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
46 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
47 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
48 0 0 42
72 0 143 0
0 0 0
0 0 0 0 100
49
0 0 0 1
0 0 0 0
0 0 0 0
0 0 100
50 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
51 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
52 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
53 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
54 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
55 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
56 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
57 0 0
0 2 0
2 0 0 0
0 0 0 0
0 100
58 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
59 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
60 0 0
0 2 0
3 0 0 0
0 1 0 0
0 100
61 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
62 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
63 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
The interrupt thread signals poll thread on incoming packets. The poll thread disables interrupt and tries to bring in as many packets as it could. This kind of inter-weaving between poll and interrupt continues throughout the system operation. This is a simple illustration of Crossbow utilising CMT architecture to drive network load.
If i'm reading the output correctly, it appears that this is a T2 based system with 8 threads per core (thus 64 "cpu"s). It allso appears that all 8 threads are on the same core (or close to it).
I seem to remember an issue with the network drivers on the T2k when it first came out. The algorithm was to put one nic on one cpu, the next nic on the next cpu, etc. Worked fine on a traditional cpu or muti-core cpu, but with the T1 (and T2) chips, this would cause the 'distribution' of interrupt handlers to be not so optimal. Basically, they all ended up on the same core.
I assume that this has been addressed so that the fanout is a bit more optimal?
Posted by john on March 10, 2008 at 09:22 PM PDT #
This is a T5220 system with 8 cores and 8threads per core. The thread bindings to different cores etc is very flexible and can be controlled through the dladm interface provided by Crossbow. Threads can be bound to any core/thread location. Interrupt assignment on such platform is equally flexible (can be spread out to different cores), though Interrupt path is used sparingly, just to signal the poll thread. Hope that answers your question..
Posted by Nitin on March 13, 2008 at 06:15 PM PDT #