Crossbow on CMT
Traditional uniprocessors are unable to drive high speed interfaces to link speed. Although there is significant improvement in uniprocessor performance, it is becoming increasingly unviable to push the limits on such uniprocessors. There is a large push towards Chip level multiprocesor architectures such as Niagara from Sun. Crossbow network stack is designed to take advantage of such CMT archictetures. It is based on assigning a group of connections to set of dedicated threads. These threads can then process messages in parallel taking advantage of the CMT architecture. Incoming packets are spread out to multiple rings and worker threads operate on these rings to process the packets. In order to demonstrate this point, I am going to show a simple example of incoming packet fanout as described above. After injecting packets on a Niagara platform, the fanout thread for UDP spreads traffic on 8 softrings. As you can see from the mpstat output below, there are 8 threads processing incoming traffic. In addition, there is a poll thread bringing in packets from the interface and a fanout thread spreading out packets to the softring.
CPU minf mjf xcal intr ithr
csw icsw migr smtx srw syscl usr sys wt idl
0
0 0 77 203 104
0 0 0 0
0 0 0 1
0 99
1 0 0
0 14246 14245 0 0
0 26447 0 0
0 63 0 37
2 0
0 0 1
0 0 0 0
0 0 0 0
0 0 100
3 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
4 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
5 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
6 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
7 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
8 0 0
0 3 0
3 0 0 0
0 3 0 0
0 100
9 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
10 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
11 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
12 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
13 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
14 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
15 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
16 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
17 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
18 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
19 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
20 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
21 0 0 4509 3474
0 6945 0 0 10262
0 0 0 43
0 57
22 0 0 3461
516 0 6922 0 0
32105 0 0
0 100 0 0
23 0
0 0 1630 0 3570
0 0 61742 0
0 0 92 0 8
24
0 0 0 14
0 34 0 0 45917
0 0 0 100 0
0
25 0 0 0
49 0 132 0
0 45610 0 0
0 100 0 0
26 0
0 0 82 0
221 0 0 45120
0 0 0 100 0
0
27 0 0 0
103 0 270 0
0 44182 0 0
0 99 0 1
28 0
0 0 813 0 1926
0 0 60564 0
0 0 96 0 4
29
0 0 0 691
0 1652 0 0 60248
0 0 0 97
0 3
30 0 0
0 611 0 1447 0
0 59956 0 0
0 98 0 2
31 0
0 0 1
0 0 0 0
0 0 0 0
0 0 100
32 1 0
0 2 0
1 0 0 0
0 203 1 0 0
99
33 0 0 0
1 0 0 0
0 0 0
0 0 0 0 100
34
0 0 0 2
1 0 0 0
0 0 0 0
0 0 100
35 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
36 0 0
0 2 0
2 0 0 0
0 0 0 1
0 99
37 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
38 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
39 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
40 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
41 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
42 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
43 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
44 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
45 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
46 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
47 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
48 0 0 42
72 0 143 0
0 0 0
0 0 0 0 100
49
0 0 0 1
0 0 0 0
0 0 0 0
0 0 100
50 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
51 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
52 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
53 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
54 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
55 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
56 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
57 0 0
0 2 0
2 0 0 0
0 0 0 0
0 100
58 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
59 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
60 0 0
0 2 0
3 0 0 0
0 1 0 0
0 100
61 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
62 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
63 0 0
0 1 0
0 0 0 0
0 0 0 0
0 100
The interrupt thread signals poll thread on incoming packets. The poll thread disables interrupt and tries to bring in as many packets as it could. This kind of inter-weaving between poll and interrupt continues throughout the system operation. This is a simple illustration of Crossbow utilising CMT architecture to drive network load.