</head> <body> <h2> Crossbow - Per Rx ring Dynamic Polling</h2> <b>As before, notice that mdb shows the 8 interrupts are bound at CPUs 38-45</b> <pre> > ::interrupts Device Shared Type MSG # State INO Mondo Pil CPU emlxs#1 no MSI 17 enbl 0x28 0x28 6 11 emlxs#1 no MSI 16 enbl 0x27 0x27 6 10 emlxs#0 no MSI 5 enbl 0x24 0x24 6 46 emlxs#0 no MSI 4 enbl 0x23 0x23 6 47 ixgbe#0 no MSI-X 15 enbl 0x22 0x22 6 36 ixgbe#0 no MSI-X 14 enbl 0x21 0x21 6 37 ixgbe#0 no MSI-X 13 enbl 0x20 0x20 6 38 ixgbe#0 no MSI-X 12 enbl 0x1f 0x1f 6 39 ixgbe#0 no MSI-X 11 enbl 0x1e 0x1e 6 40 ixgbe#0 no MSI-X 10 enbl 0x1d 0x1d 6 41 ixgbe#0 no MSI-X 9 enbl 0x1c 0x1c 6 42 ixgbe#0 no MSI-X 8 enbl 0x1b 0x1b 6 43 e1000g#1 no MSI 2 enbl 0x1a 0x1a 6 44 e1000g#0 no MSI 1 enbl 0x19 0x19 6 45 ohci#1 no Fixed --- enbl 0x17 0x17 9 50 ohci#0 no Fixed --- enbl 0x16 0x16 9 51 ehci#0 no Fixed --- enbl 0x14 0x14 9 52 mpt#0 no MSI 0 enbl 0x18 0x18 4 48 px#0 no PCIe 27 enbl 0x3b 0x3b 1 53 px#0 no PCIe 51 enbl 0x3a 0x3a 14 54 px#0 no PCIe 49 enbl 0x39 0x39 14 55 px#0 no PCIe 48 enbl 0x38 0x38 9 56 </pre> <b>And if you look at the mpstat below, you will notice that load is getting very evenly spread out. The intrstat still shows the same CPUs taking interrupts but interrupt per second are much lower even though the load serviced is twice the load of no dynamic polling case. You can also see that netstat shows a much larger packets per second that we are servicing. </b> <pre> 0 0 1 651 1544 28 1505 20 494 457 0 6180 13 26 0 61 1 0 1 420 1382 35 1456 20 438 428 0 4903 13 20 0 67 2 0 1 379 1200 31 1273 14 369 387 0 4007 12 16 0 72 3 0 1 345 1006 26 1162 11 330 369 0 3641 11 15 0 74 4 0 1 369 822 31 1180 14 343 400 0 5037 11 20 0 69 5 0 1 334 766 24 1122 14 308 350 0 3465 11 14 0 75 6 0 0 315 696 22 1058 13 278 332 0 2862 10 12 0 78 7 0 0 1210 1250 15 2255 160 287 743 0 904 4 32 0 64 8 0 0 1310 1375 30 2600 190 400 1041 0 1355 5 36 0 60 9 0 0 2851 2561 31 5707 432 551 678 0 1659 7 15 0 78 10 0 0 1880 3183 25 9493 577 525 683 0 1341 6 18 0 77 11 0 0 1217 1151 16 2164 136 248 1121 0 757 3 35 0 62 12 0 0 1116 1144 23 2194 148 308 1060 0 1063 4 33 0 63 13 0 0 2681 2271 22 5266 369 440 601 0 1401 6 13 0 80 14 0 0 1781 2944 21 8937 507 468 646 0 1210 6 17 0 77 15 0 0 1224 1171 14 2288 150 258 1009 0 778 4 32 0 64 16 0 0 1247 1308 27 2599 178 389 1016 0 1162 5 34 0 61 17 0 0 2925 2397 27 5611 398 504 663 0 1490 7 14 0 79 18 0 0 1818 3083 22 9400 544 484 686 0 1225 6 17 0 77 19 0 0 1211 1218 17 2403 149 268 915 0 832 4 29 0 67 20 0 0 1245 1140 21 2265 144 277 1166 0 787 4 37 0 59 21 0 0 2796 2272 22 5442 352 412 626 0 1275 6 13 0 81 22 0 0 1851 2969 21 9468 509 453 672 0 1172 5 17 0 77 23 0 0 1132 1075 14 2143 133 243 959 0 785 4 30 0 66 24 0 0 1211 1194 30 2390 159 367 1053 0 1280 4 36 0 60 25 0 0 2629 2212 30 5288 362 499 641 0 1537 7 14 0 79 26 0 0 1697 2760 25 8730 485 455 676 0 1181 5 17 0 78 27 0 0 1195 1185 19 2394 167 284 834 0 929 4 23 0 73 28 0 0 4363 1174 23 2351 166 285 988 0 866 4 33 0 63 29 0 0 2818 2298 24 5509 389 402 620 0 1184 5 13 0 82 30 0 0 1866 3003 23 9650 574 446 681 0 1117 5 16 0 79 31 0 0 1034 938 15 1859 113 212 1053 0 692 3 34 0 63 32 0 0 1077 1077 29 2144 136 343 1052 0 1132 4 36 0 60 33 0 0 2254 1944 28 4754 308 464 602 0 1501 6 14 0 80 34 0 0 1522 2328 22 7706 408 418 625 0 1171 5 17 0 78 35 0 0 792 972 17 2003 108 217 1009 0 742 4 31 0 66 36 0 0 854 977 19 2019 111 259 1144 0 800 3 37 0 60 37 0 0 2507 11906 10272 4267 265 326 489 0 776 4 28 0 68 38 0 0 2111 11793 9840 6503 336 314 472 0 615 3 32 0 65 39 0 0 500 10409 10164 565 7 125 174 0 1413 6 23 0 70 40 0 1 660 10423 9982 950 23 288 272 0 3834 8 34 0 58 41 0 1 658 10490 10108 847 16 238 237 0 2549 8 29 0 64 42 0 0 584 10605 10299 708 12 181 207 0 1828 7 26 0 67 43 0 0 732 10829 10559 598 9 141 193 0 1485 7 25 0 68 44 0 1 306 487 25 1091 17 282 330 0 4083 9 17 0 74 45 0 0 269 410 25 973 13 219 289 0 2792 8 12 0 80 46 0 0 623 4087 3833 639 8 136 336 0 1795 6 24 0 69 47 0 0 242 571 268 802 7 166 266 0 2135 8 9 0 83 48 0 1 364 601 34 1282 19 384 425 0 5724 9 24 0 67 49 0 1 314 512 28 1153 13 304 376 0 4088 9 17 0 74 50 0 0 277 404 22 959 10 235 306 0 3019 8 12 0 80 51 0 0 254 340 17 847 6 196 274 0 2216 8 9 0 83 52 0 1 606 502 27 1111 21 291 392 0 4770 9 20 0 72 53 0 1 266 410 23 960 13 220 327 0 3040 8 12 0 79 54 0 0 247 345 18 856 9 182 284 0 2542 8 10 0 82 55 0 0 228 321 16 818 7 168 266 0 2257 8 9 0 84 56 0 1 367 604 31 1291 21 380 433 0 5658 9 27 0 64 57 0 1 327 497 23 1115 13 298 354 0 3899 9 16 0 75 58 0 0 275 389 18 942 10 232 325 0 2776 8 11 0 80 59 0 0 240 337 15 853 9 193 276 0 2382 8 9 0 83 60 0 1 317 505 20 1128 21 291 391 0 4502 9 18 0 73 61 0 0 277 382 16 924 11 211 310 0 3148 9 13 0 78 62 0 1 244 312 14 793 7 171 283 0 2532 8 10 0 82 63 0 0 232 294 14 754 6 158 265 0 2423 8 9 0 83 device | cpu36 %tim cpu37 %tim cpu38 %tim cpu39 %tim -------------+------------------------------------------------------------ e1000g#0 | 0 0.0 0 0.0 0 0.0 0 0.0 e1000g#1 | 0 0.0 0 0.0 0 0.0 0 0.0 ehci#0 | 0 0.0 0 0.0 0 0.0 0 0.0 emlxs#0 | 0 0.0 0 0.0 0 0.0 0 0.0 ixgbe#0 | 0 0.0 15714 22.0 15748 20.0 13714 13.8 mpt#0 | 0 0.0 0 0.0 0 0.0 0 0.0 device | cpu40 %tim cpu41 %tim cpu42 %tim cpu43 %tim -------------+------------------------------------------------------------ e1000g#0 | 0 0.0 0 0.0 0 0.0 0 0.0 e1000g#1 | 0 0.0 0 0.0 0 0.0 0 0.0 ehci#0 | 0 0.0 0 0.0 0 0.0 0 0.0 emlxs#0 | 0 0.0 0 0.0 0 0.0 0 0.0 ixgbe#0 | 16598 21.1 13578 14.4 15103 19.9 15895 24.7 mpt#0 | 0 0.0 0 0.0 0 0.0 0 0.0 # netstat -ia 1 input e1000g output input (Total) output packets errs packets errs colls packets errs packets errs colls 13647 0 2168 0 0 650722196 0 1316178160 0 0 2 0 1 0 0 267619 0 522226 0 0 2 0 2 0 0 275395 0 539920 0 0 2 0 2 0 0 251023 0 482335 0 0 </pre> <b>And finally below we print some stats from the MAC per Rx ring data structure (mac_soft_ring_set_t). For each Rx ring, we track the number of packets received via interrupt path, number received via poll path, chains less than 10, chains between 10 and 50 and chains over 50 (each time we polled the Rx ring). <pre> > 60036e08680::print mac_soft_ring_set_t srs_rx |::print mac_srs_rx_t sr_poll_count sr_intr_count sr_chain_cnt_undr10 sr_chain_cnt_10to50 sr_chain_cnt_over50 sr_poll_count = 0xbf8a16 sr_intr_count = 0x6e535 sr_chain_cnt_undr10 = 0x8d38a sr_chain_cnt_10to50 = 0x14e57 sr_chain_cnt_over50 = 0x117fc > 60036e08080 sr_poll_count = 0xc25280 sr_intr_count = 0x7a05b sr_chain_cnt_undr10 = 0xc860a sr_chain_cnt_10to50 = 0x1d160 sr_chain_cnt_over50 = 0x1292b > 600391518c0 sr_poll_count = 0xc446a8 sr_intr_count = 0x7162e sr_chain_cnt_undr10 = 0xaaea8 sr_chain_cnt_10to50 = 0x184d5 sr_chain_cnt_over50 = 0x12649 > 600391512c0 sr_poll_count = 0xb8cd71 sr_intr_count = 0x780e9 sr_chain_cnt_undr10 = 0xbf2e8 sr_chain_cnt_10to50 = 0x1c395 sr_chain_cnt_over50 = 0x116b5 > 60039150cc0 sr_poll_count = 0xb2f94b sr_intr_count = 0x8d0a8 sr_chain_cnt_undr10 = 0xd03d5 sr_chain_cnt_10to50 = 0x1b7ff sr_chain_cnt_over50 = 0x10635 > 600391506c0 sr_poll_count = 0xc5ec32 sr_intr_count = 0x75df5 sr_chain_cnt_undr10 = 0xc169a sr_chain_cnt_10to50 = 0x1c589 sr_chain_cnt_over50 = 0x11eda > 600391500c0 sr_poll_count = 0xc0d01b sr_intr_count = 0x7b654 sr_chain_cnt_undr10 = 0xb1d3d sr_chain_cnt_10to50 = 0x182c4 sr_chain_cnt_over50 = 0x1129f > 60039155900 sr_poll_count = 0xbcf2f8 sr_intr_count = 0x8d596 sr_chain_cnt_undr10 = 0xca42b sr_chain_cnt_10to50 = 0x1b2f1 sr_chain_cnt