A Short Overview to PEF(Packet Event Framework)

The FireEngine networking stack is widely known as a cutting-edge technology shipped in Solaris 10, a technical white paper is available in opensolaris community. Actually it is still moving forward and the Packet Event Framework is one of those innovations.

PEF Event List

PEF expands the FireEngine packet classification architecture, it allows different protocol processing functions to link together as a sequence of events, each connection has its own event list and each event in this list is executed by the framework. With this optimized event list, we can see improvement in the following area:

• Code locality: While looking at the current TCP/IP stack, we can find some very large functions that runs the whole TCP state machine, for example tcp_rput_data(). This leads to poor code locality because every packet has to walk through these long functions, so we can split such functions into fine grain PEF events and alter the event list on the fly.
• Networking observability: It's quite easy to insert a certain event into PEF event list to trace the packet passing thought the stack, the insertion and removal of such observing events is quite light weighted.
• STREAMS interaction: FireEngine switches the Solaris networking stack from a message passing-based(STREAMS style) interface to a function call-based(BSD style) interface, PEF makes a further move, it removes the putnext() in TCP RX code path and wraps strrupt() into a PEF event.

Now, having enough knowledge about PEF event, let's move to a real life event list in the current project. In the current networking stack we have following following protocol processing functions in TCP RX code path: ip_tcp_input(), tcp_rput_data() and strrput(), which is hiding behind a putnext(), PEF wraps these three functions into a coarse grained event list, that is, per layer per event.

Network stack parallelism on CMT

CMT introduces multi-core and multiple hardware threads per CPU core, it can significantly increase system throughput by parallel processing, however, it brings impact to the current FireEngine networking stack especially on single connection throughput, that's because the current stack uses a per-CPU synchronization mechanism called vertical perimeter(aka, squeue), the vertical perimeter ensures data locality by processing the same connection in the same CPU whenever possible, so the utilization of parallel processing is limited.

PEF explores the parallelism of network stack on CMT. The events in PEF event list is executed one by one within an arbitrary vertical perimeter, thus we can achieve different parallel models by different means. One of the ideas is to dedicate some network tasks(such as TCP protocol processing) to a given CPU core or hardware thread by assigning the proper vertical perimeter to the PEF
event. For example, if we have four events in a PEF event list, and each of the event has a vertical perimeter binding to different CPU cores, then the function inside this event will be executed on different CPU cores in a pipeline fashion.

发表于 yu [Solaris] ( 六月 05, 2006 11:10 上午 ) Permalink | 评论 [2]