Synchronicity
Random thoughts from a random engineer
Archives
« December 2009
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
  
       
Today
XML
Search

Links
Referrers

Today's Page Hits: 15

All | General | Hacks
« Network Stack Simula... | Main
20090202 Monday February 02, 2009
Flows observability

Crossbow provides the ability to create flows and assign properties such as bandwidth limit and priority to them. The flowadm(1m) manpage has examples and details on how to manage flows.

Once the desired flows are created, you could verify that they are working correctly by running the following command to display per-flow statistics:

flowadm show-flow -s

The output should look something like:

FLOW        IPACKETS  RBYTES      IERRORS OPACKETS  OBYTES      OERRORS
icmpflow    4         392         0       4         392         0
tcpflow     83892     122305400   0       10506     568128      0
udpflow     0         0           0       0         0           0

The above does not tell you which processes are consuming/generating packets. To answer this question, I came up with the following dtrace script:

#!/usr/sbin/dtrace -qs

mac_rx_srs_process:entry
/(((mac_soft_ring_set_t *)args[1])->srs_type & 0x02) != 0/
{
	self->s = 1;
	self->rf = (flow_entry_t *)
	    ((mac_soft_ring_set_t *)args[1])->srs_flent;
}

sched:::wakeup
/self->s/
{
	@inbound[args[1]->pr_fname, self->rf->fe_flow_name] = count();
}

mac_rx_srs_process:return
/self->s/
{
	self->s = 0;
}

mac_flow_lookup:entry
/args[2] == 0x02/
{
	self->tfp = (flow_entry_t **)args[3];
}

mac_flow_lookup:return
/args[1] == 0 && self->tfp != NULL &&
 ((*self->tfp)->fe_type & 0x10) != 0/
{
	@outbound[execname, (*self->tfp)->fe_flow_name] = count();
	self->tfp = NULL;
}

END
{
	printf("Inbound:\n");
	printa("%-20s %-20s\n", @inbound);
	printf("\nOutbound:\n");
	printa("%-20s %-20s\n", @outbound);
}

To use the script, start some traffic-generating processes and leave the script running for a while, then Ctrl-C the script. The output should look like:

Inbound:
sched                tcpflow
in.routed            icmpflow
ping                 icmpflow
netserver            tcpflow
iperf                tcpflow
Outbound:
iperf                tcpflow
sched                icmpflow
ping                 icmpflow
netserver            tcpflow
sched                tcpflow

The script above attempts to generate a mapping between processes and flows. Due to the asynchrony of the network stack, this cannot always be reliably done. The presence of the 'sched' process indicates that part of processing was handled anonymously by a kernel thread (e.g. interrupt/worker thread). 

We will have finer-grained observability features in the next phase of crossbow and hopefully there will be simpler and more reliable ways of obtaining similar information.

 


Feb 02 2009, 12:33:13 AM PST Permalink

Comments:

Post a Comment:

Comments are closed for this entry.