Project Crossbow pre-beta release
Crossbow Flows
Crossbow pre-beta bits can be now obtained from Download. The list of new features in this release are listed in What's New.
In this entry I am going to try and discuss a bit about the administrative aspects of Flows and also list some of the known issues in the released bits (Feb 2008), for more details refer to Release Notes
A link in this text means either a physical NIC or a virtual NIC (VNIC), where necessary I'll call out one or the other. Flows can be created on a link to differentiate packets, on that link, based on the following attributes:
- transport - tcp, udp, sctp, icmpv6
- local_ip[/prefix_len] - Local IPv4/IPv6 address
- remote_ip[/prefix_len] - Remote IPv4/IPv6 address
- local_port - Local port
- remote_port - Remote port
- ip_version - 4 or 6
- maxbw - bandwidth limit
- cpus - CPU fanout list
# flowadm add-flow -l bge0 -a transport=TCP -p maxbw=500Mbps tcp-flow
If a CPU list of 0, 1 and 2 also needs to be specified, the above would change to:
# flowadm add-flow -l bge0 -a transport=TCP -p maxbw=500Mbps,cpus=0,1,2 tcp-flow
where,
- -a : comma separated list of attributes
- -p : comma separated list of properties
# flowadm show-flow tcp-flow
flow name flow attributes policy attributes
-------------------------------------------------------------------
tcp-flow v4:tcp (L) 500 Mbps
The flow restricts TCP traffic on bge0 to 500 Mbps. The flow doesn't affect packets destined for any VNICs created on bge0. Thus another flow with a different name can be created with the same attributes, but different properties on vnic1 that is created on bge0.
The flow can be modified to have a different limit, say 200 Mbps, as# flowadm set-flowprop -p maxbw=200Mbps tcp-flow
(note there is an issue in the released bits that is listed in the release notes and at the end
of this entry about modifying limits).
Statistics for a flow can be observed either by using kstat(1m)
# kstat -c flow -n tcp-flow
or netstat(1m)
# netstat -K
In subsequent releases flowadm show-flow -s [-i] will also display statistics about a flow. Additionally, we provide the facility to log information for flows and use it at a later time. To enable logging we extend acctadm(1m):
# acctadm -e extended -f logfile net
This will enable logging flow statistics into logfile. To retrieve information from logfile:
# flowadm show-usage -f logfile
E.g: (assuming logfile is /tmp/logfile)
# flowadm show-usage -f /tmp/logfile
Bytes Packets Errors Duration Bandwidth Link/Flow
_______________________________________________________
24194 153 0 20 9.68 Kbps tcp-flow
Information about a specific flow
# flowadm show-usage -f logfile flow
E.g:
# flowadm show-usage -f /tmp/logfile tcp-flow
Start Time End Time In Bytes Out Bytes Bandwidth Device
_________________________________________________________
13:40:58 13:41:18 4972 19222 9.68 Kbps tcp-flow
13:41:18 13:41:38 114 216 132.00 bps tcp-flow
Information about the dates for which logs are available:
# flowadm show-usage -d -f logfile flow
E.g:
# flowadm show-usage -d -f /tmp/logfile tcp-flow
02/15/2008
Information about the entries for a given date
# flowadm show-usage -s MM/DD/YYYY -f logfile flow
E.g:
# flowadm show-usage -s 02/15/2008 -f /tmp/logfile tcp-flow
Start Time End Time In Bytes Out Bytes Bandwidth Device
_________________________________________________________
13:40:58 13:41:18 4972 19222 9.68 Kbps tcp-flow
13:41:18 13:41:38 114 216 132.00 bps tcp-flow
13:41:38 13:41:58 60 54 45.60 bps tcp-flow
13:41:58 13:42:18 114 216 132.00 bps tcp-flow
13:42:18 13:42:38 60 54 45.60 bps tcp-flow
13:42:38 13:42:58 114 216 132.00 bps tcp-flow
13:42:58 13:43:18 0 0 0.00 bps tcp-flow
Information given a start time and end time
# flowadm show-usage -s MM/DD/YYYY,H:M:S -e MM/DD/YYYY,H:M:S -f logfile flow
E.g:
# flowadm show-usage -s 02/15/2008,13:40:58 -e 02/15/2008,13:42:58 -f /tmp/logfile
Start Time End Time In Bytes Out Bytes Bandwidth Device
____________________________________________________________
13:40:58 13:41:18 4972 19222 9.68 Kbps tcp-flow
13:41:18 13:41:38 114 216 132.00 bps tcp-flow
13:41:38 13:41:58 60 54 45.60 bps tcp-flow
13:41:58 13:42:18 114 216 132.00 bps tcp-flow
13:42:18 13:42:38 60 54 45.60 bps tcp-flow
13:42:38 13:42:58 114 216 132.00 bps tcp-flow
Generate a plot file for a specific duration
# flowadm show-usage -p -s MM/DD/YYYY,H:M:S -e MM/DD/YYYY,H:M:S -f logfile flow
E.g:
# flowadm show-usage -p -s 02/15/2008,13:40:58 -e 02/15/2008,13:42:58 -f /tmp/logfile
# Time tcp-flow
13:41:18 9.68
13:41:38 0.13
13:41:58 0.05
13:42:18 0.13
13:42:38 0.05
13:42:58 0.13
To deactivate flow logging
# acctadm -x net
The flow can be deleted using
# flowadm remove-flow tcp-flow
The following should be noted about the support for flows in the pre-beta release bits (Feb 2008):
flowadm(1m) cannot be used in the local zone. This is likely to change going forwards.
The link should be plumbed before adding flows
Flows with properties (limit and cpu list) can't be added on a VNIC. Thus flows on VNICs only aide observability. We are currently evaluating this restriction, i.e. permissible properties on flows on VNIC, so this may change going forwards. The reason for this restriction is because the administrator of a flow on a VNIC could be different than the administrator of the VNIC itself (say the VNIC is assigned to a local zone so the VNIC will be administered by the global admin and the flow on the VNIC, by the local zone admin). Since the properties of flows are dependent on the properties of the underlying link, changing properties on the underlying link may implicitly change the properties of all the flows on the link. We'd be very interested in knowing use cases/scenarios for flows on VNICs.
If a physical NIC has flows with properties, modifying the properties of the link (using dladm set-linkprop) requires that the flows be removed or properties on the flows be reset. This is related to 3 above.
Persistence of flow properties is not functional, i.e after a reboot properties on a flow are not retained.
There is an issue while modifying the bandwidth limit of a flow on a link that doesn't have a bandwidth. Modifying the limit for such a flow sets its limit to 0, which means traffic will cease for that flow. The workaround is to set a limit on the underlying link (the limit can be the speed of the link) before adding flows. After this, modifying the limit on flows will work.
