Random ramblings

Tuesday December 16, 2008

Choosing better BGP MRAI values

The IETF IDR working group is hopefully in the process of deprecating the default MRAI values given in [RFC4271]. The below text is from a defunct draft I wrote on what is known about the MRAI, with minor modifications to reflect some of the discussion on IDR. It suggests that 5s would be a better value for the general Internet, BGP MRAI (1s or lower for iBGP). Posted here for archival purposes.

Background

The proper functioning of the [BGP] routing protocol is of great importance to the Internet. Issues regarding matters of its stability and convergence have been documented widely, such as in [BGP-STAB], [bgp-converge] and [Potaroo0607].

One such issue is the effect of 'Minimum Route Advertisement Interval' (MRAI).

The MRAI Timer

The Minimum Route Advertisement Interval (MRAI) timer is specified in [RFC4271]. This timer acts to rate-limit updates, on a per-destination basis. [BGP] suggests values of 30s and 5s for this interval for eBGP and iBGP respectively. The MRAI must also be applied to withdrawals according to [RFC4271], a change from the earlier RFC1771.

Some implementations apply this rate-limiting on a per-peer basis, presumably an adequate approximation. Some implementations apply it to withdrawal methods (often called "WRATE" in the literature). Some implementations do not apply MRAI at all.

Known effects of the MRAI timer on convergence

The MRAI timer serves to suppress messages which BGP would otherwise send out to describe transitory states, and so allow BGP to converge with significantly fewer messages sent. This beneficial effect of the MRAI timer, in terms of # of messages, increases as the timer is increased until an optimum value is reached, after which the beneficial effect stabilises. [bgp-converge] [mrai-final]

In terms of convergence time, a similar beneficial effect is seen as the MRAI increases to near the same optimum value. However as the timer value is increased past this point, the convergence time increases again linearly. The scale of this increase is significantly worse with WRATE, i.e. applying the MRAI to withdrawals has an adverse effect on BGP convergence time. [bgp-converge] [mrai-final]

The optimum MRAI timer value is dependent on several factors, most particularly the topology in its layout and propagation times. The optimum value will differ between different subsets of the Internet. [mrai-final]

It is believed to be infeasible to try directly calculate this value. However a useful approximation can be made from the diameter of the topology if it is known, along with some assumptions about the the topology, such as the latency between nodes. [mrai-internet]

The interaction between extensions to BGP designed to improve convergence, such as those that allow propagation of additional and/or backup paths, and the MRAI timer is as yet unknown. However, it seems reasonable to speculate these extensions might have the effect of leading to a lower optimum MRAI than would be indicated by an approximation based on the diameter of the BGP topology. Further work on these questions would be useful.

Interaction with Flap-Damping

As the MRAI helps eliminate some updates, it interacts with flap-damping. The lower the MRAI timer, the greater the risk of crossing below the threshold of the optimum value. If that threshold is crossed, there will be an increased number of updates somewhere within the BGP system, and hence an increased risk of paths being dampened which otherwise would not.

So, in presence of significant flap-damping deployment and given the uncertainty of what the optimum is, it is reasonable to err towards selecting a value of the MRAI timer significantly higher than the optimum.

However, given that flap-damping increasingly is discouraged [RIPE-378] in Internet routing, this particular need to be conservative in the choice of MRAI timer value may be less important.

Current Status of the MRAI

The current recommended value of 30s may be far higher than is optimal for the Internet, based on observations of certain parameters related to its topology. In [mrai-final] it is suggested that the optimal value may be between 5s ('semi-safe') to 15s ('safe'). The estimation of the 'safe' value here is of no relevance if WRATE is universally deployed, as in such a case the 'semi-safe' value will be the same as the 'safe' value. Further empirical work by the same authors [mrai-internet] suggests that the optimal, Internet MRAI may be below 5s.

Further, [BGP-STAB] and [Potaroo0607] argue that operational conditions (e.g. different routers using different MRAI values) mean the MRAI is having an adverse effect even on the number of messages sent, and so further exacerbating convergence problems in the global BGP system, such as path hunting. The [BGP-STAB] document goes further still and argues that MRAI be deprecated in favour of some better way of damping BGP UPDATES, however there are no clear proposals before the IDR as of this writing for such changes to BGP.

Risk Evaluation in the Choice of MRAI Time

Though there is an optimum value for the MRAI, it's unlikely that it can be determined empirically or otherwise for the general Internet. It may even not be possible, as the optimum MRAI will differ for different subsets of the Internet. Some degree of guesstimation at a reasonable value for the MRAI is required, which is an exercise in risk; whether to err towards fast convergence at the risk of a disproportionate increase in BGP messaging, or to err to the side of an optimal number of messages at the expense of convergence.

Arguably, economising on bandwidth and control-plane processing power is of less importance than the convergence time of BGP, compared to times past. Presuming this, any new recommendations for the MRAI should seek to err slightly to the side of convergence, rather than erring towards minimising BGP traffic.

Further, if we assume most implementations apply the MRAI to withdrawals, then the Internet BGP topology effectively is WRATE-enabled, and [mrai-final] suggests there is even less benefit to erring toward a higher MRAI.

There may be risks in mismatched MRAI values between speakers in an AS as revised MRAI values are deployed. The MRAI values in [RFC4271] were deliberately specified to be lower for iBGP than for eBGP, so as to allow interior routing to converge while minimising the effect on eBGP state. So with mismatched values there is an increased risk that the external stability of an AS's routes would be affected by transient, internal states.

This last risk suggests that the existing iBGP/VPN values should be the lower-bound for any conservative revision of the eBGP MRAI value.

The most definite risk of lowering the MRAI is the increased risk of flap-damping, if the value is set too much below the optimum. Therefore, taking into account estimations of that optimum is required. That said, at least one BGP implementation by default does not apply any MRAI at all.

Recommendations on the MRAI

The suggested default values for the MinRouteAdvertisementIntervalTimer given in [RFC4271] are almost certainly too high. The author agrees with [mrai-internet] and suggests that values 5s or less for eBGP connections, and 1s or less for iBGP connections, would be more suited to Internet topologies. Opinions here differ, some think these values err too low for safety and others think the MRAI timer should be removed altogether.

These values may not be suitable for topologies which differ from the Internet, be that in scale, arrangement or otherwise. Such non-Internet, BGP topologies likely would have lower optimum values, assuming they are always significantly smaller in scale than the Internet BGP topology.

Acknowledgements

The author would like to thank Manav Bhatia for his helpful review and comments; as well as Robert Raszuk, Samita Chakrabarti, Danny McPherson and Jeffrey Haas for their useful comments; dissenting or otherwise.

The authors of the cited documents are thanked for their contributions to the understanding of BGP, of which this document is a simple summary.

References

BGP: A Border Gateway Protocol 4 (BGP-4) Y. Rekhter, T. Li and S. Hares, January 2006.

BGP-STAB: BGP Stability Improvements, T. Li and G. Huston, June 2007.

BGP-DAMP: BGP Route Flap Damping, C. Villamizar, R. Chandra and R. Govindan, Nov. 1998.

Potaroo0607: Damping BGP G. Huston, June 2007.

RIPE-378: RIPE RRG: Recommendations on Route-flap Damping P. Smith, C. Panigl, May 2006.

bgp-converge: An Experimental Analysis of BGP Convergence Time T. Griffin and B. Premore, Nov. 2001, Proc. of ICNP pages 53-61.

mrai-final: An Experimental Study of the BGP Rate-limiting Timer J. Qui, R. Hao and X. Li, June 2003, Bell Labs Technical Memo ITD-03-44604H.

mrai-internet: The Optimal Rate-Limiting Timer of BGP for Routing Convergence J. Qui, R. Hao and X. Li, IEICE TRANS. Comm. Vol.E88–B, No. 4.

Monday August 25, 2008

BGP Path Hunting

Path-hunting is the tendency of BGP, when a path is withdrawn by a remote AS, to "hunt" from one path to another, before finally converging. The problem has been described elsewhere, particularly by Geoff Huston (e.g. in this ISOC column on BGP dampening). Manav Bhatia also has a nice, graphic explanation of BGP path hunting. Tony Li and Geoff Huston additionally have authored a very interesting draft on the problem, draft-li-bgp-stability - which analogises BGP convergence with wave-front propogation, where the "front" of convergence bends with differences in path-propogation times, and expands out where ASes connect with multiple others.

This problem is very similar to[5], if not a special-case of, the well-known "count to infinity" problem inherent to the Bellman-Ford, distance-vector protocols. With infinity being constrained, by BGPs built-in split-horizon behaviour, to the number of simple paths in the topology.

Path Hunting overview

In short: given some converged subset of the BGP topology, with A and B connected by a set of one or more distinct paths {1, ..., n}, with the time taken for a message to propogate down path x of t[x] so that the set of paths has a directly corresponding set of propogation times T = {t[0], ..., t[n]}. Then for any announcement which passes through A to B, BGP will reach a "useful convergence"[1] by or within a time of:

• For an UPDATE announcing a new/updated path: minimum(T)
• For a withdrawal: maximum(T)

The latter convergence time being due to the 'hunting' behaviour of BGP, and indeed consistent with the well-known "Good news travels quickly, bad news travels slowly"[3] behaviour of distance-vector protocols.

E.g. imagine A announces some prefix to C, and imagine (for simplicity) that B does not re-announce paths to E or H, with simple paths between A and B of {ABC,ACDEB,ACDFGHB,ACDFGEB,ACDEGHB}. I.e.:

-A--C----------B
     \        / \
      D------E   H
       \      \ /
        F------G

So after a time of minimum(T), B will have received at least one UPDATE corresponding to a valid path and would be able to forward packets toward A for the prefix. Though the system is not converged, any further UPDATEs B receives can only cause it to select a better path, so the system is at least "usefully-converged" - it can route packets.

After a time of maximum(T), the state at each speakers will be fully converged, and B may then have the following paths in its RIB (arbitrarily making assumptions about preferences speakers may have), for whatever prefix it is, from among which B may pick its preferred path:

• From C, Path: CA
• From E, Path: EDCA
• From H, Path: HGFDCA

Note that B does not know about the ACDFGEB or ACDEGHB paths, however if the DE edge were withdrawn, E potentially could switch to G (and issue an update to B when it does so) and similarly with the FG edge and latter path. B can pick anyone of the above as its preferred path. I.e. though these paths are hidden from B, they can still come into play.

Imagine A now withdraws the prefix from C - so the connectivity to the prefix is now lost as far as the rest of the graph goes. After a time of minimum(T) B will receive at least one message (that might be a withdrawal from C, but it could just as well be an UPDATE from E as E hunts its preferred path over to G). Imagine it happens to come from the peer which it had selected as best, and results in B picking sa new best path via another peer (e.g. because its a withdrawal, or an UPDATE with less favourable path-attributes). Then the next update arrives from the peer newly preferred, and so B picks again, and so on. So B's best-path may well change between the following best-paths:

1. From E, Path EDCA
2. From C, Path CA
3. From H, Path HGFDCA
4. From H, Path HGFDCA
5. From E, Path EGFDCA

I.e. before B can converge on "there is no path", it potentially will first "hunt" through various paths that are already dead[2]. Yet, B *did* receive a message which was triggered by As withdrawal - several in fact! If only B could use the first message to see that, as it was A which withdrew the prefix, that therefore all paths via A must be dead, then it could short-circuit the whole hunting process!

Possible Solutions

Huston and Li propose various possible changes to BGP to mitigate hunting, such as removing the MRAI timer, band-stop filtering of updates, path-exploration damping, etc, all of which are implementable without change to the BGP protocol itself. Other proposals include Brian Dickson's second-best-backup proposal which seeks to speed up convergence by providing visibility of additional paths besides the best-path (such as the ACDFGEB path which is initially hidden to B in the example above), and also to provide a fast-path for withdrawals (this proposal requires changes to BGP). None of these proposals actually solve path-hunting, they instead try to either filter it out and/or speed it up.

Hunting can eliminated in BGP if it could be extended with mechanisms to allow BGP to recognise nodal changes in state, across paths. Other DV protocols (DSDV, AODV) solve similar problems by numbering messages/routes with a sequence-number. This sequenced-DV approach is problematic with BGP as it tends to require either that routers in an AS somehow maintain a shared counter, or that BGP must describe a more detailed router-router topology (rather than todays abstract AS-AS topology).

Some possible mechanisms that eliminate hunting:

• BGP-RCN

  A possible means to allow for convergence in O(minimum(T)) for all kinds of state-changes is described by Pei, Azuma, Massey and Zhang in "BGP-RCN: improving BGP convergence through root cause notification". Speakers include an additional attribute of information with updates/withdrawals, called the "Root Cause Notification" consisting of a list of (ASN,sequence number) tuples. This allows a receiving speaker to compare announcements received via different paths and determine relative recency, so allowing it to always act on the most recent information.

  The sequence number dependency is a significant barrier to deployment.

• AS Poisoning[4]

  This is essentially a simplified version of BGP-RCN, that does away with the problematic sequence number. When withdrawing a prefix, a speaker can include information that "poisons" nodes (or even edges). E.g. if A sends some kind of "Poison: A" attribute with its withdrawal, and if that attribute will propogate along with the withdrawal, then B can conclude from the first withdrawal it receives that all paths that go via A are defunct. BGP can now converge on a prefix-withdrawn state in minimum(T). The downside being that convergence to the prefix-announced state instead occurs in maximum(T) - though it is possible that BGP can be "usefully converged" prior to that, which would be a more useful state than path-hunting.

  So we can "fix" path-hunting, and maximise withdrawal convergence without introducing any co-dependent state with only mild changes to BGP, though at some expense to announcement convergence.

A further observation to be made is that IGP routing has switched from distance-vector to link-state (or "map-distribution") because of the same convergence limitations. Further, there are proposals for BGP security extensions which overlay a link-stateish topology onto BGP, and the BGP-RCN approach is not far from a link-state protocol either. So it is perhaps not completely unreasonable to imagine whether some kind of topological-state mechanism could be bolted onto BGP, that could be used to augment the path-selection process.

So...

Solutions to the hunting problem either are imperfect local-hueristics (which may even make convergence worse in cases where they fail), or protocol extensions with fairly significant trade-offs (either in complexity or adverse side-effects). It seems unlikely there's any magic bullet, given the length of experience we have with Bellman-Ford/DV protocols.

--

Corrections and comments would be appreciated.

Update: Added footnote 5.

1. "Useful convergence" being where speakers all have converged on some particular state, OR are in a certain semi-converged state such that any further messages still propogating can only cause speakers to transition their routes from an existing, valid path to another valid path (this can not be the case if there are any withdrawal messages still to propogate). I.e. forwarding remains loop and blackhole free.

2. This is a problem as it means that if, outside of the subset we're consider, B has other, less-preferred connections, then B will have to wait max(T) before it can decide there are no valid paths in this subset and start to consider paths outside of this subset.

3. Does anyone know the original source of this? (Douglas Comer, "Internetworking with TCP/IP", Prentice-Hall, 1991, seems to quote it).

4. Described only in private correspondence, to date.

5. In the sense that the cause of the count-to-infinity problem was due to the source event of a routing update being indeterminable. Split-horizon cut-out the count-infinity problem for direct loops between neighbours, by preventing updates from propogating endlessly. The addition of a record-of-path (e.g. AS_PATH in BGP) solved count-infinity for multi-hop loops, by preventing updates from propogating along any but simple paths. The remaining problem is of updates travelling along multiple, parallel paths with different propogation times, preventing us from recognising which message received is the most recent - fixed with sequence numbers in some DV protocols. Each modification has pared down DV protocols slightly, restricting where valid information can flow and how it is recognised.

Sunday August 10, 2008

Wholesale DSL reform: Possibilities...

Following on from my, somewhat rambling, post on wholesale DSL, here I'll speculate on possible reforms that could be made.

• Micro-peering.

  Alluded to as one of the "difficult to do" solutions. It's not at all viable today, but its interesting to consider what improvements would need to be made to internet protocols for this option to at least be possible. E.g. BGP is extremely management intensive. Even simple, non-protocol changes like better defaults could make a difference here. In terms of protocol, things like in-band transfer of policy information and auto-configuration of neighbours could simplify management greatly.

• Allow the telco's own ISP to switch IP locally, where it wishes.

  In other words, allow the telco-ISP run a single IP data-plane at the head-end for its own DSL customers, so allowing local switching of traffic within an exchange, if the telco-ISP so desires. At the moment, regulations would tend to prevent the telco discriminating between its wholesale-ISP customers and its own ISP in such a way. If it's the case that there are enough trunk-link bandwidth savings to be made to outweigh the initial costs of re-engineering and further additional managements costs, then it'll happen (at which point there'll be the tricky problem of what to do about the wholesale-ISPs not enjoying this advantage). If not, we'll know it doesn't matter - no harm done. I.e. loosen the regulations so that we can at least allow one player to obtain data on whether we need to re-think whole-sale DSL..

  This probably wouldn't work in the UK, where BT appears pretty well divided up. Might still work in Ireland, where Eircom appears still to be a single entity.

• Consolidate network layer responsibility in one company

  The benefits of the wholesale-DSL model in terms of the economic-efficiency of the network are actually negative[2]; both in terms of the congruence of IP/geography and in terms of the significant additional complexity and points-of-failure[1], which incur direct costs (engineering and sustaining) and indirect costs (more downtime for customers). Further, managing an IP network is well-understood, and the reliability of the whole-sale ISPs is reliant on the operational competence of the telco.

  The benefits of the competition introduced by the wholesale-DSL model therefore must lie primarily in customer-facing services. Therefore we could reform things by allowing the telco to manage the entire network stack (terminating PPP where it wished, assigning IPs, etc..), while leaving customer-services and further value-added services (domains, email accounts, etc..) to the "ISPs".

1. E.g. in Ireland, the few (4 odd?) "BRAS" routers operated by Eircom, which exist to dispatch PPPoE sessions to the various ISPs, via L2TP/IP. The wholesale-ISP must operate their own access-routers in addition to the "BRAS" - with obvious implications for failure rates. In a simpler model, such dispatching wouldn't be needed and PPP sessions could be terminated on cheaper (and so more numerous and perhaps more widely distributed) access-routers.

2. As per previous post: determined by comparison with single-operator networks.

Saturday August 09, 2008

Wholesale DSL: Economics of Scenic Routing

In Ireland and the UK, due to the bulk of the POTS plant being owned and managed by the incumbent, former state telco, DSL tends to be provisioned on a wholesale model, where the incumbent telco is obliged through regulation to resell access to DSL customers to independent ISPs. The resale price is usually carefully regulated to be fair to both the incumbent and the market. The customer deals with only the independent ISP, who deals with the incumbent telco to arrange provisioning, etc. So the link layer (at least, immediate to the customer, independent ISP and incumbent) is something like:

customer1----telco exchange------<telco network>------ISP1
customer2----/ |                                ------ISP2
...            |
customern-----/                                 ------ISPn

The incumbent ISP tunnels the data traffic between the independent ISP and the customer (this is known as "backhaul"). So at the IP level, it looks like a single link (over which PPP is run), and the topology looks like (the customer numbers do not correspond with above):

customer1---<PPP>----ISP access----<ISP network>----<internet>
customer2---<PPP>---- router
..                    |
customern--<PPP>------/

This scheme must have management and implementation benefits, as it is a popular model. There is a reasonably clear layering, of IP and layer-2 and corresponding separation of responsibility. Maintenance and management of each layer is reasonably well decoupled, e.g. the telco can upgrade DSL head-end equipment at the exchange without having to technically interact with ISPs and the ISPs can carry out IP layer maintenance (addressing, whether to allow a PPP session, etc) without involving the telco. The only changes which routinely need interaction are the provisioning of a link between any specific customer and ISP (i.e. setting up a new customer, or changing the level of paid-for service) - to this end the telco can provide interfaces suitable for bulk updates as well as back-end<->back-end support[1].

So far... so what?

The downside to this model is that the IP topology is quite divorced from the geographic topology.[2] E.g. Imagine two DSL customers, each attached to the same telephone exchange (so they live reasonably close to each other). The IP topology between them, to which their packets are constrained if they want to communicate in some way via the internet, likely will be (at least, if we assume the ideal of competition that drove the creation of a regulated, wholesale DSL has paid off):

customer1-------ISP1------ISP2-----customer2

Because of how ISPs tend to interconnect, the chances are they'll do so at only a few locations, such as London, Amsterdam or Frankfurt (some Irish traffic might exchange at Dublin, not all though). So packets between customer1 and customer2 will wander off to some major, European metropolitan hub, before coming back to the exact same data-plane in the exact same networking device (no doubt saying a cordial hello as they pass their fellow packets still awaiting their long excursion). E.g.:

             +------------+
customer1----|  Exchange  |
customer2----| in Glasgow |
             +------------+
                   |
             <telco network>
               |       |
              ISP1    ISP2
               |       |
              LINX (London)

Such less-than-optimal routing is not uncommon in the networking world. Packets often pass close by packets of the same flow, before going off on a long detour, maybe because there is no direct route between two physically proximate devices, or even sometimes in the same device when the relationship of the flows is obscured by abstractions in the data-layers (IP over ATM, or MPLS, etc). This is because creating links and exchanging routes has a cost, particularly so in the ongoing management of a network. Physical links require technicians to setup and maintain. Logical links require network-administrators to configure. Exchanging routing information with other organisations requires further manual configuration (filters, policies) and monitoring, adding costs that further complicate the business-case evaluation and approval processes for new inter-ISP links. Further, for intra-AS forwarding, IP is perceived as hard to manage, and organisations often prefer to tunnel IP over an abstracted and flat topology, doing their network-engineering with protocol stacks like ATM or MPLS instead.

These costs mean a less-than-optimal route in a geographical sense will actually tend to be a uneconomic route, at least in private enterprise. Economics are the overriding metric, and so we should expect that routing tends to being economically optimal.

Yawn.. obvious.. So?

So one company owns and operates the DSL networking equipment, and potentially even the inter-exchange network. Typically this would result in a single IP topology, abstracted to some degree but still vaguely congruent with the physical topology, at least in the free market. In the wholesale-DSL model, the IP layer instead is provisioned primarily according to regulatory concerns. The telco is usually constrained from discriminating between ISPs, or giving itself preferential treatment where it also operates a consumer ISP. In the UK, the regulated, local-loop telco portion of BT is effectively a seperate company called BT Openreach; in Ireland, Eircom is supposed to maintain an internal chinese-wall between its wholesale and end-user operations. This distorts the economics somewhat.

E.g. often the telco itself operates an end-user ISP division. If it operated DSL head-ends as IP access routers[3], which is not too unreasonable to imagine given the example of some cable-ISPs, then at least the traffic of its own customers would stay local to the exchange. However, in the regulated, wholesale DSL model, this traffic MUST be "backhauled" to a few central locations, like the traffic of all other ISPs.

The only way the traffic could be kept local is if the telco/ISP could offer *all* ISPs the opportunity to exchange traffic locally, by the same method the telco's ISP uses. This would be difficult to do, as things stand, without unacceptable consequences.

Point please.. Or I must kill you

Ok, ok. The point, essentially, is that the benefits of competition for ISP customer-service comes at the cost of distorted economics, and IP topologies that are far less geographically optimal than those observed in single-organisation, end-user IP access networks (assuming economic efficiency isn't substantially different for both, so that the regulatory framework for DSL is the key variable). I.e.:

• The wholesale model results in a network that must cost more (and/or delivers less) than it would have otherwise, by simple comparison of IP topology alone.
• As the DSL model still results in a single, (effective) monopoly operator of the majority of the network[5] (at various levels[6]), any economic benefits likely lie primarily in competition in customer-service amongst ISPs.
• The attractiveness of multicast is reduced by the encapsulation of IP inherent to the wholesale-DSL model. If people from m different ISPs at an exchange subscribe to a multicast stream, the load on the trunk into the exchange is m, rather than 1. (Not that inter-domain, end-user multicast deployment has seen much use).
• Locality aware caching in P2P software (e.g. Joost) can not minimise load across paths that are hidden from IP (such as exchange trunks).
• It'd be interesting to read of studies on user-behaviour (traffic patterns) for MANs or NANs ("neighbour area networks", e.g. shared cable IP networks) to see if the locality makes any difference at all (with existing protocols), as well as studies analysing P2P traffic for temporal-locality to see what, if any, potential there is for significant trunk-link bandwidth savings, such as those for IPTV[4].

In short, my argument is that we're possibly in a bad place - regulating ourselves into a more costly system of delivering broadband, particularly for popular content, which becomes ever more difficult to substantively change the more the regulation encourages investment in it, which the private market likely will not be able to crack.

An Eircom executive recently, in a speech at the IETF plenary, gently argued that Ireland's DSL regulation was a disincentive to investment by Eircom (not reported on anywhere, it seems). So I'm somewhat hopeful that there's a still a chance for Ireland[7] to improve things, and reform broadband-access regulation. That's not to say we should give Eircom exactly what they want, only that they could be a driver to initiate reform, and of that itself we should not be sceptical..

With thanks to Thomas Bridge for insights and discussion on this subject (he very likely disagrees with my conclusions though).

See follow-up post on reform possibilities.

1. I can't think of a better term for "non-consumer support".

2. Obviously, this isn't new with customer/ISP internet access (e.g. dial-up, ISDN), however what is new with the arrangements for DSL is that the telco layer is increasingly (if not overwhelmingly these days) an IP-capable network, likely managed via IP (from the ADLS head-end equipment on), with parts of (if not all) of the DSL-data-delivery occuring over an IP-forwarding network (e.g. Eircom hand-off wholesale DSL as PPPoE/L2TP over IP). Ignoring the regulatory aspects..

3. Where IP capable - often the case with more modern equipment.

4. E.g. On Next-Generation Telco-Managed P2P TV Architectures, which found locality-aware caching could contain 80% of trafffic to within a DSLAM.

5. I'd be glad to hear of examples of where telco deregulation has lead to any significant access-link, "we own the wires" competition outside of juicy, major metropolitan areas..

6. The wholesale DSL model can result in mini-ISPs that run no more than DCANs (data-centre area networks), with the state telco connecting them to their customers AND their IP transit providers (perhaps unknowingly, for the latter, via more abstraction).

7. The UK, OTOH, seems to have a lot of ISPs who rely on BT wholesale DSL, so presumably that boat would be a lot harder to turn.

Thursday August 07, 2008

Weird BIND9 AXFR error? Remove stray A6 records..

Some release after BIND 9.2.1, its parser for A6 records appears to have broken. If you've ever experimented with A6, you might to go check you've expunged all occurrences from your zone files. Resultant symptoms include:

• Slave BIND servers fail to AXFR zone updates, reporting "transfer of ..... from .... : failed while receiving responses: unexpected end of input" in their log files.
• Manual AXFRs with dig gives the errors of ";; Warning: Message parser reports malformed message packet." and ";; communications error to ....: end of file".

Background: For many years, as I saw it only between a master behind a sub-1500 MTU (PPPoE, sigh) and certain servers, I assumed it was an MTU-blackhole problem with some upstream sneakily doing firewalling. However, recently the problem started to afflict another server, after an upgrade of their BIND software, and affected AXFRs even over paths that were perfectly fine. Some brute-force testing ruled out obvious problems like size issues (not like my zones are big) or relatively new records like SSHFP, leading to some head-scratching, but eventually pinned it down to a couple of stray A6 records.

Wednesday July 23, 2008

APNICs AS-dot policy proposal

In a similar vein to my attempt to dissuade RIPE from continuing with AS-dot format, James Spenceley has submitted APNIC policy proposal 65 to try get APNIC to change their ways.

Pretty Good BGP (PGBGP)

A new, interesting, soft/side-band approach to BGP security: Pretty Good BGP.

No crypto involved, no need for extensive deployment. Just monitoring the BGP routing table and reporting anomolous updates to the operator for further investigation. What's especially interesting is that they claim this system has discovered otherwise unknown hijacks of important prefixes.

Sunday October 28, 2007

Quagga updated in Solaris

Quagga has been updated in Solaris. The following bugs are fixed in both Solaris Nevada SFW gate (build 76) and the SFW10 gate (from which any S10u5 would be built):

• 6554276 bgpd is vulnerable to a DoS if configured peers send a malformed MP_(UN)REACH_NLRI attribute
• 6578890 Quagga should be updated to 0.99.7+
• 6606039 Quagga's vty_address SMF property is not effective
• 6607647 Quagga bgpd vulnerable to DoS from configured peers sending malformed OPEN or COMMUNITY
• 6610223 vty-address should be vty_address in /usr/sbin/quaggaadm
• 6610228 /usr/sbin/quaggaadm should use vty_port

Additionally, the following bug is fixed only in SFWNV:

• 6517793 Quagga needs to be updated to match IP Instances SMF changes

which means you can now run any Quagga daemon in a zone with an exclusive IP. On S10, you can run only bgpd in a zone - we could try fix this for S10 too (all the bits are in fact putback to the gate, it'd just need some hackery to enable them..), if interested in this please register your interest on the bug.

Enjoy

Thursday October 25, 2007

AS-Dot at RIPE

Joao Damas ran through my AS-Dot Problems slide deck at RIPE-55. From the comments, which were mostly of the "Get over it" form, it's now obvious I missed the following slide:

          It's Not The Code...
          --------------------

- Changing the code is by and 
  large done.
- It's the *data*
  - router configurations
  - whois data

Ah well (or "buhuhu", as Sascha Lenz put it).

Saturday October 20, 2007

TCP RST Considered Dangerous and Obsolete

OK, I don't actually mean the title, but with news that Comcast is blocking P2P by sending fake RSTs (same technique as purportedly used by the Great Firewall of China), how long until hosts start to just ignore TCP RSTs (at least for P2P applications anyway)? Yes, it'll degrade detection of errors, but TCP will at least then be immune to such silly "filters". Other things considered obsolete: The destination port field; as things stand, future revisions of TCP may as well codify this to be a "Must Be 80" field.

The sooner the internet (TCP and its data at least) becomes opaque[1] to ISPs, the better. Good to see the ISPs are so eager to help motivate users to achieve that goal.

1. I.e. headers authenticated somehow, if possible (or else ignore possibly false errors), and all discriminatory data/fields encrypted, to the greatest extent possible.

Friday September 07, 2007

Quagga 0.99.9 released

Quagga 0.99.9 has been released, and is available, along with a full
changelog, in the usual places, such as:

        http://www.quagga.net/download/

Thanks to everyone who helped by reporting bugs and testing fixes.

                        Release notes:
                        -------------

bgpd: Low impact DoS (Mu Security)
----------------------------------

This release fixes two potential DoS conditions in bgpd, reported by Mu
Security, where a bgpd could be crashed if a peer sent a malformed OPEN
message or a malformed COMMUNITY attribute. Only configured peers can do
this, hence we consider these issues to be very low impact.


bgpd: crash with outbound route-maps
------------------------------------

This release fixes a serious regression in bgpd in Quagga 0.99.8, where use
of outbound route-maps would cause a crash.


bgpd: severe performance problems with regexes
----------------------------------------------

Operators should be aware that allowing untrusted access to the bgpd vty are
vulnerable to such untrusted users running regex commands that may cause
bgpd to block for many minutes.

To try alleviate this, bgpd now passes the 'REG_NOSUB' flag to regcomp().
This may help good regex implementations to avoid doing a lot of work when
users specify substitutions (which we will never use). Unfortunately, this
doesn't appear to have much of an effect on the platforms I have tested
(Solaris libc and GNU libc).

The 'PCRE' regex implementation however appears to be better behaved, and
does not introduce huge slow-downs when regexes with substitutions are
applied. Operators who continue to offer untrusted vty access may wish to
preload the 'libpcreposix' library (e.g. using LD_PRELOAD). Be aware however
that PCRE is not fully compatible with POSIX extended regexes, and this
workaround may adversely impact existing configurations.

bgpd: AS-Pathlimit TTL attribute support added
----------------------------------------------

This attribute allows for routes to be announced with a limited scope,
specified in terms of numbers of AS-hopcount. See the TeXinfo documentation
for further details.

isisd: Now supports Solaris

-

A short-form list of code related changes:

bgpd:
- [bgpd] low-impact DoS: crash on malformed community with debug set
- [bgpd] bug #398 Bogus free on out route-map, and assert() with rsclients
- [bgpd] Add support for AS_PATHLIMIT / draft-ietf-idr-as-pathlimit
- [bgpd] cleanup, compact and consolidate capability parsing code
- [bgpd] Dont schedule dumps multiple times for same command
- [bgpd] Pass NOSUB to regexec

ospfd:
- [ospfd] Bug #331, NSSA ASBR regression - failure to set E-bit in NSSA
areas
- Bug #362 is fixed now.
- [ospfd] Fix bad SPF calculation on some topologies - incorrect sorting

zebra:
- + fixed bug #400: adjusted rtread_sysctl.c:route_read()
- Looks like bug #320 is finally fixed now.
- Fixed ioctl_solaris.c:if_get_mtu() for IPv6'less operation
- Fixed bug #394 "RTF_DONE is ignored in rtm_read()"
- Merged own patch for bug #390 (rewrite
zebra/zebra_rib.c:nexthop_active_update())
- Use the proper field length for the peer's address
(netlink_interface_addr)
- Bugzilla #384.

isisd:
- [isisd] Add support for Solaris DLPI

Tuesday July 24, 2007

Solaris support for IS-IS

James Carlson has posted support for Solaris in isisd to the Quagga-dev list, implementing the DLPI interface isisd needed to send/receive ISO packets on Solaris. Part of the beginnings of the OpenSolaris RBridges project.

Very neat!

Wednesday July 11, 2007

BGP AS-Dot notation considered harmful The proposed As-Dot notation for BGP irretrievably breaks configuration and introduces complexity and inconsistency for no apparent gain. The BGP operator community needs to think carefully about the issues before continuing with the introduction of "asdot". [Read More] ( Jul 11 2007, 04:35:42 PM IST ) Permalink Comments [0]

Sunday May 13, 2007

Juergen Kammer's Quagga talk at RIPE-54

Juergen Kammer gave a talk on Quagga at the recent RIPE-54 meeting. Juergen implemented 4-byte ASN support for Quagga.

Sunday April 29, 2007

Mini BGP

Apparently a rather unusual pair of BGP routers have appeared at LINX