- A virtual community of network engineers
 Home  BookStore  StudyNotes  Links  Archives  StudyRooms  HelpWanted  Discounts  Login
RE: Route Reflection with Multiple POPs [7:48509] posted 07/11/2002
[Chronological Index] [Thread Index] [Top] [Date Prev][Date Next] [Thread Prev][Thread Next]

I will definitely look into some of the resources you mentioned.  Again,
thanks to all who responded.

~-----Original Message-----
~From: Peter van Oene [mailto:pvo@xxxxxxxxxxxx]
~Sent: Thursday, July 11, 2002 10:16 AM
~To: Lupi, Guy; cisco@xxxxxxxxxxxxxx
~Subject: RE: Route Reflection with Multiple POPs [7:48509]
~Not really :)  Howard's is one of the few SP oriented books on 
~the market 
~though admittedly I haven't picked it up yet (very broke :)
~The ISP resources at NANOG are helpful and contain some 
~valuable links and 
~one can glean some valuable information by looking at some of the 
~historical design oriented presentations that many have given over the 
~years.  Cisco also has a document (and recently a book based on the 
~document) from Phillip Smith which deals with ISP design tips (called 
~something like IOS Essentials: Design tips every ISP should know -- 
~something similar to that)  Some quick tips would include:
~- install only link/loopback addresses (ie no redistribution 
~into the IGP 
~of static/connected/or otherwise)
~- tune metrics for optimal SPF(usually to link delay - ie 70ms 
~= 70) - see 
~latest nanog for netflow/lots of math based method which seems cool! :)
~- pre allocate contiguous IP space to non-backbone areas such that 
~summarization _could_ take place
~- keep in mind summarization when multiple exits from an area 
~exist can 
~cause sub-optimal routing in normal states and blackholes in 
~some failure 
~- if you _really_ need to put externals in your IGP (say you 
~have a bunch 
~of dial routers and marketing gave some customers static IP's that are 
~radius assigned (shame on you marketing), use a separate 
~instance of the 
~IGP or use NSSA with no type 7/5 conversion or full external 
~filtering at 
~L1L2 border in ISIS and advertise a BGP aggregate from the 
~related ABR's.
~- authentication can't hurt
~- hard code RIDs
~- peer on lookbacks
~- hard code RIDs
~- use recognizable /32 space for loopbacks for easy router 
~- use bgp-deterministic-med (to reduce some potential for med oriented 
~oscillation) (lots of ietf chat on this on IDR group as howard 
~points out 
~from time to time)
~- build resilient RR clusters (ie two RR servers per group of clients)
~- one cluster ID per RR server (subject to some memory vs 
~potential for bad 
~routing debate) (also cisco default)
~- use communities and do community based advertisements
~- wrt to advertising to transit providers/peers, use an 
~explicit accept / 
~default reject style policy for safety (ie when you mess it 
~up, you don't 
~send anything vs messing it up and advertising transit)
~- authenticate IBGP and possibly EBGP where appropriate
~- use next-hop-self vs passive interface (helps hook mpls into 
~plane if you decide to deploy it and keeps IGP smaller)
~- don't dynamically redistribute anything in bgp - ie only 
~redist statics 
~routes for aggregation
~- try not to leak too much junk to your transit providers (ie 
~where you can without too much TE loss)
~- filter customer prefix advertisements - IRR based would be 
~ideal, but 
~manual is cool too
~- local pref customer routes over peer routes over transit routes
~- either use meds or zero them in and out (better to know what 
~is going to 
~happen than learn :)
~Lots of more stuff out there likely to cover, but thats my 10 
~minute off 
~the top of my head while my soup cooks :)  Likely lots there 
~to debate as 
~well!  Building networks generally involves a lot of personal design 
~philosophy which makes for healthy debate.
~At 09:20 AM 7/11/2002 -0400, you wrote:
~>This is where my thinking went wrong then, Phil also said 
~basically the same
~>thing.  It seems that I have a basic misunderstanding of how larger
~>providers carry their routing information.  Where would I be 
~able to get
~>information about proper addressing structures and routing protocol
~>implementation in large/service provider networks?  Are there 
~any books that
~>go into detail on these particular topics, take you step by 
~step from a
~>customer through to the core?
~>-----Original Message-----
~>From: Peter van Oene
~>To: Lupi, Guy; cisco@xxxxxxxxxxxxxx
~>Sent: 7/11/2002 8:18 AM
~>Subject: RE: Route Reflection with Multiple POPs [7:48509]
~>Hi Guy,
~>As others have pointed out, it is not necessary, nor usually 
~>run separate IGPs in your AS assuming you use your IGP properly.  A
~>properly utilized ISP IGP should carry only link and loopback 
~>thus be reasonably small.  Use of hierarchy can also help 
~larger domains
~>scale, though modern IGP implementations on reasonably powered routers
~>should be able to handle rather large single areas.
~>BGP never runs entirely on its own as one always requires 
~some method of
~>resolving BGP Next-Hops to forwarding next hops.  Static 
~routes can work
~>here to connect otherwise disconnected IGP domains, or, separate IGP
~>domains can be used for this purpose as well.
~>At 06:19 PM 7/10/2002 -0400, Lupi, Guy wrote:
~> >I know that you can run confederations and reflectors, and seperate
~> >of reflection, which Cisco refers to as "nested reflection".  Now my
~> >question is, how would you set up your bgp peering?  Due to 
~> >constraints I would imagine that the best thing to do would 
~be to have
~> >circuit from POP router 1 to core router 1, and another circuit from
~> >router 2 to core router 2.  Assuming that you are only 
~running BGP in
~> >core, and that the clients have to have a session with each 
~> >would you communicate the loopback addresses of all the 
~routers to each
~> >other?  Are static routes used in this situation?  Thanks 
~to all of the
~> >people who responded by the way, I appreciate the 
~direction.  Ever feel
~> >you know so much, only to read a book and find out that you know so
~> >:)?
~> >
~> >*-----Original Message-----
~> >*From: Peter van Oene [mailto:pvo@xxxxxxxxxxxx]
~> >*Sent: Wednesday, July 10, 2002 3:00 PM
~> >*To: cisco@xxxxxxxxxxxxxx
~> >*Subject: Re: Route Reflection with Multiple POPs [7:48509]
~> >*
~> >*
~> >*Couple thoughts.  First off, one can confederate and reflect
~> >*at the same
~> >*time (ie rr clusters inside sub-as's).  Also, keep in mind 
~that these
~> >*techniques deal with control, not forwarding and thus it is
~> >*possible, and
~> >*somewhat common, to have a RR hierarchy that does not map
~> >*directly to the
~> >*physical topology.  I tend to see an arbitrary set of core
~> >*routers (from
~> >*2-20ish) that form the central IBGP mesh with the second 
~level of the
~> >*hierarchy being formed by lead major pop routers, or 
~dedicated route
~> >*servers (ie one armed routers that simply reflect routes).  I
~> >*personally
~> >*try to keep the topology as flat as possible (ie less than 4
~> >*levels in the
~> >*hierarchy) but I don't have any particular technical driver
~> >*for that other
~> >*than to minimize complexity and aid troubleshooting.
~> >*
~> >*Of note, this assumes you are reflecting full routes 
~throughout your
~> >*backbone.  I've seen providers try and reflect partial routes
~> >*to areas of
~> >*their backbone to allow for smaller routers where RR
~> >*topologies can lead to
~> >*blackholes. In general, if you are reflecting all routes, you
~> >*shouldn't run
~> >*into issues here.
~> >*
~> >*I
~> >*
~> >*
~> >*
~> >*At 04:42 PM 7/10/2002 +0000, Lupi, Guy wrote:
~> >*>Let me preface this by saying that I am trying to learn more
~> >*about large
~> >*>scale BGP design and operation.  This question is on route
~> >*reflectors when
~> >*>you have multiple POPs in seperate IGP domains.  If you
~> >*currently have one
~> >*>POP and are going to move to 2 within the same AS, you can
~> >*either run full
~> >*>mesh (doesn't scale), reflectors, or confederations.
~> >*Assuming you don't
~> >*>currently have a central core that the POPs connect back to,
~> >*how well does
~> >*>reflection scale?  I was reading Building Service 
~Provider Networks
~> >*>[Berkowitz], and it states that iBGP doesn't scale well once
~> >*you go above
~> >*>15-20 sessions per router.  It also states that most ISPs run
~> >*reflectors
~> >*>instead of confederations, but I believe that statement is
~> >*being made under
~> >*>the assumption that the ISP will have a central core to which
~> >*the POPs will
~> >*>connect.  This would indicate to me that assuming you don't
~> >*have a central
~> >*>core, one could only connect 6 or 7 POPs (dual reflectors for
~> >*redundancy)
~> >*>together using reflection before you would have to either
~> >*create a central
~> >*>core to reduce the amount of iBGP sessions, or turn to
~> >*>Perhaps the best way to accomplish this would be to establish
~> >*a "core" in
~> >*>one of the POPs and run reflection from there, which is also
~> >*presented as a
~> >*>solution in the book?  Any opinions?  I have made an 
~attempt at ASCII
~> >*>drawing below, to me the central core solution makes more sense.
~> >*>
~> >*>                 First Scenario, no central core
~> >*>
~> >*>                   POP 1           POP 2
~> >*>                  Cluster         Cluster
~> >*>                     o---------------o
~> >*>               Full Mesh Between Reflectors
~> >*>                     o---------------o
~> >*>
~> >*>   Second Scenario, central core establised at one of the POPs
~> >*>
~> >*>                   POP 1           POP 2
~> >*>                      Core Reflectors
~> >*>             Clients o-----     -----o Clients
~> >*>                     o----- \ / -----o
~> >*>                            o o
~> >*>                   POP 3    / \     POP 4
~> >*>             Clients o-----     -----o Clients
~> >*>                     o-----     -----o
~> >*>
~> >*>
~> >*>Guy H. Lupi
~> >*>CCIE No. 9275
~> >*
~> >*
~> >*
~> >*

Message Posted at:
FAQ, list archives, and subscription info:
Report misconduct and Nondisclosure violations to abuse@xxxxxxxxxxxxxx