Routing Challenge

I was dealing with an issue the other day at work that was kind of challenging and made my brain hurt…

Scenario

Note the diagram below.

What we have is a simplified diagram of 4 routers. Rtr A and B are currently connected to our global ATT MPLS carrier and receiving routes via BGP from our other corporate BGP sites (expected). Rtr A and B are also advertising their own networks back out providing reachability to subnets behind Rtr A and B as well as C and D. End-to-end connectivity is easily done with the bidirectional exchange of route information — that is not the challenge here.

Notice that A,B,C and D are all connected to a “Domestic” MPLS network. It’s cheaper to have router C and D connected to a separate MPLS carrier within their local region of the world. Running links from both Router C and D to the nearest ATT MPLS PE router would not be cost effective due to location (Actually, there is technically about 5 more routers that will connect to the domestic MPLS carrier.)

Here is the challenge I was facing ::

For this, consider that Rtr D is closest to Rtr B, and Rtr C is closest to Rtr A. In terms of optimized path, RTT from B to D is faster than A to D. Same thing would apply to Rtr C being closer to Rtr A.

1. Traffic from other ATT MPLS sites to networks behind Router A and B should choose the closest path to that network. The solution for this is typical. For routes we advertise out Rtr A or B, set the community value on outbound routes so that the ATT PE router will choose the route based on best Local Pref (We send community to PE router which in turn adjusts Local pref for the routes based on community value i.e. sending 6501:90 would cause ATT to set local pref to 90 for that network). For Example, Traffic from ATT MPLS to a subnet behind Rtr D should enter via Rtr B. This is because Rtr D is closer and a better path from Rtr B compared to Rtr A. It’s not really hard to control the direction from ATT to the routers behind A and B.

2. What about the other direction? From Rtr D to a network on the ATT MPLS should go to Rtr B because going through Rtr A would add an additional 20-30ms. How can we control traffic from Rtr D to use Rtr B as the next hop primary path? This must be done without involving the Domestic MPLS carrier routers (P, PE, etc.). Rtr D only has one link to the domestic MPLS network so we can’t do much in terms of BGP route manipulation. And we have to maintain redundancy in case of Rtr A or B link failure to the Domestic MPLS. Since both internet and on-net traffic from C/D will use the ATT MPLS network, only default routes are sent from A and B.

Rtr D will get two default routes both pointing to the same next hop IP of the adjacent Domestic PE router. Even if the one that came from AS 6501 is given a better local pref. the next hop is still the adjacent PE router. I thought about modifying the next hop received to be that of Rtr B instead of the local PE router, but I don’t think that would work.

Obviously, having the ability to do some manipulations within the Domestic carrier is the best way to go, but I needed to consider the options without involving the carrier.

Can it be done? Think about it….

|
|
|
|
|
|
|
V

Here is what I came up with ::

When you think it’s impossible, slap a GRE tunnel on it! Now router D has two links, and with a GRE tunnel, the routing is based on the tunnel destination IP, which is now Rtr B. So I now learn default routes over the physical interface (from A and B) and over the tunnel (from B only). I adjust local pref. to prefer the tunnel interface for outbound traffic from Router D. This addresses all packets to Rtr B (based on tunnel destination IP) forcing traffic from Rtr D to go to the closest exit point being Rtr B. Don’t forget to add a static route pointing traffic destined to Rtr B interface out physical interface on Rtr D.

For traffic from ATT to Rtr D networks, we still enter via Rtr B, and Rtr B sends out physical interface (not tunnel) which only goes to Rtr D. Basically, traffic only gets put in GRE tunnel from D to B, not the other way around. This helps save on some of the overhead.

I’m sure we’ll still end up having the carrier help us out with this, but it was at least fun to play with and see if it could be done…

If you figure out a better way, let me know!

Ted

This entry was posted on January 29, 2009 at 11:00 pm and is filed under Routing&Switching, Cisco. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response or trackback from your own site.

| Print