IOS XR Teaser - BGP as PE to CE for MPLS L3VPN
Update: Congrats to Mark, our winner of 100 rack rental tokens for the first correct answer, that XR2 is missing a BGP router-id. In regular IOS, a router-id is chosen based on the highest Loopback interface. If there is no Loopback interface the highest IP address of all up/up interfaces is chosen. In the case of IOS XR however, the router-id will not be chosen from a physical link. It will only be chosen from the highest Loopback interface, or from the manual router-id command. Per the Cisco documentation:
For BGP sessions between neighbors to be established, BGP must be assigned a router ID. The router ID is sent to BGP peers in the OPEN message when a BGP session is established.
BGP attempts to obtain a router ID in the following ways (in order of preference):
- By means of the address configured using the bgp router-id command in router configuration mode.
- By using the highest IPv4 address on a loopback interface in the system if the router is booted with saved loopback address configuration.
- By using the primary IPv4 address of the first loopback address that gets configured if there are not any in the saved configuration.
If none of these methods for obtaining a router ID succeeds, BGP does not have a router ID and cannot establish any peering sessions with BGP neighbors. In such an instance, an error message is entered in the system log, and the show bgp summary command displays a router ID of 0.0.0.0.
After BGP has obtained a router ID, it continues to use it even if a better router ID becomes available. This usage avoids unnecessary flapping for all BGP sessions. However, if the router ID currently in use becomes invalid (because the interface goes down or its configuration is changed), BGP selects a new router ID (using the rules described) and all established peering sessions are reset.
Since XR2 in this case does not have a Loopback configured, the BGP process cannot initialize. The kicker with this problem is that the documentation states that when this problem occurs you should see that "an error message is entered in the system log", however in this case a Syslog was not generated about the error. At least this is the last time this problem will bite me ;)
Today while working on additional content for our CCIE Service Provider Version 3.0 Lab Workbook I had one of those epic brain fart moments. What started off as work on (what I thought was) a fairly simply design ended up as a 2 hour troubleshooting rabbit hole of rolling back config snippets one by one, debugging, and basically overall misery that can be perfectly summed up by this GIF of a guy smashing his head against his keyboard. :)
The scenario in question was a BGP peering between two IOS XR routers. One was the PE of an MPLS L3VPN network and one was the CE. As I've done this config literally hundreds of times in the past I could not for the life of me figure out why the BGP peering would not establish. The relevant snippet of the topology diagram is as follows:
Since this scenario caused me so much pleasure I am offering 100 tokens good for CCIE Service Provider Version 3.0 Rack Rentals - or any of our other Routing & Switching rack rentals & mock labs, Security rack rentals, or Voice rack rentals - to whoever the first person is that can tell me why did these neighbors not establish a BGP peering. The relevant outputs needed to troubleshoot the problem can be found below. I still haven't decided whether I'm going to leave this problem in the workbook or not since it's such a mean one :) Good luck!
<strong>RP/0/0/CPU0:XR1#show run</strong>
Fri May 11 00:34:38.563 UTC
Building configuration...
!! IOS XR Configuration 3.9.1
!! Last configuration change at Fri May 11 00:32:50 2012 by xr1
!
hostname XR1
username xr1
group root-lr
password 7 13061E010803
!
vrf ABC
address-family ipv4 unicast
import route-target
26:65001
!
export route-target
26:65001
!
!
!
line console
exec-timeout 0 0
!
ipv4 access-list PE_ROUTERS
10 permit ipv4 host 1.1.1.1 any
20 permit ipv4 host 2.2.2.2 any
30 permit ipv4 host 5.5.5.5 any
40 permit ipv4 host 19.19.19.19 any
!
interface Loopback0
ipv4 address 19.19.19.19 255.255.255.255
!
interface GigabitEthernet0/1/0/0
ipv4 address 172.19.10.19 255.255.255.0
!
interface GigabitEthernet0/1/0/1
ipv4 address 26.3.19.19 255.255.255.0
!
interface POS0/6/0/0
vrf ABC
ipv4 address 10.19.20.19 255.255.255.0
!
route-policy PASS
pass
end-policy
!
router isis 1
is-type level-2-only
net 49.0001.0000.0000.0019.00
address-family ipv4 unicast
mpls ldp auto-config
!
interface Loopback0
passive
address-family ipv4 unicast
!
!
interface GigabitEthernet0/1/0/1
point-to-point
hello-password hmac-md5 encrypted 022527722E
address-family ipv4 unicast
!
!
!
router bgp 26
address-family ipv4 unicast
!
! address-family ipv4 unicast
address-family vpnv4 unicast
!
neighbor-group PE_ROUTERS
remote-as 26
update-source Loopback0
address-family vpnv4 unicast
!
!
neighbor 1.1.1.1
use neighbor-group PE_ROUTERS
!
neighbor 2.2.2.2
use neighbor-group PE_ROUTERS
!
neighbor 5.5.5.5
use neighbor-group PE_ROUTERS
!
vrf ABC
rd 26:65001
address-family ipv4 unicast
!
neighbor 10.19.20.20
remote-as 65001
address-family ipv4 unicast
route-policy PASS in
route-policy PASS out
as-override
!
!
!
!
mpls ldp
label
allocate for PE_ROUTERS
!
!
endRP/0/0/CPU0:XR1#
<strong>RP/0/3/CPU0:XR2#show run </strong>
Fri May 11 00:35:04.932 UTC
Building configuration...
!! IOS XR Configuration 3.9.1
!! Last configuration change at Fri May 11 00:30:30 2012 by xr2
!
hostname XR2
logging console debugging
username xr2
group root-lr
password 7 00071A150754
!
cdp
line console
exec-timeout 0 0
!
interface GigabitEthernet0/4/0/0
ipv4 address 10.20.20.20 255.255.255.0
ipv6 address 2001:10:20:20::20/64
!
interface POS0/7/0/0
ipv4 address 10.19.20.20 255.255.255.0
ipv6 address 2001:10:19:20::20/64
!
route-policy PASS
pass
end-policy
!
router bgp 65001
address-family ipv4 unicast
!
neighbor 10.19.20.19
remote-as 26
address-family ipv4 unicast
route-policy PASS in
route-policy PASS out
!
!
!
endRP/0/3/CPU0:XR2#
RP/0/0/CPU0:XR1#show bgp vrf ABC ipv4 unicast summary
Fri May 11 00:34:29.712 UTC
BGP VRF ABC, state: Active
BGP Route Distinguisher: 26:65001
VRF ID: 0x60000002
BGP router identifier 19.19.19.19, local AS number 26
BGP table state: Active
Table ID: 0xe0000002
BGP main routing table version 1BGP is operating in STANDALONE mode.
Process RcvTblVer bRIB/RIB LabelVer ImportVer SendTblVer StandbyVer
Speaker 1 1 1 1 1 1Neighbor Spk AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down St/PfxRcd
10.19.20.20 0 65001 2 7 0 0 0 00:03:59 Idle
<strong>RP/0/3/CPU0:XR2#show bgp ipv4 unicast summary</strong>
Fri May 11 00:35:02.278 UTC
BGP router identifier 0.0.0.0, local AS number 65001
BGP generic scan interval 60 secs
BGP table state: Active
Table ID: 0xe0000000
BGP main routing table version 1
BGP scan interval 60 secsBGP is operating in STANDALONE mode.
Process RcvTblVer bRIB/RIB LabelVer ImportVer SendTblVer StandbyVer
Speaker 1 1 1 1 1 1Neighbor Spk AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down St/PfxRcd
10.19.20.19 0 26 2 2 0 0 0 00:04:31 Active
RP/0/0/CPU0:XR1#show bgp vrf ABC ipv4 unicast neighbors
Fri May 11 00:34:18.708 UTCBGP neighbor is 10.19.20.20, vrf ABC
Remote AS 65001, local AS 26, external link
Remote router ID 0.0.0.0
BGP state = Idle
Last read 00:00:00, Last read before reset 00:04:10
Hold time is 180, keepalive interval is 60 seconds
Configured hold time: 180, keepalive: 60, min acceptable hold time: 3
Last write 00:00:15, attempted 53, written 53
Second last write 00:01:01, attempted 53, written 53
Last write before reset 00:04:10, attempted 72, written 72
Second last write before reset 00:04:15, attempted 53, written 53
Last write pulse rcvd May 11 00:34:02.927 last full not set pulse count 9
Last write pulse rcvd before reset 00:04:10
Socket not armed for io, not armed for read, not armed for write
Last write thread event before reset 00:04:10, second last 00:04:10
Last KA expiry before reset 00:00:00, second last 00:00:00
Last KA error before reset 00:00:00, KA not sent 00:00:00
Last KA start before reset 00:00:00, second last 00:00:00
Precedence: internet
Enforcing first AS is enabled
Received 2 messages, 0 notifications, 0 in queue
Sent 7 messages, 0 notifications, 0 in queue
Minimum time between advertisement runs is 0 secsFor Address Family: IPv4 Unicast
BGP neighbor version 0
Update group: 0.2
Route refresh request: received 0, sent 0
Policy for incoming advertisements is PASS
Policy for outgoing advertisements is PASS
0 accepted prefixes, 0 are bestpaths
Cumulative no. of prefixes denied: 0.
Prefix advertised 0, suppressed 0, withdrawn 0
Maximum prefixes allowed 524288
Threshold for warning message 75%, restart interval 0 min
AS override is set
An EoR was not received during read-only mode
Last ack version 0, Last synced ack version 0
Outstanding version objects: current 0, max 0Connections established 1; dropped 1
Local host: 10.19.20.19, Local port: 19432
Foreign host: 10.19.20.20, Foreign port: 179
Last reset 00:00:15, due to Peer closing down the session
Peer reset reason: Remote closed the session (Connection timed out)
Time since last notification sent to neighbor: 00:02:11
Error Code: administrative shutdown
Notification data sent:
None
<strong>RP/0/3/CPU0:XR2#show bgp ipv4 unicast neighbors </strong>
Fri May 11 00:34:58.427 UTCBGP neighbor is 10.19.20.19
Remote AS 26, local AS 65001, external link
Remote router ID 0.0.0.0
BGP state = Active
Last read 00:00:00, Last read before reset 00:04:50
Hold time is 180, keepalive interval is 60 seconds
Configured hold time: 180, keepalive: 60, min acceptable hold time: 3
Last write 00:04:50, attempted 19, written 19
Second last write 00:04:50, attempted 53, written 53
Last write before reset 00:04:50, attempted 19, written 19
Second last write before reset 00:04:50, attempted 53, written 53
Last write pulse rcvd May 11 00:30:08.305 last full not set pulse count 4
Last write pulse rcvd before reset 00:04:50
Socket not armed for io, not armed for read, not armed for write
Last write thread event before reset 00:04:50, second last 00:04:50
Last KA expiry before reset 00:00:00, second last 00:00:00
Last KA error before reset 00:00:00, KA not sent 00:00:00
Last KA start before reset 00:04:50, second last 00:00:00
Precedence: internet
Enforcing first AS is enabled
Received 2 messages, 0 notifications, 0 in queue
Sent 2 messages, 0 notifications, 0 in queue
Minimum time between advertisement runs is 30 secsFor Address Family: IPv4 Unicast
BGP neighbor version 0
Update group: 0.2
Route refresh request: received 0, sent 0
Policy for incoming advertisements is PASS
Policy for outgoing advertisements is PASS
0 accepted prefixes, 0 are bestpaths
Cumulative no. of prefixes denied: 0.
Prefix advertised 0, suppressed 0, withdrawn 0
Maximum prefixes allowed 524288
Threshold for warning message 75%, restart interval 0 min
An EoR was not received during read-only mode
Last ack version 0, Last synced ack version 0
Outstanding version objects: current 0, max 0Connections established 1; dropped 1
Local host: 10.19.20.20, Local port: 60056
Foreign host: 10.19.20.19, Foreign port: 179
Last reset 00:02:27, due to Interface flap
Time since last notification sent to neighbor: 00:05:07
Error Code: administrative reset
Notification data sent:
None
