Update: Congrats to Mark, our winner of 100 rack rental tokens for the first correct answer, that XR2 is missing a BGP router-id. In regular IOS, a router-id is chosen based on the highest Loopback interface. If there is no Loopback interface the highest IP address of all up/up interfaces is chosen. In the case of IOS XR however, the router-id will not be chosen from a physical link. It will only be chosen from the highest Loopback interface, or from the manual router-id command. Per the Cisco documentation:
For BGP sessions between neighbors to be established, BGP must be assigned a router ID. The router ID is sent to BGP peers in the OPEN message when a BGP session is established.
BGP attempts to obtain a router ID in the following ways (in order of preference):
- By means of the address configured using the bgp router-id command in router configuration mode.
- By using the highest IPv4 address on a loopback interface in the system if the router is booted with saved loopback address configuration.
- By using the primary IPv4 address of the first loopback address that gets configured if there are not any in the saved configuration.
If none of these methods for obtaining a router ID succeeds, BGP does not have a router ID and cannot establish any peering sessions with BGP neighbors. In such an instance, an error message is entered in the system log, and the show bgp summary command displays a router ID of 0.0.0.0.
After BGP has obtained a router ID, it continues to use it even if a better router ID becomes available. This usage avoids unnecessary flapping for all BGP sessions. However, if the router ID currently in use becomes invalid (because the interface goes down or its configuration is changed), BGP selects a new router ID (using the rules described) and all established peering sessions are reset.
Since XR2 in this case does not have a Loopback configured, the BGP process cannot initialize. The kicker with this problem is that the documentation states that when this problem occurs you should see that “an error message is entered in the system log”, however in this case a Syslog was not generated about the error. At least this is the last time this problem will bite me 
Today while working on additional content for our CCIE Service Provider Version 3.0 Lab Workbook I had one of those epic brain fart moments. What started off as work on (what I thought was) a fairly simply design ended up as a 2 hour troubleshooting rabbit hole of rolling back config snippets one by one, debugging, and basically overall misery that can be perfectly summed up by this GIF of a guy smashing his head against his keyboard. 
The scenario in question was a BGP peering between two IOS XR routers. One was the PE of an MPLS L3VPN network and one was the CE. As I’ve done this config literally hundreds of times in the past I could not for the life of me figure out why the BGP peering would not establish. The relevant snippet of the topology diagram is as follows:
Since this scenario caused me so much pleasure I am offering 100 tokens good for CCIE Service Provider Version 3.0 Rack Rentals - or any of our other Routing & Switching rack rentals & mock labs, Security rack rentals, or Voice rack rentals – to whoever the first person is that can tell me why did these neighbors not establish a BGP peering. The relevant outputs needed to troubleshoot the problem can be found below. I still haven’t decided whether I’m going to leave this problem in the workbook or not since it’s such a mean one Good luck!
RP/0/0/CPU0:XR1#show run
Fri May 11 00:34:38.563 UTC
Building configuration...
!! IOS XR Configuration 3.9.1
!! Last configuration change at Fri May 11 00:32:50 2012 by xr1
!
hostname XR1
username xr1
group root-lr
password 7 13061E010803
!
vrf ABC
address-family ipv4 unicast
import route-target
26:65001
!
export route-target
26:65001
!
!
!
line console
exec-timeout 0 0
!
ipv4 access-list PE_ROUTERS
10 permit ipv4 host 1.1.1.1 any
20 permit ipv4 host 2.2.2.2 any
30 permit ipv4 host 5.5.5.5 any
40 permit ipv4 host 19.19.19.19 any
!
interface Loopback0
ipv4 address 19.19.19.19 255.255.255.255
!
interface GigabitEthernet0/1/0/0
ipv4 address 172.19.10.19 255.255.255.0
!
interface GigabitEthernet0/1/0/1
ipv4 address 26.3.19.19 255.255.255.0
!
interface POS0/6/0/0
vrf ABC
ipv4 address 10.19.20.19 255.255.255.0
!
route-policy PASS
pass
end-policy
!
router isis 1
is-type level-2-only
net 49.0001.0000.0000.0019.00
address-family ipv4 unicast
mpls ldp auto-config
!
interface Loopback0
passive
address-family ipv4 unicast
!
!
interface GigabitEthernet0/1/0/1
point-to-point
hello-password hmac-md5 encrypted 022527722E
address-family ipv4 unicast
!
!
!
router bgp 26
address-family ipv4 unicast
!
! address-family ipv4 unicast
address-family vpnv4 unicast
!
neighbor-group PE_ROUTERS
remote-as 26
update-source Loopback0
address-family vpnv4 unicast
!
!
neighbor 1.1.1.1
use neighbor-group PE_ROUTERS
!
neighbor 2.2.2.2
use neighbor-group PE_ROUTERS
!
neighbor 5.5.5.5
use neighbor-group PE_ROUTERS
!
vrf ABC
rd 26:65001
address-family ipv4 unicast
!
neighbor 10.19.20.20
remote-as 65001
address-family ipv4 unicast
route-policy PASS in
route-policy PASS out
as-override
!
!
!
!
mpls ldp
label
allocate for PE_ROUTERS
!
!
end
RP/0/0/CPU0:XR1#
RP/0/3/CPU0:XR2#show run
Fri May 11 00:35:04.932 UTC Building configuration... !! IOS XR Configuration 3.9.1 !! Last configuration change at Fri May 11 00:30:30 2012 by xr2 ! hostname XR2 logging console debugging username xr2 group root-lr password 7 00071A150754 ! cdp line console exec-timeout 0 0 ! interface GigabitEthernet0/4/0/0 ipv4 address 10.20.20.20 255.255.255.0 ipv6 address 2001:10:20:20::20/64 ! interface POS0/7/0/0 ipv4 address 10.19.20.20 255.255.255.0 ipv6 address 2001:10:19:20::20/64 ! route-policy PASS pass end-policy ! router bgp 65001 address-family ipv4 unicast ! neighbor 10.19.20.19 remote-as 26 address-family ipv4 unicast route-policy PASS in route-policy PASS out ! ! ! end RP/0/3/CPU0:XR2#
RP/0/0/CPU0:XR1#show bgp vrf ABC ipv4 unicast summary Fri May 11 00:34:29.712 UTC BGP VRF ABC, state: Active BGP Route Distinguisher: 26:65001 VRF ID: 0x60000002 BGP router identifier 19.19.19.19, local AS number 26 BGP table state: Active Table ID: 0xe0000002 BGP main routing table version 1 BGP is operating in STANDALONE mode. Process RcvTblVer bRIB/RIB LabelVer ImportVer SendTblVer StandbyVer Speaker 1 1 1 1 1 1 Neighbor Spk AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down St/PfxRcd 10.19.20.20 0 65001 2 7 0 0 0 00:03:59 Idle
RP/0/3/CPU0:XR2#show bgp ipv4 unicast summary
Fri May 11 00:35:02.278 UTC BGP router identifier 0.0.0.0, local AS number 65001 BGP generic scan interval 60 secs BGP table state: Active Table ID: 0xe0000000 BGP main routing table version 1 BGP scan interval 60 secs BGP is operating in STANDALONE mode. Process RcvTblVer bRIB/RIB LabelVer ImportVer SendTblVer StandbyVer Speaker 1 1 1 1 1 1 Neighbor Spk AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down St/PfxRcd 10.19.20.19 0 26 2 2 0 0 0 00:04:31 Active
RP/0/0/CPU0:XR1#show bgp vrf ABC ipv4 unicast neighbors
Fri May 11 00:34:18.708 UTC
BGP neighbor is 10.19.20.20, vrf ABC
Remote AS 65001, local AS 26, external link
Remote router ID 0.0.0.0
BGP state = Idle
Last read 00:00:00, Last read before reset 00:04:10
Hold time is 180, keepalive interval is 60 seconds
Configured hold time: 180, keepalive: 60, min acceptable hold time: 3
Last write 00:00:15, attempted 53, written 53
Second last write 00:01:01, attempted 53, written 53
Last write before reset 00:04:10, attempted 72, written 72
Second last write before reset 00:04:15, attempted 53, written 53
Last write pulse rcvd May 11 00:34:02.927 last full not set pulse count 9
Last write pulse rcvd before reset 00:04:10
Socket not armed for io, not armed for read, not armed for write
Last write thread event before reset 00:04:10, second last 00:04:10
Last KA expiry before reset 00:00:00, second last 00:00:00
Last KA error before reset 00:00:00, KA not sent 00:00:00
Last KA start before reset 00:00:00, second last 00:00:00
Precedence: internet
Enforcing first AS is enabled
Received 2 messages, 0 notifications, 0 in queue
Sent 7 messages, 0 notifications, 0 in queue
Minimum time between advertisement runs is 0 secs
For Address Family: IPv4 Unicast
BGP neighbor version 0
Update group: 0.2
Route refresh request: received 0, sent 0
Policy for incoming advertisements is PASS
Policy for outgoing advertisements is PASS
0 accepted prefixes, 0 are bestpaths
Cumulative no. of prefixes denied: 0.
Prefix advertised 0, suppressed 0, withdrawn 0
Maximum prefixes allowed 524288
Threshold for warning message 75%, restart interval 0 min
AS override is set
An EoR was not received during read-only mode
Last ack version 0, Last synced ack version 0
Outstanding version objects: current 0, max 0
Connections established 1; dropped 1
Local host: 10.19.20.19, Local port: 19432
Foreign host: 10.19.20.20, Foreign port: 179
Last reset 00:00:15, due to Peer closing down the session
Peer reset reason: Remote closed the session (Connection timed out)
Time since last notification sent to neighbor: 00:02:11
Error Code: administrative shutdown
Notification data sent:
None
RP/0/3/CPU0:XR2#show bgp ipv4 unicast neighbors
Fri May 11 00:34:58.427 UTC
BGP neighbor is 10.19.20.19
Remote AS 26, local AS 65001, external link
Remote router ID 0.0.0.0
BGP state = Active
Last read 00:00:00, Last read before reset 00:04:50
Hold time is 180, keepalive interval is 60 seconds
Configured hold time: 180, keepalive: 60, min acceptable hold time: 3
Last write 00:04:50, attempted 19, written 19
Second last write 00:04:50, attempted 53, written 53
Last write before reset 00:04:50, attempted 19, written 19
Second last write before reset 00:04:50, attempted 53, written 53
Last write pulse rcvd May 11 00:30:08.305 last full not set pulse count 4
Last write pulse rcvd before reset 00:04:50
Socket not armed for io, not armed for read, not armed for write
Last write thread event before reset 00:04:50, second last 00:04:50
Last KA expiry before reset 00:00:00, second last 00:00:00
Last KA error before reset 00:00:00, KA not sent 00:00:00
Last KA start before reset 00:04:50, second last 00:00:00
Precedence: internet
Enforcing first AS is enabled
Received 2 messages, 0 notifications, 0 in queue
Sent 2 messages, 0 notifications, 0 in queue
Minimum time between advertisement runs is 30 secs
For Address Family: IPv4 Unicast
BGP neighbor version 0
Update group: 0.2
Route refresh request: received 0, sent 0
Policy for incoming advertisements is PASS
Policy for outgoing advertisements is PASS
0 accepted prefixes, 0 are bestpaths
Cumulative no. of prefixes denied: 0.
Prefix advertised 0, suppressed 0, withdrawn 0
Maximum prefixes allowed 524288
Threshold for warning message 75%, restart interval 0 min
An EoR was not received during read-only mode
Last ack version 0, Last synced ack version 0
Outstanding version objects: current 0, max 0
Connections established 1; dropped 1
Local host: 10.19.20.20, Local port: 60056
Foreign host: 10.19.20.19, Foreign port: 179
Last reset 00:02:27, due to Interface flap
Time since last notification sent to neighbor: 00:05:07
Error Code: administrative reset
Notification data sent:
None
About Brian McGahan, CCIE #8593, CCDE #2013::13:
Brian McGahan was one of the youngest engineers in the world to obtain the CCIE, having achieved his first CCIE in Routing & Switching at the age of 20 in 2002. Brian has been teaching and developing CCIE training courses for over 8 years, and has assisted thousands of engineers in obtaining their CCIE certification. When not teaching or developing new products Brian consults with large ISPs and enterprise customers in the midwest region of the United States.
Find all posts by Brian McGahan, CCIE #8593, CCDE #2013::13 | Visit Website
You can leave a response, or trackback from your own site.
26 Responses to “IOS XR Teaser – BGP as PE to CE for MPLS L3VPN”
Leave a Reply
It appears that BGP could not select a router ID on XR2, with out a valid router ID BGP cannot establish any peering sessions. I’m not currently as familiar with XR as I’d like to be but I believe that should produce an error message and is indicated by the router id reported as 0.0.0.0
Congrats Mark! You were the first one to respond correctly that XR2 does not have a BGP router-id, which prevents it from establishing peerings. I’ll send you an email offline about the rack tokens.
Just throwing this out there…
Why doesn’t XR2 have a BGP router ID?
My nose says that there’s a Lo0 configured with an IPv4 address or a bgp router-id missing…
No Loopback/”bgp router-id” on XR2. Thus XR2 doesn’t have a router-id and cannot establish the bgp session.
XR2 is missing configured Loopback interfaces:
BGP attempts to obtain a router ID in the following ways (in order of preference):
•By means of the address configured using the bgp router-id command in router configuration mode.
•By using the highest IPv4 address on a loopback interface in the system if the router is booted with saved loopback address configuration.
•By using the primary IPv4 address of the first loopback address that gets configured if there are not any in the saved configuration.
If none of these methods for obtaining a router ID succeeds, BGP does not have a router ID and cannot establish any peering sessions with BGP neighbors. In such an instance, an error message is entered in the system log, and the show bgp summary command displays a router ID of 0.0.0.0
remark: XR2 is missing configured loopback interface OR router ID (bgp router-id)
Hi Brian,
I had simulated using only bgp config, the peering is up, with xr 4.2.0 without included is-is config. Is it something wrong with is-is config? However this is pe-ce scenario, it should not matter. Thanks.
XR2 has no BGP Router-ID, so it won’t establish a session with XR1. Create a loopback interface or manually specify the router-id. More here: http://www.cisco.com/en/US/docs/ios_xr_sw/iosxr_r3.0/routing/configuration/guide/rc3bgp.html#wp1082684
Router XR2 has a router id of 0.0.0.0 because there’s no loopback or router-id configured, this needs to be set.
Hi
I dont have the setup with me to test this out but i beleive in IOS-XR, you need to have a host static route otherwise BGP neighbor will not come up.
In IOS, it is created automatically and hence not required. However in IOS-XR, you need to create it manually.
Hi Brian,
I think it is because on XR2, you didn’t have loopback interface as well as BGP Router ID was configured,
To clarify, what i meant is: for BGP Neighbor to be up there are 2 solutions:
+ Setup an interface loopback 0 + an ipv4 address
or
+ Configure bgp router-id = 10.19.20.20
Hello Brian,
My name is Carlos and I was at a really usefull bootcamp with Brian Dennis in Miami a few months ago, please say hi to him.
I think the problem you’re running into has to do with a lack of a loopback interface configured in XR2 or a router-id BGP command.
Because of that you end up not having a proper BGP identifier in this device and this is clearly showing in your command: “RP/0/3/CPU0:XR2#show bgp ipv4 unicast summary” with the result “BGP router identifier 0.0.0.0″
The overall behavior is as follows:
If you do not use the bgp router-id command to configure a router ID, an IP address is not configured on any loopback interface, and no global router ID is configured, BGP neighbors remain down.
(BGP Commands on Cisco IOS XR Software)
(http://www.cisco.com/en/US/partner/docs/ios_xr_sw/iosxr_r3.7/routing/command/reference/rr37bgp.html#wp1359292)
Regards…
Carlos
More clarification is at this URL on how IOS-XR chooses its BGP Router-ID
http://www.cisco.com/en/US/docs/routers/xr12000/software/xr12k_r3.9/routing/configuration/guide/b_xr12krc39_chapter_01.html#con_1618018
BGP Router Identifier
For BGP sessions between neighbors to be established, BGP must be assigned a router ID. The router ID is sent to BGP peers in the OPEN message when a BGP session is established. BGP attempts to obtain a router ID in the following ways (in order of preference):
1. By means of the address configured using the bgp router-id command in router configuration mode.
2. By using the highest IPv4 address on a loopback interface in the system if the router is booted with saved loopback address configuration.
3. By using the primary IPv4 address of the first loopback address that gets configured if there are not any in the saved configuration.
If none of these methods for obtaining a router ID succeeds, BGP does not have a router ID and cannot establish any peering sessions with BGP neighbors. In such an instance, an error message is entered in the system log, and the show bgp summary command displays a router ID of 0.0.0.0.
After BGP has obtained a router ID, it continues to use it even if a better router ID becomes available. This usage avoids unnecessary flapping for all BGP sessions. However, if the router ID currently in use becomes invalid (because the interface goes down or its configuration is changed), BGP selects a new router ID (using the rules described) and all established peering sessions are reset.
You have to hard code the BGP router-id on XR2. By default, BGP selects the router-id based on loopback addresses only. Since there is no loopback address configured on XR2 (only physical interfaces are configured), the BGP process does not have the router-id elected automatically.
Just to clarify if my explanation was not deep enough – from the IOS XR documentation:
BGP attempts to obtain a router ID in the following ways (in order of preference):
•By means of the address configured using the bgp router-id command in router configuration mode.
•By using the highest IPv4 address on a loopback interface in the system if the router is booted with saved loopback address configuration.
•By using the primary IPv4 address of the first loopback address that gets configured if there are not any in the saved configuration.
If none of these methods for obtaining a router ID succeeds, BGP does not have a router ID and cannot establish any peering sessions with BGP neighbors. In such an instance, an error message is entered in the system log, and the show bgp summary command displays a router ID of 0.0.0.0.
Hi Brian,
The issue might be relevant to BGP Router-ID on XR2 as in accordance with the extraction, BGP Router-ID on XR2 is 0.0.0.0!
Regards,
Tuan
I saw this problem before on one of Cisco Forum 2012 presentations by Łukasz Bromirski & Piotr Jabłoński. It looks like the source of the problem is lack of IPv6 addresses on loopback interfaces which are used to establish BGP session. I don’t have an access to some hardware to verify this so please somebody confirm if that is the case.
It might be that the XR2 does not have a loopback interface to get the router-id from…
http://www.cisco.com/en/US/docs/routers/xr12000/software/xr12k_r3.9/routing/configuration/guide/b_xr12krc39_chapter_01.html#con_1618018
Brian…even though its mostly applied to MPLS Inter-AS VPN and CSC scenarios, try using the static route trick. It might work the magic. I am still to try it tho. I had this same problem while using Gigavelocity’s racks but up till now, I haven’t wrapped my head around it
Hm, it’s kinda long shot, but is it perhaps because we don’t have router-ID configured under the CE’s BGP process ? There isn’t a loopback either and I think in the XR if the router is not assigned with manual BGP ID it can pick up an IP only from a loopback interface ? =D
Brian,
in order for the neighbors to establish, need to specify the router-id or have loopbacks on both XR1 (vrf ABC) and XR2.
CCIE #25143 (RS/SP)
Brian,
I am just glad you didn’t get to the point of smashing your head on the keyboard , we still want to see the complete version of your workbook !!
Let’s get all of the CCIE SPv3 workbook content out of your head before your smash it please !
Dear all,
has any one tried the solution?
because i think something is still missing
Hmm , I think as per my understanding with XR you need explicit Static Route for any Ebgp connection , where you are trying to establish this ebgp between physical link .
I have not tried but ill try in coming weekend .
Please correct me if i’m wrong …………..I’m just small fish