May
10

Update: Congrats to Mark, our winner of 100 rack rental tokens for the first correct answer, that XR2 is missing a BGP router-id.  In regular IOS, a router-id is chosen based on the highest Loopback interface.  If there is no Loopback interface the highest IP address of all up/up interfaces is chosen.  In the case of IOS XR however, the router-id will not be chosen from a physical link.  It will only be chosen from the highest Loopback interface, or from the manual router-id command.  Per the Cisco documentation:

BGP Router Identifier

For BGP sessions between neighbors to be established, BGP must be assigned a router ID. The router ID is sent to BGP peers in the OPEN message when a BGP session is established.

BGP attempts to obtain a router ID in the following ways (in order of preference):

  • By means of the address configured using the bgp router-id command in router configuration mode.
  • By using the highest IPv4 address on a loopback interface in the system if the router is booted with saved loopback address configuration.
  • By using the primary IPv4 address of the first loopback address that gets configured if there are not any in the saved configuration.

If none of these methods for obtaining a router ID succeeds, BGP does not have a router ID and cannot establish any peering sessions with BGP neighbors. In such an instance, an error message is entered in the system log, and the show bgp summary command displays a router ID of 0.0.0.0.

After BGP has obtained a router ID, it continues to use it even if a better router ID becomes available. This usage avoids unnecessary flapping for all BGP sessions. However, if the router ID currently in use becomes invalid (because the interface goes down or its configuration is changed), BGP selects a new router ID (using the rules described) and all established peering sessions are reset.

Since XR2 in this case does not have a Loopback configured, the BGP process cannot initialize.  The kicker with this problem is that the documentation states that when this problem occurs you should see that “an error message is entered in the system log”, however in this case a Syslog was not generated about the error.  At least this is the last time this problem will bite me ;)

 


Today while working on additional content for our CCIE Service Provider Version 3.0 Lab Workbook I had one of those epic brain fart moments.  What started off as work on (what I thought was) a fairly simply design ended up as a 2 hour troubleshooting rabbit hole of rolling back config snippets one by one, debugging, and basically overall misery that can be perfectly summed up by this GIF of a guy smashing his head against his keyboard. :)

The scenario in question was a BGP peering between two IOS XR routers.  One was the PE of an MPLS L3VPN network and one was the CE.  As I’ve done this config literally hundreds of times in the past I could not for the life of me figure out why the BGP peering would not establish.  The relevant snippet of the topology diagram is as follows:

Since this scenario caused me so much pleasure I am offering 100 tokens good for CCIE Service Provider Version 3.0 Rack Rentals - or any of our other Routing & Switching rack rentals & mock labs, Security rack rentals, or Voice rack rentals – to whoever the first person is that can tell me why did these neighbors not establish a BGP peering.  The relevant outputs needed to troubleshoot the problem can be found below.  I still haven’t decided whether I’m going to leave this problem in the workbook or not since it’s such a mean one :)  Good luck!

 

 

RP/0/0/CPU0:XR1#show run
Fri May 11 00:34:38.563 UTC
Building configuration...
!! IOS XR Configuration 3.9.1
!! Last configuration change at Fri May 11 00:32:50 2012 by xr1
!
hostname XR1
username xr1
 group root-lr
 password 7 13061E010803
!
vrf ABC
 address-family ipv4 unicast
  import route-target
   26:65001
  !
  export route-target
   26:65001
  !
 !
!
line console
 exec-timeout 0 0
!
ipv4 access-list PE_ROUTERS
 10 permit ipv4 host 1.1.1.1 any
 20 permit ipv4 host 2.2.2.2 any
 30 permit ipv4 host 5.5.5.5 any
 40 permit ipv4 host 19.19.19.19 any
!
interface Loopback0
 ipv4 address 19.19.19.19 255.255.255.255
!
interface GigabitEthernet0/1/0/0
 ipv4 address 172.19.10.19 255.255.255.0
!
interface GigabitEthernet0/1/0/1
 ipv4 address 26.3.19.19 255.255.255.0
!
interface POS0/6/0/0
 vrf ABC
 ipv4 address 10.19.20.19 255.255.255.0
!
route-policy PASS
  pass
end-policy
!
router isis 1
 is-type level-2-only
 net 49.0001.0000.0000.0019.00
 address-family ipv4 unicast
  mpls ldp auto-config
 !
 interface Loopback0
  passive
  address-family ipv4 unicast
  !
 !
 interface GigabitEthernet0/1/0/1
  point-to-point
  hello-password hmac-md5 encrypted 022527722E
  address-family ipv4 unicast
  !
 !
!
router bgp 26
 address-family ipv4 unicast
 !
 ! address-family ipv4 unicast
 address-family vpnv4 unicast
 !
 neighbor-group PE_ROUTERS
  remote-as 26
  update-source Loopback0
  address-family vpnv4 unicast
  !
 !
 neighbor 1.1.1.1
  use neighbor-group PE_ROUTERS
 !
 neighbor 2.2.2.2
  use neighbor-group PE_ROUTERS
 !
 neighbor 5.5.5.5
  use neighbor-group PE_ROUTERS
 !
 vrf ABC
  rd 26:65001
  address-family ipv4 unicast
  !
  neighbor 10.19.20.20
   remote-as 65001
   address-family ipv4 unicast
    route-policy PASS in
    route-policy PASS out
    as-override
   !
  !
 !
!
mpls ldp
 label
  allocate for PE_ROUTERS
 !
!
end

RP/0/0/CPU0:XR1#
RP/0/3/CPU0:XR2#show run 
Fri May 11 00:35:04.932 UTC
Building configuration...
!! IOS XR Configuration 3.9.1
!! Last configuration change at Fri May 11 00:30:30 2012 by xr2
!
hostname XR2
logging console debugging
username xr2
 group root-lr
 password 7 00071A150754
!
cdp
line console
 exec-timeout 0 0
!
interface GigabitEthernet0/4/0/0
 ipv4 address 10.20.20.20 255.255.255.0
 ipv6 address 2001:10:20:20::20/64
!
interface POS0/7/0/0
 ipv4 address 10.19.20.20 255.255.255.0
 ipv6 address 2001:10:19:20::20/64
!
route-policy PASS
  pass
end-policy
!
router bgp 65001
 address-family ipv4 unicast
 !
 neighbor 10.19.20.19
  remote-as 26
  address-family ipv4 unicast
   route-policy PASS in
   route-policy PASS out
  !
 !
!
end

RP/0/3/CPU0:XR2#
RP/0/0/CPU0:XR1#show bgp vrf ABC ipv4 unicast summary 
Fri May 11 00:34:29.712 UTC
BGP VRF ABC, state: Active
BGP Route Distinguisher: 26:65001
VRF ID: 0x60000002
BGP router identifier 19.19.19.19, local AS number 26
BGP table state: Active
Table ID: 0xe0000002
BGP main routing table version 1

BGP is operating in STANDALONE mode.

Process       RcvTblVer   bRIB/RIB   LabelVer  ImportVer  SendTblVer  StandbyVer
Speaker               1          1          1          1           1           1

Neighbor        Spk    AS MsgRcvd MsgSent   TblVer  InQ OutQ  Up/Down  St/PfxRcd
10.19.20.20       0 65001       2       7        0    0    0 00:03:59 Idle

RP/0/3/CPU0:XR2#show bgp ipv4 unicast summary
Fri May 11 00:35:02.278 UTC
BGP router identifier 0.0.0.0, local AS number 65001
BGP generic scan interval 60 secs
BGP table state: Active
Table ID: 0xe0000000
BGP main routing table version 1
BGP scan interval 60 secs

BGP is operating in STANDALONE mode.

Process       RcvTblVer   bRIB/RIB   LabelVer  ImportVer  SendTblVer  StandbyVer
Speaker               1          1          1          1           1           1

Neighbor        Spk    AS MsgRcvd MsgSent   TblVer  InQ OutQ  Up/Down  St/PfxRcd
10.19.20.19       0    26       2       2        0    0    0 00:04:31 Active

RP/0/0/CPU0:XR1#show bgp vrf ABC ipv4 unicast neighbors 
Fri May 11 00:34:18.708 UTC

BGP neighbor is 10.19.20.20, vrf ABC
 Remote AS 65001, local AS 26, external link
 Remote router ID 0.0.0.0
  BGP state = Idle
  Last read 00:00:00, Last read before reset 00:04:10
  Hold time is 180, keepalive interval is 60 seconds
  Configured hold time: 180, keepalive: 60, min acceptable hold time: 3
  Last write 00:00:15, attempted 53, written 53
  Second last write 00:01:01, attempted 53, written 53
  Last write before reset 00:04:10, attempted 72, written 72
  Second last write before reset 00:04:15, attempted 53, written 53
  Last write pulse rcvd  May 11 00:34:02.927 last full not set pulse count 9
  Last write pulse rcvd before reset 00:04:10
  Socket not armed for io, not armed for read, not armed for write
  Last write thread event before reset 00:04:10, second last 00:04:10
  Last KA expiry before reset 00:00:00, second last 00:00:00
  Last KA error before reset 00:00:00, KA not sent 00:00:00
  Last KA start before reset 00:00:00, second last 00:00:00
  Precedence: internet
  Enforcing first AS is enabled
  Received 2 messages, 0 notifications, 0 in queue
  Sent 7 messages, 0 notifications, 0 in queue
  Minimum time between advertisement runs is 0 secs

 For Address Family: IPv4 Unicast
  BGP neighbor version 0
  Update group: 0.2
  Route refresh request: received 0, sent 0
  Policy for incoming advertisements is PASS
  Policy for outgoing advertisements is PASS
  0 accepted prefixes, 0 are bestpaths
  Cumulative no. of prefixes denied: 0.
  Prefix advertised 0, suppressed 0, withdrawn 0
  Maximum prefixes allowed 524288
  Threshold for warning message 75%, restart interval 0 min
  AS override is set
  An EoR was not received during read-only mode
  Last ack version 0, Last synced ack version 0
  Outstanding version objects: current 0, max 0

  Connections established 1; dropped 1
  Local host: 10.19.20.19, Local port: 19432
  Foreign host: 10.19.20.20, Foreign port: 179
  Last reset 00:00:15, due to Peer closing down the session
  Peer reset reason: Remote closed the session (Connection timed out)
  Time since last notification sent to neighbor: 00:02:11
  Error Code: administrative shutdown
  Notification data sent:
    None
RP/0/3/CPU0:XR2#show bgp ipv4 unicast neighbors 
Fri May 11 00:34:58.427 UTC

BGP neighbor is 10.19.20.19
 Remote AS 26, local AS 65001, external link
 Remote router ID 0.0.0.0
  BGP state = Active
  Last read 00:00:00, Last read before reset 00:04:50
  Hold time is 180, keepalive interval is 60 seconds
  Configured hold time: 180, keepalive: 60, min acceptable hold time: 3
  Last write 00:04:50, attempted 19, written 19
  Second last write 00:04:50, attempted 53, written 53
  Last write before reset 00:04:50, attempted 19, written 19
  Second last write before reset 00:04:50, attempted 53, written 53
  Last write pulse rcvd  May 11 00:30:08.305 last full not set pulse count 4
  Last write pulse rcvd before reset 00:04:50
  Socket not armed for io, not armed for read, not armed for write
  Last write thread event before reset 00:04:50, second last 00:04:50
  Last KA expiry before reset 00:00:00, second last 00:00:00
  Last KA error before reset 00:00:00, KA not sent 00:00:00
  Last KA start before reset 00:04:50, second last 00:00:00
  Precedence: internet
  Enforcing first AS is enabled
  Received 2 messages, 0 notifications, 0 in queue
  Sent 2 messages, 0 notifications, 0 in queue
  Minimum time between advertisement runs is 30 secs

 For Address Family: IPv4 Unicast
  BGP neighbor version 0
  Update group: 0.2
  Route refresh request: received 0, sent 0
  Policy for incoming advertisements is PASS
  Policy for outgoing advertisements is PASS
  0 accepted prefixes, 0 are bestpaths
  Cumulative no. of prefixes denied: 0.
  Prefix advertised 0, suppressed 0, withdrawn 0
  Maximum prefixes allowed 524288
  Threshold for warning message 75%, restart interval 0 min
  An EoR was not received during read-only mode
  Last ack version 0, Last synced ack version 0
  Outstanding version objects: current 0, max 0

  Connections established 1; dropped 1
  Local host: 10.19.20.20, Local port: 60056
  Foreign host: 10.19.20.19, Foreign port: 179
  Last reset 00:02:27, due to Interface flap
  Time since last notification sent to neighbor: 00:05:07
  Error Code: administrative reset
  Notification data sent:
    None

					
About Brian McGahan, CCIE #8593, CCDE #2013::13:

Brian McGahan was one of the youngest engineers in the world to obtain the CCIE, having achieved his first CCIE in Routing & Switching at the age of 20 in 2002. Brian has been teaching and developing CCIE training courses for over 8 years, and has assisted thousands of engineers in obtaining their CCIE certification. When not teaching or developing new products Brian consults with large ISPs and enterprise customers in the midwest region of the United States.

Find all posts by Brian McGahan, CCIE #8593, CCDE #2013::13 | Visit Website


You can leave a response, or trackback from your own site.

26 Responses to “IOS XR Teaser – BGP as PE to CE for MPLS L3VPN”

 
  1. Mark says:

    It appears that BGP could not select a router ID on XR2, with out a valid router ID BGP cannot establish any peering sessions. I’m not currently as familiar with XR as I’d like to be but I believe that should produce an error message and is indicated by the router id reported as 0.0.0.0

  2. Stejarel Veres says:

    Just throwing this out there…

    Why doesn’t XR2 have a BGP router ID? :)

    My nose says that there’s a Lo0 configured with an IPv4 address or a bgp router-id missing…

  3. Guillermo says:

    No Loopback/”bgp router-id” on XR2. Thus XR2 doesn’t have a router-id and cannot establish the bgp session.

  4. Evgeny says:

    XR2 is missing configured Loopback interfaces:

    BGP attempts to obtain a router ID in the following ways (in order of preference):

    •By means of the address configured using the bgp router-id command in router configuration mode.

    •By using the highest IPv4 address on a loopback interface in the system if the router is booted with saved loopback address configuration.

    •By using the primary IPv4 address of the first loopback address that gets configured if there are not any in the saved configuration.

    If none of these methods for obtaining a router ID succeeds, BGP does not have a router ID and cannot establish any peering sessions with BGP neighbors. In such an instance, an error message is entered in the system log, and the show bgp summary command displays a router ID of 0.0.0.0

  5. Dennis says:

    Hi Brian,

    I had simulated using only bgp config, the peering is up, with xr 4.2.0 without included is-is config. Is it something wrong with is-is config? However this is pe-ce scenario, it should not matter. Thanks.

  6. Brad Holding says:

    XR2 has no BGP Router-ID, so it won’t establish a session with XR1. Create a loopback interface or manually specify the router-id. More here: http://www.cisco.com/en/US/docs/ios_xr_sw/iosxr_r3.0/routing/configuration/guide/rc3bgp.html#wp1082684

  7. John Yeh says:

    Router XR2 has a router id of 0.0.0.0 because there’s no loopback or router-id configured, this needs to be set.

  8. Sundeep Sadhwani says:

    Hi

    I dont have the setup with me to test this out but i beleive in IOS-XR, you need to have a host static route otherwise BGP neighbor will not come up.

    In IOS, it is created automatically and hence not required. However in IOS-XR, you need to create it manually.

  9. Huy Nguyen says:

    Hi Brian,

    I think it is because on XR2, you didn’t have loopback interface as well as BGP Router ID was configured,

    • Huy Nguyen says:

      To clarify, what i meant is: for BGP Neighbor to be up there are 2 solutions:
      + Setup an interface loopback 0 + an ipv4 address
      or
      + Configure bgp router-id = 10.19.20.20

  10. Carlos Santín says:

    Hello Brian,

    My name is Carlos and I was at a really usefull bootcamp with Brian Dennis in Miami a few months ago, please say hi to him.

    I think the problem you’re running into has to do with a lack of a loopback interface configured in XR2 or a router-id BGP command.

    Because of that you end up not having a proper BGP identifier in this device and this is clearly showing in your command: “RP/0/3/CPU0:XR2#show bgp ipv4 unicast summary” with the result “BGP router identifier 0.0.0.0″

    The overall behavior is as follows:

    If you do not use the bgp router-id command to configure a router ID, an IP address is not configured on any loopback interface, and no global router ID is configured, BGP neighbors remain down.
    (BGP Commands on Cisco IOS XR Software)
    (http://www.cisco.com/en/US/partner/docs/ios_xr_sw/iosxr_r3.7/routing/command/reference/rr37bgp.html#wp1359292)

    Regards…

    Carlos

  11. Huy Nguyen says:

    More clarification is at this URL on how IOS-XR chooses its BGP Router-ID
    http://www.cisco.com/en/US/docs/routers/xr12000/software/xr12k_r3.9/routing/configuration/guide/b_xr12krc39_chapter_01.html#con_1618018

    BGP Router Identifier

    For BGP sessions between neighbors to be established, BGP must be assigned a router ID. The router ID is sent to BGP peers in the OPEN message when a BGP session is established. BGP attempts to obtain a router ID in the following ways (in order of preference):

    1. By means of the address configured using the bgp router-id command in router configuration mode.
    2. By using the highest IPv4 address on a loopback interface in the system if the router is booted with saved loopback address configuration.
    3. By using the primary IPv4 address of the first loopback address that gets configured if there are not any in the saved configuration.

    If none of these methods for obtaining a router ID succeeds, BGP does not have a router ID and cannot establish any peering sessions with BGP neighbors. In such an instance, an error message is entered in the system log, and the show bgp summary command displays a router ID of 0.0.0.0.

    After BGP has obtained a router ID, it continues to use it even if a better router ID becomes available. This usage avoids unnecessary flapping for all BGP sessions. However, if the router ID currently in use becomes invalid (because the interface goes down or its configuration is changed), BGP selects a new router ID (using the rules described) and all established peering sessions are reset.

  12. David Spudich says:

    You have to hard code the BGP router-id on XR2. By default, BGP selects the router-id based on loopback addresses only. Since there is no loopback address configured on XR2 (only physical interfaces are configured), the BGP process does not have the router-id elected automatically.

    • David Spudich says:

      Just to clarify if my explanation was not deep enough – from the IOS XR documentation:

      BGP attempts to obtain a router ID in the following ways (in order of preference):

      •By means of the address configured using the bgp router-id command in router configuration mode.

      •By using the highest IPv4 address on a loopback interface in the system if the router is booted with saved loopback address configuration.

      •By using the primary IPv4 address of the first loopback address that gets configured if there are not any in the saved configuration.

      If none of these methods for obtaining a router ID succeeds, BGP does not have a router ID and cannot establish any peering sessions with BGP neighbors. In such an instance, an error message is entered in the system log, and the show bgp summary command displays a router ID of 0.0.0.0.

  13. Tuan Nguyen says:

    Hi Brian,

    The issue might be relevant to BGP Router-ID on XR2 as in accordance with the extraction, BGP Router-ID on XR2 is 0.0.0.0!

    Regards,
    Tuan

  14. Krzysztof Ciepłucha says:

    I saw this problem before on one of Cisco Forum 2012 presentations by Łukasz Bromirski & Piotr Jabłoński. It looks like the source of the problem is lack of IPv6 addresses on loopback interfaces which are used to establish BGP session. I don’t have an access to some hardware to verify this so please somebody confirm if that is the case.

  15. hudson says:

    Brian…even though its mostly applied to MPLS Inter-AS VPN and CSC scenarios, try using the static route trick. It might work the magic. I am still to try it tho. I had this same problem while using Gigavelocity’s racks but up till now, I haven’t wrapped my head around it

  16. Daniel says:

    Hm, it’s kinda long shot, but is it perhaps because we don’t have router-ID configured under the CE’s BGP process ? There isn’t a loopback either and I think in the XR if the router is not assigned with manual BGP ID it can pick up an IP only from a loopback interface ? =D

  17. John Spaulding says:

    Brian,

    in order for the neighbors to establish, need to specify the router-id or have loopbacks on both XR1 (vrf ABC) and XR2.

    CCIE #25143 (RS/SP)

  18. Yannick, CCIE # 20413 says:

    Brian,
    I am just glad you didn’t get to the point of smashing your head on the keyboard , we still want to see the complete version of your workbook !! :-)

  19. Yannick, CCIE # 20413 says:

    Let’s get all of the CCIE SPv3 workbook content out of your head before your smash it please ! :-)

  20. Mekadoy says:

    Dear all,

    has any one tried the solution? :) because i think something is still missing

  21. Ritesh says:

    Hmm , I think as per my understanding with XR you need explicit Static Route for any Ebgp connection , where you are trying to establish this ebgp between physical link .

    I have not tried but ill try in coming weekend .

    Please correct me if i’m wrong …………..I’m just small fish :)

 

Leave a Reply

Categories

CCIE Bloggers