OSPF LFA

CSR1#sh ip route ospf
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       a - application route
       + - replicated route, % - next hop override, p - overrides from PfR

Gateway of last resort is not set

      3.0.0.0/32 is subnetted, 1 subnets
O        3.3.3.3 [110/2] via 192.168.13.3, 00:00:58, GigabitEthernet3
O     192.168.23.0/24 [110/2] via 192.168.13.3, 00:00:58, GigabitEthernet3
                      [110/2] via 192.168.12.2, 00:04:05, GigabitEthernet1
O     192.168.34.0/24 [110/2] via 192.168.14.4, 00:01:38, GigabitEthernet2
                      [110/2] via 192.168.13.3, 00:00:58, GigabitEthernet3

CSR1#sh ip cef 3.3.3.3/32
3.3.3.3/32
  nexthop 192.168.13.3 GigabitEthernet3

router ospf 1
fast-reroute per-prefix enable area 0 prefix-priority high

CSR1#sh ip cef 3.3.3.3/32
3.3.3.3/32
  nexthop 192.168.13.3 GigabitEthernet3
    repair: attached-nexthop 192.168.12.2 GigabitEthernet1

Let us try to shut down the link between CSR1 and CSR3 (which is the preferred path):

CSR1#sh ip route ospf
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       a - application route
       + - replicated route, % - next hop override, p - overrides from PfR

Gateway of last resort is not set

      3.0.0.0/32 is subnetted, 1 subnets
O        3.3.3.3 [110/3] via 192.168.14.4, 00:00:01, GigabitEthernet2
                 [110/3] via 192.168.12.2, 00:00:01, GigabitEthernet1
O     192.168.13.0/24 [110/3] via 192.168.14.4, 00:00:01, GigabitEthernet2
                      [110/3] via 192.168.12.2, 00:00:01, GigabitEthernet1
O     192.168.23.0/24 [110/2] via 192.168.12.2, 00:06:29, GigabitEthernet1
O     192.168.34.0/24 [110/2] via 192.168.14.4, 00:04:02, GigabitEthernet2

CSR1#show ip cef 3.3.3.3/32
3.3.3.3/32
  nexthop 192.168.12.2 GigabitEthernet1
    repair: attached-nexthop 192.168.14.4 GigabitEthernet2
  nexthop 192.168.14.4 GigabitEthernet2
    repair: attached-nexthop 192.168.12.2 GigabitEthernet1

CSR1#sh ip route 3.3.3.3
Routing entry for 3.3.3.3/32
  Known via "ospf 1", distance 110, metric 3, type intra area
  Last update from 192.168.12.2 on GigabitEthernet1, 00:00:25 ago
  Routing Descriptor Blocks:
  * 192.168.14.4, from 3.3.3.3, 00:00:25 ago, via GigabitEthernet2
      Route metric is 3, traffic share count is 1
      Repair Path: 192.168.12.2, via GigabitEthernet1
    192.168.12.2, from 3.3.3.3, 00:00:25 ago, via GigabitEthernet1
      Route metric is 3, traffic share count is 1
      Repair Path: 192.168.14.4, via GigabitEthernet2

CSR1#sh ip route repair-paths 3.3.3.3
Routing entry for 3.3.3.3/32
  Known via "ospf 1", distance 110, metric 3, type intra area
  Last update from 192.168.12.2 on GigabitEthernet1, 00:00:51 ago
  Routing Descriptor Blocks:
  * 192.168.14.4, from 3.3.3.3, 00:00:51 ago, via GigabitEthernet2
      Route metric is 3, traffic share count is 1
      Repair Path: 192.168.12.2, via GigabitEthernet1
    192.168.12.2, from 3.3.3.3, 00:00:51 ago, via GigabitEthernet1
      Route metric is 3, traffic share count is 1
      Repair Path: 192.168.14.4, via GigabitEthernet2

Brining back the link online:

CSR1#sh ip route ospf
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       a - application route
       + - replicated route, % - next hop override, p - overrides from PfR

Gateway of last resort is not set

      3.0.0.0/32 is subnetted, 1 subnets
O        3.3.3.3 [110/3] via 192.168.14.4, 00:03:15, GigabitEthernet2
                 [110/3] via 192.168.12.2, 00:03:15, GigabitEthernet1
O     192.168.23.0/24 [110/2] via 192.168.12.2, 00:09:43, GigabitEthernet1
O     192.168.34.0/24 [110/2] via 192.168.14.4, 00:07:16, GigabitEthernet2

Changing all costs to 10 and investigating needed outputs:

CSR1#sh ip route ospf
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       a - application route
       + - replicated route, % - next hop override, p - overrides from PfR

Gateway of last resort is not set

      3.0.0.0/32 is subnetted, 1 subnets
O        3.3.3.3 [110/11] via 192.168.13.3, 00:03:22, GigabitEthernet3
O     192.168.23.0/24 [110/20] via 192.168.13.3, 00:02:36, GigabitEthernet3
                      [110/20] via 192.168.12.2, 00:02:36, GigabitEthernet1
O     192.168.34.0/24 [110/20] via 192.168.14.4, 00:02:10, GigabitEthernet2
                      [110/20] via 192.168.13.3, 00:02:10, GigabitEthernet3

CSR1#sh ip cef 3.3.3.3
3.3.3.3/32
  nexthop 192.168.13.3 GigabitEthernet3
    repair: attached-nexthop 192.168.14.4 GigabitEthernet2

CSR1#sh ip ospf rib 3.3.3.3

            OSPF Router with ID (1.1.1.1) (Process ID 1)

                Base Topology (MTID 0)

OSPF local RIB
Codes: * - Best, > - Installed in global RIB
LSA: type/LSID/originator

*>  3.3.3.3/32, Intra, cost 11, area 0
     SPF Instance 33, age 00:02:21
     Flags: RIB, HiPrio
      via 192.168.13.3, GigabitEthernet3
       Flags: RIB
       LSA: 1/3.3.3.3/3.3.3.3
      repair path via 192.168.14.4, GigabitEthernet2, cost 21
       Flags: RIB, Repair, IntfDj, BcastDj
       LSA: 1/3.3.3.3/3.3.3.3
      repair path via 192.168.12.2, GigabitEthernet1, cost 21
       Flags: Ignore, Repair, IntfDj, BcastDj, SRLG
       LSA: 1/3.3.3.3/3.3.3.3

As can be seen from the output above , the primary path is via the direct link between CSR1 and CSR3
And the repair path is via the link between CSR1 and CSR4

CSR1#sh ip route 3.3.3.3 255.255.255.255
Routing entry for 3.3.3.3/32
  Known via "ospf 1", distance 110, metric 11, type intra area
  Last update from 192.168.13.3 on GigabitEthernet3, 00:08:33 ago
  Routing Descriptor Blocks:
  * 192.168.13.3, from 3.3.3.3, 00:08:33 ago, via GigabitEthernet3
      Route metric is 11, traffic share count is 1
      Repair Path: 192.168.14.4, via GigabitEthernet2

Let us now shut down the backup link and check if R2 satisfies the link protection inequality
The link protection inequality which is similar in terminology to EIGRP feasiable successor

D(N,D) < D(N,S) + D(S,D)

Where
D : Distance (Cost) (D outside parentheses)
S : Source
D : Destination
N : Candidate

CSR1#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
CSR1(config)#int g2
CSR1(config-if)#shut

CSR1#sh ip route 3.3.3.3 255.255.255.255
Routing entry for 3.3.3.3/32
  Known via "ospf 1", distance 110, metric 11, type intra area
  Last update from 192.168.13.3 on GigabitEthernet3, 00:00:27 ago
  Routing Descriptor Blocks:
  * 192.168.13.3, from 3.3.3.3, 00:00:27 ago, via GigabitEthernet3
      Route metric is 11, traffic share count is 1
      Repair Path: 192.168.12.2, via GigabitEthernet1

Let us bring back the link and try to modify the costs through R2 in order to be not elected for backup:
Adjusting the cost on the link between R2 and R3 to be 30

CSR1#sh ip ospf rib 3.3.3.3

            OSPF Router with ID (1.1.1.1) (Process ID 1)

                Base Topology (MTID 0)

OSPF local RIB
Codes: * - Best, > - Installed in global RIB
LSA: type/LSID/originator

*>  3.3.3.3/32, Intra, cost 11, area 0
     SPF Instance 38, age 00:12:55
     Flags: RIB, HiPrio
      via 192.168.13.3, GigabitEthernet3
       Flags: RIB
       LSA: 1/3.3.3.3/3.3.3.3
      repair path via 192.168.14.4, GigabitEthernet2, cost 21
       Flags: RIB, Repair, IntfDj, BcastDj
       LSA: 1/3.3.3.3/3.3.3.3

As can be seen , the backup path through R2 is missing , now let us shut down the link between CSR1 and CSR4 (the existing backup)

CSR1#sh ip ospf rib 3.3.3.3

            OSPF Router with ID (1.1.1.1) (Process ID 1)

                Base Topology (MTID 0)

OSPF local RIB
Codes: * - Best, > - Installed in global RIB
LSA: type/LSID/originator

*>  3.3.3.3/32, Intra, cost 11, area 0
     SPF Instance 41, age 00:14:34
     Flags: RIB, HiPrio
      via 192.168.13.3, GigabitEthernet3
       Flags: RIB
       LSA: 1/3.3.3.3/3.3.3.3

CSR1#sh ip route 3.3.3.3
Routing entry for 3.3.3.3/32
  Known via "ospf 1", distance 110, metric 11, type intra area
  Last update from 192.168.13.3 on GigabitEthernet3, 00:04:18 ago
  Routing Descriptor Blocks:
  * 192.168.13.3, from 3.3.3.3, 00:04:18 ago, via GigabitEthernet3
      Route metric is 11, traffic share count is 1

CSR1#sh ip cef 3.3.3.3/32
3.3.3.3/32
  nexthop 192.168.13.3 GigabitEthernet3

Now , no backup path is availabe

By default , OSPF treats /32 routes as high priority routes , let us check the output for abother route:

CSR1#sh ip ospf rib 192.168.23.0

            OSPF Router with ID (1.1.1.1) (Process ID 1)

                Base Topology (MTID 0)

OSPF local RIB
Codes: * - Best, > - Installed in global RIB
LSA: type/LSID/originator

*>  192.168.23.0/24, Intra, cost 40, area 0
     SPF Instance 44, age 00:32:22
     Flags: RIB
      via 192.168.12.2, GigabitEthernet1
       Flags: RIB
       LSA: 2/192.168.23.2/2.2.2.2
      via 192.168.13.3, GigabitEthernet3
       Flags: RIB
       LSA: 2/192.168.23.2/2.2.2.2

Let us try to make the route 192.168.23.0/24 of high priority

ip prefix-list MSSK seq 5 permit 192.168.23.0/24

route-map MAP permit 10
 match ip address prefix-list MSSK

router ospf 1
prefix-priority high route-map MAP

CSR1#sh ip ospf rib 192.168.23.0

            OSPF Router with ID (1.1.1.1) (Process ID 1)

                Base Topology (MTID 0)

OSPF local RIB
Codes: * - Best, > - Installed in global RIB
LSA: type/LSID/originator

*>  192.168.23.0/24, Intra, cost 40, area 0
     SPF Instance 45, age 00:36:38
     Flags: RIB, HiPrio
      via 192.168.12.2, GigabitEthernet1
       Flags: RIB
       LSA: 2/192.168.23.2/2.2.2.2
      repair path via 192.168.13.3, GigabitEthernet3, cost 40
       Flags: RIB, Repair, IntfDj, BcastDj, PrimPath, NodeProt, Downstr
       LSA: 2/192.168.23.2/2.2.2.2
      repair path via 192.168.14.4, GigabitEthernet2, cost 50
       Flags: Ignore, Repair, IntfDj, BcastDj, NodeProt
       LSA: 2/192.168.23.2/2.2.2.2
      via 192.168.13.3, GigabitEthernet3
       Flags: RIB
       LSA: 2/192.168.23.2/2.2.2.2
      repair path via 192.168.12.2, GigabitEthernet1, cost 40
       Flags: RIB, Repair, IntfDj, BcastDj, PrimPath, NodeProt, Downstr
       LSA: 2/192.168.23.2/2.2.2.2
      repair path via 192.168.14.4, GigabitEthernet2, cost 50
       Flags: Ignore, Repair, IntfDj, BcastDj
       LSA: 2/192.168.23.2/2.2.2.2

Now as there is an explicit deny , 3.3.3.3/32 is no longer high priority:

CSR1#sh ip ospf rib 3.3.3.3

            OSPF Router with ID (1.1.1.1) (Process ID 1)

                Base Topology (MTID 0)

OSPF local RIB
Codes: * - Best, > - Installed in global RIB
LSA: type/LSID/originator

*>  3.3.3.3/32, Intra, cost 11, area 0
     SPF Instance 45, age 00:50:46
     Flags: RIB
      via 192.168.13.3, GigabitEthernet3
       Flags: RIB
       LSA: 1/3.3.3.3/3.3.3.3

ip prefix-list MSSK seq 10 permit 3.3.3.3/32

CSR1#sh ip ospf rib 3.3.3.3

            OSPF Router with ID (1.1.1.1) (Process ID 1)

                Base Topology (MTID 0)

OSPF local RIB
Codes: * - Best, > - Installed in global RIB
LSA: type/LSID/originator

*>  3.3.3.3/32, Intra, cost 11, area 0
     SPF Instance 46, age 00:51:39
     Flags: RIB, HiPrio
      via 192.168.13.3, GigabitEthernet3
       Flags: RIB
       LSA: 1/3.3.3.3/3.3.3.3

We can exclude an interface from being elected as a candidate for backup using the below command:

interface gig2
ip ospf fast-reroute per-prefix candidate disable

CSR1#sh ip ospf rib 3.3.3.3

            OSPF Router with ID (1.1.1.1) (Process ID 1)

                Base Topology (MTID 0)

OSPF local RIB
Codes: * - Best, > - Installed in global RIB
LSA: type/LSID/originator

*>  3.3.3.3/32, Intra, cost 11, area 0
     SPF Instance 48, age 00:55:13
     Flags: RIB, HiPrio
      via 192.168.13.3, GigabitEthernet3
       Flags: RIB
       LSA: 1/3.3.3.3/3.3.3.3
      repair path via 192.168.14.4, GigabitEthernet2, cost 21
       Flags: RIB, Stale, Repair, IntfDj, BcastDj
       LSA: 1/3.3.3.3/3.3.3.3

CSR1#sh ip ospf rib 3.3.3.3

            OSPF Router with ID (1.1.1.1) (Process ID 1)

                Base Topology (MTID 0)

OSPF local RIB
Codes: * - Best, > - Installed in global RIB
LSA: type/LSID/originator

*>  3.3.3.3/32, Intra, cost 11, area 0
     SPF Instance 48, age 00:55:20
     Flags: RIB, HiPrio
      via 192.168.13.3, GigabitEthernet3
       Flags: RIB
       LSA: 1/3.3.3.3/3.3.3.3

Now , let us configure loopback 0 interface on CSR4 and advertise it in OSPF

CSR4:
interface loopback0
ip address 4.4.4.4 255.255.255.255
ip ospf 1 area 0

and revert back all costs to 10 on all links and checking from CSR2 (S) for outputs:

CSR2#sh ip route ospf
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       a - application route
       + - replicated route, % - next hop override, p - overrides from PfR

Gateway of last resort is not set

      1.0.0.0/32 is subnetted, 1 subnets
O        1.1.1.1 [110/11] via 192.168.12.1, 00:02:53, GigabitEthernet1
      3.0.0.0/32 is subnetted, 1 subnets
O        3.3.3.3 [110/11] via 192.168.23.3, 00:01:55, GigabitEthernet2
      4.0.0.0/32 is subnetted, 1 subnets
O        4.4.4.4 [110/21] via 192.168.23.3, 00:01:55, GigabitEthernet2
                 [110/21] via 192.168.12.1, 00:02:53, GigabitEthernet1
O     192.168.13.0/24 [110/20] via 192.168.23.3, 00:01:56, GigabitEthernet2
                      [110/20] via 192.168.12.1, 03:44:05, GigabitEthernet1
O     192.168.14.0/24 [110/20] via 192.168.12.1, 03:00:56, GigabitEthernet1
O     192.168.34.0/24 [110/20] via 192.168.23.3, 00:01:56, GigabitEthernet2

CSR2#sh ip route 4.4.4.4
Routing entry for 4.4.4.4/32
  Known via "ospf 1", distance 110, metric 21, type intra area
  Last update from 192.168.23.3 on GigabitEthernet2, 00:02:07 ago
  Routing Descriptor Blocks:
    192.168.23.3, from 4.4.4.4, 00:02:07 ago, via GigabitEthernet2
      Route metric is 21, traffic share count is 1
      Repair Path: 192.168.12.1, via GigabitEthernet1
  * 192.168.12.1, from 4.4.4.4, 00:03:05 ago, via GigabitEthernet1
      Route metric is 21, traffic share count is 1
      Repair Path: 192.168.23.3, via GigabitEthernet2

CSR2#sh ip ospf rib 4.4.4.4

            OSPF Router with ID (2.2.2.2) (Process ID 1)

                Base Topology (MTID 0)

OSPF local RIB
Codes: * - Best, > - Installed in global RIB
LSA: type/LSID/originator

*>  4.4.4.4/32, Intra, cost 21, area 0
     SPF Instance 40, age 00:02:21
     Flags: RIB, HiPrio
      via 192.168.23.3, GigabitEthernet2
       Flags: RIB
       LSA: 1/4.4.4.4/4.4.4.4
      repair path via 192.168.12.1, GigabitEthernet1, cost 21
       Flags: RIB, Repair, IntfDj, BcastDj, PrimPath, NodeProt, Downstr
       LSA: 1/4.4.4.4/4.4.4.4
      via 192.168.12.1, GigabitEthernet1
       Flags: RIB
       LSA: 1/4.4.4.4/4.4.4.4
      repair path via 192.168.23.3, GigabitEthernet2, cost 21
       Flags: RIB, Repair, IntfDj, BcastDj, PrimPath, NodeProt, Downstr
       LSA: 1/4.4.4.4/4.4.4.4

Two backup paths are installed as inequality for link protection is passed

Now , what we want to examine is node protection

CSR2#sh ip route repair 4.4.4.4
Routing entry for 4.4.4.4/32
  Known via "ospf 1", distance 110, metric 21, type intra area
  Last update from 192.168.23.3 on GigabitEthernet2, 00:04:54 ago
  Routing Descriptor Blocks:
    192.168.23.3, from 4.4.4.4, 00:04:54 ago, via GigabitEthernet2
      Route metric is 21, traffic share count is 1
      Repair Path: 192.168.12.1, via GigabitEthernet1
  * 192.168.12.1, from 4.4.4.4, 00:05:52 ago, via GigabitEthernet1
      Route metric is 21, traffic share count is 1
      Repair Path: 192.168.23.3, via GigabitEthernet2

Now , let us adjust the cost on the links between R2 and R3 to 30 , and between R1 and R4 to 15 and check the outputs

CSR2# sh ip ospf rib 4.4.4.4

            OSPF Router with ID (2.2.2.2) (Process ID 1)

                Base Topology (MTID 0)

OSPF local RIB
Codes: * - Best, > - Installed in global RIB
LSA: type/LSID/originator

*>  4.4.4.4/32, Intra, cost 26, area 0
     SPF Instance 44, age 00:00:48
     Flags: RIB, HiPrio
      via 192.168.12.1, GigabitEthernet1
       Flags: RIB
       LSA: 1/4.4.4.4/4.4.4.4
      repair path via 192.168.23.3, GigabitEthernet2, cost 41
       Flags: RIB, Repair, IntfDj, BcastDj, NodeProt, Downstr
       LSA: 1/4.4.4.4/4.4.4.4

We can see the flag of NodeProt , which means that R3 will be able to provide backup in case of the primary node (R1) failure
This was satisfied by passing Inquality of node protection:

D(N,D) < D(N,E) + D(E,D)

Where in our case:
N : CSR3
E : CSR1
D : CSR4

D(N,D) = 10
D(N,E) = 10
D(E,D) = 15

10 < 10 + 15 --> 10 < 25 Passed

Let us try to modify the costs again in order to make the inquality fails

CSR2# sh ip ospf rib 4.4.4.4

            OSPF Router with ID (2.2.2.2) (Process ID 1)


                Base Topology (MTID 0)

OSPF local RIB
Codes: * - Best, > - Installed in global RIB
LSA: type/LSID/originator

*>  4.4.4.4/32, Intra, cost 26, area 0
     SPF Instance 52, age 00:00:11
     Flags: RIB, HiPrio
      via 192.168.12.1, GigabitEthernet1
       Flags: RIB
       LSA: 1/4.4.4.4/4.4.4.4
      repair path via 192.168.23.3, GigabitEthernet2, cost 56
       Flags: RIB, Repair, IntfDj, BcastDj
       LSA: 1/4.4.4.4/4.4.4.4

As can be seen from the output above , the NodeProt disappeared

LFA Route Selection Criteria

Below are the backup prefix selection criteria with their preference in decreasing order. In the event of two backup routes available for a protected primary prefix, only one would be selected based on below mentioned ordered list of attributes they carry. Below is a brief explanation about these attributes.

Repair path selection policy tiebreaks (built-in default policy)

10 srlg
20 primary-path
30 interface-disjoint
40 lowest-metric
50 linecard-disjoint
60 node-protecting
70 broadcast-interface-disjoint
256 load-sharing

Shared risk link group (SRLG): Default LFA policy tries to avoid a path that carries same SRLG as primary path.Assume multiple routers are using the same switch ,so they all be sharing the same risk .
Primary-path: This helps in eliminating candidates that are not equal cost multiple path links or ECMPs.
Interface-Disjoint: This means that repair path is over a different interface as compared to the interface used to reach destination via primary path. In case of point-to-point links, this condition is always met.
Lowest-metric: Select a backup path with minimum cost to reach destination.
Linecard-disjoint: This prefers a backup route from an interface that is on another line card. This is also a special case of SRLG however; this does not require any special configuration and is handled automatically.
Node-protecting: Repair path all together bypasses primary path next-hop router. This ensures complete traffic protection even in the event of primary next-hop router failure.
Broadcast-interface-disjoint : This attributes helps to ensure that repair path does not make use of same broadcast network used by primary path.
Load-sharing: Traffic is load shared amongst candidate back up routes when all other checks discussed above fail to provide a unique back up path.

Let us try to modify the default built-in policy , but before that let us check outputs again and check inequality

CSR1#sh ip ospf rib 3.3.3.3

            OSPF Router with ID (1.1.1.1) (Process ID 1)

                Base Topology (MTID 0)

OSPF local RIB
Codes: * - Best, > - Installed in global RIB
LSA: type/LSID/originator

*>  3.3.3.3/32, Intra, cost 11, area 0
     SPF Instance 76, age 00:27:28
     Flags: RIB, HiPrio
      via 192.168.13.3, GigabitEthernet3
       Flags: RIB
       LSA: 1/3.3.3.3/3.3.3.3
      repair path via 192.168.12.2, GigabitEthernet1, cost 41
       Flags: RIB, Repair, IntfDj, BcastDj, CostWon
       LSA: 1/3.3.3.3/3.3.3.3
      repair path via 192.168.14.4, GigabitEthernet2, cost 51
       Flags: Ignore, Repair, IntfDj, BcastDj
       LSA: 1/3.3.3.3/3.3.3.3

CSR1#sh ip route repair-paths 3.3.3.3
Routing entry for 3.3.3.3/32
  Known via "ospf 1", distance 110, metric 11, type intra area
  Last update from 192.168.13.3 on GigabitEthernet3, 00:05:29 ago
  Routing Descriptor Blocks:
  * 192.168.13.3, from 3.3.3.3, 00:05:29 ago, via GigabitEthernet3
      Route metric is 11, traffic share count is 1
      Repair Path: 192.168.12.2, via GigabitEthernet1
    [RPR]192.168.12.2, from 3.3.3.3, 00:05:29 ago, via GigabitEthernet1
      Route metric is 41, traffic share count is 1

As can be seen from the above output , R1 best path to reach R3 is via the direct link between them
Checking the inequality for link protection:

D(N,D) < D(N,S) + D(S,D)

If we considered first backup link to be R1 - R2

D : R3
S : R1
N : R2

D(N,D) = 20
D(N,S) = 20
D(S,D) = 10

Passed!

If we considered second backup link to be R1 - R4

D : R3
S : R1
N : R4

D(N,D) = 20
D(N,S) = 30
D(S,D) = 10

Passed!

CSR1#sh ip ospf fast-reroute

            OSPF Router with ID (1.1.1.1) (Process ID 1)

Loop-free Fast Reroute protected prefixes:

           Area        Topology name   Priority   Remote LFA Enabled
              0                 Base       High                   No

  Repair path selection policy tiebreaks (built-in default policy):
     10  srlg
     20  primary-path
     30  interface-disjoint
     40  lowest-metric
     50  linecard-disjoint
     60  node-protecting
     70  broadcast-interface-disjoint
    256  load-sharing

OSPF/RIB notifications:
 Topology Base: Notification Disabled, Callback Not Registered

Last SPF calculation started 00:00:25 ago and was running for 0 ms.

fast-reroute per-prefix tie-break lowest-metric index 10
fast-reroute per-prefix tie-break srlg index 20

CSR1#sh ip ospf fast-reroute

            OSPF Router with ID (1.1.1.1) (Process ID 1)

Loop-free Fast Reroute protected prefixes:

           Area        Topology name   Priority   Remote LFA Enabled
              0                 Base       High                   No

  Repair path selection policy tiebreaks:
     10  lowest-metric
     20  srlg
    256  load-sharing

OSPF/RIB notifications:
 Topology Base: Notification Disabled, Callback Not Registered

Last SPF calculation started 00:00:56 ago and was running for 1 ms.

CSR1:
interface gig1
srlg gid 20
interface gig3
srlg gid 20

CSR1#sh ip ospf rib 3.3.3.3

            OSPF Router with ID (1.1.1.1) (Process ID 1)


                Base Topology (MTID 0)

OSPF local RIB
Codes: * - Best, > - Installed in global RIB
LSA: type/LSID/originator

*>  3.3.3.3/32, Intra, cost 11, area 0
     SPF Instance 86, age 00:55:42
     Flags: RIB, HiPrio
      via 192.168.13.3, GigabitEthernet3
       Flags: RIB
       LSA: 1/3.3.3.3/3.3.3.3
      repair path via 192.168.12.2, GigabitEthernet1, cost 41
       Flags: RIB, Repair, IntfDj, BcastDj, SRLG, CostWon
       LSA: 1/3.3.3.3/3.3.3.3
      repair path via 192.168.14.4, GigabitEthernet2, cost 51
       Flags: Ignore, Repair, IntfDj, BcastDj
       LSA: 1/3.3.3.3/3.3.3.3

MPLS TE Auto Tunnel Mesh

R1:
interface Loopback0
 ip address 1.1.1.1 255.255.255.255

interface FastEthernet1/0
 ip address 192.168.12.1 255.255.255.0
 speed 100
 duplex full

router rip
 version 2
 network 1.0.0.0
 network 192.168.12.0
 no auto-summary

R2:
interface Loopback0
 ip address 2.2.2.2 255.255.255.255

interface FastEthernet1/0
 ip address 192.168.23.2 255.255.255.0
 speed 100
 duplex full
 mpls traffic-eng tunnels

interface FastEthernet1/1
 ip address 192.168.24.2 255.255.255.0
 speed 100
 duplex full
 mpls traffic-eng tunnels

ip vrf MSSK
 rd 1:1
 route-target export 1:1
 route-target import 1:1

interface FastEthernet2/0
 ip vrf forwarding MSSK
 ip address 192.168.12.2 255.255.255.0
 speed 100
 duplex full

mpls traffic-eng tunnels
mpls traffic-eng auto-tunnel mesh

router ospf 1
 router-id 2.2.2.2
 network 2.2.2.2 0.0.0.0 area 0
 network 192.168.23.2 0.0.0.0 area 0
 network 192.168.24.2 0.0.0.0 area 0
 mpls traffic-eng router-id Loopback0
 mpls traffic-eng area 0
 mpls traffic-eng mesh-group 10 Loopback0 area 0

router rip
 address-family ipv4 vrf MSSK
  redistribute bgp 1 metric 2
  network 192.168.12.0
  no auto-summary
  version 2
 exit-address-family

router bgp 1
 bgp log-neighbor-changes
 no bgp default ipv4-unicast
 neighbor 6.6.6.6 remote-as 1
 neighbor 6.6.6.6 update-source Loopback0

 address-family vpnv4
  neighbor 6.6.6.6 activate
  neighbor 6.6.6.6 send-community both
 exit-address-family

 address-family ipv4 vrf MSSK
  redistribute rip
 exit-address-family

interface Auto-Template1
 ip unnumbered Loopback0
 tunnel mode mpls traffic-eng
 tunnel destination mesh-group 10
 tunnel mpls traffic-eng autoroute announce
 tunnel mpls traffic-eng path-option 10 dynamic

R3:
interface Loopback0
 ip address 3.3.3.3 255.255.255.255

interface FastEthernet1/0
 ip address 192.168.23.3 255.255.255.0
 speed 100
 duplex full
 mpls traffic-eng tunnels

interface FastEthernet1/1
 ip address 192.168.36.3 255.255.255.0
 speed 100
 duplex full
 mpls traffic-eng tunnels

interface FastEthernet2/0
 ip address 192.168.35.3 255.255.255.0
 speed 100
 duplex full
 mpls traffic-eng tunnels

mpls traffic-eng tunnels
mpls traffic-eng auto-tunnel mesh

router ospf 1
 router-id 3.3.3.3
 network 3.3.3.3 0.0.0.0 area 0
 network 192.168.23.3 0.0.0.0 area 0
 network 192.168.35.3 0.0.0.0 area 0
 network 192.168.36.3 0.0.0.0 area 0
 mpls traffic-eng router-id Loopback0
 mpls traffic-eng area 0
 mpls traffic-eng mesh-group 10 Loopback0 area 0

R4:
interface Loopback0
 ip address 4.4.4.4 255.255.255.255

interface FastEthernet1/0
 ip address 192.168.24.4 255.255.255.0
 speed 100
 duplex full
 mpls traffic-eng tunnels

interface FastEthernet1/1
 ip address 192.168.45.4 255.255.255.0
 speed 100
 duplex full
 mpls traffic-eng tunnels

mpls traffic-eng tunnels
mpls traffic-eng auto-tunnel mesh

router ospf 1
 router-id 4.4.4.4
 network 4.4.4.4 0.0.0.0 area 0
 network 192.168.24.4 0.0.0.0 area 0
 network 192.168.45.4 0.0.0.0 area 0
 mpls traffic-eng router-id Loopback0
 mpls traffic-eng area 0
 mpls traffic-eng mesh-group 10 Loopback0 area 0

R5:
interface Loopback0
 ip address 5.5.5.5 255.255.255.255

interface FastEthernet1/0
 ip address 192.168.56.5 255.255.255.0
 speed 100
 duplex full
 mpls traffic-eng tunnels

interface FastEthernet1/1
 ip address 192.168.45.5 255.255.255.0
 speed 100
 duplex full
 mpls traffic-eng tunnels

interface FastEthernet2/0
 ip address 192.168.35.5 255.255.255.0
 speed 100
 duplex full
 mpls traffic-eng tunnels

mpls traffic-eng tunnels
mpls traffic-eng auto-tunnel mesh

router ospf 1
 router-id 5.5.5.5
 network 5.5.5.5 0.0.0.0 area 0
 network 192.168.35.5 0.0.0.0 area 0
 network 192.168.45.5 0.0.0.0 area 0
 network 192.168.56.5 0.0.0.0 area 0
 mpls traffic-eng router-id Loopback0
 mpls traffic-eng area 0
 mpls traffic-eng mesh-group 10 Loopback0 area 0

R6:
interface Loopback0
 ip address 6.6.6.6 255.255.255.255

interface FastEthernet1/0
 ip address 192.168.36.6 255.255.255.0
 speed 100
 duplex full
 mpls traffic-eng tunnels

interface FastEthernet1/1
 ip address 192.168.56.6 255.255.255.0
 speed 100
 duplex full
 mpls traffic-eng tunnels

ip vrf MSSK
 rd 1:1
 route-target export 1:1
 route-target import 1:1

interface FastEthernet2/0
 ip vrf forwarding MSSK
 ip address 192.168.67.6 255.255.255.0
 speed 100
 duplex full

mpls traffic-eng tunnels
mpls traffic-eng auto-tunnel mesh

router ospf 1
 router-id 6.6.6.6
 network 6.6.6.6 0.0.0.0 area 0
 network 192.168.36.6 0.0.0.0 area 0
 network 192.168.56.6 0.0.0.0 area 0
 mpls traffic-eng router-id Loopback0
 mpls traffic-eng area 0
 mpls traffic-eng mesh-group 10 Loopback0 area 0

router rip
 address-family ipv4 vrf MSSK
  redistribute bgp 1 metric 2
  network 192.168.67.0
  no auto-summary
  version 2
 exit-address-family

router bgp 1
 bgp log-neighbor-changes
 no bgp default ipv4-unicast
 neighbor 2.2.2.2 remote-as 1
 neighbor 2.2.2.2 update-source Loopback0

 address-family vpnv4
  neighbor 2.2.2.2 activate
  neighbor 2.2.2.2 send-community both
 exit-address-family

 address-family ipv4 vrf MSSK
  redistribute rip
 exit-address-family

interface Auto-Template1
 ip unnumbered Loopback0
 tunnel mode mpls traffic-eng
 tunnel destination mesh-group 10
 tunnel mpls traffic-eng autoroute announce
 tunnel mpls traffic-eng path-option 10 dynamic

R7:
interface Loopback0
 ip address 7.7.7.7 255.255.255.255

interface FastEthernet1/0
 ip address 192.168.67.7 255.255.255.0
 speed 100
 duplex full

router rip
 version 2
 network 7.0.0.0
 network 192.168.67.0
 no auto-summary

Verification:

Let us check CE1 routing table and test connectivity to CE2:
R1#sh ip route rip
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       + - replicated route, % - next hop override

Gateway of last resort is not set

      7.0.0.0/32 is subnetted, 1 subnets
R        7.7.7.7 [120/2] via 192.168.12.2, 00:00:16, FastEthernet1/0
R     192.168.67.0/24 [120/2] via 192.168.12.2, 00:00:16, FastEthernet1/0

R1#ping 7.7.7.7 source loopback 0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 7.7.7.7, timeout is 2 seconds:
Packet sent with a source address of 1.1.1.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 76/88/104 ms

R1#traceroute 7.7.7.7 source lo0 numeric
Type escape sequence to abort.
Tracing the route to 7.7.7.7
VRF info: (vrf in name/id, vrf out name/id)
  1 192.168.12.2 56 msec 72 msec 68 msec
  2 192.168.23.3 [MPLS: Labels 18/17 Exp 0] 76 msec 52 msec 76 msec
  3 192.168.67.6 [MPLS: Label 17 Exp 0] 116 msec 124 msec 88 msec
  4 192.168.67.7 116 msec 152 msec 136 msec

R2#show mpls traffic-eng topology brief
My_System_id: 2.2.2.2 (ospf 1  area 0)

Signalling error holddown: 10 sec Global Link Generation 18

IGP Id: 2.2.2.2, MPLS TE Id:2.2.2.2 Router Node  (ospf 1  area 0)
      link[0]: Broadcast, DR: 192.168.23.2, nbr_node_id:1, gen:8
      frag_id: 2, Intf Address: 192.168.23.2
      TE metric: 1, IGP metric: 1, attribute flags: 0x0
      SRLGs: None

      link[1]: Broadcast, DR: 192.168.24.2, nbr_node_id:2, gen:8
      frag_id: 3, Intf Address: 192.168.24.2
      TE metric: 1, IGP metric: 1, attribute flags: 0x0
      SRLGs: None

IGP Id: 3.3.3.3, MPLS TE Id:3.3.3.3 Router Node  (ospf 1  area 0)
Area mg-id's:
: mg-id 10    3.3.3.3 :
      link[0]: Broadcast, DR: 192.168.23.2, nbr_node_id:1, gen:11
      frag_id: 2, Intf Address: 192.168.23.3
      TE metric: 1, IGP metric: 1, attribute flags: 0x0
      SRLGs: None

      link[1]: Broadcast, DR: 192.168.36.6, nbr_node_id:4, gen:11
      frag_id: 3, Intf Address: 192.168.36.3
      TE metric: 1, IGP metric: 1, attribute flags: 0x0
      SRLGs: None

      link[2]: Broadcast, DR: 192.168.35.3, nbr_node_id:3, gen:11
      frag_id: 4, Intf Address: 192.168.35.3
      TE metric: 1, IGP metric: 1, attribute flags: 0x0
      SRLGs: None

IGP Id: 4.4.4.4, MPLS TE Id:4.4.4.4 Router Node  (ospf 1  area 0)
Area mg-id's:
: mg-id 10    4.4.4.4 :
      link[0]: Broadcast, DR: 192.168.24.2, nbr_node_id:2, gen:13
      frag_id: 2, Intf Address: 192.168.24.4
      TE metric: 1, IGP metric: 1, attribute flags: 0x0
      SRLGs: None

      link[1]: Broadcast, DR: 192.168.45.4, nbr_node_id:5, gen:13
      frag_id: 3, Intf Address: 192.168.45.4
      TE metric: 1, IGP metric: 1, attribute flags: 0x0
      SRLGs: None

IGP Id: 5.5.5.5, MPLS TE Id:5.5.5.5 Router Node  (ospf 1  area 0)
Area mg-id's:
: mg-id 10    5.5.5.5 :
      link[0]: Broadcast, DR: 192.168.56.6, nbr_node_id:6, gen:16
      frag_id: 2, Intf Address: 192.168.56.5
      TE metric: 1, IGP metric: 1, attribute flags: 0x0
      SRLGs: None

      link[1]: Broadcast, DR: 192.168.45.4, nbr_node_id:5, gen:16
      frag_id: 3, Intf Address: 192.168.45.5
      TE metric: 1, IGP metric: 1, attribute flags: 0x0
      SRLGs: None

      link[2]: Broadcast, DR: 192.168.35.3, nbr_node_id:3, gen:16
      frag_id: 4, Intf Address: 192.168.35.5
      TE metric: 1, IGP metric: 1, attribute flags: 0x0
      SRLGs: None

IGP Id: 6.6.6.6, MPLS TE Id:6.6.6.6 Router Node  (ospf 1  area 0)
Area mg-id's:
: mg-id 10    6.6.6.6 :
      link[0]: Broadcast, DR: 192.168.36.6, nbr_node_id:4, gen:18
      frag_id: 2, Intf Address: 192.168.36.6
      TE metric: 1, IGP metric: 1, attribute flags: 0x0
      SRLGs: None

      link[1]: Broadcast, DR: 192.168.56.6, nbr_node_id:6, gen:18
      frag_id: 3, Intf Address: 192.168.56.6
      TE metric: 1, IGP metric: 1, attribute flags: 0x0
      SRLGs: None

IGP Id: 192.168.23.2, Network Node  (ospf 1  area 0)
      link[0]: Broadcast, Nbr IGP Id: 2.2.2.2, nbr_node_id:7, gen:1

      link[1]: Broadcast, Nbr IGP Id: 3.3.3.3, nbr_node_id:8, gen:1

IGP Id: 192.168.24.2, Network Node  (ospf 1  area 0)
      link[0]: Broadcast, Nbr IGP Id: 2.2.2.2, nbr_node_id:7, gen:2

      link[1]: Broadcast, Nbr IGP Id: 4.4.4.4, nbr_node_id:9, gen:2

IGP Id: 192.168.35.3, Network Node  (ospf 1  area 0)
      link[0]: Broadcast, Nbr IGP Id: 3.3.3.3, nbr_node_id:8, gen:3

      link[1]: Broadcast, Nbr IGP Id: 5.5.5.5, nbr_node_id:10, gen:3

IGP Id: 192.168.36.6, Network Node  (ospf 1  area 0)
      link[0]: Broadcast, Nbr IGP Id: 6.6.6.6, nbr_node_id:11, gen:4

      link[1]: Broadcast, Nbr IGP Id: 3.3.3.3, nbr_node_id:8, gen:4

IGP Id: 192.168.45.4, Network Node  (ospf 1  area 0)
      link[0]: Broadcast, Nbr IGP Id: 4.4.4.4, nbr_node_id:9, gen:5

      link[1]: Broadcast, Nbr IGP Id: 5.5.5.5, nbr_node_id:10, gen:5

IGP Id: 192.168.56.6, Network Node  (ospf 1  area 0)
      link[0]: Broadcast, Nbr IGP Id: 6.6.6.6, nbr_node_id:11, gen:6

      link[1]: Broadcast, Nbr IGP Id: 5.5.5.5, nbr_node_id:10, gen:6

R2# sh ip int bri | ex down
Interface              IP-Address      OK? Method Status                Protocol
FastEthernet1/0        192.168.23.2    YES manual up                    up
FastEthernet1/1        192.168.24.2    YES manual up                    up
FastEthernet2/0        192.168.12.2    YES manual up                    up
Auto-Template1         2.2.2.2         YES TFTP   up                    up
Loopback0              2.2.2.2         YES manual up                    up
Tunnel64336            2.2.2.2         YES TFTP   up                    up
Tunnel64337            2.2.2.2         YES TFTP   up                    up
Tunnel64338            2.2.2.2         YES TFTP   up                    up
Tunnel64339            2.2.2.2         YES TFTP   up                    up

R2#show mpls forwarding-table
Local      Outgoing   Prefix           Bytes Label   Outgoing   Next Hop
Label      Label      or Tunnel Id     Switched      interface
16         No Label   192.168.12.0/24[V]   \
                                       0             aggregate/MSSK
17         No Label   1.1.1.1/32[V]    3768          Fa2/0      192.168.12.1

R3#show mpls forwarding-table
Local      Outgoing   Prefix           Bytes Label   Outgoing   Next Hop
Label      Label      or Tunnel Id     Switched      interface
16         Pop Label  2.2.2.2 64338 [7768]   \
                                       0             Fa2/0      192.168.35.5
17         Pop Label  6.6.6.6 64336 [2762]   \
                                       6801          Fa1/0      192.168.23.2
18         Pop Label  2.2.2.2 64339 [3399]   \
                                       4961          Fa1/1      192.168.36.6
R2#show ip route | inc Tunnel
O        3.3.3.3 [110/2] via 3.3.3.3, 00:19:55, Tunnel64336
O        4.4.4.4 [110/2] via 4.4.4.4, 00:19:55, Tunnel64337
O        5.5.5.5 [110/3] via 5.5.5.5, 00:19:45, Tunnel64338
O        6.6.6.6 [110/3] via 6.6.6.6, 00:19:45, Tunnel64339
O     192.168.35.0/24 [110/2] via 3.3.3.3, 00:19:55, Tunnel64336
O     192.168.36.0/24 [110/2] via 3.3.3.3, 00:19:55, Tunnel64336
O     192.168.45.0/24 [110/2] via 4.4.4.4, 00:19:55, Tunnel64337
O     192.168.56.0/24 [110/3] via 6.6.6.6, 00:19:45, Tunnel64339
                               [110/3] via 5.5.5.5, 00:19:45, Tunnel64338

R2#show mpls traffic-eng auto-tunnel mesh

Auto-Template1:

 Using mesh-group 10 to clone the following tunnel interfaces:

  Destination         Interface
  -----------         ---------

  3.3.3.3             Tunnel64336
  4.4.4.4             Tunnel64337
  5.5.5.5             Tunnel64338
  6.6.6.6             Tunnel64339

Mesh tunnel interface numbers: min 64336 max 65335

R6#show mpls traffic-eng auto-tunnel mesh

Auto-Template1:

 Using mesh-group 10 to clone the following tunnel interfaces:

  Destination         Interface
  -----------         ---------

  2.2.2.2             Tunnel64336
  3.3.3.3             Tunnel64337
  4.4.4.4             Tunnel64338
  5.5.5.5             Tunnel64339

Mesh tunnel interface numbers: min 64336 max 65335

If we checked the OSPF database for one of the PE routers (we are using two , one for the MPLS TE and one for the mesh-group)

  LS age: 1695
  Options: (No TOS-capability, DC)
  LS Type: Opaque Area Link
  Link State ID: 4.0.0.0
  Opaque Type: 4
  Opaque ID: 0
  Advertising Router: 2.2.2.2
  LS Seq Number: 80000001
  Checksum: 0xE129
  Length: 32

    Capability Type: Mesh-group
    Length: 8
    Value:

    0000 000A 0202 0202

  LS age: 1686
  Options: (No TOS-capability, DC)
  LS Type: Opaque Area Link
  Link State ID: 4.0.0.0
  Opaque Type: 4
  Opaque ID: 0
  Advertising Router: 3.3.3.3
  LS Seq Number: 80000001
  Checksum: 0xF50D
  Length: 32

    Capability Type: Mesh-group
    Length: 8
    Value:

    0000 000A 0303 0303

  LS age: 1681
  Options: (No TOS-capability, DC)
  LS Type: Opaque Area Link
  Link State ID: 4.0.0.0
  Opaque Type: 4
  Opaque ID: 0
  Advertising Router: 4.4.4.4
  LS Seq Number: 80000001
  Checksum: 0xAF0
  Length: 32

    Capability Type: Mesh-group
    Length: 8
    Value:

    0000 000A 0404 0404

  LS age: 1678
  Options: (No TOS-capability, DC)
  LS Type: Opaque Area Link
  Link State ID: 4.0.0.0
  Opaque Type: 4
  Opaque ID: 0
  Advertising Router: 5.5.5.5
  LS Seq Number: 80000001
  Checksum: 0x1ED4
  Length: 32

    Capability Type: Mesh-group
    Length: 8
    Value:

    0000 000A 0505 0505

  LS age: 1661
  Options: (No TOS-capability, DC)
  LS Type: Opaque Area Link
  Link State ID: 4.0.0.0
  Opaque Type: 4
  Opaque ID: 0
  Advertising Router: 6.6.6.6
  LS Seq Number: 80000001
  Checksum: 0x32B8
  Length: 32

    Capability Type: Mesh-group
    Length: 8
    Value:

    0000 000A 0606 0606

As can be seen the 0000 000A refers to the TLV which when decoded back to decimal equals to 10 (which is the mesh group number configured in the command mpls traffic-eng mesh-group 10 Loopback0 area 0 under the OSPF process)

PBB EVPN

PBB stands for Provide Backbone Bridging , PBB EVPN is one of the next generation VPNs which solved a lot of limitations previously faced by deploying what so called Q-in-Q (PB : Provider Bridging)

One of the well know limitations in deploying Q-in-Q is the limited number of supported S-VLANs which is 4096 (even if we can map multiple C-VLANs to one S-VLANs) , as well , the MAC addresses of customers end stations will be flooded across the provider network which can overwhelm resources and can cause broadcast storms among the provider network , and not to forget , demarcation point is not clear and this caveat was solved with PBB using what so called i-SID (Instance Service Identifier) 

Basic configuration to be listed below which includes:

• IGP configuration (OSPF area 0) in order to build LDP neighborships properly
• iBGP neighborship between MPLS PEs under the respective address-family (L2VPN EVPN)
• Two bridge groups are required : first bridge group contains the PBB Edge bridge domains, these are the customer facing BD , every Edge bridge domain must have its own I-SID (Instance Service Identifier) to distinguish  endpoints, the Edge bridge domain must also be attached to what so called Core bridge domain , the second bridge group contains the PBB Core bridge domains which are the MPLS core / EVPN facing BDs , this only requires EVI (Ethernet VPN Instance) ID in basic configuration

 XR1:

router ospf 1
 router-id 10.10.10.10
 area 0
  interface Loopback0

  interface GigabitEthernet0/0/0/0

mpls ldp
 router-id 10.10.10.10
 interface GigabitEthernet0/0/0/0

router bgp 1
 address-family l2vpn evpn

 neighbor 20.20.20.20
  remote-as 1
  update-source Loopback0
  address-family l2vpn evpn

l2vpn
 pbb
  backbone-source-mac 0001.0001.0001

 bridge group BG_CORE
  bridge-domain BD_CORE
   pbb core
    evpn evi 1500

 bridge group BG1_CUST
  bridge-domain BD1_CUST
   interface GigabitEthernet0/0/0/2.100

   pbb edge i-sid 1000 core-bridge BD_CORE

XR2:

router ospf 1
 router-id 20.20.20.20
 area 0
  interface Loopback0

  interface GigabitEthernet0/0/0/0

mpls ldp
 router-id 20.20.20.20
 interface GigabitEthernet0/0/0/0
router bgp 1
 address-family l2vpn evpn

 neighbor 10.10.10.10
  remote-as 1
  update-source Loopback0
  address-family l2vpn evpn

l2vpn
 pbb
  backbone-source-mac 0002.0002.0002

 bridge group BG_CORE
  bridge-domain BD_CORE
   pbb core
    evpn evi 1500

 bridge group BG1_CUST
  bridge-domain BD1_CUST
   interface GigabitEthernet0/0/0/2.100

   pbb edge i-sid 1000 core-bridge BD_CORE
 

Verification:

 RP/0/0/CPU0:XR1#sh bgp l2vpn evpn summary
Wed Jan 25 14:31:38.956 UTC
BGP router identifier 10.10.10.10, local AS number 1
BGP generic scan interval 60 secs
Non-stop routing is enabled
BGP table state: Active
Table ID: 0x0   RD version: 0
BGP main routing table version 7
BGP NSR Initial initsync version 1 (Reached)
BGP NSR/ISSU Sync-Group versions 0/0
BGP scan interval 60 secs

BGP is operating in STANDALONE mode.


Process       RcvTblVer   bRIB/RIB   LabelVer  ImportVer  SendTblVer  StandbyVer
Speaker               7          7          7          7           7           0

Neighbor        Spk    AS MsgRcvd MsgSent   TblVer  InQ OutQ  Up/Down  St/PfxRcd
20.20.20.20       0     1      38      42        7    0    0 00:34:10          2


RP/0/0/CPU0:XR1#sh bgp l2vpn evpn
Wed Jan 25 14:31:45.196 UTC
BGP router identifier 10.10.10.10, local AS number 1
BGP generic scan interval 60 secs
Non-stop routing is enabled
BGP table state: Active
Table ID: 0x0   RD version: 0
BGP main routing table version 7
BGP NSR Initial initsync version 1 (Reached)
BGP NSR/ISSU Sync-Group versions 0/0
BGP scan interval 60 secs

Status codes: s suppressed, d damped, h history, * valid, > best
              i - internal, r RIB-failure, S stale, N Nexthop-discard
Origin codes: i - IGP, e - EGP, ? - incomplete
   Network            Next Hop            Metric LocPrf Weight Path
Route Distinguisher: 10.10.10.10:1500 (default for vrf BD_CORE)
*> [2][0][48][0001.0001.0001][0]/104
                      0.0.0.0                                0 i
*>i[2][0][48][0002.0002.0002][0]/104
                      20.20.20.20                   100      0 i
*> [3][1000][32][10.10.10.10]/80
                      0.0.0.0                                0 i
*>i[3][1000][32][20.20.20.20]/80
                      20.20.20.20                   100      0 i
Route Distinguisher: 20.20.20.20:1500
*>i[2][0][48][0002.0002.0002][0]/104
                      20.20.20.20                   100      0 i
*>i[3][1000][32][20.20.20.20]/80
                      20.20.20.20                   100      0 i

Processed 6 prefixes, 6 paths

RP/0/0/CPU0:XR1#show l2vpn pbb backbone-source-mac
Wed Jan 25 14:32:03.955 UTC
Backbone Source MAC: 0001.0001.0001
Chassis MAC        : 0b16.212c.3742


RP/0/0/CPU0:XR1#show l2vpn forwarding bridge-domain pbb edge detail location 0$
Wed Jan 25 14:32:47.772 UTC

Bridge-domain name: BG1_CUST:BD1_CUST, id: 0, state: up
 Type: pbb-edge, I-SID: 1000
 Core-bridge: NULL
 MAC learning: enabled
 MAC port down flush: enabled
 Flooding:
   Broadcast & Multicast: enabled
   Unknown unicast: enabled
 MAC aging time: 300 s, Type: inactivity
 MAC limit: 4000, Action: none, Notification: syslog
 MAC limit reached: no
 MAC Secure: disabled, Logging: disabled
 DHCPv4 snooping: profile not known on this node
 Dynamic ARP Inspection: disabled, Logging: disabled
 IP Source Guard: disabled, Logging: disabled
 IGMP snooping: disabled, flooding: enabled
 MLD snooping: disabled, flooding: disabled
 MMRP Flood Optimization: disabled
 Storm control: disabled
 P2MP PW: disabled
 Bridge MTU: 1500 bytes
 Number of bridge ports: 2
 Number of MAC addresses: 0
 Multi-spanning tree instance: 0
 MIRP-lite: received 0, sent 0

  PBB Edge, state: Up
    Number of MAC: 0

  GigabitEthernet0/0/0/2.100, state: down
    Number of MAC: 0


RP/0/0/CPU0:XR1#show l2vpn forwarding bridge-domain mac-address location 0/0/C$
Wed Jan 25 14:33:11.430 UTC
Mac Address    Type    Learned from/Filtered on    LC learned Age                Mapped to
--------------------------------------------------------------------------------
0001.0001.0001 S-BMAC  BD id: 1                    N/A        N/A                N/A
0002.0002.0002 BMAC    BD id: 1                    N/A        N/A                N/A

Note : Am using simulator to illustrate the concept

Seamless MPLS

In this post , we are going to examine what so called Seamless MPLS and the beinift from such a feature
We will start at the begining by doing usual MPLS L3VPN where R1 and R5 are MPLS PEs and all routers are running OSPF area 0 as their IGP

R1#show bgp vpnv4 unicast all
BGP table version is 4, local router ID is 1.1.1.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
              x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
Route Distinguisher: 1:1 (default for vrf MSSK)
 *>  10.10.10.0/24    0.0.0.0                  0         32768 i
 *>i 10.10.20.0/24    5.5.5.5                  0    100      0 i

PC1> ping 10.10.20.10
84 bytes from 10.10.20.10 icmp_seq=1 ttl=59 time=98.006 ms
84 bytes from 10.10.20.10 icmp_seq=2 ttl=59 time=104.006 ms
84 bytes from 10.10.20.10 icmp_seq=3 ttl=59 time=75.005 ms
84 bytes from 10.10.20.10 icmp_seq=4 ttl=59 time=142.008 ms
84 bytes from 10.10.20.10 icmp_seq=5 ttl=59 time=86.005 ms

After checking end to end connectivity and before we go into Seamless MPLS , let us check the MPLS forwarding table on one of the PEs and on the Ps for later comparsion:

R1#sh ip bgp vpnv4 all labels
   Network          Next Hop      In label/Out label
Route Distinguisher: 1:1 (MSSK)
   10.10.10.0/24    0.0.0.0         23/nolabel(MSSK)
   10.10.20.0/24    5.5.5.5         nolabel/23

R1#show mpls forwarding-table
Local      Outgoing   Prefix           Bytes Label   Outgoing   Next Hop
Label      Label      or Tunnel Id     Switched      interface
16         Pop Label  2.2.2.2/32       0             Fa1/0      192.168.12.2
17         Pop Label  192.168.23.0/24  0             Fa1/0      192.168.12.2
18         17         3.3.3.3/32       0             Fa1/0      192.168.12.2
19         18         192.168.34.0/24  0             Fa1/0      192.168.12.2
20         19         4.4.4.4/32       0             Fa1/0      192.168.12.2
21         20         192.168.45.0/24  0             Fa1/0      192.168.12.2
22         21         5.5.5.5/32       0             Fa1/0      192.168.12.2
23         No Label   10.10.10.0/24[V] 686           aggregate/MSSK

R3#sh mpls forwarding-table
Local      Outgoing   Prefix           Bytes Label   Outgoing   Next Hop
Label      Label      or Tunnel Id     Switched      interface
16         Pop Label  2.2.2.2/32       0             Fa1/0      192.168.23.2
17         16         1.1.1.1/32       2173          Fa1/0      192.168.23.2
18         Pop Label  192.168.12.0/24  0             Fa1/0      192.168.23.2
19         Pop Label  4.4.4.4/32       0             Fa1/1      192.168.34.4
20         Pop Label  192.168.45.0/24  0             Fa1/1      192.168.34.4
21         21         5.5.5.5/32       2179          Fa1/1      192.168.34.4

Now , let us divide the network illustrated in the above diagram into layers as per common design :

R2 - R3 - R4 are within the core layer , R1 - R2 and R4 - R5 are distribution layer and PCs (CEs) connections to their respective PEs are access layer
Seamless router roughly speaking aims to allow our distribution to expand smoothly and conserve the MPLS forwarding table to contain only what assist in establishing end to end LSP

We are going to modify the IGP to be divided into three routing processes instead of one process , we will use OSPF PID 12 between R1 and R2 , we will use OSPF PID 1 within our core and we will use OSPF PID 45 between R4 and R5 

Now , as soon we do this , we will loose our end to end LSP , which means we will not be able to maintain connectivity between our PEs and as a result the VPNv4 iBGP session will be IDLE

The idea of Seamless MPLS is to divide the provider network as we did in the previous and to establish IPv4 iBGP with label sening capability (which means we will rely on BGP to assign labels among the LSP)

So , the first thing we will do is to leak R2 Loopback address inside OSPF PID 12 and leak R4 Loopback address inside OSPF PID 45

R2:
ip prefix-list R2LOOP seq 5 permit 2.2.2.2/32

route-map MAP permit 10
match ip address prefix R2LOOP

router ospf 12
redistribute ospf 1 subnets route-map MAP

Note :  the same to be done on R4

Next , we will establish IPv4 iBGP sessions between R1 and R2 , R2 and R4 , R4 and R5 with send-label capability 

Note :  we will consider both R2 and R4 as route reflectors for the respective address-family (IPv4) and we will have to modify the next-hop using the command next-hop-self all attached to neighbor statement under the address family (we need all as we are establishing iBGP relations)

R1:
router bgp 1
 bgp log-neighbor-changes
 no bgp default ipv4-unicast
 neighbor 2.2.2.2 remote-as 1
 neighbor 2.2.2.2 update-source Loopback0
 neighbor 5.5.5.5 remote-as 1

 address-family ipv4
  network 1.1.1.1 mask 255.255.255.255
  neighbor 2.2.2.2 activate
  neighbor 2.2.2.2 send-label
 exit-address-family

 address-family vpnv4
  neighbor 5.5.5.5 activate
  neighbor 5.5.5.5 send-community both
 exit-address-family

 address-family ipv4 vrf MSSK
  network 10.10.10.0 mask 255.255.255.0
 exit-address-family

R2:
router bgp 1
 bgp log-neighbor-changes
 no bgp default ipv4-unicast
 neighbor 1.1.1.1 remote-as 1
 neighbor 1.1.1.1 update-source Loopback0
 neighbor 4.4.4.4 remote-as 1
 neighbor 4.4.4.4 update-source Loopback0

 address-family ipv4
  neighbor 1.1.1.1 activate
  neighbor 1.1.1.1 route-reflector-client
  neighbor 1.1.1.1 next-hop-self all
  neighbor 1.1.1.1 send-label
  neighbor 4.4.4.4 activate
  neighbor 4.4.4.4 route-reflector-client
  neighbor 4.4.4.4 next-hop-self all
  neighbor 4.4.4.4 send-label
 exit-address-family

R4:
router bgp 1
 bgp log-neighbor-changes
 no bgp default ipv4-unicast
 neighbor 2.2.2.2 remote-as 1
 neighbor 2.2.2.2 update-source Loopback0
 neighbor 5.5.5.5 remote-as 1
 neighbor 5.5.5.5 update-source Loopback0

 address-family ipv4
  neighbor 2.2.2.2 activate
  neighbor 2.2.2.2 route-reflector-client
  neighbor 2.2.2.2 next-hop-self all
  neighbor 2.2.2.2 send-label
  neighbor 5.5.5.5 activate
  neighbor 5.5.5.5 route-reflector-client
  neighbor 5.5.5.5 next-hop-self all
  neighbor 5.5.5.5 send-label
 exit-address-family

R5:
router bgp 1
 bgp log-neighbor-changes
 no bgp default ipv4-unicast
 neighbor 1.1.1.1 remote-as 1
 neighbor 1.1.1.1 update-source Loopback0
 neighbor 4.4.4.4 remote-as 1
 neighbor 4.4.4.4 update-source Loopback0

 address-family ipv4
  network 5.5.5.5 mask 255.255.255.255
  neighbor 4.4.4.4 activate
  neighbor 4.4.4.4 send-label
 exit-address-family

 address-family vpnv4
  neighbor 1.1.1.1 activate
  neighbor 1.1.1.1 send-community both
 exit-address-family

 address-family ipv4 vrf MSSK
  network 10.10.20.0 mask 255.255.255.0
 exit-address-family
 
PC2> ping 10.10.20.5
84 bytes from 10.10.20.5 icmp_seq=1 ttl=255 time=51.003 ms
84 bytes from 10.10.20.5 icmp_seq=2 ttl=255 time=55.003 ms
84 bytes from 10.10.20.5 icmp_seq=3 ttl=255 time=60.003 ms
84 bytes from 10.10.20.5 icmp_seq=4 ttl=255 time=79.005 ms
84 bytes from 10.10.20.5 icmp_seq=5 ttl=255 time=39.002 ms

R1#sh ip bgp vpnv4 all labels
   Network          Next Hop      In label/Out label
Route Distinguisher: 1:1 (MSSK)
   10.10.10.0/24    0.0.0.0         23/nolabel(MSSK)
   10.10.20.0/24    5.5.5.5         nolabel/24

Now , let us have a look at the MPLS forwarding-table:

R1#show mpls forwarding-table
Local      Outgoing   Prefix           Bytes Label   Outgoing   Next Hop
Label      Label      or Tunnel Id     Switched      interface
16         Pop Label  2.2.2.2/32       0             Fa1/0      192.168.12.2
23         No Label   10.10.10.0/24[V] 0             aggregate/MSSK

R3#show mpls forwarding-table
Local      Outgoing   Prefix           Bytes Label   Outgoing   Next Hop
Label      Label      or Tunnel Id     Switched      interface
16         Pop Label  2.2.2.2/32       2925          Fa1/0      192.168.23.2
19         Pop Label  4.4.4.4/32       3017          Fa1/1      192.168.34.4

As can be seen , the difference in the number of entries is obvious , which means we conserved our resources and we gave ability to new PEs to connect and server customers smoothly

 
 

MPLS CBTS

We are going to examine MPLS Class Based Tunnel Selection (CBTS)
This feature aims to allow certian traffic which associated with different precedence (EXP) values to follow specfic tunnels (we are using MPLS TE in order to maintain connectivity between the CEs with RIPv2 is the PE-CE routing protocol)
Tunnel 1 is supposed to handle precedences 0 , 1 , 2 , 4 , 6 and 7
Tunnel 2 is supposed to handle precedence 3
Tunnel 3 is supposed to handle precedence 5

Configurations:

R1
interface Loopback0
 ip address 1.1.1.1 255.255.255.255

interface FastEthernet1/0
 ip address 192.168.12.1 255.255.255.0
 speed 100
 duplex full

router rip
 version 2
 network 1.0.0.0
 network 192.168.12.0
 no auto-summary

R2
interface Loopback0
 ip address 2.2.2.2 255.255.255.255
 ip router isis 1

 mpls traffic-eng tunnels

interface FastEthernet1/0
 ip address 192.168.23.2 255.255.255.0
 ip router isis 1
 speed 100
 duplex full
 mpls traffic-eng tunnels
 ip rsvp bandwidth

interface FastEthernet1/1
 ip address 192.168.24.2 255.255.255.0
 ip router isis 1
 speed 100
 duplex full
 mpls traffic-eng tunnels
 ip rsvp bandwidth

ip vrf MSSK
rd 1:1
route-target import 1:1
route-target export 1:1

interface FastEthernet2/0
 ip vrf forwarding MSSK
 ip address 192.168.12.2 255.255.255.0
 speed 100
 duplex full

router isis 1
 net 49.0001.0000.0000.0002.00
 is-type level-2-only
 metric-style wide
 mpls traffic-eng router-id Loopback0
 mpls traffic-eng level-2

router rip
 address-family ipv4 vrf MSSK
  redistribute bgp 1 metric 2
  network 192.168.12.0
  no auto-summary
  version 2
 exit-address-family

router bgp 1
 bgp log-neighbor-changes
 no bgp default ipv4-unicast
 neighbor 5.5.5.5 remote-as 1
 neighbor 5.5.5.5 update-source Loopback0

 address-family vpnv4
  neighbor 5.5.5.5 activate
  neighbor 5.5.5.5 send-community both
 exit-address-family

 address-family ipv4 vrf MSSK
  redistribute rip
 exit-address-family

R3
interface Loopback0
 ip address 3.3.3.3 255.255.255.255
 ip router isis 1

 mpls traffic-eng tunnels

interface FastEthernet1/0
 ip address 192.168.23.3 255.255.255.0
 ip router isis 1
 speed 100
 duplex full
 mpls traffic-eng tunnels
 ip rsvp bandwidth

interface FastEthernet1/1
 ip address 192.168.35.3 255.255.255.0
 ip router isis 1
 speed 100
 duplex full
 mpls traffic-eng tunnels
 ip rsvp bandwidth

router isis 1
 net 49.0001.0000.0000.0003.00
 is-type level-2-only
 metric-style wide
 mpls traffic-eng router-id Loopback0
 mpls traffic-eng level-2

R4
interface Loopback0
 ip address 4.4.4.4 255.255.255.255
 ip router isis 1
 mpls traffic-eng tunnels

interface FastEthernet1/0
 ip address 192.168.24.4 255.255.255.0
 ip router isis 1
 speed 100
 duplex full
 mpls traffic-eng tunnels
 ip rsvp bandwidth

interface FastEthernet1/1
 ip address 192.168.45.4 255.255.255.0
 ip router isis 1
 speed 100
 duplex full
 mpls traffic-eng tunnels
 ip rsvp bandwidth

router isis 1
 net 49.0001.0000.0000.0004.00
 is-type level-2-only
 metric-style wide
 mpls traffic-eng router-id Loopback0
 mpls traffic-eng level-2


R5
interface Loopback0
 ip address 5.5.5.5 255.255.255.255
 ip router isis 1

 mpls traffic-eng tunnels

interface FastEthernet1/0
 ip address 192.168.35.5 255.255.255.0
 ip router isis 1
 speed 100
 duplex full
 mpls traffic-eng tunnels
 ip rsvp bandwidth

interface FastEthernet1/1
 ip address 192.168.45.5 255.255.255.0
 ip router isis 1
 speed 100
 duplex full
 mpls traffic-eng tunnels
 ip rsvp bandwidth

ip vrf MSSK
rd 1:1
route-target import 1:1
route-target export 1:1

interface FastEthernet2/0
 ip vrf forwarding MSSK
 ip address 192.168.56.5 255.255.255.0
 speed 100
 duplex full

router isis 1
 net 49.0001.0000.0000.0005.00
 is-type level-2-only
 metric-style wide
 mpls traffic-eng router-id Loopback0
 mpls traffic-eng level-2

router rip
 address-family ipv4 vrf MSSK
  redistribute bgp 1 metric 2
  network 192.168.56.0
  no auto-summary
  version 2
 exit-address-family

router bgp 1
 bgp log-neighbor-changes
 no bgp default ipv4-unicast
 neighbor 2.2.2.2 remote-as 1
 neighbor 2.2.2.2 update-source Loopback0

 address-family ipv4
 exit-address-family

 address-family vpnv4
  neighbor 2.2.2.2 activate
  neighbor 2.2.2.2 send-community both
 exit-address-family

 address-family ipv4 vrf MSSK
  redistribute rip
 exit-address-family
 
R6
interface Loopback0
 ip address 6.6.6.6 255.255.255.255

interface FastEthernet1/0
 ip address 192.168.56.6 255.255.255.0
 speed 100
 duplex full

router rip
 version 2
 network 6.0.0.0
 network 192.168.56.0
 no auto-summary

The relevant Tunnels configurations:

R2
ip explicit-path name TUN1_PATH enable
 next-address 192.168.23.3
 next-address 192.168.35.5
 next-address 5.5.5.5

ip explicit-path name TUN2_PATH enable
 next-address 192.168.25.5
 next-address 5.5.5.5

ip explicit-path name TUN3_PATH enable
 next-address 192.168.24.4
 next-address 192.168.45.5
 next-address 5.5.5.5
 
interface Tunnel1
 ip unnumbered Loopback0
 tunnel mode mpls traffic-eng
 tunnel destination 5.5.5.5
 tunnel mpls traffic-eng autoroute announce
 tunnel mpls traffic-eng path-option 1 explicit name TUN1_PATH
 tunnel mpls traffic-eng exp 0

interface Tunnel2
 ip unnumbered Loopback0
 tunnel mode mpls traffic-eng
 tunnel destination 5.5.5.5
 tunnel mpls traffic-eng autoroute announce
 tunnel mpls traffic-eng path-option 1 explicit name TUN2_PATH
 tunnel mpls traffic-eng exp 3

interface Tunnel3
 ip unnumbered Loopback0
 tunnel mode mpls traffic-eng
 tunnel destination 5.5.5.5
 tunnel mpls traffic-eng autoroute announce
 tunnel mpls traffic-eng path-option 1 explicit name TUN3_PATH
 tunnel mpls traffic-eng exp 5
 
 And we have to configure what so called the master tunnel interface:

interface Tunnel0
 ip unnumbered Loopback0
 tunnel mode mpls traffic-eng
 tunnel destination 5.5.5.5
 tunnel mpls traffic-eng autoroute announce
 tunnel mpls traffic-eng exp-bundle master
 tunnel mpls traffic-eng exp-bundle member Tunnel1
 tunnel mpls traffic-eng exp-bundle member Tunnel2
 tunnel mpls traffic-eng exp-bundle member Tunnel3

Verifications:

R2#show mpls traffic-eng exp

Destination: 5.5.5.5

   Master: Tunnel0              Status: up

   Members         Status               Conf Exp        Actual Exp
   Tunnel1         up  (Active)         0               0 1 2 4 6 7
   Tunnel2         up  (Active)         3               3
   Tunnel3         up  (Active)         5               5


(D) : Destination is different
(NE): Exp values not configured on tunnel

Let us now check connectivity between CEs:

R1#ping 6.6.6.6 source lo0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 6.6.6.6, timeout is 2 seconds:
Packet sent with a source address of 1.1.1.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 80/104/124 ms

Let us now turn on debug for MPLS packets on three routers (R2 , R3 and R4)

R2#debug mpls packet
Packet debugging is on

R3#debug mpls packet
Packet debugging is on

R4#debug mpls packet
Packet debugging is on

R1#ping 6.6.6.6 source lo0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 6.6.6.6, timeout is 2 seconds:
Packet sent with a source address of 1.1.1.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 72/108/140 ms

R3#
*Jan 15 15:02:10.451: MPLS turbo: Fa1/0: rx: Len 122 Stack {16 0 254} {19 0 254} - ipv4 data s:1.1.1.1 d:6.6.6.6 ttl:254 tos:0 prot:1
*Jan 15 15:02:10.455: MPLS turbo: Fa1/1: tx: Len 118 Stack {19 0 253} - ipv4 data s:1.1.1.1 d:6.6.6.6 ttl:254 tos:0 prot:1
*Jan 15 15:02:10.511: MPLS turbo: Fa1/1: rx: Len 122 Stack {17 0 254} {18 0 254} - ipv4 data s:6.6.6.6 d:1.1.1.1 ttl:254 tos:0 prot:1
*Jan 15 15:02:10.515: MPLS turbo: Fa1/0: tx: Len 118 Stack {18 0 253} - ipv4 data s:6.6.6.6 d:1.1.1.1 ttl:254 tos:0 prot:1
*Jan 15 15:02:10.615: MPLS turbo: Fa1/0: rx: Len 122 Stack {16 0 254} {19 0 254} - ipv4 data s:1.1.1.1 d:6.6.6.6 ttl:254 tos:0 prot:1
*Jan 15 15:02:10.619: MPLS turbo: Fa1/1: tx: Len 118 Stack {19 0 253} - ipv4 data s:1.1.1.1 d:6.6.6.6 ttl:254 tos:0 prot:1
*Jan 15 15:02:10.675: MPLS turbo: Fa1/1: rx: Len 122 Stack {17 0 254} {18 0 254} - ipv4 data s:6.6.6.6 d:1.1.1.1 ttl:254 tos:0 prot:1
*Jan 15 15:02:10.679: MPLS turbo: Fa1/0: tx: Len 118 Stack {18 0 253} - ipv4 data s:6

R1#ping 6.6.6.6 source lo0 tos 32
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 6.6.6.6, timeout is 2 seconds:
Packet sent with a source address of 1.1.1.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 76/87/100 ms

R3#
*Jan 15 15:02:24.239: MPLS turbo: Fa1/0: tx: Len 118 Stack {18 1 253} - ipv4 data s:6.6.6.6 d:1.1.1.1 ttl:254 tos:20 prot:1
*Jan 15 15:02:24.327: MPLS turbo: Fa1/0: rx: Len 122 Stack {16 1 254} {19 1 254} - ipv4 data s:1.1.1.1 d:6.6.6.6 ttl:254 tos:20 prot:1
*Jan 15 15:02:24.331: MPLS turbo: Fa1/1: tx: Len 118 Stack {19 1 253} - ipv4 data s:1.1.1.1 d:6.6.6.6 ttl:254 tos:20 prot:1
*Jan 15 15:02:24.407: MPLS turbo: Fa1/1: rx: Len 122 Stack {17 1 254} {18 1 254} - ipv4 data s:6.6.6.6 d:1.1.1.1 ttl:254 tos:20 prot:1
*Jan 15 15:02:24.411: MPLS turbo: Fa1/0: tx: Len 118 Stack {18 1 253} - ipv4 data s:6.6.6.6 d:1.1.1.1 ttl:254 tos:20 prot:1

R1#ping 6.6.6.6 source lo0 tos 96
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 6.6.6.6, timeout is 2 seconds:
Packet sent with a source address of 1.1.1.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 72/85/104 ms

R2#555: MPLS turbo: Fa2/1: rx: Len 118 Stack {18 3 254} - ipv4 data s:6.6.6.6 d:1.1.1.1 ttl:254 tos:60 prot:1

R1#ping 6.6.6.6 source lo0 tos 160
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 6.6.6.6, timeout is 2 seconds:
Packet sent with a source address of 1.1.1.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 72/87/112 ms

R4# Fa1/0: tx: Len 118 Stack {18 5 253} - ipv4 data s:6.6.6.6 d:1.1.1.1 ttl:254 tos:A0 prot:1
*Jan 15 15:02:54.707: MPLS turbo: Fa1/0: rx: Len 122 Stack {16 5 254} {19 5 254} - ipv4 data s:1.1.1.1 d:6.6.6.6 ttl:254 tos:A0 prot:1
*Jan 15 15:02:54.711: MPLS turbo: Fa1/1: tx: Len 118 Stack {19 5 253} - ipv4 data s:1.1.1.1 d:6.6.6.6 ttl:254 tos:A0 prot:1
*Jan 15 15:02:54.791: MPLS turbo: Fa1/1: rx: Len 122 Stack {17 5 254} {18 5 254} - ipv4 data s:6.6.6.6 d:1.1.1.1 ttl:254 tos:A0 prot:1
*Jan 15 15:02:54.795: MPLS turbo: Fa1/0: tx: Len 118 Stack {18 5 253} - ipv4 data s:6.6.6.6 d:1.1.1.1 ttl:254 tos:A0 prot:1

We can also check by doing traceroute MPLS command on R2 as below:

R2#traceroute mpls traffic-eng tunnel 1 exp 0
Tracing MPLS TE Label Switched Path on Tunnel1, timeout is 2 seconds

Codes: '!' - success, 'Q' - request not sent, '.' - timeout,
  'L' - labeled output interface, 'B' - unlabeled output interface,
  'D' - DS Map mismatch, 'F' - no FEC mapping, 'f' - FEC mismatch,
  'M' - malformed request, 'm' - unsupported tlvs, 'N' - no label entry,
  'P' - no rx intf label prot, 'p' - premature termination of LSP,
  'R' - transit router, 'I' - unknown upstream index,
  'l' - Label switched with FEC change, 'd' - see DDMAP for return code,
  'X' - unknown return code, 'x' - return code 0

Type escape sequence to abort.
  0 192.168.23.2 MRU 1500 [Labels: 16 Exp: 0]
L 1 192.168.23.3 MRU 1504 [Labels: implicit-null Exp: 0] 56 ms
! 2 192.168.35.5 68 ms

R2#traceroute mpls traffic-eng tunnel 1 exp 1
Tracing MPLS TE Label Switched Path on Tunnel1, timeout is 2 seconds

Codes: '!' - success, 'Q' - request not sent, '.' - timeout,
  'L' - labeled output interface, 'B' - unlabeled output interface,
  'D' - DS Map mismatch, 'F' - no FEC mapping, 'f' - FEC mismatch,
  'M' - malformed request, 'm' - unsupported tlvs, 'N' - no label entry,
  'P' - no rx intf label prot, 'p' - premature termination of LSP,
  'R' - transit router, 'I' - unknown upstream index,
  'l' - Label switched with FEC change, 'd' - see DDMAP for return code,
  'X' - unknown return code, 'x' - return code 0

Type escape sequence to abort.
  0 192.168.23.2 MRU 1500 [Labels: 16 Exp: 1]
L 1 192.168.23.3 MRU 1504 [Labels: implicit-null Exp: 1] 68 ms
! 2 192.168.35.5 64 ms

R2#traceroute mpls traffic-eng tunnel 1 exp 2
Tracing MPLS TE Label Switched Path on Tunnel1, timeout is 2 seconds

Codes: '!' - success, 'Q' - request not sent, '.' - timeout,
  'L' - labeled output interface, 'B' - unlabeled output interface,
  'D' - DS Map mismatch, 'F' - no FEC mapping, 'f' - FEC mismatch,
  'M' - malformed request, 'm' - unsupported tlvs, 'N' - no label entry,
  'P' - no rx intf label prot, 'p' - premature termination of LSP,
  'R' - transit router, 'I' - unknown upstream index,
  'l' - Label switched with FEC change, 'd' - see DDMAP for return code,
  'X' - unknown return code, 'x' - return code 0

Type escape sequence to abort.
  0 192.168.23.2 MRU 1500 [Labels: 16 Exp: 2]
L 1 192.168.23.3 MRU 1504 [Labels: implicit-null Exp: 2] 68 ms
! 2 192.168.35.5 84 ms

R2#traceroute mpls traffic-eng tunnel 1 exp 4
Tracing MPLS TE Label Switched Path on Tunnel1, timeout is 2 seconds

Codes: '!' - success, 'Q' - request not sent, '.' - timeout,
  'L' - labeled output interface, 'B' - unlabeled output interface,
  'D' - DS Map mismatch, 'F' - no FEC mapping, 'f' - FEC mismatch,
  'M' - malformed request, 'm' - unsupported tlvs, 'N' - no label entry,
  'P' - no rx intf label prot, 'p' - premature termination of LSP,
  'R' - transit router, 'I' - unknown upstream index,
  'l' - Label switched with FEC change, 'd' - see DDMAP for return code,
  'X' - unknown return code, 'x' - return code 0

Type escape sequence to abort.
  0 192.168.23.2 MRU 1500 [Labels: 16 Exp: 4]
L 1 192.168.23.3 MRU 1504 [Labels: implicit-null Exp: 4] 76 ms
! 2 192.168.35.5 68 ms

R2#traceroute mpls traffic-eng tunnel 1 exp 6
Tracing MPLS TE Label Switched Path on Tunnel1, timeout is 2 seconds

Codes: '!' - success, 'Q' - request not sent, '.' - timeout,
  'L' - labeled output interface, 'B' - unlabeled output interface,
  'D' - DS Map mismatch, 'F' - no FEC mapping, 'f' - FEC mismatch,
  'M' - malformed request, 'm' - unsupported tlvs, 'N' - no label entry,
  'P' - no rx intf label prot, 'p' - premature termination of LSP,
  'R' - transit router, 'I' - unknown upstream index,
  'l' - Label switched with FEC change, 'd' - see DDMAP for return code,
  'X' - unknown return code, 'x' - return code 0

Type escape sequence to abort.
  0 192.168.23.2 MRU 1500 [Labels: 16 Exp: 6]
L 1 192.168.23.3 MRU 1504 [Labels: implicit-null Exp: 6] 68 ms
! 2 192.168.35.5 104 ms

R2#traceroute mpls traffic-eng tunnel 1 exp 7
Tracing MPLS TE Label Switched Path on Tunnel1, timeout is 2 seconds

Codes: '!' - success, 'Q' - request not sent, '.' - timeout,
  'L' - labeled output interface, 'B' - unlabeled output interface,
  'D' - DS Map mismatch, 'F' - no FEC mapping, 'f' - FEC mismatch,
  'M' - malformed request, 'm' - unsupported tlvs, 'N' - no label entry,
  'P' - no rx intf label prot, 'p' - premature termination of LSP,
  'R' - transit router, 'I' - unknown upstream index,
  'l' - Label switched with FEC change, 'd' - see DDMAP for return code,
  'X' - unknown return code, 'x' - return code 0

Type escape sequence to abort.
  0 192.168.23.2 MRU 1500 [Labels: 16 Exp: 7]
L 1 192.168.23.3 MRU 1504 [Labels: implicit-null Exp: 7] 60 ms
! 2 192.168.35.5 64 ms

R2#traceroute mpls traffic-eng tunnel 2 exp 3
Tracing MPLS TE Label Switched Path on Tunnel2, timeout is 2 seconds

Codes: '!' - success, 'Q' - request not sent, '.' - timeout,
  'L' - labeled output interface, 'B' - unlabeled output interface,
  'D' - DS Map mismatch, 'F' - no FEC mapping, 'f' - FEC mismatch,
  'M' - malformed request, 'm' - unsupported tlvs, 'N' - no label entry,
  'P' - no rx intf label prot, 'p' - premature termination of LSP,
  'R' - transit router, 'I' - unknown upstream index,
  'l' - Label switched with FEC change, 'd' - see DDMAP for return code,
  'X' - unknown return code, 'x' - return code 0

Type escape sequence to abort.
  0 192.168.25.2 MRU 1500 [Labels: implicit-null Exp: 3]
! 1 192.168.25.5 92 ms

R2#traceroute mpls traffic-eng tunnel 3 exp 5
Tracing MPLS TE Label Switched Path on Tunnel3, timeout is 2 seconds

Codes: '!' - success, 'Q' - request not sent, '.' - timeout,
  'L' - labeled output interface, 'B' - unlabeled output interface,
  'D' - DS Map mismatch, 'F' - no FEC mapping, 'f' - FEC mismatch,
  'M' - malformed request, 'm' - unsupported tlvs, 'N' - no label entry,
  'P' - no rx intf label prot, 'p' - premature termination of LSP,
  'R' - transit router, 'I' - unknown upstream index,
  'l' - Label switched with FEC change, 'd' - see DDMAP for return code,
  'X' - unknown return code, 'x' - return code 0

Type escape sequence to abort.
  0 192.168.24.2 MRU 1500 [Labels: 16 Exp: 5]
L 1 192.168.24.4 MRU 1504 [Labels: implicit-null Exp: 5] 68 ms
! 2 192.168.45.5 88 ms