Category Archives: Cisco

Integrated IS-IS Basics

Intermediate System to Intermediate System (IS-IS)

Intermediate System-to-Intermediate System (IS-IS) Protocol is an intra-domain Open System Interconnection (OSI) dynamic routing protocol specified in International Organization for Standardization (ISO) 10589. The protocol is designed to operate in OSI Connectionless Network Service (CLNS).

Connectionless Network Service (CLNS)

OSI CLNS is a network layer service similar to bare IP service. A CLNS entity communicates over Connectionless Network Protocol (CLNP) with its peer CLNS entity.

Integrated IS-IS

IS-IS was originally designed for use as a dynamic routing protocol for ISO CLNP and later adapted to carry IP prefixes in addition to CLNP (known as Integrated or Dual IS-IS) as described in RFC 1195.

IS-IS Terminology
  • CLNP – Connection-Less Network Protocol (ISO 8473, the OSI connectionless network layer protocol, very similar to IP)
  • ES – End System (The OSI term for a host)
  • ES-IS – End System to Intermediate System Routeing Exchange Protocol (ISO 9542 – OSI protocol between routers and end systems)
  • IS – Intermediate System (The OSI term for a router)
  • IS-IS – Intermediate System to Intermediate System Routeing Exchange Protocol (The ISO protocol for routing within a single routing domain)
  • IP – Internetwork Protocol (an Internet Standard Network Layer Protocol)
  • IS-IS Hello – An Hello packet defined by the IS-IS protocol (a type of packet used by the IS-IS protocol)
  • ISH – An Hello packet defined by ISO 9542 (ES-IS protocol). Not the same as IS-IS Hello.
  • LSP – Link State Packet (a type of packet used by the IS-IS protocol)
  • NSAP – Network Service Access Point (a conceptual interface point at which the network service is made available)
  • SEL – NSAP Selector (the last octet of NSAP addresses, also called NSEL)
IS-IS and OSPF

Both Link State Protocols use the Dijkstra SPF Algorithm to calculate loop-free routes. OSPF is used purely within the TCP/IP environment and used in both Enterprise and ISP environment. IS-IS is predominantly used in ISP environment.

  • OSPF uses IP protocol 89 as transport.
  • IS-IS is directly encapsulated in Layer 2.
IS-IS Network Types
Broadcast
  • It is used on multi-access interfaces like Ethernet.
  • Designated Intermediate System is elected (DIS) within a brodacast network, similar to OSPF DR but there is no backup DIS is elected in IS-IS.
  • DIS is elected based on Highest priority and Highest SNPA (MAC) address.
  • DIS election is preemptive.
  • DIS performs flooding of LSA updates to a multicast address. That address is different depending on the Level:
    • 01-80-C2-00-00-15 for Level 2 adjacencies
    • 01-80-C2-00-00-14 for Level 1 adjacencies
  • Flooded LSP are not ACK in a broadcast network. The DIS periodically sends out the CNSP (Complete) update. By default it’s 10 seconds and can be changed using the command isis csnp-interval.
Point-to-Point
  • Used where two routers are connected in point-to-point fashion i.e Serial link. It can be used on an Ethernet link also when only two routers are connected.
  • Cisco recommends using point to point type for better convergence.
  • Each LSP has to be ACKed by PSNP. If one is not ACKed, the retransmit timer will trigger a retransmit.
  • L1 and L2 LSPs are directly sent to the neighbor.
  • No DIS election is done on point to point.
IS-IS Areas

Both IS-IS and OSPF use areas to create a hierarchical topology, but there is a fundamental difference exists in the way in which the two protocols define their areas. OSPF area borders are marked by routers where some interfaces of a router are in one area, and other interfaces are in another area. When an OSPF router has interfaces in more than one area, it is an Area Border Router (ABR).

With IS-IS, an individual router is in only one area, and the border between areas is on the link that connects two routers that are in different areas. This is in contrast to OSPF, in which the area borders are within the Area Border Routers (ABRs). The reason for this difference is that an IS-IS router generally has one network service access point (NSAP) address, and an IP router generally has multiple IP addresses.

An intermediate system can be a level 1 (L1) router, a level 2 (L2) router, or both (L1/L2). L1 routers are analogous to OSPF nonbackbone Internal Routers, L2 routers are analogous to OSPF backbone routers, and L1/L2 routers are analogous to OSPF ABRs.

With OSPF, inter-area traffic must traverse Backbone Area (Area 0) to prevent inter-area routing loops. All routers in the area maintain an identical link-state database for their own Area and Area Border Routers (ABRs) advertise LSA (Links State Advertisement) for destinations outside of its own area.

With ISIS, Every L1 router within an area (including the area’s L1/L2 routers) maintains an identical link-state database. Unlike OSPF ABRs, L1/L2 routers do not by default advertise L2 routes to L1 routers. Therefore, an L1 router has no knowledge of destinations outside of its own area. In this sense, an L1 area is similar to an OSPF totally stubby area. To route a packet to another area, an L1 router must forward the packet to an L1/L2 router. When an L1/L2 router sends its level 1 LSP into an area, it signals other L1 routers that it can reach another area by setting a bit known as the Attached (ATT) bit in the LSP.

Network Entity Title

IS-IS can be used to route TCP/IP, but IS-IS is still an ISO CLNP protocol. The packets by which IS-IS communicates with its peers are CLNS PDUs, which in turn means that even in an IP-only environment, an IS-IS router must have an ISO address. The ISO address is a network address, known as Network Entity Title (NET). While IP addresses are 32 bits long and are normally written in dotted quad notation (such as 192.168.1.1), NETs can be 8 to 20 bytes long, but are generally 10 bytes long and are written as shown in this example:

49.0001.1921.6800.1001.00

The NET address consists of three parts:

Area identifier: The first three bytes are the area ID. The first byte of this example — 49 is the address family identifier (AFI) of the authority, which is equivalent to the IP address space that is assigned to an autonomous system. The AFI value 49 is what IS-IS uses for private addressing, which is the equivalent of RFC 1918 address space for IP protocols.

The second two bytes of the area ID – 0001 represent the IS-IS area number. In this example, the area number is 1.

System identifier: The next six bytes identify the node (that is, the router) on the network. Although you can choose any value for the system identifier, a commonly used method is to use binary-coded decimal (BCD) which involves taking the router’s IP address (the address you assigned to the loopback interface), filling in all leading zeros, and then re-positioning the decimal points to form three two-byte numbers.

In this example, if you pad the IP address 192.168.1.1 with zeros, the result is 192.168.001.001. Rearranging the decimal points gives you 1921.6800.1001.

NET selector: The final two bytes are the NET selector (NSEL). For IS-IS, they must always be 00, to indicate “this system.”

Neighbors and Adjacencies

IS-IS routers discover neighbors and form adjacencies by exchanging IS-IS Hello PDUs. The type of router (L1-only, L2-only, or L1/L2) influences the type of adjacency that is formed, and the area IDs configured on the two neighbors in question.
The following rules apply:

  • Two L1-only routers form an L1 adjacency only if their AIDs match.
  • Two L2-only routers form an L2 adjacency, even if their AIDs are different.
  • An L1-only router forms an L1 adjacency with an L1/L2 router only if their AIDs match.
  • An L2-only router forms an L2 adjacency with an L1/L2 router even if their AIDs are different.
  • Two L1/L2 routers form both L1 and L2 adjacencies if their AIDs match.
  • Two L1/L2 routers form only an L2 adjacency if their AIDs do not match.
IS-IS Metric

IS-IS supports four different metric values – Default Metric, Delay, Expense, Error. Cisco IOS supports only default metric which has a default value of 10. The default metric is always set to 10 irrespective of the interface type (GigE or Serial etc.).

IS-IS Metric Styles
Narrow
  • Default metric type.
  • Metric default is 10 for each interface.
  • Range for narrow metric range for an interface is 1 to 63 (6 bits).
  • The maximum total value of all hops can be 1023 (10 bits)
Wide
  • Wide metric type expands the metric to use 24 bits for TLV-22.
  • The new metric style, link metrics can have a maximum value of 16,777,215 with a total path metric of 4,261,412,864.
  • Cisco recommends running the wide metric style.
  • If there is a mismatch between metric styles, the adjacency will be maintained, but routes will not be accepted with different metric style.
  • Wide metric is required for MPLS TE and Multi topology configuration.
Transitional
  • One way to transition to a different metric-style is by configuring all routers to metric-style transitional.
  • Once all routers are sent you can change to either narrow or wide style.
  • During the transitional phase, both types will be recognized.
IS-IS Topology Types

Single Topology – Single Topology IS-IS is used when multiple protocol stacks, such as IPv4 and IPv6, are configured in an identical 1:1 basis on all interfaces in the topology. It allows a single SPF calculation to apply to both protocol stacks at the same time, simplifying the database calculation and protocol overhead of IS-IS.  By default, IS-IS instances runs in  Single Topology mode with regular Cisco IOS, and Multi Topology mode with Cisco IOS XR. These modes are not compatible with each other and must be configured to match, or to run in transition mode.

Multi Topology – This method is used when transitioning to IPv6 or when different routing topologies are necessary for IPv4 and IPv6 address family.  Each protocol stack maintain separate database structures and use separate SPF runs, which means that one topology is independent of another.

IS-IS Authentication
  • Clear Text
  • MD5

Cisco Group Encrypted Transport VPN – GET VPN

Cisco Group Encrypted Transport VPN

GET VPN is a Cisco solution to encrypt the traffic across private WAN. In recent years government regulations, such as Health Insurance Portability and Accountability Act (HIPAA), Gramm-Leach-Bliley Act (GLBA), and Payment Card Industry Data Security Standard (PCI DSS), mandate encryption even over private IP networks. GET VPN is common deployed over private WAN topologies such as MPLS VPN.

Cisco IOS offers several IP security (IPsec) tunnel-based encryption solutions (for example, Site to Site IPsec, IPsec/GRE, and Dynamic Multipoint VPN (DMVPN) that can be deployed over an MPLS VPN, VPLS or shared IP networks. Traditional tunnel-based encryption solutions are point-to-point.

GET VPN uses IPSec to encrypt the traffic but the main concept of GET VPN is to use group security association (SA) as opposed to the standard LAN to LAN tunnels where the SA is created in a point to point fashion.

Traditional point-to-point IPsec tunneling solutions suffer from multicast replication issues because multicast replication must be performed before tunnel encapsulation and encryption at the IPsec CE (customer edge) router closest to the multicast source. Multicast replication cannot be performed in the provider network because encapsulated multicasts appear to the core network as unicast data.

GET VPN provides a tunnel-less VPN solution. It is tunnel-less as it retains the original IP header of the packet and encrypts only the data payload. To retain the original IP header, the original header is copied and placed before the IPSEC header. GET VPN does not rely on a point-to-point VPN mechanism and can further expand with the capability of scaling any-to-any intersite VPN connectivity. It takes advantage of underlying IP VPN routing infrastructure and does not require an overlay routing control plane.

Cisco’s Group Encrypted Transport VPN (GET VPN) introduces the concept of a trusted group to eliminate point-to-point tunnels and their associated overlay routing. All group members (GMs) share a common security association (SA), also known as a group SA. This enables GMs to decrypt traffic that was encrypted by any other GM. The CE router acts as a GM. In GET VPN networks, there is no need to negotiate point-to- point IPsec tunnels between the members of a group, because GET VPN is “tunnel-less.”

GET VPN Comparison

GET VPN Components
Group Domain of Interpretation – GDOI

The GDOI group key management protocol is used to provide a set of cryptographic keys and policies to a group of devices. In a GET VPN network, GDOI is used to distribute common IPsec keys to a group of enterprise VPN gateways that must communicate securely. These keys are periodically refreshed and are updated on all the VPN gateways using a process called “rekey.” GDOI protocol uses UDP port 848. GDOI is documented in RFC3547.

The GDOI protocol is protected by a Phase 1 Internet Key Exchange (IKE) SA. The participating VPN gateways authenticate themselves to the device providing keys using IKE. Authentication can be performed with a pre-shared key (PSK) or through a public key infrastructure (PKI). After the VPN gateways have been authenticated and provided with the appropriate security keys via the IKE SA, the IKE SA will expire and GDOI is then used to update the GMs in a more scalable and efficient manner.

Key Server – KS

A key server (KS) is an IOS device responsible for creating and maintaining the GET VPN control plane. All encryption policies, such as interesting traffic, encryption protocols, security association, rekey timers, and so on, are centrally defined on the KS and are pushed down to all GMs at registration time. The KS sends two types of keys:

  1. Traffic Encryption Key – TEK
  2. Key Encryption Key – KEK

The TEK becomes the IPsec SA, which is used to communicate with group members within the same group.

The KEK is used to encrypt the rekey messages and is used by the group members (GMs) to decrypt the incoming rekey messages from the key server (KS).

Cooperative Key Servers – COOP KSs

The KS is the most important entity in the GET VPN network because the KS maintains the control plane. Therefore, a single KS is a single point of failure for an entire GET VPN network. Because redundancy is an important consideration for KSs, GET VPN supports multiple KSs, called cooperative (COOP) KSs, to ensure seamless fault recovery if a KS fails or becomes unreachable.

A GM can be configured to register to any available KS from a list of all COOP KSs. GM configuration determines the registration order. The KS defined first is contacted first, followed by the second defined KS, and so on.

When COOP KSs boot, all KSs assume a “secondary” role and begin an election process. One KS, typically the one having the highest priority, is elected as a “primary” KS. The other KSs remain in the secondary state. The primary KS is responsible for creating and distributing group policies to all GMs, and to periodically synchronize the COOP KSs.

Group Member – GM

The group member (GM) is the router that registers with the key server (KS) to get the IPsec SA to communicate with other devices in the group.

Group SA

Unlike traditional IPsec encryption solutions, GET VPN uses the concept of group SA. All members in the GET VPN group can communicate with each other using a common encryption policy and a shared SA. With a common encryption policy and a shared SA, there is no need to negotiate IPsec between GMs.

Group Member ACL

Traffic that requires encryption is statically defined on the key server (KS) through an access control list (ACL). This policy is defined for both unicast and multicast traffic. This information is sent to all authenticated group members (GMs) to create a trusted domain of communication.

It is a best practice to summarize interesting traffic to as few permit entries as possible, and to build symmetric policies. For example, an enterprise network is using subnets from a class A major IP network 10.0.0.0/8 for all LAN interfaces behind the group members. You can define one ACL permit statement “access-list 101 permit ip 10.0.0.0 0.255.255.255 10.0.0.0 0.255.255.255” to represent all the subnets in your network.

Unicast Rekey

The KS will generate a rekey message and send multiple copies of this message, one copy to each GM. The GM will then ACK this rekey message to the KS. The ACK mechanism keeps the list of GMs at the KS current and ensures that the rekey message is only sent to active GMs.

A KS can be configured to re-transmit rekey messages to overcome reachability issues in the network. If a GM does not send an ACK for three consecutive rekey messages, the KS will remove the GM from the active GM database and stop sending rekey messages to that GM.

Multicast Rekey

In the multicast rekey process, a single copy of the rekey message is sent to a multicast group which the GMs will have joined. Each GM joins this multicast group at registration, each GM will therefore receive this rekey message.

The multicast rekey process does not have an ACK mechanism and the KS does not keep a list of active GMs. The KS can be configured to retransmit rekey messages. The CPU overhead is a lot less with multicast rekeying since only a single messages needs to be sent compared to having to replicating it in unicast mode to potentially hundreds of routers.

Multicast must be enabled in the core network for multicast rekey to work in the GET VPN control plane.

Time Based Anti-Replay

Traditional IPSec solutions have anti replay capabilities to prevent a malicious third party from capturing IPSec packets and relaying those packets at a later time to perform an attack against the IPSec endpoints. This is normally done by having a counter based sliding window where the sender sends a packet with a sequence number and the receiver will use the sliding window to determine if the packet is acceptable or if it has arrived out of sequence and outside the window of acceptable packets.

This mechanism is not useful in GET VPN because it uses a group SA. GET VPN therefore uses a time based anti replay function where the KS uses a pseudo time clock. Because the KS uses a pseudo time clock, there is no need to synchronize the time with NTP for this reason.

The primary KS will keep this pseudo time synchronized on all GMs with rekey updates. Every GM will include its pseudo time as a time stamp in the data packets. The receiving VPN gateway will then compare the time stamp of the received packet with the GM reference pseudo time clock it maintains for the group. If the packet is too late it will be dropped.

How GET VPN works
  • GET VPN uses the keying protocol GDOI combined with IPsec standards encryption to encrypt and decrypt the packets, thereby providing an efficient mechanism to secure native (non-tunneled) IP unicast and multicast traffic.
  • Each group member (GM) sends  a registration request to the key server (KS). Using the GDOI protocol, the key server authenticates and authorizes the group member and sends the IPsec policy and the keys, which are required  to encrypt and decrypt IP unicast and multicast packets.
  • After the group member is registered with the IPsec SA, and upon receiving the respective keys, group members can directly exchange encrypted IP unicast and multicast packets with each other, bypassing the key server.
  • As needed, the key server sends a rekey message to all the group members within the group. The rekey message contains the new IPsec policy and keys that are used when the outdated IPsec SA expires.
Implementing Cisco GET VPN

We will use following topology for GET VPN implementation.

Configurations
MPLS Provider Network Configuration
R1 - PE Router

hostname R1
!
vrf definition CUST-A
 rd 100:1
 !
 address-family ipv4
  route-target export 100:1
  route-target import 100:1
 exit-address-family
!
ip cef
!
interface Loopback0
 ip address 10.1.0.1 255.255.255.255
 ip ospf 1 area 0
!
interface FastEthernet0/0
 ip address 10.1.15.1 255.255.255.0
 ip ospf network point-to-point
 ip ospf 1 area 0
 mpls ip
!
interface FastEthernet0/1
 vrf forwarding CUST-A
 ip address 172.16.16.1 255.255.255.0
!
router ospf 1
 router-id 10.1.0.1
!
router bgp 100
 bgp router-id 10.1.0.1
 bgp log-neighbor-changes
 no bgp default ipv4-unicast
 neighbor 10.1.0.5 remote-as 100
 neighbor 10.1.0.5 update-source Loopback0
 !
 address-family ipv4
 exit-address-family
 !
 address-family vpnv4
  neighbor 10.1.0.5 activate
  neighbor 10.1.0.5 send-community extended
 exit-address-family
 !
 address-family ipv4 vrf CUST-A
  redistribute connected
  neighbor 172.16.16.6 remote-as 65001
  neighbor 172.16.16.6 activate
  neighbor 172.16.16.6 as-override
 exit-address-family
!

R2 - PE Router

hostname R2
!
vrf definition CUST-A
 rd 100:1
 !
 address-family ipv4
  route-target export 100:1
  route-target import 100:1
 exit-address-family
!
ip cef
!
interface Loopback0
 ip address 10.1.0.2 255.255.255.255
 ip ospf 1 area 0
!
interface FastEthernet0/0
 ip address 10.1.25.2 255.255.255.0
 ip ospf network point-to-point
 ip ospf 1 area 0
 mpls ip
!
interface FastEthernet0/1
 vrf forwarding CUST-A
 ip address 172.16.27.2 255.255.255.0
!
router ospf 1
 router-id 10.1.0.2
!
router bgp 100
 bgp router-id 10.1.0.2
 bgp log-neighbor-changes
 no bgp default ipv4-unicast
 neighbor 10.1.0.5 remote-as 100
 neighbor 10.1.0.5 update-source Loopback0
 !
 address-family ipv4
 exit-address-family
 !
 address-family vpnv4
  neighbor 10.1.0.5 activate
  neighbor 10.1.0.5 send-community extended
 exit-address-family
 !
 address-family ipv4 vrf CUST-A
  redistribute connected
  neighbor 172.16.27.7 remote-as 65001
  neighbor 172.16.27.7 activate
  neighbor 172.16.27.7 as-override
 exit-address-family
!

R3 - PE Router

hostname R3
!
vrf definition CUST-A
 rd 100:1
 !
 address-family ipv4
  route-target export 100:1
  route-target import 100:1
 exit-address-family
!
ip cef
!
interface Loopback0
 ip address 10.1.0.3 255.255.255.255
 ip ospf 1 area 0
!
interface FastEthernet0/0
 ip address 10.1.35.3 255.255.255.0
 ip ospf network point-to-point
 ip ospf 1 area 0
 mpls ip
!
interface FastEthernet0/1
 vrf forwarding CUST-A
 ip address 172.16.38.3 255.255.255.0
!
router ospf 1
 router-id 10.1.0.3
!
router bgp 100
 bgp router-id 10.1.0.3
 bgp log-neighbor-changes
 no bgp default ipv4-unicast
 neighbor 10.1.0.5 remote-as 100
 neighbor 10.1.0.5 update-source Loopback0
 !
 address-family ipv4
 exit-address-family
 !
 address-family vpnv4
  neighbor 10.1.0.5 activate
  neighbor 10.1.0.5 send-community extended
 exit-address-family
 !
 address-family ipv4 vrf CUST-A
  redistribute connected
  neighbor 172.16.38.8 remote-as 65001
  neighbor 172.16.38.8 activate
  neighbor 172.16.38.8 as-override
 exit-address-family
!

R4 - PE Router

hostname R4
!
vrf definition CUST-A
 rd 100:1
 !
 address-family ipv4
  route-target export 100:1
  route-target import 100:1
 exit-address-family
!
ip cef
!
interface Loopback0
 ip address 10.1.0.4 255.255.255.255
 ip ospf 1 area 0
!
interface FastEthernet0/0
 ip address 10.1.45.4 255.255.255.0
 ip ospf network point-to-point
 ip ospf 1 area 0
 mpls ip
!
interface FastEthernet0/1
 vrf forwarding CUST-A
 ip address 172.16.49.4 255.255.255.0
!
router ospf 1
 router-id 10.1.0.4
!
router bgp 100
 bgp router-id 10.1.0.4
 bgp log-neighbor-changes
 no bgp default ipv4-unicast
 neighbor 10.1.0.5 remote-as 100
 neighbor 10.1.0.5 update-source Loopback0
 !
 address-family ipv4
 exit-address-family
 !
 address-family vpnv4
  neighbor 10.1.0.5 activate
  neighbor 10.1.0.5 send-community extended
 exit-address-family
 !
 address-family ipv4 vrf CUST-A
  redistribute connected
  neighbor 172.16.49.9 remote-as 65001
  neighbor 172.16.49.9 activate
  neighbor 172.16.49.9 as-override
 exit-address-family
!

R5 - P Router

hostname R5
!
ip cef
!
interface Loopback0
 ip address 10.1.0.5 255.255.255.255
 ip ospf 1 area 0
!
interface FastEthernet0/0
 ip address 10.1.15.5 255.255.255.0
 ip ospf network point-to-point
 ip ospf 1 area 0
 mpls ip
!
interface FastEthernet0/1
 ip address 10.1.25.5 255.255.255.0
 ip ospf network point-to-point
 ip ospf 1 area 0
 mpls ip
!
interface FastEthernet1/0
 ip address 10.1.35.5 255.255.255.0
 ip ospf network point-to-point
 ip ospf 1 area 0
 mpls ip
!
interface FastEthernet1/1
 ip address 10.1.45.5 255.255.255.0
 ip ospf network point-to-point
 ip ospf 1 area 0
 mpls ip
!
router ospf 1
 router-id 10.1.0.5
!
router bgp 100
 bgp router-id 10.1.0.5
 bgp log-neighbor-changes
 no bgp default ipv4-unicast
 neighbor PE-ROUTERS peer-group
 neighbor PE-ROUTERS remote-as 100
 neighbor PE-ROUTERS update-source Loopback0
 neighbor 10.1.0.1 peer-group PE-ROUTERS
 neighbor 10.1.0.2 peer-group PE-ROUTERS
 neighbor 10.1.0.3 peer-group PE-ROUTERS
 neighbor 10.1.0.4 peer-group PE-ROUTERS
 !
 address-family ipv4
 exit-address-family
 !
 address-family vpnv4
  neighbor PE-ROUTERS send-community extended
  neighbor PE-ROUTERS route-reflector-client
  neighbor 10.1.0.1 activate
  neighbor 10.1.0.2 activate
  neighbor 10.1.0.3 activate
  neighbor 10.1.0.4 activate
 exit-address-family
!
Customer Sites Configuration

We need to generate RSA key before key server (KS) configurations. RSA keys must be generated on any KS. All KSs must share the same keys, so these keys must be generated with an “exportable” tag. The keys are then imported on the remaining KSs. These keys do not need to be imported on the GMs.

!! -- Generate RSA key on Primary Key Server Router R10 -- !!

R10-Primary-KS#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R10-Primary-KS(config)#crypto key generate rsa general-keys label getvpn-export-general modulus 1024 exportable        
The name for the keys will be: getvpn-export-general

% The key modulus size is 1024 bits
% Generating 1024 bit RSA keys, keys will be exportable...
[OK] (elapsed time was 1 seconds)

R10-Primary-KS(config)#

!! -- Export this key to the terminal -- !!

R10-Primary-KS(config)#crypto key export rsa getvpn-export-general pem terminal 3des myrsakeypasswd
% Key name: getvpn-export-general
   Usage: General Purpose Key
   Key data:
-----BEGIN PUBLIC KEY-----
MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQC0W2WX/cvI41mAJ+AbDgOMpRbi
l6FPtOIiw6SDYa3z4GixsYqem5pt7SLk/spxLeHFNsYaD7XqzRAj/hmDWpfrHOmE
IANfiCwfhnh4/Ep1a0+dBKCqK7C90qtQQswdDJfdX8Eunbp93E73caSws4Qkpwqs
iEigKWoAauIvFlDV6QIDAQAB
-----END PUBLIC KEY-----
-----BEGIN RSA PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED
DEK-Info: DES-EDE3-CBC,DAAC1EB9DAA8FA6D

HqkzonxyuD6bXNJZD6+Zblyscwuvbfskhs9mLnYo1CflKMUxHv1XU/5Ctdvmirp2
AVzlWFgR/FFriVYofQoT1BbCtoCJXsT4nPbF07K/rsBsP7HXX0UEsh6zx/Tagv6Q
EBOjk15Tfx+yP3twIpK7jN3hp/81Yjybg+hDV2sfVElqpVShHr8HrAqpiQdtuQiC
MNUtC3/GJy95CO0Wpg/qAKknG4H/wEQqG4wXU0yZ8qUE71xlO7p9Z2GqGMjcRmwZ
ADsIKvGBwNQSg77VZ+Bq2zYLosThz63r1kn7oVxt2dU4wNSpu3CvcAlVhxf7TJoW
mKCr7H99Yk8upFH6Nh+HUG6+BhFZEkkDOjIU1CJeFFO1qEToQA1U4ue0OMt0IQnU
KMA+1IfNCIoH05L4+x2TRHu+uEk1tgu5smU+QmJYSwn6drj74poXsuRcJgNs4VWO
0B7phKvaoXXUdkXiyaOMgHacnnCHL8LEzmp6BzVUbDCbBtYomeaFCI6B7BaOvwDv
sr1Z85xgynXhm1vGWJNd45Lk7+SSVaOp6qp/IIAcFTbA1OfqtWayMMcDHRsc1m3O
iyn5fa4AUlx6smkhkBe7lh1FCQh+C1DrZdrKVWmo7iuDfAxdC6IW+XiUW1gwlnKd
taw4Fy+AALlhMc/pKHqx2WVX1s6SAWgxwhsjxbFIZXSDdRfhkLdl9QAd6G0zVsBq
U7J1rPwZ1dMtZTDfyj/KSl2DbPpv90vJYGHMcU+h49XfBNF2QsizIkx8mjegmX0O
+ep0LsrsqV5XnMeotYzCTPNmb6PoE/fN+YdhzjWJHmTOt6HAdhRsKw==
-----END RSA PRIVATE KEY-----

R10-Primary-KS(config)#

!! -- Import this key using cut-and-paste -- !!
!! -- to other Key Server (KS) Router R11 -- !!

R11-Backup-KS#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R11-Backup-KS(config)#crypto key import rsa getvpn-export-general pem exportable terminal myrsakeypasswd    
% Enter PEM-formatted public General Purpose key or certificate.
% End with a blank line or "quit" on a line by itself.
-----BEGIN PUBLIC KEY-----
MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQC0W2WX/cvI41mAJ+AbDgOMpRbi
l6FPtOIiw6SDYa3z4GixsYqem5pt7SLk/spxLeHFNsYaD7XqzRAj/hmDWpfrHOmE
IANfiCwfhnh4/Ep1a0+dBKCqK7C90qtQQswdDJfdX8Eunbp93E73caSws4Qkpwqs
iEigKWoAauIvFlDV6QIDAQAB
-----END PUBLIC KEY-----
quit
% Enter PEM-formatted encrypted private General Purpose key.
% End with "quit" on a line by itself.
-----BEGIN RSA PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED
DEK-Info: DES-EDE3-CBC,DAAC1EB9DAA8FA6D

HqkzonxyuD6bXNJZD6+Zblyscwuvbfskhs9mLnYo1CflKMUxHv1XU/5Ctdvmirp2
AVzlWFgR/FFriVYofQoT1BbCtoCJXsT4nPbF07K/rsBsP7HXX0UEsh6zx/Tagv6Q
EBOjk15Tfx+yP3twIpK7jN3hp/81Yjybg+hDV2sfVElqpVShHr8HrAqpiQdtuQiC
MNUtC3/GJy95CO0Wpg/qAKknG4H/wEQqG4wXU0yZ8qUE71xlO7p9Z2GqGMjcRmwZ
ADsIKvGBwNQSg77VZ+Bq2zYLosThz63r1kn7oVxt2dU4wNSpu3CvcAlVhxf7TJoW
mKCr7H99Yk8upFH6Nh+HUG6+BhFZEkkDOjIU1CJeFFO1qEToQA1U4ue0OMt0IQnU
KMA+1IfNCIoH05L4+x2TRHu+uEk1tgu5smU+QmJYSwn6drj74poXsuRcJgNs4VWO
0B7phKvaoXXUdkXiyaOMgHacnnCHL8LEzmp6BzVUbDCbBtYomeaFCI6B7BaOvwDv
sr1Z85xgynXhm1vGWJNd45Lk7+SSVaOp6qp/IIAcFTbA1OfqtWayMMcDHRsc1m3O
iyn5fa4AUlx6smkhkBe7lh1FCQh+C1DrZdrKVWmo7iuDfAxdC6IW+XiUW1gwlnKd
taw4Fy+AALlhMc/pKHqx2WVX1s6SAWgxwhsjxbFIZXSDdRfhkLdl9QAd6G0zVsBq
U7J1rPwZ1dMtZTDfyj/KSl2DbPpv90vJYGHMcU+h49XfBNF2QsizIkx8mjegmX0O
+ep0LsrsqV5XnMeotYzCTPNmb6PoE/fN+YdhzjWJHmTOt6HAdhRsKw==
-----END RSA PRIVATE KEY-----
quit
% Key pair import succeeded.

R11-Backup-KS(config)#
Configuration
!! -- R10 Primary Key Server Router -- !!

hostname R10-Primary-KS
!
!! -- IKE Phase 1 Configuration -- !!
crypto isakmp policy 10
 encr aes 256
 authentication pre-share
 group 2
!! --Defined Pre-shared key for peers network range 172.16.0.0/16--!! 
crypto isakmp key $3cur3dG3tVpNK3Y address 172.16.0.0
crypto isakmp keepalive 10 periodic     
!
!! -- Transform Set Configuration -- !!
crypto ipsec transform-set GDOI-TRANS esp-aes 256 esp-sha256-hmac 
 mode transport
!
!! -- IPsec Profile Configuration -- !!
crypto ipsec profile GDOI-PROFILE
 set security-association lifetime seconds 7200
 set transform-set GDOI-TRANS 
!
!! -- GDOI Group Configuration -- !!
crypto gdoi group GETVPN-GROUP
 identity number 1234
 !! -- Local keyword identified this router as Key Server -- !!
 server local
  rekey retransmit 10 number 2
  !! -- RSA Key -- !!
  rekey authentication mypubkey rsa getvpn-export-general
  !! -- Rekeying through unicast transport -- !!
  rekey transport unicast
  sa ipsec 1
   !! -- Transform Set for Group Members -- !!
   profile GDOI-PROFILE
   !! -- Policies defining traffic to be encrypted -- !!
   match address ipv4 GETVPN-ACL
   replay time window-size 5
  address ipv4 172.16.16.10
  !! -- Enable Cooperative Key Server feature -- !!
  !! -- High Priority server decides the Primary Key Server Role -- !!
  !! -- All other Key Server(s) must be configured with as peers -- !!
  redundancy
   local priority 100
   peer address ipv4 172.16.27.11
!
interface FastEthernet0/0
 ip address 172.16.16.10 255.255.255.0
!
ip route 0.0.0.0 0.0.0.0 172.16.16.6
!
ip access-list extended GETVPN-ACL
 permit ip 192.168.0.0 0.0.255.255 192.168.0.0 0.0.255.255
!

!! -- R11 Backup Key Server Router -- !!

hostname R11-Backup-KS
!
crypto isakmp policy 10
 encr aes 256
 authentication pre-share
 group 2
!
crypto isakmp key $3cur3dG3tVpNK3Y address 172.16.0.0  
crypto isakmp keepalive 10 periodic   
!
crypto ipsec transform-set GDOI-TRANS esp-aes 256 esp-sha256-hmac 
 mode transport
!
crypto ipsec profile GDOI-PROFILE
 set security-association lifetime seconds 7200
 set transform-set GDOI-TRANS 
!
crypto gdoi group GETVPN-GROUP
 identity number 1234
 server local
  rekey retransmit 10 number 2
  rekey authentication mypubkey rsa getvpn-export-general
  rekey transport unicast
  sa ipsec 1
   profile GDOI-PROFILE
   match address ipv4 GETVPN-ACL
   replay time window-size 5
  address ipv4 172.16.27.11
  redundancy
   local priority 75
   peer address ipv4 172.16.16.10
!
interface FastEthernet0/0
 ip address 172.16.27.11 255.255.255.0
!
ip route 0.0.0.0 0.0.0.0 172.16.27.7
!
ip access-list extended GETVPN-ACL
 permit ip 192.168.0.0 0.0.255.255 192.168.0.0 0.0.255.255
!

!! -- R6 CE, GET VPN Group Member Router -- !!

hostname R6-CE-GM
!
!! -- IKE Phase 1 Configuration -- !!
crypto isakmp policy 10
 encr aes 256
 authentication pre-share
 group 2
!
!! -- Defined Pre-shared key for Key Server Routers -- !!
crypto isakmp key $3cur3dG3tVpNK3Y address 172.16.16.10   
crypto isakmp key $3cur3dG3tVpNK3Y address 172.16.27.11   
!
!! -- GDOI Group Configuration -- !!
crypto gdoi group GETVPN-GROUP
 identity number 1234
 server address ipv4 172.16.16.10
 server address ipv4 172.16.27.11
!
!! -- Crypto MAP Configuration -- !!
crypto map GETVPN-MAP 10 gdoi 
 set group GETVPN-GROUP
!
!! -- Associate Crypto MAP to the WAN Interfaces -- !!
interface FastEthernet0/0
 description To MPLS Provider
 ip address 172.16.16.6 255.255.255.0
 crypto map GETVPN-MAP
!
interface FastEthernet0/1
 description To LAN
 ip address 192.168.6.6 255.255.255.0
!
router bgp 65001
 bgp router-id 172.16.16.6
 bgp log-neighbor-changes
 network 192.168.0.6 mask 255.255.255.255
 network 192.168.6.0
 neighbor 172.16.16.1 remote-as 100
!

!! -- R7 CE, GET VPN Group Member Router -- !!

hostname R7-CE-GM
!
crypto isakmp policy 10
 encr aes 256
 authentication pre-share
 group 2
!
crypto isakmp key $3cur3dG3tVpNK3Y address 172.16.16.10   
crypto isakmp key $3cur3dG3tVpNK3Y address 172.16.27.11   
!
crypto gdoi group GETVPN-GROUP
 identity number 1234
 server address ipv4 172.16.16.10
 server address ipv4 172.16.27.11
!
crypto map GETVPN-MAP 10 gdoi 
 set group GETVPN-GROUP
!
interface FastEthernet0/0
 description To MPLS Provider
 ip address 172.16.27.7 255.255.255.0
 crypto map GETVPN-MAP
!
interface FastEthernet0/1
 description To LAN
 ip address 192.168.7.7 255.255.255.0
!
router bgp 65001
 bgp router-id 172.16.27.7
 bgp log-neighbor-changes
 network 192.168.0.7 mask 255.255.255.255
 network 192.168.7.0
 neighbor 172.16.27.2 remote-as 100
!

!! -- R8 CE, GET VPN Group Member Router -- !!

hostname R8-CE-GM
!
crypto isakmp policy 10
 encr aes 256
 authentication pre-share
 group 2
!
crypto isakmp key $3cur3dG3tVpNK3Y address 172.16.16.10   
crypto isakmp key $3cur3dG3tVpNK3Y address 172.16.27.11   
!
crypto gdoi group GETVPN-GROUP
 identity number 1234
 server address ipv4 172.16.16.10
 server address ipv4 172.16.27.11
!
crypto map GETVPN-MAP 10 gdoi 
 set group GETVPN-GROUP
!
interface FastEthernet0/0
 description To MPLS Provider
 ip address 172.16.38.8 255.255.255.0
 crypto map GETVPN-MAP
!
interface FastEthernet0/1
 description To LAN
 ip address 192.168.8.8 255.255.255.0
!
router bgp 65001
 bgp router-id 192.168.8.8
 bgp log-neighbor-changes
 network 192.168.8.0
 neighbor 172.16.38.3 remote-as 100
!

!! -- R9 CE, GET VPN Group Member Router -- !!

hostname R9-CE-GM
!
crypto isakmp policy 10
 encr aes 256
 authentication pre-share
 group 2
!
crypto isakmp key $3cur3dG3tVpNK3Y address 172.16.16.10   
crypto isakmp key $3cur3dG3tVpNK3Y address 172.16.27.11   
!
crypto gdoi group GETVPN-GROUP
 identity number 1234
 server address ipv4 172.16.16.10
 server address ipv4 172.16.27.11
!
crypto map GETVPN-MAP 10 gdoi 
 set group GETVPN-GROUP
!
interface FastEthernet0/0
 description To MPLS Provider
 ip address 172.16.49.9 255.255.255.0
 crypto map GETVPN-MAP
!
interface FastEthernet0/1
 description To LAN
 ip address 192.168.9.9 255.255.255.0
!
router bgp 65001
 bgp router-id 192.168.9.9
 bgp log-neighbor-changes
 network 192.168.9.0
 neighbor 172.16.49.4 remote-as 100
!
Verification and Testing
!! -- Routing on all Customer sites Routers -- !!

R6-CE-GM#show ip route | beg Gate
Gateway of last resort is not set

      172.16.0.0/16 is variably subnetted, 5 subnets, 2 masks
C        172.16.16.0/24 is directly connected, FastEthernet0/0
L        172.16.16.6/32 is directly connected, FastEthernet0/0
B        172.16.27.0/24 [20/0] via 172.16.16.1, 09:25:04
B        172.16.38.0/24 [20/0] via 172.16.16.1, 09:25:04
B        172.16.49.0/24 [20/0] via 172.16.16.1, 09:24:55
      192.168.6.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.6.0/24 is directly connected, FastEthernet0/1
L        192.168.6.6/32 is directly connected, FastEthernet0/1
B     192.168.7.0/24 [20/0] via 172.16.16.1, 09:25:04
B     192.168.8.0/24 [20/0] via 172.16.16.1, 09:25:04
B     192.168.9.0/24 [20/0] via 172.16.16.1, 09:23:48


R7-CE-GM#show ip route | beg Gate
Gateway of last resort is not set

      172.16.0.0/16 is variably subnetted, 5 subnets, 2 masks
B        172.16.16.0/24 [20/0] via 172.16.27.2, 09:25:25
C        172.16.27.0/24 is directly connected, FastEthernet0/0
L        172.16.27.7/32 is directly connected, FastEthernet0/0
B        172.16.38.0/24 [20/0] via 172.16.27.2, 09:25:25
B        172.16.49.0/24 [20/0] via 172.16.27.2, 09:25:16
B     192.168.6.0/24 [20/0] via 172.16.27.2, 09:25:25
      192.168.7.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.7.0/24 is directly connected, FastEthernet0/1
L        192.168.7.7/32 is directly connected, FastEthernet0/1
B     192.168.8.0/24 [20/0] via 172.16.27.2, 09:25:25
B     192.168.9.0/24 [20/0] via 172.16.27.2, 09:24:09

R8-CE-GM#show ip route | beg Gate
Gateway of last resort is not set

      172.16.0.0/16 is variably subnetted, 5 subnets, 2 masks
B        172.16.16.0/24 [20/0] via 172.16.38.3, 09:25:35
B        172.16.27.0/24 [20/0] via 172.16.38.3, 09:25:35
C        172.16.38.0/24 is directly connected, FastEthernet0/0
L        172.16.38.8/32 is directly connected, FastEthernet0/0
B        172.16.49.0/24 [20/0] via 172.16.38.3, 09:25:26
B     192.168.6.0/24 [20/0] via 172.16.38.3, 09:25:35
B     192.168.7.0/24 [20/0] via 172.16.38.3, 09:25:35
      192.168.8.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.8.0/24 is directly connected, FastEthernet0/1
L        192.168.8.8/32 is directly connected, FastEthernet0/1
B     192.168.9.0/24 [20/0] via 172.16.38.3, 09:24:19

R9-CE-GM#show ip route | beg Gate
Gateway of last resort is not set

      172.16.0.0/16 is variably subnetted, 5 subnets, 2 masks
B        172.16.16.0/24 [20/0] via 172.16.49.4, 09:24:31
B        172.16.27.0/24 [20/0] via 172.16.49.4, 09:24:31
B        172.16.38.0/24 [20/0] via 172.16.49.4, 09:24:31
C        172.16.49.0/24 is directly connected, FastEthernet0/0
L        172.16.49.9/32 is directly connected, FastEthernet0/0
B     192.168.6.0/24 [20/0] via 172.16.49.4, 09:24:31
B     192.168.7.0/24 [20/0] via 172.16.49.4, 09:24:31
B     192.168.8.0/24 [20/0] via 172.16.49.4, 09:24:31
      192.168.9.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.9.0/24 is directly connected, FastEthernet0/1
L        192.168.9.9/32 is directly connected, FastEthernet0/1

R10-Primary-KS#show ip route | beg Gate
Gateway of last resort is 172.16.16.6 to network 0.0.0.0

S*    0.0.0.0/0 [1/0] via 172.16.16.6
      172.16.0.0/16 is variably subnetted, 2 subnets, 2 masks
C        172.16.16.0/24 is directly connected, FastEthernet0/0
L        172.16.16.10/32 is directly connected, FastEthernet0/0

R11-Backup-KS#show ip route | beg Gate
Gateway of last resort is 172.16.27.7 to network 0.0.0.0

S*    0.0.0.0/0 [1/0] via 172.16.27.7
      172.16.0.0/16 is variably subnetted, 2 subnets, 2 masks
C        172.16.27.0/24 is directly connected, FastEthernet0/0
L        172.16.27.11/32 is directly connected, FastEthernet0/0


!! -- When only Primary Key Server is configured -- !!
!! -- KS would participate in the Election with Secondary role -- !! 

*Apr 19 15:21:13.519: %CRYPTO-6-GDOI_ON_OFF: GDOI is ON
*Apr 19 15:21:47.923: %GDOI-5-KS_REKEY_TRANS_2_UNI: Group GETVPN-GROUP transitioned to Unicast Rekey.
*Apr 19 15:22:55.623: %GDOI-5-COOP_KS_ADD: 172.16.27.11 added as COOP Key Server in group GETVPN-GROUP.
*Apr 19 15:23:35.671: %GDOI-5-COOP_KS_ELECTION: KS entering election mode in group GETVPN-GROUP (Previous Primary = NONE)

R10-Primary-KS#sh crypto gdoi ks coop 
Crypto Gdoi Group Name :GETVPN-GROUP 
        Group handle: 2147483650, Local Key Server handle: 2147483650

        Local Address: 172.16.16.10 
        Local Priority: 100      
        Local KS Role: Secondary , Local KS Status: Alive     
        Local KS version: 1.0.4 
        Secondary Timers: 
                Sec Primary Periodic Time: 30 
                Remaining Time: 2, Retries: 2
                Invalid ANN PST recvd: 0
                New GM Temporary Blocking Enforced?: No
                Antireplay Sequence Number: 3

        Peer Sessions:
        Session 1:
                Server handle: 2147483651
                Peer Address: 172.16.27.11
                Peer Version: 0.0.0 
                Peer Priority: Unknown 
                Peer KS Role: Secondary , Peer KS Status: Unknown   
                Antireplay Sequence Number: 0

                IKE status: In Progress
                Counters:
                    Ann msgs sent: 0
                    Ann msgs sent with reply request: 0
                    Ann msgs recv: 0 
                    Ann msgs recv with reply request: 0
                    Packet sent drops: 3 
                    Packet Recv drops: 0 
                    Total bytes sent: 0 
                    Total bytes recv: 0
                
!! -- It keep trying to reach peer KS but we have not confiugured--!! 
!! -- Backup KS (Peer KS) so when Primary KS do not receive response!! 
!! -- from any other KS, it will become Primary and declare -- !!
!! -- Peer KS as dead -- !!

*Apr 19 15:25:05.775: %GDOI-5-COOP_KS_TRANS_TO_PRI: KS 172.16.16.10 in group GETVPN-GROUP transitioned to Primary (Previous Primary = NONE)
*Apr 19 15:25:05.815: %GDOI-3-COOP_KS_UNREACH: Cooperative KS 172.16.27.11 Unreachable in group GETVPN-GROUP. IKE SA Status = Failed to establish.

R10-Primary-KS#sh crypto gdoi ks coop 
Crypto Gdoi Group Name :GETVPN-GROUP 
        Group handle: 2147483650, Local Key Server handle: 2147483650

        Local Address: 172.16.16.10 
        Local Priority: 100      
        Local KS Role: Primary   , Local KS Status: Alive     
        Local KS version: 1.0.4 
        Primary Timers: 
                Primary Refresh Policy Time: 20 
                Remaining Time: 18
                Antireplay Sequence Number: 5

        Peer Sessions:
        Session 1:
                Server handle: 2147483651
                Peer Address: 172.16.27.11
                Peer Version: 0.0.0 
                Peer Priority: Unknown 
                Peer KS Role: Secondary , Peer KS Status: Dead      
                Antireplay Sequence Number: 0

                IKE status: In Progress
                Counters:
                    Ann msgs sent: 0
                    Ann msgs sent with reply request: 0
                    Ann msgs recv: 0 
                    Ann msgs recv with reply request: 0
                    Packet sent drops: 5 
                    Packet Recv drops: 0 
                    Total bytes sent: 0 
                    Total bytes recv: 0

!! -- When we configure Backup Key Server, it will also participate-!!
!! -- in the Election with Secondary KS Role -- !!
!! -- When It knows Primary KS with higher priority is active -- !!
!! -- It remains in the Secondary Role -- !! 

R11-Backup-KS# 
*Apr 19 15:26:17.627: %CRYPTO-6-GDOI_ON_OFF: GDOI is ON
*Apr 19 15:27:24.143: %GDOI-5-COOP_KS_ADD: 172.16.16.10 added as COOP Key Server in group GETVPN-GROUP.
*Apr 19 15:27:24.183: %GDOI-5-COOP_KS_ELECTION: KS entering election mode in group GETVPN-GROUP (Previous Primary = NONE)
*Apr 19 15:27:29.527: %GDOI-4-GDOI_ANN_TIMESTAMP_LARGE: COOP_KS ANN received from KS 172.16.16.10 in group GETVPN-GROUP has PST bigger than myself. Adjust to new PST:
my_old_pst is 71 sec, peer_pst is 375 sec
*Apr 19 15:27:34.543: %GDOI-5-COOP_KS_TRANS_TO_PRI: KS 172.16.16.10 in group GETVPN-GROUP transitioned to Primary (Previous Primary = NONE)

R11-Backup-KS#sh crypto gdoi ks coop                 
Crypto Gdoi Group Name :GETVPN-GROUP 
        Group handle: 2147483650, Local Key Server handle: 2147483650

        Local Address: 172.16.27.11 
        Local Priority: 75       
        Local KS Role: Secondary , Local KS Status: Alive     
        Local KS version: 1.0.4 
        Secondary Timers: 
                Sec Primary Periodic Time: 30 
                Remaining Time: 27, Retries: 0
                Invalid ANN PST recvd: 0
                New GM Temporary Blocking Enforced?: No
                Antireplay Sequence Number: 1

        Peer Sessions:
        Session 1:
                Server handle: 2147483651
                Peer Address: 172.16.16.10
                Peer Version: 1.0.4 
                Peer Priority: 100             
                Peer KS Role: Primary   , Peer KS Status: Alive     
                Antireplay Sequence Number: 42

                IKE status: Established
                Counters:
                    Ann msgs sent: 0
                    Ann msgs sent with reply request: 1
                    Ann msgs recv: 29 
                    Ann msgs recv with reply request: 0
                    Packet sent drops: 0 
                    Packet Recv drops: 0 
                    Total bytes sent: 152 
                    Total bytes recv: 18415

!! -- Now IKE phase-1 is UP between Primary and Backup KS -- !!

R10-Primary-KS#sh cry isa sa
IPv4 Crypto ISAKMP SA
dst             src             state          conn-id status
172.16.27.11    172.16.16.10    GDOI_IDLE         1001 ACTIVE

R11-Backup-KS#sh cry isa sa
IPv4 Crypto ISAKMP SA
dst             src             state          conn-id status
172.16.27.11    172.16.16.10    GDOI_IDLE         1001 ACTIVE

!! -- Verify interesting traffic ACL added into GETVPN domain -- !!

R10-Primary-KS#sh crypto gdoi ks acl 
Group Name: GETVPN-GROUP
 Configured ACL:
   access-list GETVPN-ACL  permit ip 192.168.0.0 0.0.255.255 192.168.0.0 0.0.255.255

!! -- Verify GDOI Policy on KS, which would be pushed to the -- !!
!! -- Group Members (GM) -- !!

R10-Primary-KS#sh crypto gdoi ks policy 
Key Server Policy:
For group GETVPN-GROUP (handle: 2147483650) server 172.16.16.10 (handle: 2147483650):

  # of teks : 2  Seq num : 1
  KEK POLICY (transport type : Unicast)
    spi : 0x25C92085E5951A54B8182918A13D10B4
    management alg     : disabled    encrypt alg       : 3DES      
    crypto iv length   : 8           key size          : 24      
    orig life(sec): 86400       remaining life(sec): 79722     
    sig hash algorithm : enabled     sig key length    : 162     
    sig size           : 128       
    sig key name       : getvpn-export-general

  TEK POLICY (encaps : ENCAPS_TRANSPORT)
    spi                : 0xF22F8F3B
    access-list        : GETVPN-ACL
    transform          : esp-256-aes esp-sha256-hmac 
    alg key size       : 32            sig key size          : 32        
    orig life(sec)     : 7200          remaining life(sec)   : 523       
    tek life(sec)      : 7200          elapsed time(sec)     : 6677      
    override life (sec): 0             antireplay window size: 5         

          
  TEK POLICY (encaps : ENCAPS_TRANSPORT)
    spi                : 0xBB9CF351
    access-list        : GETVPN-ACL
    transform          : esp-256-aes esp-sha256-hmac 
    alg key size       : 32            sig key size          : 32        
    orig life(sec)     : 7200          remaining life(sec)   : 6978      
    tek life(sec)      : 7200          elapsed time(sec)     : 222       
    override life (sec): 0             antireplay window size: 5         

  Replay Value 6909.39 secs 
For group GETVPN-GROUP (handle: 2147483650) server 172.16.27.11 (handle: 2147483651):

!! -- After configuration of Group Member (GM) Router R6 -- !!
!! -- It will register to all key servers (KS) in order -- !!
!! -- of KS configuration list on GM -- !!
!! -- But GM will have ISAKMP SA only with Primary KS -- !!
!! -- and Primary KS will push the GDOI policies to GM -- !!

R6-CE-GM(config-if)#
*Apr 19 15:41:54.175: %CRYPTO-5-GM_REGSTER: Start registration to KS 172.16.16.10 for group GETVPN-GROUP using address 172.16.16.6
*Apr 19 15:41:54.191: %CRYPTO-6-GDOI_ON_OFF: GDOI is ON
R6-CE-GM(config-if)#
*Apr 19 15:41:54.719: %GDOI-5-GM_REKEY_TRANS_2_UNI: Group GETVPN-GROUP transitioned to Unicast Rekey.
*Apr 19 15:41:54.723: %GDOI-5-SA_KEK_UPDATED: SA KEK was updated
*Apr 19 15:41:54.727: %GDOI-5-SA_TEK_UPDATED: SA TEK was updated
*Apr 19 15:41:54.863: %GDOI-5-GM_REGS_COMPL: Registration to KS 172.16.16.10 complete for group GETVPN-GROUP using address 172.16.16.6
*Apr 19 15:41:54.883: %GDOI-5-GM_INSTALL_POLICIES_SUCCESS: SUCCESS: Installation of Reg/Rekey policies from KS 172.16.16.10 for group GETVPN-GROUP & gm identity  172.16.16.6
R6-CE-GM(config-if)#end
R6-CE-GM#

R6-CE-GM#sh cry isa sa
IPv4 Crypto ISAKMP SA
dst             src             state          conn-id status
172.16.16.10    172.16.16.6     GDOI_IDLE         1001 ACTIVE
172.16.16.6     172.16.16.10    GDOI_REKEY        1002 ACTIVE

!! -- Similarily configure Group Member Router R7 -- !!

R7-CE-GM(config-if)#
*Apr 19 15:46:30.231: %CRYPTO-5-GM_REGSTER: Start registration to KS 172.16.16.10 for group GETVPN-GROUP using address 172.16.27.7
*Apr 19 15:46:30.247: %CRYPTO-6-GDOI_ON_OFF: GDOI is ON
*Apr 19 15:46:31.643: %GDOI-5-GM_REKEY_TRANS_2_UNI: Group GETVPN-GROUP transitioned to Unicast Rekey.
*Apr 19 15:46:31.647: %GDOI-5-SA_KEK_UPDATED: SA KEK was updated
*Apr 19 15:46:31.651: %GDOI-5-SA_TEK_UPDATED: SA TEK was updated
*Apr 19 15:46:31.851: %GDOI-5-GM_REGS_COMPL: Registration to KS 172.16.16.10 complete for group GETVPN-GROUP using address 172.16.27.7
*Apr 19 15:46:31.871: %GDOI-5-GM_INSTALL_POLICIES_SUCCESS: SUCCESS: Installation of Reg/Rekey policies from KS 172.16.16.10 for group GETVPN-GROUP & gm identity  172.16.27.7

R7-CE-GM#
*Apr 19 15:46:33.451: %SYS-5-CONFIG_I: Configured from console by console
R7-CE-GM#sh cry isa sa
IPv4 Crypto ISAKMP SA
dst             src             state          conn-id status
172.16.16.10    172.16.27.7     GDOI_IDLE         1001 ACTIVE
172.16.27.7     172.16.16.10    GDOI_REKEY        1002 ACTIVE

!! -- Configure Group Member Router R8 -- !!

R8-CE-GM(config-if)#
*Apr 19 16:11:57.187: %CRYPTO-5-GM_REGSTER: Start registration to KS 172.16.16.10 for group GETVPN-GROUP using address 172.16.38.8
R8-CE-GM(config-if)#end
R8-CE-GM#
*Apr 19 16:11:57.207: %CRYPTO-6-GDOI_ON_OFF: GDOI is ON
R8-CE-GM#
*Apr 19 16:11:58.739: %SYS-5-CONFIG_I: Configured from console by console
*Apr 19 16:11:58.795: %GDOI-5-GM_REKEY_TRANS_2_UNI: Group GETVPN-GROUP transitioned to Unicast Rekey.
*Apr 19 16:11:58.799: %GDOI-5-SA_KEK_UPDATED: SA KEK was updated
*Apr 19 16:11:58.803: %GDOI-5-SA_TEK_UPDATED: SA TEK was updated
*Apr 19 16:11:59.131: %GDOI-5-GM_REGS_COMPL: Registration to KS 172.16.16.10 complete for group GETVPN-GROUP using address 172.16.38.8
*Apr 19 16:11:59.151: %GDOI-5-GM_INSTALL_POLICIES_SUCCESS: SUCCESS: Installation of Reg/Rekey policies from KS 172.16.16.10 for group GETVPN-GROUP & gm identity  172.16.38.8
R8-CE-GM#sh cry isa sa
IPv4 Crypto ISAKMP SA
dst             src             state          conn-id status
172.16.38.8     172.16.16.10    GDOI_REKEY        1002 ACTIVE
172.16.16.10    172.16.38.8     GDOI_IDLE         1001 ACTIVE

!! -- Configure Group Member Router R9 -- !!

R9-CE-GM(config-if)#
*Apr 19 16:13:31.651: %CRYPTO-5-GM_REGSTER: Start registration to KS 172.16.16.10 for group GETVPN-GROUP using address 172.16.49.9
R9-CE-GM(config-if)#
*Apr 19 16:13:31.667: %CRYPTO-6-GDOI_ON_OFF: GDOI is ON
R9-CE-GM(config-if)#
*Apr 19 16:13:33.055: %GDOI-5-GM_REKEY_TRANS_2_UNI: Group GETVPN-GROUP transitioned to Unicast Rekey.
*Apr 19 16:13:33.059: %GDOI-5-SA_KEK_UPDATED: SA KEK was updated
*Apr 19 16:13:33.063: %GDOI-5-SA_TEK_UPDATED: SA TEK was updated
*Apr 19 16:13:33.279: %GDOI-5-GM_REGS_COMPL: Registration to KS 172.16.16.10 complete for group GETVPN-GROUP using address 172.16.49.9
*Apr 19 16:13:33.295: %GDOI-5-GM_INSTALL_POLICIES_SUCCESS: SUCCESS: Installation of Reg/Rekey policies from KS 172.16.16.10 for group GETVPN-GROUP & gm identity  172.16.49.9
R9-CE-GM(config-if)#end
R9-CE-GM#
*Apr 19 16:13:35.263: %SYS-5-CONFIG_I: Configured from console by console
R9-CE-GM#sh cry isa sa
IPv4 Crypto ISAKMP SA
dst             src             state          conn-id status
172.16.49.9     172.16.16.10    GDOI_REKEY        1002 ACTIVE
172.16.16.10    172.16.49.9     GDOI_IDLE         1001 ACTIVE

!! -- After configuration of all GM -- !!

R10-Primary-KS#sh cry isa sa
IPv4 Crypto ISAKMP SA
dst             src             state          conn-id status
172.16.16.10    172.16.27.7     GDOI_IDLE         1003 ACTIVE
172.16.16.10    172.16.49.9     GDOI_IDLE         1005 ACTIVE
172.16.16.10    172.16.16.6     GDOI_IDLE         1002 ACTIVE
172.16.27.11    172.16.16.10    GDOI_IDLE         1001 ACTIVE
172.16.16.10    172.16.38.8     GDOI_IDLE         1004 ACTIVE

R11-Backup-KS#sh cry isa sa
IPv4 Crypto ISAKMP SA
dst             src             state          conn-id status
172.16.27.11    172.16.16.10    GDOI_IDLE         1001 ACTIVE

!! -- Verify GDOI status on Group Member (GM) Routers -- !!

R6-CE-GM#show crypto gdoi 
GROUP INFORMATION

    Group Name               : GETVPN-GROUP
    Group Identity           : 1234
    Crypto Path              : ipv4
    Key Management Path      : ipv4
    Rekeys received          : 1
    IPSec SA Direction       : Both

     Group Server list       : 172.16.16.10
                               172.16.27.11
                               
    Group member             : 172.16.16.6      vrf: None
       Version               : 1.0.4 
       Registration status   : Registered
       Registered with       : 172.16.16.10
       Re-registers in       : 6947 sec
       Succeeded registration: 1
       Attempted registration: 1
       Last rekey from       : 172.16.16.10
       Last rekey seq num    : 1
       Unicast rekey received: 1
       Rekey ACKs sent       : 1
       Rekey Rcvd(hh:mm:ss)  : 00:00:31
       allowable rekey cipher: any
       allowable rekey hash  : any
       allowable transformtag: any ESP

    Rekeys cumulative
       Total received        : 1
       After latest register : 1
       Rekey Acks sents      : 1

 ACL Downloaded From KS 172.16.16.10:
   access-list   permit ip 192.168.0.0 0.0.255.255 192.168.0.0 0.0.255.255

KEK POLICY:
    Rekey Transport Type     : Unicast
    Lifetime (secs)          : 79912
    Encrypt Algorithm        : 3DES
    Key Size                 : 192     
    Sig Hash Algorithm       : HMAC_AUTH_SHA
    Sig Key Length (bits)    : 1024    

TEK POLICY for the current KS-Policy ACEs Downloaded:
  FastEthernet0/0:
    IPsec SA:
        spi: 0xF22F8F3B(4063203131)
        transform: esp-256-aes esp-sha256-hmac 
        sa timing:remaining key lifetime (sec): (714)
        Anti-Replay(Time Based) : 5 sec interval

    IPsec SA:
        spi: 0xBB9CF351(3147625297)
        transform: esp-256-aes esp-sha256-hmac 
        sa timing:remaining key lifetime (sec): (7169)
        Anti-Replay(Time Based) : 5 sec interval

!! -- No traffic is exchanged between sites yet -- !!
!! -- encap/decap and encryp/decrypt packet count is zero -- !!

R8-CE-GM#sh cry ipsec sa

interface: FastEthernet0/0
    Crypto map tag: GETVPN-MAP, local addr 172.16.38.8

   protected vrf: (none)
   local  ident (addr/mask/prot/port): (192.168.0.0/255.255.0.0/0/0)
   remote ident (addr/mask/prot/port): (192.168.0.0/255.255.0.0/0/0)
   current_peer 0.0.0.0 port 848
     PERMIT, flags={}
    #pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0
    #pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0
    #pkts compressed: 0, #pkts decompressed: 0
    #pkts not compressed: 0, #pkts compr. failed: 0
    #pkts not decompressed: 0, #pkts decompress failed: 0
    #send errors 0, #recv errors 0

     local crypto endpt.: 172.16.38.8, remote crypto endpt.: 0.0.0.0
     path mtu 1500, ip mtu 1500, ip mtu idb FastEthernet0/0
     current outbound spi: 0xBB9CF351(3147625297)
     PFS (Y/N): N, DH group: none

     inbound esp sas:
      spi: 0xBB9CF351(3147625297)
        transform: esp-256-aes esp-sha256-hmac ,
        in use settings ={Transport, }
        conn id: 3, flow_id: SW:3, sibling_flags 80000000, crypto map: GETVPN-MAP
        sa timing: remaining key lifetime (sec): (6387)
        Kilobyte Volume Rekey has been disabled
        IV size: 16 bytes
        replay detection support: Y  replay window size: 5
        Status: ACTIVE(ACTIVE)

     inbound ah sas:

     inbound pcp sas:

     outbound esp sas:
      spi: 0xBB9CF351(3147625297)
        transform: esp-256-aes esp-sha256-hmac ,
        in use settings ={Transport, }
        conn id: 4, flow_id: SW:4, sibling_flags 80000000, crypto map: GETVPN-MAP
        sa timing: remaining key lifetime (sec): (6387)
        Kilobyte Volume Rekey has been disabled
        IV size: 16 bytes
        replay detection support: Y  replay window size: 5
        Status: ACTIVE(ACTIVE)

     outbound ah sas:

     outbound pcp sas:

!! -- Send traffic between Site-3 and Site-4 LAN -- !!

R8-CE-GM#ping 192.168.9.9 so 192.168.8.8
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.9.9, timeout is 2 seconds:
Packet sent with a source address of 192.168.8.8 
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 184/202/228 ms
R8-CE-GM#sh cry ipsec sa                

interface: FastEthernet0/0
    Crypto map tag: GETVPN-MAP, local addr 172.16.38.8

   protected vrf: (none)
   local  ident (addr/mask/prot/port): (192.168.0.0/255.255.0.0/0/0)
   remote ident (addr/mask/prot/port): (192.168.0.0/255.255.0.0/0/0)
   current_peer 0.0.0.0 port 848
     PERMIT, flags={}
    #pkts encaps: 5, #pkts encrypt: 5, #pkts digest: 5
    #pkts decaps: 5, #pkts decrypt: 5, #pkts verify: 5
    #pkts compressed: 0, #pkts decompressed: 0
    #pkts not compressed: 0, #pkts compr. failed: 0
    #pkts not decompressed: 0, #pkts decompress failed: 0
    #send errors 0, #recv errors 0

     local crypto endpt.: 172.16.38.8, remote crypto endpt.: 0.0.0.0
     path mtu 1500, ip mtu 1500, ip mtu idb FastEthernet0/0
     current outbound spi: 0xBB9CF351(3147625297)
     PFS (Y/N): N, DH group: none

     inbound esp sas:
      spi: 0xBB9CF351(3147625297)
        transform: esp-256-aes esp-sha256-hmac ,
        in use settings ={Transport, }
        conn id: 3, flow_id: SW:3, sibling_flags 80000000, crypto map: GETVPN-MAP
        sa timing: remaining key lifetime (sec): (6373)
        Kilobyte Volume Rekey has been disabled
        IV size: 16 bytes
        replay detection support: Y  replay window size: 5
        Status: ACTIVE(ACTIVE)

     inbound ah sas:

     inbound pcp sas:

     outbound esp sas:
      spi: 0xBB9CF351(3147625297)
        transform: esp-256-aes esp-sha256-hmac ,
        in use settings ={Transport, }
        conn id: 4, flow_id: SW:4, sibling_flags 80000000, crypto map: GETVPN-MAP
        sa timing: remaining key lifetime (sec): (6373)
        Kilobyte Volume Rekey has been disabled
        IV size: 16 bytes
        replay detection support: Y  replay window size: 5
        Status: ACTIVE(ACTIVE)

     outbound ah sas:

     outbound pcp sas:

!! -- Send traffic between Site-1 and Site-3 LAN -- !!

R8-CE-GM#ping 192.168.6.6 so 192.168.8.8
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.6.6, timeout is 2 seconds:
Packet sent with a source address of 192.168.8.8 
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 168/188/224 ms

!! -- Send traffic between Site-2 and Site-3 LAN -- !!

R8-CE-GM#ping 192.168.7.7 so 192.168.8.8
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.7.7, timeout is 2 seconds:
Packet sent with a source address of 192.168.8.8 
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 176/194/212 ms

!! -- Traffic between each site LAN is ecnrypted now -- !!
!! -- GM has downloaded the ACL from KS, which covers LAN subnets --!!
!! -- of all sites -- !!

R8-CE-GM#sh cry ipsec sa                

interface: FastEthernet0/0
    Crypto map tag: GETVPN-MAP, local addr 172.16.38.8

   protected vrf: (none)
   local  ident (addr/mask/prot/port): (192.168.0.0/255.255.0.0/0/0)
   remote ident (addr/mask/prot/port): (192.168.0.0/255.255.0.0/0/0)
   current_peer 0.0.0.0 port 848
     PERMIT, flags={}
    #pkts encaps: 15, #pkts encrypt: 15, #pkts digest: 15
    #pkts decaps: 15, #pkts decrypt: 15, #pkts verify: 15
    #pkts compressed: 0, #pkts decompressed: 0
    #pkts not compressed: 0, #pkts compr. failed: 0
    #pkts not decompressed: 0, #pkts decompress failed: 0
    #send errors 0, #recv errors 0

     local crypto endpt.: 172.16.38.8, remote crypto endpt.: 0.0.0.0
     path mtu 1500, ip mtu 1500, ip mtu idb FastEthernet0/0
     current outbound spi: 0xBB9CF351(3147625297)
     PFS (Y/N): N, DH group: none

     inbound esp sas:
      spi: 0xBB9CF351(3147625297)
        transform: esp-256-aes esp-sha256-hmac ,
        in use settings ={Transport, }
        conn id: 3, flow_id: SW:3, sibling_flags 80000000, crypto map: GETVPN-MAP
        sa timing: remaining key lifetime (sec): (6341)
        Kilobyte Volume Rekey has been disabled
        IV size: 16 bytes
        replay detection support: Y  replay window size: 5
        Status: ACTIVE(ACTIVE)

     inbound ah sas:

     inbound pcp sas:

     outbound esp sas:
      spi: 0xBB9CF351(3147625297)
        transform: esp-256-aes esp-sha256-hmac ,
        in use settings ={Transport, }
        conn id: 4, flow_id: SW:4, sibling_flags 80000000, crypto map: GETVPN-MAP
        sa timing: remaining key lifetime (sec): (6341)
        Kilobyte Volume Rekey has been disabled
        IV size: 16 bytes
        replay detection support: Y  replay window size: 5
        Status: ACTIVE(ACTIVE)

     outbound ah sas:

     outbound pcp sas:

Reference:

http://www.cisco.com/c/dam/en/us/products/collateral/security/group-encrypted-transport-vpn/GETVPN_DIG_version_1_0_External.pdf

 

IPv6 over MPLS – 6PE/6VPE

IPv6 over MPLS

We have a standard MPLS service provider network running with IPv4 IGP, LDP and MP-BGP. Now we want to connect customer’s IPv6 sites over MPLS service provider network. One option is to run IPv6 IGP, LDPv6 (LDP for IPv6 routes) and MP-BGP in service provider network. Currently LDPv6 is in draft stage.

Another approach is to maintain the MPLS network as it stands, but implement mechanisms on the PE routers that allow to transport IPv6 packets as normal labeled packets on the P routers. This is exactly what the 6PE and 6VPE solutions do.

Both 6PE and 6VPE exploit the fact that as long as a packet somehow can be forwarded along an LSP from ingress to egress PE, P routers do not care about anything but the transport label. When using a BGP route in an IPv4 MPLS VPN (or just IPv4 over MPLS), the top label is found by looking at the BGP next hop of the route. The ingress looks at this IPv4 next hop, finds the label associated with it, and by using this label, the packet will be forwarded to the egress PE.

The difference between 6PE and 6VPE is whether the IPv6 routes are in the global routing table or in VRFs. 6PE serves the same role as plain IPv4 over MPLS, and 6VPE is the equivalent of an MPLS VPN.

6PE Design
  • 6PE is defined in RFC4798
  • Service provider network is with IPv4 IGP, LDP and MP-BGP
  • Service provider core (P Routers) is BGP free and not running IPv6
  • PE routers are dual stack, running both IPv4 and IPv6
  • PE routers stores IPv6 routing information in global routing table from all IPv6 customers
6VPE Design
  • 6VPE is defined in RFC4659
  • Service provider network is with IPv4 IGP, LDP and MP-BGP
  • Service provider core is BGP free and not running IPv4
  • PE routers are dual stack, running both IPv4 and IPv6
  • PE routers stores IPv6 routing information in separate VRF for each IPv6 customer, providing MPLS VPN for IPv6

We will use following network topology to configure 6PE and 6VPE.

6PE Configuration
When using 6PE, a CE router is connected to an interface on the PE router, which is in the global IPv6 routing table. There is an MPLS network with BGP, an IGP and LDP between PE routers. The IGP and LDP only needs to be IPv4 capable. However, the BGP session need to be activated for the IPv6 unicast address family to make it possible to advertise IPv6 prefixes from PE to PE. Additionally, BGP needs to attach a label to each IPv6 prefix with the ‘send label’ command. 

R1 - PE1 Router

hostname R1
!
ip cef
ipv6 unicast-routing
ipv6 cef
!
interface Loopback0
 ip address 10.1.0.1 255.255.255.255
 ip ospf 1 area 0
!
interface FastEthernet0/0
 ip address 10.1.13.1 255.255.255.0
 ip ospf network point-to-point
 ip ospf 1 area 0
 mpls ip
!
interface FastEthernet1/0
 no ip address
 ipv6 address 2001:192:168:14::1/64
!
interface FastEthernet1/1
 no ip address
 ipv6 address 2001:172:16:15::1/64
!
router ospf 1
 router-id 10.1.0.1
!
router bgp 100
 bgp router-id 10.1.0.1
 bgp log-neighbor-changes
 no bgp default ipv4-unicast
 neighbor 10.1.0.2 remote-as 100
 neighbor 10.1.0.2 update-source Loopback0
 neighbor 2001:172:16:15::5 remote-as 65005
 neighbor 2001:192:168:14::4 remote-as 65004
 !
 address-family ipv4
 exit-address-family
 !
 address-family ipv6
  redistribute connected
  neighbor 10.1.0.2 activate
  neighbor 10.1.0.2 send-label
  neighbor 2001:172:16:15::5 activate
  neighbor 2001:172:16:15::5 route-map CUST-XYZ out
  neighbor 2001:192:168:14::4 activate
  neighbor 2001:192:168:14::4 route-map CUST-ABC out
 exit-address-family
!
ipv6 prefix-list CUST-ABC seq 5 permit 2001:192::/32 le 64
!
ipv6 prefix-list CUST-XYZ seq 5 permit 2001:172::/32 le 64
!
route-map CUST-XYZ permit 10
 match ipv6 address prefix-list CUST-XYZ
!
route-map CUST-ABC permit 10
 match ipv6 address prefix-list CUST-ABC
!

R2 - PE2 Router

hostname R2
!
no ip domain lookup
ip cef
ipv6 unicast-routing
ipv6 cef
!
interface Loopback0
 ip address 10.1.0.2 255.255.255.255
 ip ospf 1 area 0
!
interface FastEthernet0/0
 ip address 10.1.23.2 255.255.255.0
 ip ospf network point-to-point
 ip ospf 1 area 0
 mpls ip
!
interface FastEthernet1/0
 no ip address
 ipv6 address 2001:192:168:26::2/64
!
interface FastEthernet1/1
 no ip address
 ipv6 address 2001:172:16:27::2/64
!
router ospf 1
 router-id 10.1.0.2
!
router bgp 100
 bgp router-id 10.1.0.2
 bgp log-neighbor-changes
 no bgp default ipv4-unicast
 neighbor 10.1.0.1 remote-as 100
 neighbor 10.1.0.1 update-source Loopback0
 neighbor 2001:172:16:27::7 remote-as 65007
 neighbor 2001:192:168:26::6 remote-as 65006
 !
 address-family ipv4
 exit-address-family
 !
 address-family ipv6
  redistribute connected
  neighbor 10.1.0.1 activate
  neighbor 10.1.0.1 send-label
  neighbor 2001:172:16:27::7 activate
  neighbor 2001:172:16:27::7 route-map CUST-XYZ out
  neighbor 2001:192:168:26::6 activate
  neighbor 2001:192:168:26::6 route-map CUST-ABC out
 exit-address-family
!
ipv6 prefix-list CUST-ABC seq 5 permit 2001:192::/32 le 64
!
ipv6 prefix-list CUST-XYZ seq 5 permit 2001:172::/32 le 64
!
route-map CUST-XYZ permit 10
 match ipv6 address prefix-list CUST-XYZ
!
route-map CUST-ABC permit 10
 match ipv6 address prefix-list CUST-ABC
!

R3 - P Router

hostname R3
!
ip cef
!
interface Loopback0
 ip address 10.1.0.3 255.255.255.255
 ip ospf 1 area 0
!
interface FastEthernet0/0
 ip address 10.1.13.3 255.255.255.0
 ip ospf network point-to-point
 ip ospf 1 area 0
 mpls ip
!
interface FastEthernet0/1
 ip address 10.1.23.3 255.255.255.0
 ip ospf network point-to-point
 ip ospf 1 area 0
 mpls ip
!
router ospf 1
 router-id 10.1.0.3
!

R4 - Customer ABC Site 1

hostname R4
!
ipv6 unicast-routing
ipv6 cef
!
interface FastEthernet0/0
 no ip address
 ipv6 address 2001:192:168:14::4/64
!
interface FastEthernet0/1
 no ip address
 ipv6 address 2001:192:168:4::4/64
!
router bgp 65004
 bgp router-id 4.4.4.4
 bgp log-neighbor-changes
 no bgp default ipv4-unicast
 neighbor 2001:192:168:14::1 remote-as 100
 !
 address-family ipv4
 exit-address-family
 !
 address-family ipv6
  network 2001:192:168:4::/64
  neighbor 2001:192:168:14::1 activate
 exit-address-family
!

R5 - Customer XYZ Site 1

hostname R5
!
ipv6 unicast-routing
ipv6 cef
!
interface FastEthernet0/0
 no ip address
 ipv6 address 2001:172:16:15::5/64
!
interface FastEthernet0/1
 no ip address
 ipv6 address 2001:172:16:5::5/64
!
router bgp 65005
 bgp router-id 5.5.5.5
 bgp log-neighbor-changes
 no bgp default ipv4-unicast
 neighbor 2001:172:16:15::1 remote-as 100
 !
 address-family ipv4
 exit-address-family
 !
 address-family ipv6
  network 2001:172:16:5::/64
  neighbor 2001:172:16:15::1 activate
 exit-address-family
!

R6 - Customer ABC Site 2

hostname R6
!
ipv6 unicast-routing
ipv6 cef
!
interface FastEthernet0/0
 no ip address
 ipv6 address 2001:192:168:26::6/64
!
interface FastEthernet0/1
 no ip address
 ipv6 address 2001:192:168:6::6/64
!
router bgp 65006
 bgp router-id 6.6.6.6
 bgp log-neighbor-changes
 no bgp default ipv4-unicast
 neighbor 2001:192:168:26::2 remote-as 100
 !
 address-family ipv4
 exit-address-family
 !
 address-family ipv6
  network 2001:192:168:6::/64
  neighbor 2001:192:168:26::2 activate
 exit-address-family
!

R7 - Customer XYZ Site 2

hostname R7
!
ipv6 unicast-routing
ipv6 cef
!
interface FastEthernet0/0
 no ip address
 ipv6 address 2001:172:16:27::7/64
!
interface FastEthernet0/1
 no ip address
 ipv6 address 2001:172:16:7::7/64
!
router bgp 65007
 bgp router-id 7.7.7.7
 bgp log-neighbor-changes
 no bgp default ipv4-unicast
 neighbor 2001:172:16:27::2 remote-as 100
 !
 address-family ipv4
 exit-address-family
 !
 address-family ipv6
  network 2001:172:16:7::/64
  neighbor 2001:172:16:27::2 activate
 exit-address-family
!
Verification
PE Router BGP Peering

!! -- PE1 (R1) has IPv6 BGP+Label peering with PE2 (R2) -- !!
!! -- PE1 (R1) has IPv6 BGP peering with CE4 (R4) -- !!
!! -- PE1 (R1) has IPv6 BGP peering with CE5 (R5) -- !!

R1#show bgp ipv6 unicast summary 
BGP router identifier 10.1.0.1, local AS number 100
BGP table version is 9, main routing table version 9
8 network entries using 1376 bytes of memory
8 path entries using 704 bytes of memory
6/6 BGP path/bestpath attribute entries using 816 bytes of memory
4 BGP AS-PATH entries using 96 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 2992 total bytes of memory
BGP activity 8/0 prefixes, 8/0 paths, scan interval 60 secs

Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
10.1.0.2        4          100     179     181        9    0    0 02:21:48        4
2001:172:16:15::5
                4        65005     161     178        9    0    0 02:22:05        1
2001:192:168:14::4
                4        65004     163     176        9    0    0 02:22:03        1

The PE1 router will receive IPv6 routes with a label and a next hop of ::FFFF:10.1.0.2. This next hop is an “IPv4-mapped IPv6 address” and it tells the router that if it were to use that IPv6 BGP route, it should use the transport label for the IPv4 address 10.1.0.2.

Similarly, PE2 router will receive IPv6 routes with a label and a next hop of ::FFFF:10.1.0.1.

R1#show bgp ipv6 unicast
BGP table version is 9, local router ID is 10.1.0.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, 
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, 
              x best-external, a additional-path, c RIB-compressed, 
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *>  2001:172:16:5::/64
                       2001:172:16:15::5
                                                0             0 65005 i
 *>i 2001:172:16:7::/64
                       ::FFFF:10.1.0.2          0    100      0 65007 i
 *>  2001:172:16:15::/64
                       ::                       0         32768 ?
 *>i 2001:172:16:27::/64
                       ::FFFF:10.1.0.2          0    100      0 ?
 *>  2001:192:168:4::/64
                       2001:192:168:14::4
                                                0             0 65004 i
 *>i 2001:192:168:6::/64
                       ::FFFF:10.1.0.2          0    100      0 65006 i
 *>  2001:192:168:14::/64
                       ::                       0         32768 ?
 *>i 2001:192:168:26::/64
                       ::FFFF:10.1.0.2          0    100      0 ?

R2#show bgp ipv6 unicast summary
BGP router identifier 10.1.0.2, local AS number 100
BGP table version is 9, main routing table version 9
8 network entries using 1376 bytes of memory
8 path entries using 704 bytes of memory
6/6 BGP path/bestpath attribute entries using 816 bytes of memory
4 BGP AS-PATH entries using 96 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 2992 total bytes of memory
BGP activity 8/0 prefixes, 8/0 paths, scan interval 60 secs

Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
10.1.0.1        4          100     183     181        9    0    0 02:23:38        4
2001:172:16:27::7
                4        65007     165     176        9    0    0 02:23:51        1
2001:192:168:26::6
                4        65006     163     175        9    0    0 02:23:50        1

R2#show bgp ipv6 unicast        
BGP table version is 9, local router ID is 10.1.0.2
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, 
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, 
              x best-external, a additional-path, c RIB-compressed, 
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *>i 2001:172:16:5::/64
                       ::FFFF:10.1.0.1          0    100      0 65005 i
 *>  2001:172:16:7::/64
                       2001:172:16:27::7
                                                0             0 65007 i
 *>i 2001:172:16:15::/64
                       ::FFFF:10.1.0.1          0    100      0 ?
 *>  2001:172:16:27::/64
                       ::                       0         32768 ?
 *>i 2001:192:168:4::/64
                       ::FFFF:10.1.0.1          0    100      0 65004 i
 *>  2001:192:168:6::/64
                       2001:192:168:26::6
                                                0             0 65006 i
 *>i 2001:192:168:14::/64
                       ::FFFF:10.1.0.1          0    100      0 ?
 *>  2001:192:168:26::/64
                       ::                       0         32768 ?

!! -- Verify BGP Label Assignments -- !!

R1#show bgp ipv6 unicast labels 
   Network          Next Hop      In label/Out label
   2001:172:16:5::/64
                    2001:172:16:15::5
                                    19/nolabel
   2001:172:16:7::/64
                    ::FFFF:10.1.0.2 nolabel/19
   2001:172:16:15::/64
                    ::              20/nolabel
   2001:172:16:27::/64
                    ::FFFF:10.1.0.2 nolabel/20
   2001:192:168:4::/64
                    2001:192:168:14::4
                                    22/nolabel
   2001:192:168:6::/64
                    ::FFFF:10.1.0.2 nolabel/22
   2001:192:168:14::/64
                    ::              21/nolabel
   2001:192:168:26::/64
                    ::FFFF:10.1.0.2 nolabel/21

R2#show bgp ipv6 unicast labels
   Network          Next Hop      In label/Out label
   2001:172:16:5::/64
                    ::FFFF:10.1.0.1 nolabel/19
   2001:172:16:7::/64
                    2001:172:16:27::7
                                    19/nolabel
   2001:172:16:15::/64
                    ::FFFF:10.1.0.1 nolabel/20
   2001:172:16:27::/64
                    ::              20/nolabel
   2001:192:168:4::/64
                    ::FFFF:10.1.0.1 nolabel/22
   2001:192:168:6::/64
                    2001:192:168:26::6
                                    22/nolabel
   2001:192:168:14::/64
                    ::FFFF:10.1.0.1 nolabel/21
   2001:192:168:26::/64
                    ::              21/nolabel

!! -- Customer ABC Site-2 LAN IPv6 network reachability from PE1-R1 via MPLS path -- !!

R1#show bgp ipv6 unicast 2001:192:168:6::/64
BGP routing table entry for 2001:192:168:6::/64, version 9
Paths: (1 available, best #1, table default)
  Advertised to update-groups:
     3         
  Refresh Epoch 5
  65006
    ::FFFF:10.1.0.2 (metric 3) from 10.1.0.2 (10.1.0.2)
      Origin IGP, metric 0, localpref 100, valid, internal, best
      mpls labels in/out nolabel/22
R1#sh ipv6 cef 2001:192:168:6::/64 detail 
2001:192:168:6::/64, epoch 0, flags rib defined all labels
  recursive via 10.1.0.2 label 22
    nexthop 10.1.13.3 FastEthernet0/0 label 16

!! -- Ping/Traceroute from R4 Customer ABC Site-1 to Site-2 -- !!
!! -- via MPLS provider network -- !!

R4#show ipv6 route
IPv6 Routing Table - default - 7 entries
Codes: C - Connected, L - Local, S - Static, U - Per-user Static route
       B - BGP, HA - Home Agent, MR - Mobile Router, R - RIP
       H - NHRP, I1 - ISIS L1, I2 - ISIS L2, IA - ISIS interarea
       IS - ISIS summary, D - EIGRP, EX - EIGRP external, NM - NEMO
       ND - ND Default, NDp - ND Prefix, DCE - Destination, NDr - Redirect
       O - OSPF Intra, OI - OSPF Inter, OE1 - OSPF ext 1, OE2 - OSPF ext 2
       ON1 - OSPF NSSA ext 1, ON2 - OSPF NSSA ext 2, l - LISP
C   2001:192:168:4::/64 [0/0]
     via FastEthernet0/1, directly connected
L   2001:192:168:4::4/128 [0/0]
     via FastEthernet0/1, receive
B   2001:192:168:6::/64 [20/0]
     via FE80::C803:31FF:FE5C:1C, FastEthernet0/0
C   2001:192:168:14::/64 [0/0]
     via FastEthernet0/0, directly connected
L   2001:192:168:14::4/128 [0/0]
     via FastEthernet0/0, receive
B   2001:192:168:26::/64 [20/0]
     via FE80::C803:31FF:FE5C:1C, FastEthernet0/0
L   FF00::/8 [0/0]
     via Null0, receive


R4#ping 2001:192:168:6::6 source 2001:192:168:4::4
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 2001:192:168:6::6, timeout is 2 seconds:
Packet sent with a source address of 2001:192:168:4::4
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 76/93/104 ms

R4#traceroute 
Protocol [ip]: ipv6
Target IPv6 address: 2001:192:168:6::6
Source address: 2001:192:168:4::4
Insert source routing header? [no]: 
Numeric display? [no]: 
Timeout in seconds [3]: 
Probe count [3]: 
Minimum Time to Live [1]: 
Maximum Time to Live [30]: 
Priority [0]: 
Port Number [0]: 
Type escape sequence to abort.
Tracing the route to 2001:192:168:6::6

  1 2001:192:168:14::1 [AS 100] 72 msec 44 msec 24 msec
  2 ::FFFF:10.1.13.3 [MPLS: Labels 16/22 Exp 0] 92 msec 88 msec 88 msec
  3 2001:192:168:26::2 [AS 100] [MPLS: Label 22 Exp 0] 76 msec 56 msec 68 msec
  4 2001:192:168:26::6 [AS 100] 92 msec 96 msec 88 msec


6VPE Configuration
!! -- Only PE routers configuration will be changed -- !!
!! -- CE facing interface on PE router will be in a VRF -- !!

R1 - PE1 Router

hostname R1
!
vrf definition CUST-ABC
 rd 100:1
 !
 address-family ipv6
  route-target export 100:1
  route-target import 100:1
 exit-address-family
!
vrf definition CUST-XYZ
 rd 100:2
 !
 address-family ipv6
  route-target export 100:2
  route-target import 100:2
 exit-address-family
!
ip cef
ipv6 unicast-routing
ipv6 cef
!
interface Loopback0
 ip address 10.1.0.1 255.255.255.255
 ip ospf 1 area 0
!
interface FastEthernet0/0
 ip address 10.1.13.1 255.255.255.0
 ip ospf network point-to-point
 ip ospf 1 area 0
 mpls ip
!
interface FastEthernet1/0
 vrf forwarding CUST-ABC
 no ip address
 ipv6 address 2001:192:168:14::1/64
!
interface FastEthernet1/1
 vrf forwarding CUST-XYZ
 no ip address
 ipv6 address 2001:172:16:15::1/64
!
router ospf 1
 router-id 10.1.0.1
!
router bgp 100
 bgp router-id 10.1.0.1
 bgp log-neighbor-changes
 no bgp default ipv4-unicast
 neighbor 10.1.0.2 remote-as 100
 neighbor 10.1.0.2 update-source Loopback0
 !
 address-family ipv4
 exit-address-family
 !
 address-family vpnv6
  neighbor 10.1.0.2 activate
  neighbor 10.1.0.2 send-community extended
 exit-address-family
 !
 address-family ipv6 vrf CUST-ABC
  redistribute connected
  neighbor 2001:192:168:14::4 remote-as 65004
  neighbor 2001:192:168:14::4 activate
 exit-address-family
 !
 address-family ipv6 vrf CUST-XYZ
  redistribute connected
  neighbor 2001:172:16:15::5 remote-as 65005
  neighbor 2001:172:16:15::5 activate
 exit-address-family
!

R2 - PE2 Router

hostname R2
!
vrf definition CUST-ABC
 rd 100:1
 !
 address-family ipv6
  route-target export 100:1
  route-target import 100:1
 exit-address-family
!
vrf definition CUST-XYZ
 rd 100:2
 !
 address-family ipv6
  route-target export 100:2
  route-target import 100:2
 exit-address-family
!
ip cef
ipv6 unicast-routing
ipv6 cef
!
interface Loopback0
 ip address 10.1.0.2 255.255.255.255
 ip ospf 1 area 0
!
interface FastEthernet0/0
 ip address 10.1.23.2 255.255.255.0
 ip ospf network point-to-point
 ip ospf 1 area 0
 mpls ip
!
interface FastEthernet1/0
 vrf forwarding CUST-ABC
 no ip address
 ipv6 address 2001:192:168:26::2/64
!
interface FastEthernet1/1
 vrf forwarding CUST-XYZ
 no ip address
 ipv6 address 2001:172:16:27::2/64
!
router ospf 1
 router-id 10.1.0.2
!
router bgp 100
 bgp router-id 10.1.0.2
 bgp log-neighbor-changes
 no bgp default ipv4-unicast
 neighbor 10.1.0.1 remote-as 100
 neighbor 10.1.0.1 update-source Loopback0
 !
 address-family ipv4
 exit-address-family
 !
 address-family vpnv6
  neighbor 10.1.0.1 activate
  neighbor 10.1.0.1 send-community extended
 exit-address-family
 !
 address-family ipv6 vrf CUST-ABC
  redistribute connected
  neighbor 2001:192:168:26::6 remote-as 65006
  neighbor 2001:192:168:26::6 activate
 exit-address-family
 !
 address-family ipv6 vrf CUST-XYZ
  redistribute connected
  neighbor 2001:172:16:27::7 remote-as 65007
  neighbor 2001:172:16:27::7 activate
 exit-address-family
!
Verification
!! -- VRF status on PE Routers -- !!

R1#show vrf 
  Name                             Default RD          Protocols   Interfaces
  CUST-ABC                         100:1               ipv6        Fa1/0
  CUST-XYZ                         100:2               ipv6        Fa1/1

R2#show vrf 
  Name                             Default RD          Protocols   Interfaces
  CUST-ABC                         100:1               ipv6        Fa1/0
  CUST-XYZ                         100:2               ipv6        Fa1/1

!! -- BGP Peering on PE Routers -- !!
R1#show bgp vpnv6 unicast all summary 
BGP router identifier 10.1.0.1, local AS number 100
BGP table version is 13, main routing table version 13
8 network entries using 1536 bytes of memory
8 path entries using 736 bytes of memory
10/8 BGP path/bestpath attribute entries using 1440 bytes of memory
4 BGP AS-PATH entries using 96 bytes of memory
2 BGP extended community entries using 48 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 3856 total bytes of memory
BGP activity 16/8 prefixes, 16/8 paths, scan interval 60 secs

Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
10.1.0.2        4          100      28      29       13    0    0 00:19:42        4
2001:172:16:15::5
                4        65005      29      30       13    0    0 00:22:35        1
2001:192:168:14::4
                4        65004      30      32       13    0    0 00:23:04        1

R2#show bgp vpnv6 unicast all summary 
BGP router identifier 10.1.0.2, local AS number 100
BGP table version is 13, main routing table version 13
8 network entries using 1536 bytes of memory
8 path entries using 736 bytes of memory
10/8 BGP path/bestpath attribute entries using 1440 bytes of memory
4 BGP AS-PATH entries using 96 bytes of memory
2 BGP extended community entries using 48 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 3856 total bytes of memory
BGP activity 8/0 prefixes, 8/0 paths, scan interval 60 secs

Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
10.1.0.1        4          100      30      28       13    0    0 00:19:57        4
2001:172:16:27::7
                4        65007      25      28       13    0    0 00:18:58        1
2001:192:168:26::6
                4        65006      26      27       13    0    0 00:19:23        1

!! -- BGP Table on PE Routers -- !!

R1#show bgp vpnv6 unicast all
BGP table version is 13, local router ID is 10.1.0.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, 
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, 
              x best-external, a additional-path, c RIB-compressed, 
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
Route Distinguisher: 100:1 (default for vrf CUST-ABC)
 *>  2001:192:168:4::/64
                       2001:192:168:14::4
                                                0             0 65004 i
 *>i 2001:192:168:6::/64
                       ::FFFF:10.1.0.2          0    100      0 65006 i
 *>  2001:192:168:14::/64
                       ::                       0         32768 ?
 *>i 2001:192:168:26::/64
                       ::FFFF:10.1.0.2          0    100      0 ?
Route Distinguisher: 100:2 (default for vrf CUST-XYZ)
 *>  2001:172:16:5::/64
                       2001:172:16:15::5
                                                0             0 65005 i
 *>i 2001:172:16:7::/64
                       ::FFFF:10.1.0.2          0    100      0 65007 i
 *>  2001:172:16:15::/64
                       ::                       0         32768 ?
 *>i 2001:172:16:27::/64
                       ::FFFF:10.1.0.2          0    100      0 ?

R2#show bgp vpnv6 unicast all
BGP table version is 13, local router ID is 10.1.0.2
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, 
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, 
              x best-external, a additional-path, c RIB-compressed, 
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
Route Distinguisher: 100:1 (default for vrf CUST-ABC)
 *>i 2001:192:168:4::/64
                       ::FFFF:10.1.0.1          0    100      0 65004 i
 *>  2001:192:168:6::/64
                       2001:192:168:26::6
                                                0             0 65006 i
 *>i 2001:192:168:14::/64
                       ::FFFF:10.1.0.1          0    100      0 ?
 *>  2001:192:168:26::/64
                       ::                       0         32768 ?
Route Distinguisher: 100:2 (default for vrf CUST-XYZ)
 *>i 2001:172:16:5::/64
                       ::FFFF:10.1.0.1          0    100      0 65005 i
 *>  2001:172:16:7::/64
                       2001:172:16:27::7
                                                0             0 65007 i
 *>i 2001:172:16:15::/64
                       ::FFFF:10.1.0.1          0    100      0 ?
 *>  2001:172:16:27::/64
                       ::                       0         32768 ?

!! -- Customer IPv6 routes are not in Global RIB -- !!

R1#show ipv6 route
IPv6 Routing Table - default - 1 entries
Codes: C - Connected, L - Local, S - Static, U - Per-user Static route
       B - BGP, HA - Home Agent, MR - Mobile Router, R - RIP
       H - NHRP, I1 - ISIS L1, I2 - ISIS L2, IA - ISIS interarea
       IS - ISIS summary, D - EIGRP, EX - EIGRP external, NM - NEMO
       ND - ND Default, NDp - ND Prefix, DCE - Destination, NDr - Redirect
       O - OSPF Intra, OI - OSPF Inter, OE1 - OSPF ext 1, OE2 - OSPF ext 2
       ON1 - OSPF NSSA ext 1, ON2 - OSPF NSSA ext 2, l - LISP
L   FF00::/8 [0/0]
     via Null0, receive

R2#show ipv6 route
IPv6 Routing Table - default - 1 entries
Codes: C - Connected, L - Local, S - Static, U - Per-user Static route
       B - BGP, HA - Home Agent, MR - Mobile Router, R - RIP
       H - NHRP, I1 - ISIS L1, I2 - ISIS L2, IA - ISIS interarea
       IS - ISIS summary, D - EIGRP, EX - EIGRP external, NM - NEMO
       ND - ND Default, NDp - ND Prefix, DCE - Destination, NDr - Redirect
       O - OSPF Intra, OI - OSPF Inter, OE1 - OSPF ext 1, OE2 - OSPF ext 2
       ON1 - OSPF NSSA ext 1, ON2 - OSPF NSSA ext 2, l - LISP
L   FF00::/8 [0/0]
     via Null0, receive

!! -- MPLS VPN Labels on PE Routers -- !!

R1#sh mpls forwarding-table vrf CUST-ABC
Local      Outgoing   Prefix           Bytes Label   Outgoing   Next Hop    
Label      Label      or Tunnel Id     Switched      interface              
23         No Label   2001:192:168:14::/64[V]   \
                                       0             aggregate/CUST-ABC 
24         No Label   2001:192:168:4::/64[V]   \
                                       1922          Fa1/0      FE80::C805:19FF:FEAC:8

R1#sh mpls forwarding-table vrf CUST-XYZ
Local      Outgoing   Prefix           Bytes Label   Outgoing   Next Hop    
Label      Label      or Tunnel Id     Switched      interface              
25         No Label   2001:172:16:15::/64[V]   \
                                       0             aggregate/CUST-XYZ 
26         No Label   2001:172:16:5::/64[V]   \
                                       0             Fa1/1      FE80::C800:17FF:FE58:8

R2#sh mpls forwarding-table vrf CUST-ABC
Local      Outgoing   Prefix           Bytes Label   Outgoing   Next Hop    
Label      Label      or Tunnel Id     Switched      interface              
19         No Label   2001:192:168:26::/64[V]   \
                                       0             aggregate/CUST-ABC 
20         No Label   2001:192:168:6::/64[V]   \
                                       1374          Fa1/0      FE80::C801:17FF:FE58:8
  
R2#sh mpls forwarding-table vrf CUST-XYZ
Local      Outgoing   Prefix           Bytes Label   Outgoing   Next Hop    
Label      Label      or Tunnel Id     Switched      interface              
21         No Label   2001:172:16:7::/64[V]   \
                                       0             Fa1/1      FE80::C802:2BFF:FEF0:8
22         No Label   2001:172:16:27::/64[V]   \
                                       0             aggregate/CUST-XYZ 

!! -- Customer ABC Site-2 LAN IPv6 network reachability from PE1 -- !!

R1#show bgp vpnv6 unicast all 2001:192:168:6::/64        
BGP routing table entry for [100:1]2001:192:168:6::/64, version 7
Paths: (1 available, best #1, table CUST-ABC)
  Advertised to update-groups:
     1         
  Refresh Epoch 1
  65006
    ::FFFF:10.1.0.2 (metric 3) from 10.1.0.2 (10.1.0.2)
      Origin IGP, metric 0, localpref 100, valid, internal, best
      Extended Community: RT:100:1
      mpls labels in/out nolabel/20   <<<<< VPN Label

R1#show ipv6 cef vrf CUST-ABC 2001:192:168:6::/64 detail 
2001:192:168:6::/64, epoch 0, flags rib defined all labels
  recursive via 10.1.0.2 label 20    
    nexthop 10.1.13.3 FastEthernet0/0 label 16   <<<<< Transport Label

!! -- Ping/Traceroute from Customer ABC Site 1 to Site 2 LAN --!!

R4#ping 2001:192:168:6::6 source 2001:192:168:4::4
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 2001:192:168:6::6, timeout is 2 seconds:
Packet sent with a source address of 2001:192:168:4::4
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 112/129/136 ms

R4#traceroute 
Protocol [ip]: ipv6
Target IPv6 address: 2001:192:168:6::6
Source address: 2001:192:168:4::4
Insert source routing header? [no]: 
Numeric display? [no]: 
Timeout in seconds [3]: 
Probe count [3]: 
Minimum Time to Live [1]: 
Maximum Time to Live [30]: 
Priority [0]: 
Port Number [0]: 
Type escape sequence to abort.
Tracing the route to 2001:192:168:6::6

  1 2001:192:168:14::1 [AS 100] 44 msec 40 msec 36 msec
  2 ::FFFF:10.1.13.3 [MPLS: Labels 16/20 Exp 0] 120 msec 136 msec 132 msec
  3  * 
    2001:192:168:26::2 108 msec 104 msec
  4 2001:192:168:26::6 [AS 100] 136 msec 128 msec 132 msec

*VPN Label = 20
*Transport Label = 16 

 

DMVPN Phase 3

DMVPN Phase 3

The Shortcut Switching Enhancements for NHRP in DMVPN Phase 3 provides a more scalable alternative to the previous NHRP model. Routers in a Dynamic Multipoint VPN (DMVPN) Phase 3 network use Next Hop Resolution Protocol (NHRP) Shortcut Switching to discover shorter paths to a destination network after receiving an NHRP redirect message from the hub. This allows the routers to communicate directly with each other without the need for an intermediate hop.

Benefits of DMVPN Phase 3 Design:

  • Allows summarization of routing protocol updates from hub to spokes. The spokes no longer need to have an individual route with an IP next-hop of the tunnel IP address of the remote spoke for the networks behind all the other spokes. The spokes can use summarized routes or specific routes with an IP next-hop of the tunnel IP address of the hub and still be able to build spoke-to-spoke tunnels.
  • Allows for hierarchical (greater than one level) and more complex tree-based DMVPN network topologies. Tree-based topologies allow the capability to build DMVPN networks with regional hubs that are spokes of central hubs. This architecture
    allows the regional hub to handle the data and NHRP control traffic for its regional spokes, but still allows spoke-to-spoke tunnels to be built between any spokes within the DMVPN network, whether they are in the same region or not. This architecture also allows the DMVPN network layout to more closely match regional or hierarchical data flow patterns.

Check these links for

Here is the network topology for DMVPN phase 3 discussion and configuration.

The DMVPN Phase 2 design requires to maintain full routing table on all spoke routers.  Each route for remote spoke networks needs to be a specific route with the next hop pointing to the remote spoke’s tunnel address. This prevents the hub from being able to send down a summarized route to the spokes for a more concise routing table.

Phase 3 overcomes this restriction using NHRP traffic indication messages from the hub to signal to the spokes that a better path exists to reach the target network. This functionality is enabled by configuring ip nhrp redirect on the hub and ip nhrp shortcut on the spokes. The redirect command tells the hub to send the NHRP traffic indication message while the shortcut command tells the spokes to accept the redirect and install the shortcut route.

DMVPN Phase 3 configuration with EIGRP
Configuration:
----------
R1 - Hub:
----------

hostname R1
!
ip cef
!
crypto isakmp policy 10
 encr aes
 authentication pre-share
 group 2
crypto isakmp key DmVpNpR3$h@r3dK3Y address 0.0.0.0        
!
!
crypto ipsec transform-set TRANSFORM-SET esp-aes esp-sha-hmac 
 mode transport
!
crypto ipsec profile PROTECT-GRE
 set transform-set TRANSFORM-SET 
!
interface Tunnel0
 bandwidth 4096
 ip address 192.168.1.1 255.255.255.0
 no ip redirects
 ip mtu 1400
 ip nhrp authentication NhRp@UtH
 ip nhrp map multicast dynamic
 ip nhrp network-id 100
 ip nhrp redirect
 ip summary-address eigrp 100 10.10.0.0 255.255.0.0
 ip tcp adjust-mss 1360
 tunnel source FastEthernet0/0
 tunnel mode gre multipoint
 tunnel key 123
 tunnel protection ipsec profile PROTECT-GRE
!
interface FastEthernet0/0
 ip address 1.1.14.1 255.255.255.0
!
interface FastEthernet0/1
 ip address 10.10.1.1 255.255.255.0
!
router eigrp 100
 network 10.10.1.1 0.0.0.0
 network 192.168.1.1 0.0.0.0
!
ip route 0.0.0.0 0.0.0.0 1.1.14.4

------------
R2 - Spoke:
------------

hostname R2
!
ip cef
!
crypto isakmp policy 10
 encr aes
 authentication pre-share
 group 2
crypto isakmp key DmVpNpR3$h@r3dK3Y address 0.0.0.0        
!
!
crypto ipsec transform-set TRANSFORM-SET esp-aes esp-sha-hmac 
 mode transport
!
crypto ipsec profile PROTECT-GRE
 set transform-set TRANSFORM-SET 
!
interface Tunnel0
 bandwidth 4096
 ip address 192.168.1.2 255.255.255.0
 no ip redirects
 ip mtu 1400
 ip nhrp authentication NhRp@UtH
 ip nhrp map multicast 1.1.14.1
 ip nhrp map 192.168.1.1 1.1.14.1
 ip nhrp network-id 100
 ip nhrp nhs 192.168.1.1
 ip nhrp shortcut
 ip tcp adjust-mss 1360
 tunnel source FastEthernet0/0
 tunnel mode gre multipoint
 tunnel key 123
 tunnel protection ipsec profile PROTECT-GRE
!
interface FastEthernet0/0
 ip address 1.1.24.2 255.255.255.0
!
interface FastEthernet0/1
 ip address 10.10.2.2 255.255.255.0
!
router eigrp 100
 network 10.10.2.2 0.0.0.0
 network 192.168.1.2 0.0.0.0
!
ip route 0.0.0.0 0.0.0.0 1.1.24.4

------------
R3 - Spoke:
------------

hostname R3
!
ip cef
!
crypto isakmp policy 10
 encr aes
 authentication pre-share
 group 2
crypto isakmp key DmVpNpR3$h@r3dK3Y address 0.0.0.0        
!
!
crypto ipsec transform-set TRANSFORM-SET esp-aes esp-sha-hmac 
 mode transport
!
crypto ipsec profile PROTECT-GRE
 set transform-set TRANSFORM-SET 
!
interface Tunnel0
 bandwidth 4096
 ip address 192.168.1.3 255.255.255.0
 no ip redirects
 ip mtu 1400
 ip nhrp authentication NhRp@UtH
 ip nhrp map multicast 1.1.14.1
 ip nhrp map 192.168.1.1 1.1.14.1
 ip nhrp network-id 100
 ip nhrp nhs 192.168.1.1
 ip nhrp shortcut
 ip tcp adjust-mss 1360
 tunnel source FastEthernet0/0
 tunnel mode gre multipoint
 tunnel key 123
 tunnel protection ipsec profile PROTECT-GRE
!
interface FastEthernet0/0
 ip address 1.1.34.3 255.255.255.0
!
interface FastEthernet0/1
 ip address 10.10.3.3 255.255.255.0
!
router eigrp 100
 network 10.10.3.3 0.0.0.0
 network 192.168.1.3 0.0.0.0
!
ip route 0.0.0.0 0.0.0.0 1.1.34.4

---------------
R4 - Internet:
---------------

hostname R4
!
interface FastEthernet0/0
 ip address 1.1.14.4 255.255.255.0
!
interface FastEthernet0/1
 ip address 1.1.24.4 255.255.255.0
!
interface FastEthernet1/0
 ip address 1.1.34.4 255.255.255.0
!
Verification:
- Hub and Spokes routers are using mGRE tunnels

R1#show int t0
Tunnel0 is up, line protocol is up 
  Hardware is Tunnel
  Internet address is 192.168.1.1/24
  MTU 17870 bytes, BW 4096 Kbit/sec, DLY 50000 usec, 
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation TUNNEL, loopback not set
  Keepalive not set
  Tunnel source 1.1.14.1 (FastEthernet0/0)
   Tunnel Subblocks:
      src-track:
         Tunnel0 source tracking subblock associated with FastEthernet0/0
          Set of tunnels with source FastEthernet0/0, 1 member (includes iterators), on interface <OK>
  Tunnel protocol/transport multi-GRE/IP

R2#show int t0
Tunnel0 is up, line protocol is up 
  Hardware is Tunnel
  Internet address is 192.168.1.2/24
  MTU 17870 bytes, BW 4096 Kbit/sec, DLY 50000 usec, 
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation TUNNEL, loopback not set
  Keepalive not set
  Tunnel source 1.1.24.2 (FastEthernet0/0)
   Tunnel Subblocks:
      src-track:
         Tunnel0 source tracking subblock associated with FastEthernet0/0
          Set of tunnels with source FastEthernet0/0, 1 member (includes iterators), on interface <OK>
  Tunnel protocol/transport multi-GRE/IP

R3#show int t0
Tunnel0 is up, line protocol is up 
  Hardware is Tunnel
  Internet address is 192.168.1.3/24
  MTU 17870 bytes, BW 4096 Kbit/sec, DLY 50000 usec, 
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation TUNNEL, loopback not set
  Keepalive not set
  Tunnel source 1.1.34.3 (FastEthernet0/0)
   Tunnel Subblocks:
      src-track:
         Tunnel0 source tracking subblock associated with FastEthernet0/0
          Set of tunnels with source FastEthernet0/0, 1 member (includes iterators), on interface <OK>
  Tunnel protocol/transport multi-GRE/IP

- Routing adjacencies (EIGRP neighborship) are between Hub and Spokes only 

R1#show ip eigrp neighbors
EIGRP-IPv4 Neighbors for AS(100)
H   Address                 Interface              Hold Uptime   SRTT   RTO  Q  Seq
                                                   (sec)         (ms)       Cnt Num
1   192.168.1.2             Tu0                      10 00:28:43  188  1128  0  3
0   192.168.1.3             Tu0                      14 00:29:10  174  1044  0  3

R2#show ip eigrp neighbors
EIGRP-IPv4 Neighbors for AS(100)
H   Address                 Interface              Hold Uptime   SRTT   RTO  Q  Seq
                                                   (sec)         (ms)       Cnt Num
0   192.168.1.1             Tu0                      14 00:28:45 1282  5000  0  5

R3#show ip eigrp neighbors
EIGRP-IPv4 Neighbors for AS(100)
H   Address                 Interface              Hold Uptime   SRTT   RTO  Q  Seq
                                                   (sec)         (ms)       Cnt Num
0   192.168.1.1             Tu0                      11 00:29:16  200  1200  0  2

- Routing Table
- Hub router advertised a summary route (10.10.0.0/16) to spoke EIGRP routers


R1#show ip route eigrp | beg Gate
Gateway of last resort is 1.1.14.4 to network 0.0.0.0

      10.0.0.0/8 is variably subnetted, 5 subnets, 3 masks
D        10.10.0.0/16 is a summary, 00:30:13, Null0
D        10.10.2.0/24 [90/1907456] via 192.168.1.2, 00:29:34, Tunnel0
D        10.10.3.0/24 [90/1907456] via 192.168.1.3, 00:30:04, Tunnel0

R2#show ip route eigrp | beg Gate
Gateway of last resort is 1.1.24.4 to network 0.0.0.0

      10.0.0.0/8 is variably subnetted, 3 subnets, 3 masks
D        10.10.0.0/16 [90/1907456] via 192.168.1.1, 00:29:38, Tunnel0

R3#show ip route eigrp | beg Gate
Gateway of last resort is 1.1.34.4 to network 0.0.0.0

      10.0.0.0/8 is variably subnetted, 3 subnets, 3 masks
D        10.10.0.0/16 [90/1907456] via 192.168.1.1, 00:30:10, Tunnel0

- DMVPN Status
- Initially there is no spoke-to-spoke communication

R1#show dmvpn
Legend: Attrb --> S - Static, D - Dynamic, I - Incomplete
        N - NATed, L - Local, X - No Socket
        # Ent --> Number of NHRP entries with same NBMA peer
        NHS Status: E --> Expecting Replies, R --> Responding, W --> Waiting
        UpDn Time --> Up or Down Time for a Tunnel
====================================================================
Interface: Tunnel0, IPv4 NHRP Details 
Type:Hub, NHRP Peers:2, 

 # Ent  Peer NBMA Addr Peer Tunnel Add State  UpDn Tm Attrb
 ----- --------------- --------------- ----- -------- -----
     1 1.1.24.2            192.168.1.2    UP 00:30:53     D
     1 1.1.34.3            192.168.1.3    UP 00:31:22     D

R2#show dmvpn
Legend: Attrb --> S - Static, D - Dynamic, I - Incomplete
        N - NATed, L - Local, X - No Socket
        # Ent --> Number of NHRP entries with same NBMA peer
        NHS Status: E --> Expecting Replies, R --> Responding, W --> Waiting
        UpDn Time --> Up or Down Time for a Tunnel
======================================================================
Interface: Tunnel0, IPv4 NHRP Details 
Type:Spoke, NHRP Peers:1, 

 # Ent  Peer NBMA Addr Peer Tunnel Add State  UpDn Tm Attrb
 ----- --------------- --------------- ----- -------- -----
     1 1.1.14.1            192.168.1.1    UP 00:30:55     S

R3#show dmvpn
Legend: Attrb --> S - Static, D - Dynamic, I - Incomplete
        N - NATed, L - Local, X - No Socket
        # Ent --> Number of NHRP entries with same NBMA peer
        NHS Status: E --> Expecting Replies, R --> Responding, W --> Waiting
        UpDn Time --> Up or Down Time for a Tunnel
======================================================================
Interface: Tunnel0, IPv4 NHRP Details 
Type:Spoke, NHRP Peers:1, 

 # Ent  Peer NBMA Addr Peer Tunnel Add State  UpDn Tm Attrb
 ----- --------------- --------------- ----- -------- -----
     1 1.1.14.1            192.168.1.1    UP 00:31:26     S

- Traffic between Spoke R2 and Spoke R3
- First packet would traverse via Hub and it will trigger NHRP shortcut switching process then traffic will start using direct spoke-to-spoke tunnel

R2#trace 10.10.3.3 so 10.10.2.2
Type escape sequence to abort.
Tracing the route to 10.10.3.3
VRF info: (vrf in name/id, vrf out name/id)
  1 192.168.1.1 132 msec 136 msec 160 msec    << Hub R1 Tunnel IP  
  2 192.168.1.3 236 msec                      << Spoke R3 Tunnel IP

Following events occur for NHRP shortcut switching:

When the traffic between Spokes R2 & R3 arrives at Hub-R1 router, it will send redirect message (due to "ip nhrp redirect") to spokes to tell them, as both of you are DMVPN spokes and there would be a better path if you can build a direct spoke-to-spoke tunnel.

=> Hub-R1 sends NHRP redirect to Spoke-R2
R1#
*Mar  1 23:24:47.063: NHRP: Send Traffic Indication via Tunnel0 vrf 0, packet size: 100
*Mar  1 23:24:47.071:  src: 192.168.1.1, dst: 10.10.2.2
*Mar  1 23:24:47.079:  (F) afn: AF_IP(1), type: IP(800), hop: 255, ver: 1
*Mar  1 23:24:47.079:      shtl: 4(NSAP), sstl: 0(NSAP)
*Mar  1 23:24:47.083:      pktsz: 100 extoff: 68
*Mar  1 23:24:47.087:  (M) traffic code: redirect(0)
*Mar  1 23:24:47.087:      src NBMA: 1.1.14.1
*Mar  1 23:24:47.087:      src protocol: 192.168.1.1, dst protocol: 10.10.2.2
*Mar  1 23:24:47.087:      Contents of nhrp traffic indication packet:
*Mar  1 23:24:47.087:         45 00 00 64 00 00 00 00 FE 01 A3 80 0A 0A 02 02 
*Mar  1 23:24:47.087:         0A 0A 03 03 08 00 A8 A1 00 00 00 

=> Hub-R1 sends NHRP redirect to Spoke-R3
R1#
*Mar  1 23:24:47.139: NHRP: Send Traffic Indication via Tunnel0 vrf 0, packet size: 100
*Mar  1 23:24:47.143:  src: 192.168.1.1, dst: 10.10.3.3
*Mar  1 23:24:47.151:  (F) afn: AF_IP(1), type: IP(800), hop: 255, ver: 1
*Mar  1 23:24:47.155:      shtl: 4(NSAP), sstl: 0(NSAP)
*Mar  1 23:24:47.155:      pktsz: 100 extoff: 68
*Mar  1 23:24:47.159:  (M) traffic code: redirect(0)
*Mar  1 23:24:47.163:      src NBMA: 1.1.14.1
*Mar  1 23:24:47.163:      src protocol: 192.168.1.1, dst protocol: 10.10.3.3
*Mar  1 23:24:47.171:      Contents of nhrp traffic indication packet:
*Mar  1 23:24:47.175:         45 00 00 64 00 00 00 00 FE 01 A3 80 0A 0A 03 03 
*Mar  1 23:24:47.179:         0A 0A 02 02 00 00 B0 A1 00 00 00 

=> Spoke-R2 receives NHRP redirect from Hub-R1
R2#
*Mar  1 23:24:47.135: NHRP: Receive Traffic Indication via Tunnel0 vrf 0, packet size: 100
*Mar  1 23:24:47.139:  (F) afn: AF_IP(1), type: IP(800), hop: 255, ver: 1
*Mar  1 23:24:47.139:      shtl: 4(NSAP), sstl: 0(NSAP)
*Mar  1 23:24:47.139:      pktsz: 100 extoff: 68
*Mar  1 23:24:47.139:  (M) traffic code: redirect(0)
*Mar  1 23:24:47.139:      src NBMA: 1.1.14.1
*Mar  1 23:24:47.143:      src protocol: 192.168.1.1, dst protocol: 10.10.2.2
*Mar  1 23:24:47.143:      Contents of nhrp traffic indication packet:
*Mar  1 23:24:47.143:         45 00 00 64 00 00 00 00 FE 01 A3 80 0A 0A 02 02 
*Mar  1 23:24:47.143:         0A 0A 03 03 08 00 A8 A1 00 00 00 

=> Spoke-R2 will accept this message and want to build a shortcut path (direct spoke-to-spoke tunnel) because of "ip nhrp shortcut" command.
=> Spoke-R2 will send a NHRP resolution request to Spoke-R3 via Hub-R1, to find NBMA IP of Spoke-R3 to build a direct spoke-to-spoke tunnel   
 
R2#
*Mar  1 23:24:47.163: NHRP: Send Resolution Request via Tunnel0 vrf 0, packet size: 88
*Mar  1 23:24:47.167:  src: 192.168.1.2, dst: 10.10.3.3
*Mar  1 23:24:47.167:  (F) afn: AF_IP(1), type: IP(800), hop: 255, ver: 1
*Mar  1 23:24:47.167:      shtl: 4(NSAP), sstl: 0(NSAP)
*Mar  1 23:24:47.167:      pktsz: 88 extoff: 52
*Mar  1 23:24:47.167:  (M) flags: "router auth src-stable nat ", reqid: 2 
*Mar  1 23:24:47.167:      src NBMA: 1.1.24.2
*Mar  1 23:24:47.167:      src protocol: 192.168.1.2, dst protocol: 10.10.3.3
*Mar  1 23:24:47.167:  (C-1) code: no error(0)
*Mar  1 23:24:47.167:        prefix: 32, mtu: 17870, hd_time: 7200
*Mar  1 23:24:47.167:        addr_len: 0(NSAP), subaddr_len: 0(NSAP), proto_len: 0, pref: 0

=> Similar events happens on Spoke-R3.
=> Spoke-R3 receives NHRP redirect from Hub-R1
=> Spoke-R3 will also send a NHRP resolution request to Spoke-R2 via Hub-R1, to find NBMA IP of Spoke-R2 to build a direct spoke-to-spoke tunnel

R3#
*Mar  1 23:24:47.263: NHRP: Receive Traffic Indication via Tunnel0 vrf 0, packet size: 100
*Mar  1 23:24:47.267:  (F) afn: AF_IP(1), type: IP(800), hop: 255, ver: 1
*Mar  1 23:24:47.271:      shtl: 4(NSAP), sstl: 0(NSAP)
*Mar  1 23:24:47.275:      pktsz: 100 extoff: 68
*Mar  1 23:24:47.275:  (M) traffic code: redirect(0)
*Mar  1 23:24:47.275:      src NBMA: 1.1.14.1
*Mar  1 23:24:47.275:      src protocol: 192.168.1.1, dst protocol: 10.10.3.3
*Mar  1 23:24:47.275:      Contents of nhrp traffic indication packet:
*Mar  1 23:24:47.275:         45 00 00 64 00 00 00 00 FE 01 A3 80 0A 0A 03 03 
*Mar  1 23:24:47.275:         0A 0A 02 02 00 00 B0 A1 00 00 00 

R3#
*Mar  1 23:24:47.307: NHRP: Send Resolution Request via Tunnel0 vrf 0, packet size: 88
*Mar  1 23:24:47.311:  src: 192.168.1.3, dst: 10.10.2.2
*Mar  1 23:24:47.319:  (F) afn: AF_IP(1), type: IP(800), hop: 255, ver: 1
*Mar  1 23:24:47.323:      shtl: 4(NSAP), sstl: 0(NSAP)
*Mar  1 23:24:47.323:      pktsz: 88 extoff: 52

R3#*Mar  1 23:24:47.327:  (M) flags: "router auth src-stable nat ", reqid: 2 
*Mar  1 23:24:47.331:      src NBMA: 1.1.34.3
*Mar  1 23:24:47.335:      src protocol: 192.168.1.3, dst protocol: 10.10.2.2
*Mar  1 23:24:47.335:  (C-1) code: no error(0)
*Mar  1 23:24:47.335:        prefix: 32, mtu: 17870, hd_time: 7200
*Mar  1 23:24:47.335:        addr_len: 0(NSAP), subaddr_len: 0(NSAP), proto_len: 0, pref: 0

=> Hub-R1 receives NHRP resolution requests and forward it to both the spokes (R2 & R3)

R1#
*Mar  1 23:24:47.211: NHRP: Receive Resolution Request via Tunnel0 vrf 0, packet size: 88
*Mar  1 23:24:47.219:  (F) afn: AF_IP(1), type: IP(800), hop: 255, ver: 1
*Mar  1 23:24:47.219:      shtl: 4(NSAP), sstl: 0(NSAP)
*Mar  1 23:24:47.219:      pktsz: 88 extoff: 52
*Mar  1 23:24:47.219:  (M) flags: "router auth src-stable nat ", reqid: 2 
*Mar  1 23:24:47.219:      src NBMA: 1.1.24.2
*Mar  1 23:24:47.219:      src protocol: 192.168.1.2, dst protocol: 10.10.3.3
*Mar  1 23:24:47.219:  (C-1) code: no error(0)
*Mar  1 23:24:47.219:   
R1#     prefix: 32, mtu: 17870, hd_time: 7200
*Mar  1 23:24:47.219:        addr_len: 0(NSAP), subaddr_len: 0(NSAP), proto_len: 0, pref: 0
*Mar  1 23:24:47.223: NHRP: Forwarding Resolution Request via Tunnel0 vrf 0, packet size: 108
*Mar  1 23:24:47.223:  src: 192.168.1.1, dst: 10.10.3.3
*Mar  1 23:24:47.223:  (F) afn: AF_IP(1), type: IP(800), hop: 254, ver: 1
*Mar  1 23:24:47.223:      shtl: 4(NSAP), sstl: 0(NSAP)
*Mar  1 23:24:47.223:      pktsz: 108 extoff: 52
*Mar  1 23:24:47.223:  (M) flags: "router auth src-stable nat ", reqid: 2 
*Mar  1 23:24:47.223:      src NBMA: 1.1.24.2
*Mar  1 23:24:47.223:      src protocol: 192.168.1.2, dst protocol: 10.10.3.3
*Mar  1 23:24:47.223:  (C-1) code: no error(0)
*Mar  1 23:24:47.223:        prefix: 32, mtu: 17870, hd_time: 7200
*Mar  1 23:24:47.223:        addr_len: 0(NSAP), subaddr_len: 0(NSAP), proto_len: 0, pref: 0

R1#
*Mar  1 23:24:47.387: NHRP: Receive Resolution Request via Tunnel0 vrf 0, packet size: 88
*Mar  1 23:24:47.395:  (F)
R1# afn: AF_IP(1), type: IP(800), hop: 255, ver: 1
*Mar  1 23:24:47.399:      shtl: 4(NSAP), sstl: 0(NSAP)
*Mar  1 23:24:47.399:      pktsz: 88 extoff: 52
*Mar  1 23:24:47.403:  (M) flags: "router auth src-stable nat ", reqid: 2 
*Mar  1 23:24:47.403:      src NBMA: 1.1.34.3
*Mar  1 23:24:47.403:      src protocol: 192.168.1.3, dst protocol: 10.10.2.2
*Mar  1 23:24:47.403:  (C-1) code: no error(0)
*Mar  1 23:24:47.403:        prefix: 32, mtu: 17870, hd_time: 7200
*Mar  1 23:24:47.403:        addr_len: 0(NSAP), subaddr_len: 0(NSAP), proto_len: 0, pref: 0
*Mar  1 23:24:47.407: NHRP: Forwarding Resolution Request via Tunnel0 vrf 0, packet size: 108
*Mar  1 23:24:47.407:  src: 192.168.1.1, dst: 10.10.2.2
*Mar  1 23:24:47.407:  (F) afn: AF_IP(1), type: IP(800), hop: 254, ver: 1
*Mar  1 23:24:47.407:      shtl: 4(NSAP), sstl: 0(NSAP)
*Mar  1 23:24:47.407:      pktsz: 108 extoff: 52
*Mar  1 23:24:47.407:  (M) flags: "router auth src-stable nat ", reqid: 2 
*Mar  1 23:24:47.407:      src NBMA: 1.1.34.3
*Mar  1 23:24:47.407:      src protocol: 192.168.1.3, dst protocol: 10.10.2.2
*Mar  1 23:24:47.407:  (C-1) code: no error(0)
*Mar  1 23:24:47.407:        prefix: 32, mtu: 17870, hd_time: 7200
*Mar  1 23:24:47.407:        addr_len: 0(NSAP), subaddr_len: 0(NSAP), proto_len: 0, pref: 0

=> Both Spoke-R2 & Spoke-R3 send NHRP resolution reply via Hub-R1

R2#
*Mar  1 23:24:48.431: NHRP: Send Resolution Reply via Tunnel0 vrf 0, packet size: 136
*Mar  1 23:24:48.435:  src: 192.168.1.2, dst: 192.168.1.3

R3#
*Mar  1 23:24:48.267: NHRP: Send Resolution Reply via Tunnel0 vrf 0, packet size: 136
*Mar  1 23:24:48.271:  src: 192.168.1.3, dst: 192.168.1.2

=> Finally, both Spokes receives each other's NHRP resolution reply 

R2#
*Mar  1 23:24:48.311: NHRP: Receive Resolution Reply via Tunnel0 vrf 0, packet size: 136
*Mar  1 23:24:48.319:  (F) afn: AF_IP(1), type: IP(800), hop: 255, ver: 1
*Mar  1 23:24:48.323:      shtl: 4(NSAP), sstl: 0(NSAP)
*Mar  1 23:24:48.323:      pktsz: 136 extoff: 60
*Mar  1 23:24:48.327:  (M) flags: "router auth dst-stable unique src-stable nat ", reqid: 2 
*Mar  1 23:24:48.331:      src NBMA: 1.1.24.2
*Mar  1 23:24:48.335:      src protocol: 192.168.1.2, dst protocol: 10.10.3.3
*Mar  1 23:24:48.335:  (C-1) code: no error(0)
*Mar  1 23:24:48.335:        prefix: 24, mtu: 17870, hd_time: 7199
*Mar  1 23:24:48.335:        addr_len: 4(NSAP), subaddr_len: 0(NSAP), proto_len: 4, pref: 0
*Mar  1 23:24:48.335:        client NBMA: 1.1.34.3
*Mar  1 23:24:48.335:        client protocol: 192.168.1.3

R3#
*Mar  1 23:24:48.551: NHRP: Receive Resolution Reply via Tunnel0 vrf 0, packet size: 136
*Mar  1 23:24:48.559:  (F) afn: AF_IP(1), type: IP(800), hop: 255, ver: 1
*Mar  1 23:24:48.563:      shtl: 4(NSAP), sstl: 0(NSAP)
*Mar  1 23:24:48.563:      pktsz: 136 extoff: 60
*Mar  1 23:24:48.567:  (M) flags: "router auth dst-stable unique src-stable nat ", reqid: 2 
*Mar  1 23:24:48.571:      src NBMA: 1.1.34.3
*Mar  1 23:24:48.575:      src protocol: 192.168.1.3, dst protocol: 10.10.2.2
*Mar  1 23:24:48.579:  (C-1) code: no error(0)
*Mar  1 23:24:48.579:        prefix: 24, mtu: 17870, hd_time: 7200
*Mar  1 23:24:48.579:        addr_len: 4(NSAP), subaddr_len: 0(NSAP), proto_len: 4, pref: 0
*Mar  1 23:24:48.579:        client NBMA: 1.1.24.2
*Mar  1 23:24:48.579:        client protocol: 192.168.1.2

=> Now both spokes know each other's NBMA & Tunnel IP address, and can build a direct spoke-to-spoke tunnel
=> And both spokes add a NHRP shortcut route entry to reach each other's LAN subnet

R2#
*Mar  1 23:24:48.455: NHRP: Adding route entry for 10.10.3.0/24 to RIB
*Mar  1 23:24:48.455: NHRP: Route addition to RIB Successful 

R3#
*Mar  1 23:24:48.587: NHRP: Adding route entry for 10.10.2.0/24 to RIB
*Mar  1 23:24:48.595: NHRP: Route addition to RIB Successful 

- Let's test communication between Spoke-R2 and Spoke-R3 now

R2#ping 10.10.3.3 so 10.10.2.2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.10.3.3, timeout is 2 seconds:
Packet sent with a source address of 10.10.2.2 
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 116/136/152 ms

- Now traffic between Spoke-R2 and Spoke-R3 is using direct DMVPN tunnel

R2#trace 10.10.3.3 so 10.10.2.2
Type escape sequence to abort.
Tracing the route to 10.10.3.3
VRF info: (vrf in name/id, vrf out name/id)
  1 192.168.1.3 148 msec *  116 msec        << Spoke-R3 Tunnel IP

- Spokes routing table would show a new NHRP routing entry to reach subnet behind other spoke router  

R2#show ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area 
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       + - replicated route, % - next hop override

Gateway of last resort is 1.1.24.4 to network 0.0.0.0

S*    0.0.0.0/0 [1/0] via 1.1.24.4
      1.0.0.0/8 is variably subnetted, 2 subnets, 2 masks
C        1.1.24.0/24 is directly connected, FastEthernet0/0
L        1.1.24.2/32 is directly connected, FastEthernet0/0
      10.0.0.0/8 is variably subnetted, 4 subnets, 3 masks
D        10.10.0.0/16 [90/1907456] via 192.168.1.1, 01:31:44, Tunnel0
C        10.10.2.0/24 is directly connected, FastEthernet0/1
L        10.10.2.2/32 is directly connected, FastEthernet0/1
H        10.10.3.0/24 [250/1] via 192.168.1.3, 00:05:13, Tunnel0
      192.168.1.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.1.0/24 is directly connected, Tunnel0
L        192.168.1.2/32 is directly connected, Tunnel0

R3#show ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area 
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       + - replicated route, % - next hop override

Gateway of last resort is 1.1.34.4 to network 0.0.0.0

S*    0.0.0.0/0 [1/0] via 1.1.34.4
      1.0.0.0/8 is variably subnetted, 2 subnets, 2 masks
C        1.1.34.0/24 is directly connected, FastEthernet0/0
L        1.1.34.3/32 is directly connected, FastEthernet0/0
      10.0.0.0/8 is variably subnetted, 4 subnets, 3 masks
D        10.10.0.0/16 [90/1907456] via 192.168.1.1, 01:32:12, Tunnel0
H        10.10.2.0/24 [250/1] via 192.168.1.2, 00:06:01, Tunnel0
C        10.10.3.0/24 is directly connected, FastEthernet0/1
L        10.10.3.3/32 is directly connected, FastEthernet0/1
      192.168.1.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.1.0/24 is directly connected, Tunnel0
L        192.168.1.3/32 is directly connected, Tunnel0

- DMVPN Status

R1#show dmvpn
Legend: Attrb --> S - Static, D - Dynamic, I - Incomplete
        N - NATed, L - Local, X - No Socket
        # Ent --> Number of NHRP entries with same NBMA peer
        NHS Status: E --> Expecting Replies, R --> Responding, W --> Waiting
        UpDn Time --> Up or Down Time for a Tunnel
======================================================================
Interface: Tunnel0, IPv4 NHRP Details 
Type:Hub, NHRP Peers:2, 

 # Ent  Peer NBMA Addr Peer Tunnel Add State  UpDn Tm Attrb
 ----- --------------- --------------- ----- -------- -----
     1 1.1.24.2            192.168.1.2    UP 01:37:55     D
     1 1.1.34.3            192.168.1.3    UP 01:37:36     D


R2#show dmvpn
Legend: Attrb --> S - Static, D - Dynamic, I - Incomplete
        N - NATed, L - Local, X - No Socket
        # Ent --> Number of NHRP entries with same NBMA peer
        NHS Status: E --> Expecting Replies, R --> Responding, W --> Waiting
        UpDn Time --> Up or Down Time for a Tunnel
======================================================================
Interface: Tunnel0, IPv4 NHRP Details 
Type:Spoke, NHRP Peers:2, 

 # Ent  Peer NBMA Addr Peer Tunnel Add State  UpDn Tm Attrb
 ----- --------------- --------------- ----- -------- -----
     2 1.1.34.3            192.168.1.3    UP 00:11:26   DT1
                           192.168.1.3    UP 00:11:26     D
     1 1.1.14.1            192.168.1.1    UP 01:37:59     S
*T1 - Route Installed

R3#show dmvpn
Legend: Attrb --> S - Static, D - Dynamic, I - Incomplete
        N - NATed, L - Local, X - No Socket
        # Ent --> Number of NHRP entries with same NBMA peer
        NHS Status: E --> Expecting Replies, R --> Responding, W --> Waiting
        UpDn Time --> Up or Down Time for a Tunnel
======================================================================
Interface: Tunnel0, IPv4 NHRP Details 
Type:Spoke, NHRP Peers:2, 

 # Ent  Peer NBMA Addr Peer Tunnel Add State  UpDn Tm Attrb
 ----- --------------- --------------- ----- -------- -----
     2 1.1.24.2            192.168.1.2    UP 00:11:29   DT1
                           192.168.1.2    UP 00:11:29     D
     1 1.1.14.1            192.168.1.1    UP 01:37:43     S
*T1 - Route Installed

- NHRP Table

R1#show ip nhrp
192.168.1.2/32 via 192.168.1.2
   Tunnel0 created 01:38:52, expire 01:41:01
   Type: dynamic, Flags: unique registered used 
   NBMA address: 1.1.24.2 
192.168.1.3/32 via 192.168.1.3
   Tunnel0 created 01:38:33, expire 01:41:00
   Type: dynamic, Flags: unique registered used 
   NBMA address: 1.1.34.3

R2#show ip nhrp
10.10.2.0/24 via 192.168.1.2
   Tunnel0 created 00:12:24, expire 01:47:35
   Type: dynamic, Flags: router unique local 
   NBMA address: 1.1.24.2 
    (no-socket) 
10.10.3.0/24 via 192.168.1.3
   Tunnel0 created 00:12:24, expire 01:47:35
   Type: dynamic, Flags: router used rib 
   NBMA address: 1.1.34.3 
192.168.1.1/32 via 192.168.1.1
   Tunnel0 created 01:39:13, never expire 
   Type: static, Flags: used 
   NBMA address: 1.1.14.1 
192.168.1.3/32 via 192.168.1.3
   Tunnel0 created 00:12:25, expire 01:47:35
   Type: dynamic, Flags: router implicit used 
   NBMA address: 1.1.34.3 

R3#show ip nhrp
10.10.2.0/24 via 192.168.1.2
   Tunnel0 created 00:12:26, expire 01:47:33
   Type: dynamic, Flags: router used rib 
   NBMA address: 1.1.24.2 
10.10.3.0/24 via 192.168.1.3
   Tunnel0 created 00:12:26, expire 01:47:33
   Type: dynamic, Flags: router unique local 
   NBMA address: 1.1.34.3 
    (no-socket) 
192.168.1.1/32 via 192.168.1.1
   Tunnel0 created 01:39:16, never expire 
   Type: static, Flags: used 
   NBMA address: 1.1.14.1 
192.168.1.2/32 via 192.168.1.2
   Tunnel0 created 00:12:27, expire 01:47:33
   Type: dynamic, Flags: router implicit used 
   NBMA address: 1.1.24.2 


DMVPN Phase 3 configuration with OSPF
  • Tunnel mode mGRE on Hub and Spoke routers
  • OSPF network type would be point-to-multipoint on Hub and Spoke Routers
  • Route summarization is not feasible in single OSPF area, this is one of the main reason that OSPF is not an ideal routing protocol for DMVPN phase 3 designs
  • Shortcut route is marked with “%” symbol which indicates next-hop override feature. It means router is not using next-hop IP from OSPF RIB entry, instead it is using DMVPN tunnel IP of remote spoke to route traffic over direct DMVPN tunnel.
Configuration:
----------
R1 - Hub:
----------

hostname R1
!
ip cef
!
crypto isakmp policy 10
 encr aes
 authentication pre-share
 group 2
crypto isakmp key DmVpNpR3$h@r3dK3Y address 0.0.0.0        
!
!
crypto ipsec transform-set TRANSFORM-SET esp-aes esp-sha-hmac 
 mode transport
!
crypto ipsec profile PROTECT-GRE
 set transform-set TRANSFORM-SET 
!
interface Tunnel0
 bandwidth 4096
 ip address 192.168.1.1 255.255.255.0
 no ip redirects
 ip mtu 1400
 ip nhrp authentication NhRp@UtH
 ip nhrp map multicast dynamic
 ip nhrp network-id 100
 ip nhrp redirect
 ip tcp adjust-mss 1360
 ip ospf network point-to-multipoint
 tunnel source FastEthernet0/0
 tunnel mode gre multipoint
 tunnel key 123
 tunnel protection ipsec profile PROTECT-GRE
!
interface FastEthernet0/0
 ip address 1.1.14.1 255.255.255.0
!
interface FastEthernet0/1
 ip address 10.10.1.1 255.255.255.0
!
router ospf 1
 network 10.10.1.1 0.0.0.0 area 0
 network 192.168.1.1 0.0.0.0 area 0
!
ip route 0.0.0.0 0.0.0.0 1.1.14.4


------------
R2 - Spoke:
------------

hostname R2
!
ip cef
!
crypto isakmp policy 10
 encr aes
 authentication pre-share
 group 2
crypto isakmp key DmVpNpR3$h@r3dK3Y address 0.0.0.0        
!
!
crypto ipsec transform-set TRANSFORM-SET esp-aes esp-sha-hmac 
 mode transport
!
crypto ipsec profile PROTECT-GRE
 set transform-set TRANSFORM-SET 
!
interface Tunnel0
 bandwidth 4096
 ip address 192.168.1.2 255.255.255.0
 no ip redirects
 ip mtu 1400
 ip nhrp authentication NhRp@UtH
 ip nhrp map multicast 1.1.14.1
 ip nhrp map 192.168.1.1 1.1.14.1
 ip nhrp network-id 100
 ip nhrp nhs 192.168.1.1
 ip nhrp shortcut
 ip tcp adjust-mss 1360
 ip ospf network point-to-multipoint
 tunnel source FastEthernet0/0
 tunnel mode gre multipoint
 tunnel key 123
 tunnel protection ipsec profile PROTECT-GRE
!
interface FastEthernet0/0
 ip address 1.1.24.2 255.255.255.0
!
interface FastEthernet0/1
 ip address 10.10.2.2 255.255.255.0
!
router ospf 1
 network 10.10.2.2 0.0.0.0 area 0
 network 192.168.1.2 0.0.0.0 area 0
!
ip route 0.0.0.0 0.0.0.0 1.1.24.4

------------
R3 - Spoke:
------------

hostname R3
!
ip cef
!
crypto isakmp policy 10
 encr aes
 authentication pre-share
 group 2
crypto isakmp key DmVpNpR3$h@r3dK3Y address 0.0.0.0        
!
!
crypto ipsec transform-set TRANSFORM-SET esp-aes esp-sha-hmac 
 mode transport
!
crypto ipsec profile PROTECT-GRE
 set transform-set TRANSFORM-SET 
!
interface Tunnel0
 bandwidth 4096
 ip address 192.168.1.3 255.255.255.0
 no ip redirects
 ip mtu 1400
 ip nhrp authentication NhRp@UtH
 ip nhrp map multicast 1.1.14.1
 ip nhrp map 192.168.1.1 1.1.14.1
 ip nhrp network-id 100
 ip nhrp nhs 192.168.1.1
 ip nhrp shortcut
 ip tcp adjust-mss 1360
 ip ospf network point-to-multipoint
 tunnel source FastEthernet0/0
 tunnel mode gre multipoint
 tunnel key 123
 tunnel protection ipsec profile PROTECT-GRE
!
interface FastEthernet0/0
 ip address 1.1.34.3 255.255.255.0
!
interface FastEthernet0/1
 ip address 10.10.3.3 255.255.255.0
!
router ospf 1
 network 10.10.3.3 0.0.0.0 area 0
 network 192.168.1.3 0.0.0.0 area 0
!
ip route 0.0.0.0 0.0.0.0 1.1.34.4
Verification:
- Tunnel OSPF Network Type 

R1#show ip ospf int t0
Tunnel0 is up, line protocol is up 
  Internet Address 192.168.1.1/24, Area 0, Attached via Network Statement
  Process ID 1, Router ID 192.168.1.1, Network Type POINT_TO_MULTIPOINT, Cost: 24
  Topology-MTID    Cost    Disabled    Shutdown      Topology Name
        0           24        no          no            Base
  Transmit Delay is 1 sec, State POINT_TO_MULTIPOINT
  Timer intervals configured, Hello 30, Dead 120, Wait 120, Retransmit 5
    oob-resync timeout 120
    Hello due in 00:00:19
  Supports Link-local Signaling (LLS)
  Cisco NSF helper support enabled
  IETF NSF helper support enabled
  Index 2/2, flood queue length 0
  Next 0x0(0)/0x0(0)
  Last flood scan length is 1, maximum is 1
  Last flood scan time is 4 msec, maximum is 4 msec
  Neighbor Count is 2, Adjacent neighbor count is 2 
    Adjacent with neighbor 192.168.1.3
    Adjacent with neighbor 192.168.1.2
  Suppress hello for 0 neighbor(s)

R2#show ip ospf int t0
Tunnel0 is up, line protocol is up 
  Internet Address 192.168.1.2/24, Area 0, Attached via Network Statement
  Process ID 1, Router ID 192.168.1.2, Network Type POINT_TO_MULTIPOINT, Cost: 24
  Topology-MTID    Cost    Disabled    Shutdown      Topology Name
        0           24        no          no            Base
  Transmit Delay is 1 sec, State POINT_TO_MULTIPOINT
  Timer intervals configured, Hello 30, Dead 120, Wait 120, Retransmit 5
    oob-resync timeout 120
    Hello due in 00:00:15
  Supports Link-local Signaling (LLS)
  Cisco NSF helper support enabled
  IETF NSF helper support enabled
  Index 2/2, flood queue length 0
  Next 0x0(0)/0x0(0)
  Last flood scan length is 1, maximum is 1
  Last flood scan time is 0 msec, maximum is 4 msec
  Neighbor Count is 1, Adjacent neighbor count is 1 
    Adjacent with neighbor 192.168.1.1
  Suppress hello for 0 neighbor(s)

R3#show ip ospf int t0
Tunnel0 is up, line protocol is up 
  Internet Address 192.168.1.3/24, Area 0, Attached via Network Statement
  Process ID 1, Router ID 192.168.1.3, Network Type POINT_TO_MULTIPOINT, Cost: 24
  Topology-MTID    Cost    Disabled    Shutdown      Topology Name
        0           24        no          no            Base
  Transmit Delay is 1 sec, State POINT_TO_MULTIPOINT
  Timer intervals configured, Hello 30, Dead 120, Wait 120, Retransmit 5
    oob-resync timeout 120
    Hello due in 00:00:09
  Supports Link-local Signaling (LLS)
  Cisco NSF helper support enabled
  IETF NSF helper support enabled
  Index 2/2, flood queue length 0
  Next 0x0(0)/0x0(0)
  Last flood scan length is 1, maximum is 1
  Last flood scan time is 4 msec, maximum is 4 msec
  Neighbor Count is 1, Adjacent neighbor count is 1 
    Adjacent with neighbor 192.168.1.1
  Suppress hello for 0 neighbor(s)

- OSPF Adjacencies

R1#show ip ospf neighbor 

Neighbor ID     Pri   State           Dead Time   Address         Interface
192.168.1.3       0   FULL/  -        00:01:37    192.168.1.3     Tunnel0
192.168.1.2       0   FULL/  -        00:01:55    192.168.1.2     Tunnel0

R2#show ip ospf neighbor 

Neighbor ID     Pri   State           Dead Time   Address         Interface
192.168.1.1       0   FULL/  -        00:01:50    192.168.1.1     Tunnel0

R3#show ip ospf neighbor 

Neighbor ID     Pri   State           Dead Time   Address         Interface
192.168.1.1       0   FULL/  -        00:01:48    192.168.1.1     Tunnel0

- Traffic between Spoke-R2 and Spoke-R3 is via direct spoke-to-spoke DMVPN tunnel

R2#ping 10.10.3.3 so 10.10.2.2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.10.3.3, timeout is 2 seconds:
Packet sent with a source address of 10.10.2.2 
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 120/134/148 ms

R2#trace 10.10.3.3 so 10.10.2.2
Type escape sequence to abort.
Tracing the route to 10.10.3.3
VRF info: (vrf in name/id, vrf out name/id)
  1 192.168.1.3 128 msec *  152 msec          << Spoke-R3 Tunnel IP

- DMVPN Status

R1#show dmvpn
Legend: Attrb --> S - Static, D - Dynamic, I - Incomplete
        N - NATed, L - Local, X - No Socket
        # Ent --> Number of NHRP entries with same NBMA peer
        NHS Status: E --> Expecting Replies, R --> Responding, W --> Waiting
        UpDn Time --> Up or Down Time for a Tunnel
======================================================================
Interface: Tunnel0, IPv4 NHRP Details 
Type:Hub, NHRP Peers:2, 

 # Ent  Peer NBMA Addr Peer Tunnel Add State  UpDn Tm Attrb
 ----- --------------- --------------- ----- -------- -----
     1 1.1.24.2            192.168.1.2    UP 01:33:58     D
     1 1.1.34.3            192.168.1.3    UP 01:33:48     D

R2#show dmvpn
Legend: Attrb --> S - Static, D - Dynamic, I - Incomplete
        N - NATed, L - Local, X - No Socket
        # Ent --> Number of NHRP entries with same NBMA peer
        NHS Status: E --> Expecting Replies, R --> Responding, W --> Waiting
        UpDn Time --> Up or Down Time for a Tunnel
======================================================================
Interface: Tunnel0, IPv4 NHRP Details 
Type:Spoke, NHRP Peers:2, 

 # Ent  Peer NBMA Addr Peer Tunnel Add State  UpDn Tm Attrb
 ----- --------------- --------------- ----- -------- -----
     2 1.1.34.3            192.168.1.3    UP 00:01:42   DT2
                           192.168.1.3    UP 00:01:42     D
     1 1.1.14.1            192.168.1.1    UP 01:34:01     S
*T2 - Nexthop-override

R3#show dmvpn
Legend: Attrb --> S - Static, D - Dynamic, I - Incomplete
        N - NATed, L - Local, X - No Socket
        # Ent --> Number of NHRP entries with same NBMA peer
        NHS Status: E --> Expecting Replies, R --> Responding, W --> Waiting
        UpDn Time --> Up or Down Time for a Tunnel
======================================================================
Interface: Tunnel0, IPv4 NHRP Details 
Type:Spoke, NHRP Peers:2, 

 # Ent  Peer NBMA Addr Peer Tunnel Add State  UpDn Tm Attrb
 ----- --------------- --------------- ----- -------- -----
     2 1.1.24.2            192.168.1.2    UP 00:01:44   DT2
                           192.168.1.2    UP 00:01:44     D
     1 1.1.14.1            192.168.1.1    UP 01:33:54     S
*T2 - Nexthop-override

- Routing Table

R1#show ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area 
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       + - replicated route, % - next hop override

Gateway of last resort is 1.1.14.4 to network 0.0.0.0

S*    0.0.0.0/0 [1/0] via 1.1.14.4
      1.0.0.0/8 is variably subnetted, 2 subnets, 2 masks
C        1.1.14.0/24 is directly connected, FastEthernet0/0
L        1.1.14.1/32 is directly connected, FastEthernet0/0
      10.0.0.0/8 is variably subnetted, 4 subnets, 2 masks
C        10.10.1.0/24 is directly connected, FastEthernet0/1
L        10.10.1.1/32 is directly connected, FastEthernet0/1
O        10.10.2.0/24 [110/25] via 192.168.1.2, 01:36:33, Tunnel0
O        10.10.3.0/24 [110/25] via 192.168.1.3, 01:36:03, Tunnel0
      192.168.1.0/24 is variably subnetted, 4 subnets, 2 masks
C        192.168.1.0/24 is directly connected, Tunnel0
L        192.168.1.1/32 is directly connected, Tunnel0
O        192.168.1.2/32 [110/24] via 192.168.1.2, 01:36:33, Tunnel0
O        192.168.1.3/32 [110/24] via 192.168.1.3, 01:36:03, Tunnel0

R2#show ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area 
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       + - replicated route, % - next hop override

Gateway of last resort is 1.1.24.4 to network 0.0.0.0

S*    0.0.0.0/0 [1/0] via 1.1.24.4
      1.0.0.0/8 is variably subnetted, 2 subnets, 2 masks
C        1.1.24.0/24 is directly connected, FastEthernet0/0
L        1.1.24.2/32 is directly connected, FastEthernet0/0
      10.0.0.0/8 is variably subnetted, 4 subnets, 2 masks
O        10.10.1.0/24 [110/25] via 192.168.1.1, 01:36:38, Tunnel0
C        10.10.2.0/24 is directly connected, FastEthernet0/1
L        10.10.2.2/32 is directly connected, FastEthernet0/1
O   %    10.10.3.0/24 [110/49] via 192.168.1.1, 01:35:58, Tunnel0
      192.168.1.0/24 is variably subnetted, 4 subnets, 2 masks
C        192.168.1.0/24 is directly connected, Tunnel0
O        192.168.1.1/32 [110/24] via 192.168.1.1, 01:36:38, Tunnel0
L        192.168.1.2/32 is directly connected, Tunnel0
O        192.168.1.3/32 [110/48] via 192.168.1.1, 01:35:58, Tunnel0

R3#show ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area 
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       + - replicated route, % - next hop override

Gateway of last resort is 1.1.34.4 to network 0.0.0.0

S*    0.0.0.0/0 [1/0] via 1.1.34.4
      1.0.0.0/8 is variably subnetted, 2 subnets, 2 masks
C        1.1.34.0/24 is directly connected, FastEthernet0/0
L        1.1.34.3/32 is directly connected, FastEthernet0/0
      10.0.0.0/8 is variably subnetted, 4 subnets, 2 masks
O        10.10.1.0/24 [110/25] via 192.168.1.1, 01:36:10, Tunnel0
O   %    10.10.2.0/24 [110/49] via 192.168.1.1, 01:36:10, Tunnel0
C        10.10.3.0/24 is directly connected, FastEthernet0/1
L        10.10.3.3/32 is directly connected, FastEthernet0/1
      192.168.1.0/24 is variably subnetted, 4 subnets, 2 masks
C        192.168.1.0/24 is directly connected, Tunnel0
O        192.168.1.1/32 [110/24] via 192.168.1.1, 01:36:10, Tunnel0
O        192.168.1.2/32 [110/48] via 192.168.1.1, 01:36:10, Tunnel0
L        192.168.1.3/32 is directly connected, Tunnel0

- NHRP Table

R1#show ip nhrp
192.168.1.2/32 via 192.168.1.2
   Tunnel0 created 01:45:40, expire 01:33:54
   Type: dynamic, Flags: unique registered used 
   NBMA address: 1.1.24.2 
192.168.1.3/32 via 192.168.1.3
   Tunnel0 created 01:45:30, expire 01:33:54
   Type: dynamic, Flags: unique registered used 
   NBMA address: 1.1.34.3 

R2#show ip nhrp
10.10.2.0/24 via 192.168.1.2
   Tunnel0 created 00:13:24, expire 01:46:35
   Type: dynamic, Flags: router unique local 
   NBMA address: 1.1.24.2 
    (no-socket) 
10.10.3.0/24 via 192.168.1.3
   Tunnel0 created 00:13:23, expire 01:46:36
   Type: dynamic, Flags: router used rib nho << next-hop override
   NBMA address: 1.1.34.3 
192.168.1.1/32 via 192.168.1.1
   Tunnel0 created 01:46:18, never expire 
   Type: static, Flags: used 
   NBMA address: 1.1.14.1 
192.168.1.3/32 via 192.168.1.3
   Tunnel0 created 00:13:24, expire 01:46:35
   Type: dynamic, Flags: router implicit 
   NBMA address: 1.1.34.3 

R3#show ip nhrp
10.10.2.0/24 via 192.168.1.2
   Tunnel0 created 00:13:30, expire 01:46:28
   Type: dynamic, Flags: router used rib nho << next-hop override
   NBMA address: 1.1.24.2 
10.10.3.0/24 via 192.168.1.3
   Tunnel0 created 00:13:30, expire 01:46:29
   Type: dynamic, Flags: router unique local 
   NBMA address: 1.1.34.3 
    (no-socket) 
192.168.1.1/32 via 192.168.1.1
   Tunnel0 created 01:46:24, never expire 
   Type: static, Flags: used 
   NBMA address: 1.1.14.1 
192.168.1.2/32 via 192.168.1.2
   Tunnel0 created 00:13:31, expire 01:46:29
   Type: dynamic, Flags: router implicit 
   NBMA address: 1.1.24.2 
DMVPN Phase 3 configuration with BGP
  • Hub and Spoke routers are using mGRE tunnels
  • Hub router(s) act as route-reflector server(s)
  • Spoke routers are route-reflector clients
  • Hub router can send summary route to Spoke routers
  • eBGP can also be used by configuring Hub and Spokes in different ASNs. Here we are using iBGP with route-reflector
Configuration:
----------
R1 - Hub:
----------

hostname R1
!
ip cef
!
crypto isakmp policy 10
 encr aes
 authentication pre-share
 group 2
crypto isakmp key DmVpNpR3$h@r3dK3Y address 0.0.0.0        
!
crypto ipsec transform-set TRANSFORM-SET esp-aes esp-sha-hmac 
 mode transport
!
crypto ipsec profile PROTECT-GRE
 set transform-set TRANSFORM-SET 
!
interface Tunnel0
 bandwidth 4096
 ip address 192.168.1.1 255.255.255.0
 no ip redirects
 ip mtu 1400
 ip nhrp authentication NhRp@UtH
 ip nhrp map multicast dynamic
 ip nhrp network-id 100
 ip nhrp redirect
 ip tcp adjust-mss 1360
 tunnel source FastEthernet0/0
 tunnel mode gre multipoint
 tunnel key 123
 tunnel protection ipsec profile PROTECT-GRE
!
interface FastEthernet0/0
 ip address 1.1.14.1 255.255.255.0
!
interface FastEthernet0/1
 ip address 10.10.1.1 255.255.255.0
!
router bgp 65001
 bgp router-id 192.168.1.1
 bgp log-neighbor-changes
 bgp listen range 192.168.1.0/24 peer-group DMVPN-SPOKES
 bgp listen limit 50
 network 10.10.1.0 mask 255.255.255.0
 aggregate-address 10.10.0.0 255.255.0.0 summary-only
 neighbor DMVPN-SPOKES peer-group
 neighbor DMVPN-SPOKES remote-as 65001
 neighbor DMVPN-SPOKES route-reflector-client
!
ip route 0.0.0.0 0.0.0.0 1.1.14.4

------------
R2 - Spoke:
------------

hostname R2
!
ip cef
!
crypto isakmp policy 10
 encr aes
 authentication pre-share
 group 2
crypto isakmp key DmVpNpR3$h@r3dK3Y address 0.0.0.0        
!
crypto ipsec transform-set TRANSFORM-SET esp-aes esp-sha-hmac 
 mode transport
!
crypto ipsec profile PROTECT-GRE
 set transform-set TRANSFORM-SET 
!
interface Tunnel0
 bandwidth 4096
 ip address 192.168.1.2 255.255.255.0
 no ip redirects
 ip mtu 1400
 ip nhrp authentication NhRp@UtH
 ip nhrp map multicast 1.1.14.1
 ip nhrp map 192.168.1.1 1.1.14.1
 ip nhrp network-id 100
 ip nhrp nhs 192.168.1.1
 ip nhrp shortcut
 ip tcp adjust-mss 1360
 tunnel source FastEthernet0/0
 tunnel mode gre multipoint
 tunnel key 123
 tunnel protection ipsec profile PROTECT-GRE
!
interface FastEthernet0/0
 ip address 1.1.24.2 255.255.255.0
!
interface FastEthernet0/1
 ip address 10.10.2.2 255.255.255.0
!
router bgp 65001
 bgp router-id 192.168.1.2
 bgp log-neighbor-changes
 network 10.10.2.0 mask 255.255.255.0
 neighbor 192.168.1.1 remote-as 65001
!
ip route 0.0.0.0 0.0.0.0 1.1.24.4

------------
R3 - Spoke:
------------

hostname R3
!
ip cef
!
crypto isakmp policy 10
 encr aes
 authentication pre-share
 group 2
crypto isakmp key DmVpNpR3$h@r3dK3Y address 0.0.0.0        
!
crypto ipsec transform-set TRANSFORM-SET esp-aes esp-sha-hmac 
 mode transport
!
crypto ipsec profile PROTECT-GRE
 set transform-set TRANSFORM-SET 
!
interface Tunnel0
 bandwidth 4096
 ip address 192.168.1.3 255.255.255.0
 no ip redirects
 ip mtu 1400
 ip nhrp authentication NhRp@UtH
 ip nhrp map multicast 1.1.14.1
 ip nhrp map 192.168.1.1 1.1.14.1
 ip nhrp network-id 100
 ip nhrp nhs 192.168.1.1
 ip nhrp shortcut
 ip tcp adjust-mss 1360
 tunnel source FastEthernet0/0
 tunnel mode gre multipoint
 tunnel key 123
 tunnel protection ipsec profile PROTECT-GRE
!
interface FastEthernet0/0
 ip address 1.1.34.3 255.255.255.0
!
interface FastEthernet0/1
 ip address 10.10.3.3 255.255.255.0
!
router bgp 65001
 bgp router-id 192.168.1.3
 bgp log-neighbor-changes
 network 10.10.3.0 mask 255.255.255.0
 neighbor 192.168.1.1 remote-as 65001
!
ip route 0.0.0.0 0.0.0.0 1.1.34.4
Verification:
- BGP peering between Hub and Spokes only, No spoke to spoke peering

R1#show ip bgp summary 
BGP router identifier 192.168.1.1, local AS number 65001
BGP table version is 8, main routing table version 8
4 network entries using 592 bytes of memory
4 path entries using 256 bytes of memory
3/3 BGP path/bestpath attribute entries using 408 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 1256 total bytes of memory
BGP activity 4/0 prefixes, 4/0 paths, scan interval 60 secs

Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
*192.168.1.2    4        65001      18      20        8    0    0 00:12:43        1
*192.168.1.3    4        65001      19      18        8    0    0 00:12:45        1
* Dynamically created based on a listen range command
Dynamically created neighbors: 2, Subnet ranges: 1

BGP peergroup DMVPN-SPOKES listen range group members: 
  192.168.1.0/24 

Total dynamically created neighbors: 2/(50 max), Subnet ranges: 1

R2#show ip bgp summary 
BGP router identifier 192.168.1.2, local AS number 65001
BGP table version is 3, main routing table version 3
2 network entries using 296 bytes of memory
2 path entries using 128 bytes of memory
2/2 BGP path/bestpath attribute entries using 272 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 696 total bytes of memory
BGP activity 2/0 prefixes, 2/0 paths, scan interval 60 secs

Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
192.168.1.1     4        65001      20      18        3    0    0 00:12:50        1

R3#show ip bgp summary 
BGP router identifier 192.168.1.3, local AS number 65001
BGP table version is 3, main routing table version 3
2 network entries using 296 bytes of memory
2 path entries using 128 bytes of memory
2/2 BGP path/bestpath attribute entries using 272 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 696 total bytes of memory
BGP activity 2/0 prefixes, 2/0 paths, scan interval 60 secs

Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
192.168.1.1     4        65001      19      19        3    0    0 00:12:54        1

- Traffic between Spoke-R2 and Spoke-R3

R2#ping 10.10.3.3 so 10.10.2.2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.10.3.3, timeout is 2 seconds:
Packet sent with a source address of 10.10.2.2 
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 124/132/140 ms

R2#trace 10.10.3.3 so 10.10.2.2
Type escape sequence to abort.
Tracing the route to 10.10.3.3
VRF info: (vrf in name/id, vrf out name/id)
  1 192.168.1.3 144 msec *  124 msec         << Spoke R3 Tunnel IP

- DMVPN Status

R1#show dmvpn
Legend: Attrb --> S - Static, D - Dynamic, I - Incomplete
        N - NATed, L - Local, X - No Socket
        # Ent --> Number of NHRP entries with same NBMA peer
        NHS Status: E --> Expecting Replies, R --> Responding, W --> Waiting
        UpDn Time --> Up or Down Time for a Tunnel
======================================================================
Interface: Tunnel0, IPv4 NHRP Details 
Type:Hub, NHRP Peers:2, 

 # Ent  Peer NBMA Addr Peer Tunnel Add State  UpDn Tm Attrb
 ----- --------------- --------------- ----- -------- -----
     1 1.1.24.2            192.168.1.2    UP 00:15:42     D
     1 1.1.34.3            192.168.1.3    UP 00:15:42     D

R2#show dmvpn
Legend: Attrb --> S - Static, D - Dynamic, I - Incomplete
        N - NATed, L - Local, X - No Socket
        # Ent --> Number of NHRP entries with same NBMA peer
        NHS Status: E --> Expecting Replies, R --> Responding, W --> Waiting
        UpDn Time --> Up or Down Time for a Tunnel
======================================================================
Interface: Tunnel0, IPv4 NHRP Details 
Type:Spoke, NHRP Peers:2, 

 # Ent  Peer NBMA Addr Peer Tunnel Add State  UpDn Tm Attrb
 ----- --------------- --------------- ----- -------- -----
     2 1.1.34.3            192.168.1.3    UP 00:01:20   DT1
                           192.168.1.3    UP 00:01:20     D
     1 1.1.14.1            192.168.1.1    UP 00:15:47     S
*T1 - Route Installed

R3#show dmvpn
Legend: Attrb --> S - Static, D - Dynamic, I - Incomplete
        N - NATed, L - Local, X - No Socket
        # Ent --> Number of NHRP entries with same NBMA peer
        NHS Status: E --> Expecting Replies, R --> Responding, W --> Waiting
        UpDn Time --> Up or Down Time for a Tunnel
======================================================================
Interface: Tunnel0, IPv4 NHRP Details 
Type:Spoke, NHRP Peers:2, 

 # Ent  Peer NBMA Addr Peer Tunnel Add State  UpDn Tm Attrb
 ----- --------------- --------------- ----- -------- -----
     2 1.1.24.2            192.168.1.2    UP 00:01:23   DT1
                           192.168.1.2    UP 00:01:23     D
     1 1.1.14.1            192.168.1.1    UP 00:15:49     S
*T1 - Route Installed

- BGP Table

R1#show ip bgp
BGP table version is 8, local router ID is 192.168.1.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, 
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, 
              x best-external, a additional-path, c RIB-compressed, 
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *>  10.10.0.0/16     0.0.0.0                            32768 i
 s>  10.10.1.0/24     0.0.0.0                  0         32768 i
 s>i 10.10.2.0/24     192.168.1.2              0    100      0 i
 s>i 10.10.3.0/24     192.168.1.3              0    100      0 i

R2#show ip bgp
BGP table version is 3, local router ID is 192.168.1.2
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, 
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, 
              x best-external, a additional-path, c RIB-compressed, 
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *>i 10.10.0.0/16     192.168.1.1              0    100      0 i
 *>  10.10.2.0/24     0.0.0.0                  0         32768 i

R3#show ip bgp
BGP table version is 3, local router ID is 192.168.1.3
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, 
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, 
              x best-external, a additional-path, c RIB-compressed, 
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *>i 10.10.0.0/16     192.168.1.1              0    100      0 i
 *>  10.10.3.0/24     0.0.0.0                  0         32768 i

- Routing Table

R1#show ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area 
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       + - replicated route, % - next hop override

Gateway of last resort is 1.1.14.4 to network 0.0.0.0

S*    0.0.0.0/0 [1/0] via 1.1.14.4
      1.0.0.0/8 is variably subnetted, 2 subnets, 2 masks
C        1.1.14.0/24 is directly connected, FastEthernet0/0
L        1.1.14.1/32 is directly connected, FastEthernet0/0
      10.0.0.0/8 is variably subnetted, 5 subnets, 3 masks
B        10.10.0.0/16 [200/0] via 0.0.0.0, 00:17:13, Null0
C        10.10.1.0/24 is directly connected, FastEthernet0/1
L        10.10.1.1/32 is directly connected, FastEthernet0/1
B        10.10.2.0/24 [200/0] via 192.168.1.2, 00:16:05
B        10.10.3.0/24 [200/0] via 192.168.1.3, 00:16:57
      192.168.1.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.1.0/24 is directly connected, Tunnel0
L        192.168.1.1/32 is directly connected, Tunnel0

R2#show ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area 
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       + - replicated route, % - next hop override

Gateway of last resort is 1.1.24.4 to network 0.0.0.0

S*    0.0.0.0/0 [1/0] via 1.1.24.4
      1.0.0.0/8 is variably subnetted, 2 subnets, 2 masks
C        1.1.24.0/24 is directly connected, FastEthernet0/0
L        1.1.24.2/32 is directly connected, FastEthernet0/0
      10.0.0.0/8 is variably subnetted, 4 subnets, 3 masks
B        10.10.0.0/16 [200/0] via 192.168.1.1, 00:16:11
C        10.10.2.0/24 is directly connected, FastEthernet0/1
L        10.10.2.2/32 is directly connected, FastEthernet0/1
H        10.10.3.0/24 [250/1] via 192.168.1.3, 00:02:52, Tunnel0
      192.168.1.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.1.0/24 is directly connected, Tunnel0
L        192.168.1.2/32 is directly connected, Tunnel0

R3#show ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area 
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       + - replicated route, % - next hop override

Gateway of last resort is 1.1.34.4 to network 0.0.0.0

S*    0.0.0.0/0 [1/0] via 1.1.34.4
      1.0.0.0/8 is variably subnetted, 2 subnets, 2 masks
C        1.1.34.0/24 is directly connected, FastEthernet0/0
L        1.1.34.3/32 is directly connected, FastEthernet0/0
      10.0.0.0/8 is variably subnetted, 4 subnets, 3 masks
B        10.10.0.0/16 [200/0] via 192.168.1.1, 00:17:06
H        10.10.2.0/24 [250/1] via 192.168.1.2, 00:02:55, Tunnel0
C        10.10.3.0/24 is directly connected, FastEthernet0/1
L        10.10.3.3/32 is directly connected, FastEthernet0/1
      192.168.1.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.1.0/24 is directly connected, Tunnel0
L        192.168.1.3/32 is directly connected, Tunnel0

- NHRP Table

R1#show ip nhrp
192.168.1.2/32 via 192.168.1.2
   Tunnel0 created 00:17:49, expire 01:42:10
   Type: dynamic, Flags: unique registered used 
   NBMA address: 1.1.24.2 
192.168.1.3/32 via 192.168.1.3
   Tunnel0 created 00:17:49, expire 01:42:11
   Type: dynamic, Flags: unique registered used 
   NBMA address: 1.1.34.3 

R2#show ip nhrp
10.10.2.0/24 via 192.168.1.2
   Tunnel0 created 00:03:27, expire 01:56:32
   Type: dynamic, Flags: router unique local 
   NBMA address: 1.1.24.2 
    (no-socket) 
10.10.3.0/24 via 192.168.1.3
   Tunnel0 created 00:03:27, expire 01:56:31
   Type: dynamic, Flags: router used rib 
   NBMA address: 1.1.34.3 
192.168.1.1/32 via 192.168.1.1
   Tunnel0 created 00:18:28, never expire 
   Type: static, Flags: used 
   NBMA address: 1.1.14.1 
192.168.1.3/32 via 192.168.1.3
   Tunnel0 created 00:03:28, expire 01:56:32
   Type: dynamic, Flags: router implicit used 
   NBMA address: 1.1.34.3 

R3#show ip nhrp
10.10.2.0/24 via 192.168.1.2
   Tunnel0 created 00:03:29, expire 01:56:30
   Type: dynamic, Flags: router used rib 
   NBMA address: 1.1.24.2 
10.10.3.0/24 via 192.168.1.3
   Tunnel0 created 00:03:30, expire 01:56:29
   Type: dynamic, Flags: router unique local 
   NBMA address: 1.1.34.3 
    (no-socket) 
192.168.1.1/32 via 192.168.1.1
   Tunnel0 created 00:18:31, never expire 
   Type: static, Flags: used 
   NBMA address: 1.1.14.1 
192.168.1.2/32 via 192.168.1.2
   Tunnel0 created 00:03:31, expire 01:56:29
   Type: dynamic, Flags: router implicit used 
   NBMA address: 1.1.24.2 

 

DMVPN Phase 2

Phase 2 – Spoke to Spoke Design

DMVPN Phase 2 design introduced the ability for dynamic spoke-to-spoke tunnels without having the traffic go through the hub. Phase 2 improved on Phase 1 by allowing spokes to build a spoke-to-spoke tunnel on demand with these restrictions:

  • Spokes must use mGRE (Multipoint GRE) tunnels
  • Spokes must receive specific routes for all remote spoke subnets
  • The next hop of the entry in the routing table must list the remote spoke as the next hop

Check these links for

Here is the network topology for DMVPN phase 2 discussion and configuration.

DMVPN Phase 2 configuration with EIGRP

EIGRP Split Horizon Rule – The split horizon rule prohibits a router from advertising a route through an interface that the router itself uses to reach the destination. In DMVPN, Hub router learn route(s) from one spoke via Tunnel0 interface and have to advertise it to other spoke via same Tunnel0 interface. Thus, in order for DMVPN to work in Phase 2 with EIGRP, split horizon must be disabled on the tunnel interface using the “no ip split-horizon eigrp <asn>” command.

Next-hop self in EIGRP – The next hop for all of the routes must point to the remote spoke. This is the key to triggering the generation of a spoke-to-spoke tunnel. To instruct EIGRP to use the received next hop rather than itself, use the “no ip next-hop-self eigrp <asn>” command.

Configuration:
----------
R1 - Hub:
----------

hostname R1
!
ip cef
!
crypto isakmp policy 10
 encr aes
 authentication pre-share
 group 2
crypto isakmp key DmVpNpR3$h@r3dK3Y address 0.0.0.0        
!
!
crypto ipsec transform-set TRANSFORM-SET esp-aes esp-sha-hmac 
 mode transport
!
crypto ipsec profile PROTECT-GRE
 set transform-set TRANSFORM-SET 
!
interface Tunnel0
 bandwidth 4096
 ip address 192.168.1.1 255.255.255.0
 no ip redirects
 ip mtu 1400
 no ip next-hop-self eigrp 100
 no ip split-horizon eigrp 100
 ip nhrp authentication NhRp@UtH
 ip nhrp map multicast dynamic
 ip nhrp network-id 100
 ip tcp adjust-mss 1360
 tunnel source FastEthernet0/0
 tunnel mode gre multipoint
 tunnel key 123
 tunnel protection ipsec profile PROTECT-GRE
!
interface FastEthernet0/0
 ip address 1.1.14.1 255.255.255.0
!
interface FastEthernet0/1
 ip address 10.10.1.1 255.255.255.0
!
router eigrp 100
 network 10.10.1.1 0.0.0.0
 network 192.168.1.1 0.0.0.0
!
ip route 0.0.0.0 0.0.0.0 1.1.14.4

------------
R2 - Spoke:
------------

hostname R2
!
ip cef
!
crypto isakmp policy 10
 encr aes
 authentication pre-share
 group 2
crypto isakmp key DmVpNpR3$h@r3dK3Y address 0.0.0.0        
!
!
crypto ipsec transform-set TRANSFORM-SET esp-aes esp-sha-hmac 
 mode transport
!
crypto ipsec profile PROTECT-GRE
 set transform-set TRANSFORM-SET 
!
interface Tunnel0
 bandwidth 4096
 ip address 192.168.1.2 255.255.255.0
 no ip redirects
 ip mtu 1400
 ip nhrp authentication NhRp@UtH
 ip nhrp map multicast 1.1.14.1
 ip nhrp map 192.168.1.1 1.1.14.1
 ip nhrp network-id 100
 ip nhrp nhs 192.168.1.1
 ip tcp adjust-mss 1360
 tunnel source FastEthernet0/0
 tunnel mode gre multipoint
 tunnel key 123
 tunnel protection ipsec profile PROTECT-GRE
!
interface FastEthernet0/0
 ip address 1.1.24.2 255.255.255.0
!
interface FastEthernet0/1
 ip address 10.10.2.2 255.255.255.0
!
router eigrp 100
 network 10.10.2.2 0.0.0.0
 network 192.168.1.2 0.0.0.0
!
ip route 0.0.0.0 0.0.0.0 1.1.24.4

------------
R3 - Spoke:
------------

hostname R3
!
ip cef
!
crypto isakmp policy 10
 encr aes
 authentication pre-share
 group 2
crypto isakmp key DmVpNpR3$h@r3dK3Y address 0.0.0.0        
!
!
crypto ipsec transform-set TRANSFORM-SET esp-aes esp-sha-hmac 
 mode transport
!
crypto ipsec profile PROTECT-GRE
 set transform-set TRANSFORM-SET 
!
interface Tunnel0
 bandwidth 4096
 ip address 192.168.1.3 255.255.255.0
 no ip redirects
 ip mtu 1400
 ip nhrp authentication NhRp@UtH
 ip nhrp map multicast 1.1.14.1
 ip nhrp map 192.168.1.1 1.1.14.1
 ip nhrp network-id 100
 ip nhrp nhs 192.168.1.1
 ip tcp adjust-mss 1360
 tunnel source FastEthernet0/0
 tunnel mode gre multipoint
 tunnel key 123
 tunnel protection ipsec profile PROTECT-GRE
!
interface FastEthernet0/0
 ip address 1.1.34.3 255.255.255.0
!
interface FastEthernet0/1
 ip address 10.10.3.3 255.255.255.0
!
router eigrp 100
 network 10.10.3.3 0.0.0.0
 network 192.168.1.3 0.0.0.0
!
ip route 0.0.0.0 0.0.0.0 1.1.34.4

---------------
R4 - Internet:
---------------

hostname R4
!
interface FastEthernet0/0
 ip address 1.1.14.4 255.255.255.0
!
interface FastEthernet0/1
 ip address 1.1.24.4 255.255.255.0
!
interface FastEthernet1/0
 ip address 1.1.34.4 255.255.255.0
!
Verification:
- Hub and Spokes routers are using mGRE tunnels

R1#show int t0
Tunnel0 is up, line protocol is up 
  Hardware is Tunnel
  Internet address is 192.168.1.1/24
  MTU 17870 bytes, BW 4096 Kbit/sec, DLY 50000 usec, 
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation TUNNEL, loopback not set
  Keepalive not set
  Tunnel source 1.1.14.1 (FastEthernet0/0)
   Tunnel Subblocks:
      src-track:
         Tunnel0 source tracking subblock associated with FastEthernet0/0
          Set of tunnels with source FastEthernet0/0, 1 member (includes iterators), on interface <OK>
  Tunnel protocol/transport multi-GRE/IP

R2#show int t0
Tunnel0 is up, line protocol is up 
  Hardware is Tunnel
  Internet address is 192.168.1.2/24
  MTU 17870 bytes, BW 4096 Kbit/sec, DLY 50000 usec, 
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation TUNNEL, loopback not set
  Keepalive not set
  Tunnel source 1.1.24.2 (FastEthernet0/0)
   Tunnel Subblocks:
      src-track:
         Tunnel0 source tracking subblock associated with FastEthernet0/0
          Set of tunnels with source FastEthernet0/0, 1 member (includes iterators), on interface <OK>
  Tunnel protocol/transport multi-GRE/IP

R3#show int t0
Tunnel0 is up, line protocol is up 
  Hardware is Tunnel
  Internet address is 192.168.1.3/24
  MTU 17870 bytes, BW 4096 Kbit/sec, DLY 50000 usec, 
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation TUNNEL, loopback not set
  Keepalive not set
  Tunnel source 1.1.34.3 (FastEthernet0/0)
   Tunnel Subblocks:
      src-track:
         Tunnel0 source tracking subblock associated with FastEthernet0/0
          Set of tunnels with source FastEthernet0/0, 1 member (includes iterators), on interface <OK>
  Tunnel protocol/transport multi-GRE/IP

- Routing adjacency (EIGRP neighborship) is between Hub and Spokes only 

R1#show ip eigrp neighbors
EIGRP-IPv4 Neighbors for AS(100)
H   Address                 Interface              Hold Uptime   SRTT   RTO  Q  Seq
                                                   (sec)         (ms)       Cnt Num
1   192.168.1.2             Tu0                      13 05:09:22  201  1206  0  3
0   192.168.1.3             Tu0                      13 05:09:32  167  1002  0  4

R2#show ip eigrp neighbors
EIGRP-IPv4 Neighbors for AS(100)
H   Address                 Interface              Hold Uptime   SRTT   RTO  Q  Seq
                                                   (sec)         (ms)       Cnt Num
0   192.168.1.1             Tu0                      13 05:09:37  140   840  0  6

R3#show ip eigrp neighbors
EIGRP-IPv4 Neighbors for AS(100)
H   Address                 Interface              Hold Uptime   SRTT   RTO  Q  Seq
                                                   (sec)         (ms)       Cnt Num
0   192.168.1.1             Tu0                      11 05:09:49  156   936  0  6

- Routing Table (see next-hop IP on spoke routers to reach other spoke subnets) 

R1#show ip route eigrp | beg Gate
Gateway of last resort is 1.1.14.4 to network 0.0.0.0

      10.0.0.0/8 is variably subnetted, 4 subnets, 2 masks
D        10.10.2.0/24 [90/1907456] via 192.168.1.2, 05:11:20, Tunnel0
D        10.10.3.0/24 [90/1907456] via 192.168.1.3, 05:11:30, Tunnel0

R2#show ip route eigrp | beg Gate
Gateway of last resort is 1.1.24.4 to network 0.0.0.0

      10.0.0.0/8 is variably subnetted, 4 subnets, 2 masks
D        10.10.1.0/24 [90/1907456] via 192.168.1.1, 05:11:31, Tunnel0
D        10.10.3.0/24 [90/3187456] via 192.168.1.3, 05:11:31, Tunnel0

R3#show ip route eigrp | beg Gate
Gateway of last resort is 1.1.34.4 to network 0.0.0.0

      10.0.0.0/8 is variably subnetted, 4 subnets, 2 masks
D        10.10.1.0/24 [90/1907456] via 192.168.1.1, 05:11:42, Tunnel0
D        10.10.2.0/24 [90/3187456] via 192.168.1.2, 05:11:32, Tunnel0

- DMVPN Table
- Hub shows dynamic entries, spokes registered themselves with hub
- Spokes show static entries, static entry defined for hub

R1#show dmvpn
Legend: Attrb --> S - Static, D - Dynamic, I - Incomplete
        N - NATed, L - Local, X - No Socket
        # Ent --> Number of NHRP entries with same NBMA peer
        NHS Status: E --> Expecting Replies, R --> Responding, W --> Waiting
        UpDn Time --> Up or Down Time for a Tunnel
======================================================================
Interface: Tunnel0, IPv4 NHRP Details 
Type:Hub, NHRP Peers:2, 

 # Ent  Peer NBMA Addr Peer Tunnel Add State  UpDn Tm Attrb
 ----- --------------- --------------- ----- -------- -----
     1 1.1.24.2            192.168.1.2    UP 00:51:08     D
     1 1.1.34.3            192.168.1.3    UP 00:51:08     D

R2#show dmvpn
Legend: Attrb --> S - Static, D - Dynamic, I - Incomplete
        N - NATed, L - Local, X - No Socket
        # Ent --> Number of NHRP entries with same NBMA peer
        NHS Status: E --> Expecting Replies, R --> Responding, W --> Waiting
        UpDn Time --> Up or Down Time for a Tunnel
======================================================================
Interface: Tunnel0, IPv4 NHRP Details 
Type:Spoke, NHRP Peers:1, 

 # Ent  Peer NBMA Addr Peer Tunnel Add State  UpDn Tm Attrb
 ----- --------------- --------------- ----- -------- -----
     1 1.1.14.1            192.168.1.1    UP 00:51:28     S

R3#show dmvpn
Legend: Attrb --> S - Static, D - Dynamic, I - Incomplete
        N - NATed, L - Local, X - No Socket
        # Ent --> Number of NHRP entries with same NBMA peer
        NHS Status: E --> Expecting Replies, R --> Responding, W --> Waiting
        UpDn Time --> Up or Down Time for a Tunnel
======================================================================
Interface: Tunnel0, IPv4 NHRP Details 
Type:Spoke, NHRP Peers:1, 

 # Ent  Peer NBMA Addr Peer Tunnel Add State  UpDn Tm Attrb
 ----- --------------- --------------- ----- -------- -----
     1 1.1.14.1            192.168.1.1    UP 00:51:30     S

- Traffic between Spoke R2 and Spoke R3

Routing entry on R2 shows R3 subnet (10.10.3.0/24) is reachable via next-hop IP 192.168.1.3 (R3 Tunnel IP)

D        10.10.3.0/24 [90/3187456] via 192.168.1.3, 00:05:12, Tunnel0

Lets check CEF adjacency for next-hop IP 192.168.1.3

R2#show adjacency 192.168.1.3
Protocol Interface                 Address
IP       Tunnel0                   192.168.1.3(5) (incomplete)

The adjacency is showing incomplete. The incomplete adjacency triggers a CEF punt to the CPU for further processing (to resolve the address).

R2#show ip cef 192.168.1.3 internal 
192.168.1.0/24, epoch 0, flags attached, connected, cover dependents, need deagg, RIB[C], refcount 5, per-destination sharing
  sources: RIB 
  feature space:
   IPRM: 0x0003800C
  subblocks:
   gsb Connected chain head(1): 0x6A36ADFC
   Covered dependent prefixes: 3
     need deagg: 2
     notify cover updated: 1
  ifnums:
   Tunnel0(6)
  path 695C0994, path list 6A90BBB8, share 1/1, type connected prefix, for IPv4
  connected to Tunnel0, adjacency punt
  output chain: punt

This causes R2(Spoke) to send a resolution request to R1(Hub) for Spoke-R3’s NBMA address. The request gets forwarded from R1(Hub) to Spoke-R3. Spoke-R3 replies directly to Spoke-R2 with its mapping information. During this process, R2(Spoke) will send the actual data packet to R1(Hub) to be delivered to R3(Spoke) as a last-ditch effort to not drop the traffic. The first traceroute will look as below, the traffic traverse via hub.

R2#traceroute 10.10.3.3 so 10.10.2.2
Type escape sequence to abort.
Tracing the route to 10.10.3.3
VRF info: (vrf in name/id, vrf out name/id)
  1 192.168.1.1 180 msec 128 msec 152 msec    << Hub R1 Tunnel IP
  2 192.168.1.3 252 msec

- Debug nhrp packet shows following events:

Spoke-R2 sends resolution request via Tunnel0, it will reach to Hub-R1 as this is the only active tunnel on R2 at this moment. 
R2#
*Mar  1 00:29:49.567: NHRP: Send Resolution Request via Tunnel0 vrf 0, packet size: 88
*Mar  1 00:29:49.575:  src: 192.168.1.2, dst: 192.168.1.3
*Mar  1 00:29:49.583:  (F) afn: AF_IP(1), type: IP(800), hop: 255, ver: 1
*Mar  1 00:29:49.583:      shtl: 4(NSAP), sstl: 0(NSAP)
*Mar  1 00:29:49.587:      pktsz: 88 extoff: 52
*Mar  1 00:29:49.591:  (M) flags: "router auth src-stable nat ", reqid: 2 
*Mar  1 00:29:49.595:      src NBMA: 1.1.24.2
*Mar  1 00:29:49.595:      src protocol: 192.168.1.2, dst protocol: 192.168.1.3
*Mar  1 00:29:49.603:  (C-1) code: no error(0)
*Mar  1 00:29:49.603:        prefix: 32, mtu: 17870, hd_time: 7200
*Mar  1 00:29:49.603:        addr_len: 0(NSAP), subaddr_len: 0(NSAP), proto_len: 0, pref: 0

This request arrives at Hub-R1. Hub-R1 finds it is destined for destination IP 192.168.1.3 and Hub-R1 will forward it to Spoke-R3.  
R1#
*Mar  1 00:29:49.683: NHRP: Receive Resolution Request via Tunnel0 vrf 0, packet size: 88
*Mar  1 00:29:49.687:  (F) afn: AF_IP(1), type: IP(800), hop: 255, ver: 1
*Mar  1 00:29:49.691:      shtl: 4(NSAP), sstl: 0(NSAP)
*Mar  1 00:29:49.695:      pktsz: 88 extoff: 52
*Mar  1 00:29:49.695:  (M) flags: "router auth src-stable nat ", reqid: 2 
*Mar  1 00:29:49.699:      src NBMA: 1.1.24.2
*Mar  1 00:29:49.703:      src protocol: 192.168.1.2, dst protocol: 192.168.1.3
*Mar  1 00:29:49.707:  (C-1) code: no error(0)
*Mar  1 00:29:49.707:        prefix: 32, mtu: 17870, hd_time: 7200
*Mar  1 00:29:49.707:        addr_len: 0(NSAP), subaddr_len: 0(NSAP), proto_len: 0, pref: 0
*Mar  1 00:29:49.715: NHRP: Forwarding Resolution Request via Tunnel0 vrf 0, packet size: 108
*Mar  1 00:29:49.719:  src: 192.168.1.1, dst: 192.168.1.3
*Mar  1 00:29:49.723:  (F) afn: AF_IP(1), type: IP(800), hop: 254, ver: 1
*Mar  1 00:29:49.723:      shtl: 4(NSAP), sstl: 0(NSAP)
*Mar  1 00:29:49.723:     
R1# pktsz: 108 extoff: 52
*Mar  1 00:29:49.723:  (M) flags: "router auth src-stable nat ", reqid: 2 
*Mar  1 00:29:49.723:      src NBMA: 1.1.24.2
*Mar  1 00:29:49.723:      src protocol: 192.168.1.2, dst protocol: 192.168.1.3
*Mar  1 00:29:49.723:  (C-1) code: no error(0)
*Mar  1 00:29:49.723:        prefix: 32, mtu: 17870, hd_time: 7200
*Mar  1 00:29:49.723:        addr_len: 0(NSAP), subaddr_len: 0(NSAP), proto_len: 0, pref: 0

Then Spoke-R3 receives NHRP resolution request. This request contains Spoke-R2's details including NBMA and Tunnel IP of Spoke-R2. Spoke-R3 would initiate a direct DMVPN tunnel to R2 and send reply to R2.

R3#
*Mar  1 00:29:49.671: NHRP: Receive Resolution Request via Tunnel0 vrf 0, packet size: 108
*Mar  1 00:29:49.675:  (F) afn: AF_IP(1), type: IP(800), hop: 254, ver: 1
*Mar  1 00:29:49.675:      shtl: 4(NSAP), sstl: 0(NSAP)
*Mar  1 00:29:49.675:      pktsz: 108 extoff: 52
*Mar  1 00:29:49.675:  (M) flags: "router auth src-stable nat ", reqid: 2 
*Mar  1 00:29:49.675:      src NBMA: 1.1.24.2
*Mar  1 00:29:49.675:      src protocol: 192.168.1.2, dst protocol: 192.168.1.3
*Mar  1 00:29:49.675:  (C-1) code: no error(0)
*Mar  1 00:29:49.675:        prefix: 32, mtu: 17870, hd_time: 7200
*Mar  1 00:29:49.675:        addr_len: 0(NSAP), subaddr_len: 0(NSAP), proto_len: 0, pref: 0
*Mar  1 00:29:50.487: NHRP: Send Resolution Reply via Tunnel0 vrf 0, packet size: 136
*Mar  1 00:29:50.487:  src: 192.168.1.3, dst: 192.168.1.2
*Mar  1 00:29:50.487:  (F) afn: AF_IP(1), type: IP(800), hop: 255, ver: 1
*Mar  1 00:29:50.487:      shtl: 4(NSAP), sstl: 0(NSAP)
*Mar  1 00:29:50.487:      pktsz
R3#: 136 extoff: 60
*Mar  1 00:29:50.487:  (M) flags: "router auth dst-stable unique src-stable nat ", reqid: 2 
*Mar  1 00:29:50.487:      src NBMA: 1.1.24.2
*Mar  1 00:29:50.487:      src protocol: 192.168.1.2, dst protocol: 192.168.1.3
*Mar  1 00:29:50.487:  (C-1) code: no error(0)
*Mar  1 00:29:50.487:        prefix: 32, mtu: 17870, hd_time: 7200
*Mar  1 00:29:50.487:        addr_len: 4(NSAP), subaddr_len: 0(NSAP), proto_len: 4, pref: 0
*Mar  1 00:29:50.487:        client NBMA: 1.1.34.3
*Mar  1 00:29:50.487:        client protocol: 192.168.1.3

R2 receives reply and at this moment both R2 and R3 knows each other's NBMA IP and have direct dmvpn tunnel.

R2#
*Mar  1 00:29:50.635: NHRP: Receive Resolution Reply via Tunnel0 vrf 0, packet size: 136
*Mar  1 00:29:50.643:  (F) afn: AF_IP(1), type: IP(800), hop: 255, ver: 1
*Mar  1 00:29:50.647:      shtl: 4(NSAP), sstl: 0(NSAP)
*Mar  1 00:29:50.647:      pktsz: 136 extoff: 60
*Mar  1 00:29:50.651:  (M) flags: "router auth dst-stable unique src-stable nat ", reqid: 2 
*Mar  1 00:29:50.655:      src NBMA: 1.1.24.2
*Mar  1 00:29:50.655:      src protocol: 192.168.1.2, dst protocol: 192.168.1.3
*Mar  1 00:29:50.663:  (C-1) code: no error(0)
*Mar  1 00:29:50.663:        prefix: 32, mtu: 17870, hd_time: 7200
*Mar  1 00:29:50.663:        addr_len: 4(NSAP), subaddr_len: 0(NSAP), proto_len: 4, pref: 0
*Mar  1 00:29:50.663:        client NBMA: 1.1.34.3
*Mar  1 00:29:50.663:        client protocol: 192.168.1.3 *  140 msec

- Check DMVPN status now

R1#show dmvpn
Legend: Attrb --> S - Static, D - Dynamic, I - Incomplete
        N - NATed, L - Local, X - No Socket
        # Ent --> Number of NHRP entries with same NBMA peer
        NHS Status: E --> Expecting Replies, R --> Responding, W --> Waiting
        UpDn Time --> Up or Down Time for a Tunnel
=====================================================================
Interface: Tunnel0, IPv4 NHRP Details 
Type:Hub, NHRP Peers:2, 

 # Ent  Peer NBMA Addr Peer Tunnel Add State  UpDn Tm Attrb
 ----- --------------- --------------- ----- -------- -----
     1 1.1.24.2            192.168.1.2    UP 05:15:09     D
     1 1.1.34.3            192.168.1.3    UP 05:15:19     D

R2#show dmvpn
Legend: Attrb --> S - Static, D - Dynamic, I - Incomplete
        N - NATed, L - Local, X - No Socket
        # Ent --> Number of NHRP entries with same NBMA peer
        NHS Status: E --> Expecting Replies, R --> Responding, W --> Waiting
        UpDn Time --> Up or Down Time for a Tunnel
======================================================================
Interface: Tunnel0, IPv4 NHRP Details 
Type:Spoke, NHRP Peers:2, 

 # Ent  Peer NBMA Addr Peer Tunnel Add State  UpDn Tm Attrb
 ----- --------------- --------------- ----- -------- -----
     1 1.1.14.1            192.168.1.1    UP 05:15:26     S
     1 1.1.34.3            192.168.1.3    UP 00:34:02     D

R3#show dmvpn
Legend: Attrb --> S - Static, D - Dynamic, I - Incomplete
        N - NATed, L - Local, X - No Socket
        # Ent --> Number of NHRP entries with same NBMA peer
        NHS Status: E --> Expecting Replies, R --> Responding, W --> Waiting
        UpDn Time --> Up or Down Time for a Tunnel
======================================================================
Interface: Tunnel0, IPv4 NHRP Details 
Type:Spoke, NHRP Peers:2, 

 # Ent  Peer NBMA Addr Peer Tunnel Add State  UpDn Tm Attrb
 ----- --------------- --------------- ----- -------- -----
     1 1.1.14.1            192.168.1.1    UP 05:15:51     S
     1 1.1.24.2            192.168.1.2    UP 00:34:17     D

- NHRP State

R1#show ip nhrp
192.168.1.2/32 via 192.168.1.2
   Tunnel0 created 05:16:06, expire 01:23:19
   Type: dynamic, Flags: unique registered used 
   NBMA address: 1.1.24.2 
192.168.1.3/32 via 192.168.1.3
   Tunnel0 created 05:16:16, expire 01:23:19
   Type: dynamic, Flags: unique registered used 
   NBMA address: 1.1.34.3 

R2#show ip nhrp               
192.168.1.1/32 via 192.168.1.1
   Tunnel0 created 05:17:28, never expire 
   Type: static, Flags: used 
   NBMA address: 1.1.14.1 
192.168.1.2/32 via 192.168.1.2
   Tunnel0 created 00:35:19, expire 01:24:40
   Type: dynamic, Flags: router unique local 
   NBMA address: 1.1.24.2 
    (no-socket) 
192.168.1.3/32 via 192.168.1.3
   Tunnel0 created 00:35:20, expire 01:24:40
   Type: dynamic, Flags: router used 
   NBMA address: 1.1.34.3 

R3#show ip nhrp
192.168.1.1/32 via 192.168.1.1
   Tunnel0 created 05:17:14, never expire 
   Type: static, Flags: used 
   NBMA address: 1.1.14.1 
192.168.1.2/32 via 192.168.1.2
   Tunnel0 created 00:35:06, expire 01:24:34
   Type: dynamic, Flags: router used 
   NBMA address: 1.1.24.2 
192.168.1.3/32 via 192.168.1.3
   Tunnel0 created 00:35:05, expire 01:24:34
   Type: dynamic, Flags: router unique local 
   NBMA address: 1.1.34.3 
    (no-socket)

- Ping and traceroute between Spoke-R2 and Spoke-R3
- Now traffic is going through direct spoke-to-spoke tunnel

R2#ping 10.10.3.3 so 10.10.2.2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.10.3.3, timeout is 2 seconds:
Packet sent with a source address of 10.10.2.2 
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 116/128/136 ms

R2#trace 10.10.3.3 so 10.10.2.2
Type escape sequence to abort.
Tracing the route to 10.10.3.3
VRF info: (vrf in name/id, vrf out name/id)
  1 192.168.1.3 156 msec *  144 msec         << R3 Tunnel0 IP

- CEF would look as below after direct spoke-to-spoke DMVPN tunnel creation

R2#show adjacency 192.168.1.3
Protocol Interface                 Address
IP       Tunnel0                   192.168.1.3(11)

R2#show ip cef 192.168.1.3 internal
192.168.1.3/32, epoch 0, flags attached, refcount 5, per-destination sharing
  sources: Adj 
  subblocks:
   Adj source: IP midchain out of Tunnel0, addr 192.168.1.3 6AFD37A0
    Dependent covered prefix type adjfib, cover 192.168.1.0/24
  ifnums:
   Tunnel0(6): 192.168.1.3
  path 6AFD6068, path list 6AFD78A0, share 1/1, type adjacency prefix, for IPv4
  attached to Tunnel0, adjacency IP midchain out of Tunnel0, addr 192.168.1.3 6AFD37A0
  output chain: IP midchain out of Tunnel0, addr 192.168.1.3 6AFD37A0 IP adj out of FastEthernet0/0, addr 1.1.24.4 69237240
R2#
DMVPN Phase 2 configuration with OSPF
  • Configure Hub router as DR. If there is 2nd hub for redundancy, 2nd hub will be BDR.
  • Configure Spokes routers with ospf priority as ZERO so that they should not participate in DR election and always act as DROTHER.
  • All spoke routers will have OSPF adjacencies with DR and BDR only.
Configuration:
---------
R1 - Hub:
---------

hostname R1
!
ip cef
!
crypto isakmp policy 10
 encr aes
 authentication pre-share
 group 2
crypto isakmp key DmVpNpR3$h@r3dK3Y address 0.0.0.0        
!
!
crypto ipsec transform-set TRANSFORM-SET esp-aes esp-sha-hmac 
 mode transport
!
crypto ipsec profile PROTECT-GRE
 set transform-set TRANSFORM-SET 
!
interface Tunnel0
 bandwidth 4096
 ip address 192.168.1.1 255.255.255.0
 no ip redirects
 ip mtu 1400
 ip nhrp authentication NhRp@UtH
 ip nhrp map multicast dynamic
 ip nhrp network-id 100
 ip tcp adjust-mss 1360
 ip ospf network broadcast
 ip ospf priority 255
 tunnel source FastEthernet0/0
 tunnel mode gre multipoint
 tunnel key 123
 tunnel protection ipsec profile PROTECT-GRE
!
interface FastEthernet0/0
 ip address 1.1.14.1 255.255.255.0
!
interface FastEthernet0/1
 ip address 10.10.1.1 255.255.255.0
!
router ospf 1
 network 10.10.1.1 0.0.0.0 area 0
 network 192.168.1.1 0.0.0.0 area 0
!
ip route 0.0.0.0 0.0.0.0 1.1.14.4

------------
R2 - Spoke:
------------

hostname R2
!
ip cef
!
crypto isakmp policy 10
 encr aes
 authentication pre-share
 group 2
crypto isakmp key DmVpNpR3$h@r3dK3Y address 0.0.0.0        
!
!
crypto ipsec transform-set TRANSFORM-SET esp-aes esp-sha-hmac 
 mode transport
!
crypto ipsec profile PROTECT-GRE
 set transform-set TRANSFORM-SET 
!
interface Tunnel0
 bandwidth 4096
 ip address 192.168.1.2 255.255.255.0
 no ip redirects
 ip mtu 1400
 ip nhrp authentication NhRp@UtH
 ip nhrp map multicast 1.1.14.1
 ip nhrp map 192.168.1.1 1.1.14.1
 ip nhrp network-id 100
 ip nhrp nhs 192.168.1.1
 ip tcp adjust-mss 1360
 ip ospf network broadcast
 ip ospf priority 0
 tunnel source FastEthernet0/0
 tunnel mode gre multipoint
 tunnel key 123
 tunnel protection ipsec profile PROTECT-GRE
!
interface FastEthernet0/0
 ip address 1.1.24.2 255.255.255.0
!
interface FastEthernet0/1
 ip address 10.10.2.2 255.255.255.0
!
router ospf 1
 network 10.10.2.2 0.0.0.0 area 0
 network 192.168.1.2 0.0.0.0 area 0
!
ip route 0.0.0.0 0.0.0.0 1.1.24.4

------------
R3 - Spoke:
------------

hostname R3
!
ip cef
!
crypto isakmp policy 10
 encr aes
 authentication pre-share
 group 2
crypto isakmp key DmVpNpR3$h@r3dK3Y address 0.0.0.0        
!
!
crypto ipsec transform-set TRANSFORM-SET esp-aes esp-sha-hmac 
 mode transport
!
crypto ipsec profile PROTECT-GRE
 set transform-set TRANSFORM-SET 
!
interface Tunnel0
 bandwidth 4096
 ip address 192.168.1.3 255.255.255.0
 no ip redirects
 ip mtu 1400
 ip nhrp authentication NhRp@UtH
 ip nhrp map multicast 1.1.14.1
 ip nhrp map 192.168.1.1 1.1.14.1
 ip nhrp network-id 100
 ip nhrp nhs 192.168.1.1
 ip tcp adjust-mss 1360
 ip ospf network broadcast
 ip ospf priority 0
 tunnel source FastEthernet0/0
 tunnel mode gre multipoint
 tunnel key 123
 tunnel protection ipsec profile PROTECT-GRE
!
interface FastEthernet0/0
 ip address 1.1.34.3 255.255.255.0
!
interface FastEthernet0/1
 ip address 10.10.3.3 255.255.255.0
!
router ospf 1
 network 10.10.3.3 0.0.0.0 area 0
 network 192.168.1.3 0.0.0.0 area 0
!
ip route 0.0.0.0 0.0.0.0 1.1.34.4
!
Verification:
- OSPF Tunnel Network Type

R1#show ip ospf int t0
Tunnel0 is up, line protocol is up 
  Internet Address 192.168.1.1/24, Area 0, Attached via Network Statement
  Process ID 1, Router ID 192.168.1.1, Network Type BROADCAST, Cost: 24
  Topology-MTID    Cost    Disabled    Shutdown      Topology Name
        0           24        no          no            Base
  Transmit Delay is 1 sec, State DR, Priority 255
  Designated Router (ID) 192.168.1.1, Interface address 192.168.1.1
  No backup designated router on this network
  Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5
    oob-resync timeout 40
    Hello due in 00:00:04
  Supports Link-local Signaling (LLS)
  Cisco NSF helper support enabled
  IETF NSF helper support enabled
  Index 2/2, flood queue length 0
  Next 0x0(0)/0x0(0)
  Last flood scan length is 1, maximum is 1
  Last flood scan time is 4 msec, maximum is 4 msec
  Neighbor Count is 2, Adjacent neighbor count is 2 
    Adjacent with neighbor 192.168.1.2
    Adjacent with neighbor 192.168.1.3
  Suppress hello for 0 neighbor(s)

R2#show ip ospf int t0
Tunnel0 is up, line protocol is up 
  Internet Address 192.168.1.2/24, Area 0, Attached via Network Statement
  Process ID 1, Router ID 192.168.1.2, Network Type BROADCAST, Cost: 24
  Topology-MTID    Cost    Disabled    Shutdown      Topology Name
        0           24        no          no            Base
  Transmit Delay is 1 sec, State DROTHER, Priority 0
  Designated Router (ID) 192.168.1.1, Interface address 192.168.1.1
  No backup designated router on this network
  Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5
    oob-resync timeout 40
    Hello due in 00:00:01
  Supports Link-local Signaling (LLS)
  Cisco NSF helper support enabled
  IETF NSF helper support enabled
  Index 2/2, flood queue length 0
  Next 0x0(0)/0x0(0)
  Last flood scan length is 1, maximum is 1
  Last flood scan time is 4 msec, maximum is 4 msec
  Neighbor Count is 1, Adjacent neighbor count is 1 
    Adjacent with neighbor 192.168.1.1  (Designated Router)
  Suppress hello for 0 neighbor(s)

R3#show ip ospf int t0
Tunnel0 is up, line protocol is up 
  Internet Address 192.168.1.3/24, Area 0, Attached via Network Statement
  Process ID 1, Router ID 192.168.1.3, Network Type BROADCAST, Cost: 24
  Topology-MTID    Cost    Disabled    Shutdown      Topology Name
        0           24        no          no            Base
  Transmit Delay is 1 sec, State DROTHER, Priority 0
  Designated Router (ID) 192.168.1.1, Interface address 192.168.1.1
  No backup designated router on this network
  Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5
    oob-resync timeout 40
    Hello due in 00:00:07
  Supports Link-local Signaling (LLS)
  Cisco NSF helper support enabled
  IETF NSF helper support enabled
  Index 2/2, flood queue length 0
  Next 0x0(0)/0x0(0)
  Last flood scan length is 1, maximum is 1
  Last flood scan time is 4 msec, maximum is 4 msec
  Neighbor Count is 1, Adjacent neighbor count is 1 
    Adjacent with neighbor 192.168.1.1  (Designated Router)
  Suppress hello for 0 neighbor(s)

- OSPF Adjacencies

R1#show ip ospf neighbor 
Neighbor ID     Pri   State           Dead Time   Address         Interface
192.168.1.2       0   FULL/DROTHER    00:00:32    192.168.1.2     Tunnel0
192.168.1.3       0   FULL/DROTHER    00:00:32    192.168.1.3     Tunnel0

R2#show ip ospf neighbor 
Neighbor ID     Pri   State           Dead Time   Address         Interface
192.168.1.1     255   FULL/DR         00:00:31    192.168.1.1     Tunnel0

R3#show ip ospf neighbor 
Neighbor ID     Pri   State           Dead Time   Address         Interface
192.168.1.1     255   FULL/DR         00:00:39    192.168.1.1     Tunnel0

- Routing Table

R1#show ip route ospf | beg Gate
Gateway of last resort is 1.1.14.4 to network 0.0.0.0

      10.0.0.0/8 is variably subnetted, 4 subnets, 2 masks
O        10.10.2.0/24 [110/25] via 192.168.1.2, 00:25:23, Tunnel0
O        10.10.3.0/24 [110/25] via 192.168.1.3, 00:25:23, Tunnel0

R2#show ip route ospf | beg Gate
Gateway of last resort is 1.1.24.4 to network 0.0.0.0

      10.0.0.0/8 is variably subnetted, 4 subnets, 2 masks
O        10.10.1.0/24 [110/25] via 192.168.1.1, 00:25:28, Tunnel0
O        10.10.3.0/24 [110/25] via 192.168.1.3, 00:25:18, Tunnel0

R3#show ip route ospf | beg Gate
Gateway of last resort is 1.1.34.4 to network 0.0.0.0

      10.0.0.0/8 is variably subnetted, 4 subnets, 2 masks
O        10.10.1.0/24 [110/25] via 192.168.1.1, 00:25:30, Tunnel0
O        10.10.2.0/24 [110/25] via 192.168.1.2, 00:25:30, Tunnel0

- Traffic between Spoke R2 and Spoke R3

R2#ping 10.10.3.3 so 10.10.2.2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.10.3.3, timeout is 2 seconds:
Packet sent with a source address of 10.10.2.2 
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 116/123/128 ms

R2#trace 10.10.3.3 so 10.10.2.2
Type escape sequence to abort.
Tracing the route to 10.10.3.3
VRF info: (vrf in name/id, vrf out name/id)
  1 192.168.1.3 144 msec *  128 msec         << R3 Tunnel IP
DMVPN Phase 2 configuration with BGP
  • Hub router would be BGP Route-Reflector server
  • Spoke routers would be BGP Route-Reflector clients
  • All tunnels would be mGRE
  • You can use eBGP also, where Hub and Spoke routers could be configured in different autonomous system number (ASN). Here we will use iBGP only.
-----------
R1 - Hub:
-----------

hostname R1
!
ip cef
!
crypto isakmp policy 10
 encr aes
 authentication pre-share
 group 2
crypto isakmp key DmVpNpR3$h@r3dK3Y address 0.0.0.0        
!
!
crypto ipsec transform-set TRANSFORM-SET esp-aes esp-sha-hmac 
 mode transport
!
crypto ipsec profile PROTECT-GRE
 set transform-set TRANSFORM-SET 
!
interface Tunnel0
 bandwidth 4096
 ip address 192.168.1.1 255.255.255.0
 no ip redirects
 ip mtu 1400
 ip nhrp authentication NhRp@UtH
 ip nhrp map multicast dynamic
 ip nhrp network-id 100
 ip tcp adjust-mss 1360
 tunnel source FastEthernet0/0
 tunnel mode gre multipoint
 tunnel key 123
 tunnel protection ipsec profile PROTECT-GRE
!
interface FastEthernet0/0
 ip address 1.1.14.1 255.255.255.0
!
interface FastEthernet0/1
 ip address 10.10.1.1 255.255.255.0
!
router bgp 65001
 bgp router-id 192.168.1.1
 bgp log-neighbor-changes
 bgp listen range 192.168.1.0/24 peer-group DMVPN-SPOKES
 bgp listen limit 50
 network 10.10.1.0 mask 255.255.255.0
 neighbor DMVPN-SPOKES peer-group
 neighbor DMVPN-SPOKES remote-as 65001
 neighbor DMVPN-SPOKES route-reflector-client
!
ip route 0.0.0.0 0.0.0.0 1.1.14.4

------------
R2 - Spoke:
------------

hostname R2
!
ip cef
!
crypto isakmp policy 10
 encr aes
 authentication pre-share
 group 2
crypto isakmp key DmVpNpR3$h@r3dK3Y address 0.0.0.0        
!
!
crypto ipsec transform-set TRANSFORM-SET esp-aes esp-sha-hmac 
 mode transport
!
crypto ipsec profile PROTECT-GRE
 set transform-set TRANSFORM-SET 
!
interface Tunnel0
 bandwidth 4096
 ip address 192.168.1.2 255.255.255.0
 no ip redirects
 ip mtu 1400
 ip nhrp authentication NhRp@UtH
 ip nhrp map multicast 1.1.14.1
 ip nhrp map 192.168.1.1 1.1.14.1
 ip nhrp network-id 100
 ip nhrp nhs 192.168.1.1
 ip tcp adjust-mss 1360
 tunnel source FastEthernet0/0
 tunnel mode gre multipoint
 tunnel key 123
 tunnel protection ipsec profile PROTECT-GRE
!
interface FastEthernet0/0
 ip address 1.1.24.2 255.255.255.0
!
interface FastEthernet0/1
 ip address 10.10.2.2 255.255.255.0
!
router bgp 65001
 bgp router-id 192.168.1.2
 bgp log-neighbor-changes
 network 10.10.2.0 mask 255.255.255.0
 neighbor 192.168.1.1 remote-as 65001
!
ip route 0.0.0.0 0.0.0.0 1.1.24.4

------------
R3 - Spoke:
------------

hostname R3
!
ip cef
!
crypto isakmp policy 10
 encr aes
 authentication pre-share
 group 2
crypto isakmp key DmVpNpR3$h@r3dK3Y address 0.0.0.0        
!
!
crypto ipsec transform-set TRANSFORM-SET esp-aes esp-sha-hmac 
 mode transport
!
crypto ipsec profile PROTECT-GRE
 set transform-set TRANSFORM-SET 
!
interface Tunnel0
 bandwidth 4096
 ip address 192.168.1.3 255.255.255.0
 no ip redirects
 ip mtu 1400
 ip nhrp authentication NhRp@UtH
 ip nhrp map multicast 1.1.14.1
 ip nhrp map 192.168.1.1 1.1.14.1
 ip nhrp network-id 100
 ip nhrp nhs 192.168.1.1
 ip tcp adjust-mss 1360
 tunnel source FastEthernet0/0
 tunnel mode gre multipoint
 tunnel key 123
 tunnel protection ipsec profile PROTECT-GRE
!
interface FastEthernet0/0
 ip address 1.1.34.3 255.255.255.0
!
interface FastEthernet0/1
 ip address 10.10.3.3 255.255.255.0
!
router bgp 65001
 bgp router-id 192.168.1.3
 bgp log-neighbor-changes
 network 10.10.3.0 mask 255.255.255.0
 neighbor 192.168.1.1 remote-as 65001
!
ip route 0.0.0.0 0.0.0.0 1.1.34.4
Verification:
- BGP Neighborship between Hub and Spokes only

R1#show ip bgp sum
BGP router identifier 192.168.1.1, local AS number 65001
BGP table version is 4, main routing table version 4
3 network entries using 444 bytes of memory
3 path entries using 192 bytes of memory
2/2 BGP path/bestpath attribute entries using 272 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 908 total bytes of memory
BGP activity 3/0 prefixes, 3/0 paths, scan interval 60 secs

Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
*192.168.1.2    4        65001      11      15        4    0    0 00:06:11        1
*192.168.1.3    4        65001      12      13        4    0    0 00:06:21        1
* Dynamically created based on a listen range command
Dynamically created neighbors: 2, Subnet ranges: 1

BGP peergroup DMVPN-SPOKES listen range group members: 
  192.168.1.0/24 

Total dynamically created neighbors: 2/(50 max), Subnet ranges: 1

R2#show ip bgp sum
BGP router identifier 192.168.1.2, local AS number 65001
BGP table version is 4, main routing table version 4
3 network entries using 444 bytes of memory
3 path entries using 192 bytes of memory
2/2 BGP path/bestpath attribute entries using 272 bytes of memory
1 BGP rrinfo entries using 24 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 932 total bytes of memory
BGP activity 3/0 prefixes, 3/0 paths, scan interval 60 secs

Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
192.168.1.1     4        65001      15      11        4    0    0 00:06:17        2

R3#show ip bgp sum
BGP router identifier 192.168.1.3, local AS number 65001
BGP table version is 4, main routing table version 4
3 network entries using 444 bytes of memory
3 path entries using 192 bytes of memory
2/2 BGP path/bestpath attribute entries using 272 bytes of memory
1 BGP rrinfo entries using 24 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 932 total bytes of memory
BGP activity 3/0 prefixes, 3/0 paths, scan interval 60 secs

Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
192.168.1.1     4        65001      13      12        4    0    0 00:06:29        2

- BGP Table

R1#show ip bgp
BGP table version is 4, local router ID is 192.168.1.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, 
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, 
              x best-external, a additional-path, c RIB-compressed, 
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *>  10.10.1.0/24     0.0.0.0                  0         32768 i
 *>i 10.10.2.0/24     192.168.1.2              0    100      0 i
 *>i 10.10.3.0/24     192.168.1.3              0    100      0 i

R2#show ip bgp
BGP table version is 4, local router ID is 192.168.1.2
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, 
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, 
              x best-external, a additional-path, c RIB-compressed, 
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *>i 10.10.1.0/24     192.168.1.1              0    100      0 i
 *>  10.10.2.0/24     0.0.0.0                  0         32768 i
 *>i 10.10.3.0/24     192.168.1.3              0    100      0 i

R3#show ip bgp
BGP table version is 4, local router ID is 192.168.1.3
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, 
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, 
              x best-external, a additional-path, c RIB-compressed, 
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *>i 10.10.1.0/24     192.168.1.1              0    100      0 i
 *>i 10.10.2.0/24     192.168.1.2              0    100      0 i
 *>  10.10.3.0/24     0.0.0.0                  0         32768 i

- Routing Table

R1#show ip route bgp | beg Gate
Gateway of last resort is 1.1.14.4 to network 0.0.0.0

      10.0.0.0/8 is variably subnetted, 4 subnets, 2 masks
B        10.10.2.0/24 [200/0] via 192.168.1.2, 00:07:09
B        10.10.3.0/24 [200/0] via 192.168.1.3, 00:08:11

R2#show ip route bgp | beg Gate
Gateway of last resort is 1.1.24.4 to network 0.0.0.0

      10.0.0.0/8 is variably subnetted, 4 subnets, 2 masks
B        10.10.1.0/24 [200/0] via 192.168.1.1, 00:07:16
B        10.10.3.0/24 [200/0] via 192.168.1.3, 00:07:16

R3#show ip route bgp | beg Gate
Gateway of last resort is 1.1.34.4 to network 0.0.0.0

      10.0.0.0/8 is variably subnetted, 4 subnets, 2 masks
B        10.10.1.0/24 [200/0] via 192.168.1.1, 00:08:20
B        10.10.2.0/24 [200/0] via 192.168.1.2, 00:07:18

- Traffic between Spoke R2 and Spoke R3

R2#ping 10.10.3.3 so 10.10.2.2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.10.3.3, timeout is 2 seconds:
Packet sent with a source address of 10.10.2.2 
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 108/120/128 ms

R2#trace 10.10.3.3 so 10.10.2.2
Type escape sequence to abort.
Tracing the route to 10.10.3.3
VRF info: (vrf in name/id, vrf out name/id)
  1 192.168.1.3 132 msec *  120 msec        << R3 Tunnel IP

DMVPN Phase 2 deployment provides direct spoke-to-spoke tunnels, but one of the limitations is maintaining full routing tables on the spokes. Each route for remote spoke networks needs to be a specific route with the next hop pointing to the remote spoke’s tunnel address. This prevents the hub from being able to send down a summarized route to the spokes for a more concise routing table. This limitation is addressed in DMVPN Phase 3 design, which we will discuss later.