--- title: "Cisco NX-OS iBGP VXLAN EVPN Configurations" slug: "cisco-nx-os-ibgp-vxlan-evpn-configurations" description: "Explore NX-OS configurations for VAST EVPN/VXLAN fabric, covering OSPF, VLAN mapping, and PIM for optimized network performance and reliability." updated: 2026-10-27T19:37:32Z published: 2026-10-28T18:36:57Z canonical: "docs.vastdata.com/cisco-nx-os-ibgp-vxlan-evpn-configurations" --- > ## Documentation Index > Fetch the complete documentation index at: https://docs.vastdata.com/llms.txt > Use this file to discover all available pages before exploring further. # Converged Cisco NX-OS - iBGP EVPN VXLAN ## NX-OS reference configurations for the converged VAST EVPN/VXLAN fabric. Replace placeholders (`<asn>`, `<tenant-vrf>`, `<vip-prefix>`, `<vtep>`, `<spine-lo0>`, `Ethernet<x>`, etc.) per site. --- ## Platform features Enable on all switches before overlay config: ```plaintext feature nxapi nv overlay evpn feature ospf feature bgp feature pim feature interface-vlan feature vn-segment-vlan-based feature bfd feature nv overlay ``` MTU **9216** on fabric and host-facing ports. OSPF process `UNDERLAY` with BFD on spine-leaf /30 links. --- ## VLAN / VNI mapping (leaves) ```plaintext vlan 10  name VAST-Storage_VLAN_10  vn-segment 30010 vlan 69  name VAST-Storage-Network  vn-segment 30069 vlan 100  name vip-pool  vn-segment 100 vlan 2000  vn-segment 50000 ``` | VLAN | VNI | Purpose | | --- | --- | --- | | 10 | 30010 | Native on storage trunks (discovery/infra) | | 69 | 30069 | RoCEv2 inter-EBox backend | | 100 | 100 | VIP pool + Anycast SVI | | 2000 | 50000 | L3 VRF (`associate-vrf` on NVE) | ```plaintext evpn  vni 100 l2    rd auto    route-target import auto    route-target export auto  vni 30010 l2    rd auto    route-target import auto    route-target export auto  vni 30069 l2    rd auto    route-target import auto    route-target export auto ``` Spines carry VLAN 2000 / VNI 50000 only (L3 VNI). L2 VNIs terminate on leaves. --- ## OSPF underlay OSPF carries loopback and /30 reachability only - not VIP subnets or client prefixes. ```plaintext router ospf UNDERLAY  bfd  router-id interface loopback0  ip address /32  ip router ospf UNDERLAY area 0.0.0.0  ip pim sparse-mode interface loopback1  description VTEP  ip address /32  ip router ospf UNDERLAY area 0.0.0.0  ip pim sparse-mode interface Ethernet          ! spine uplink  mtu 9216  ip address /30  ip ospf network point-to-point  ip router ospf UNDERLAY area 0.0.0.0  ip pim sparse-mode  priority-flow-control mode on  service-policy type qos input QOS_CLASSIFICATION  service-policy type queuing output QOS_EGRESS_PORT ``` Out of OSPF: tenant VRF, VIP-pool SVI, client ports, mgmt0. --- ## NVE / VXLAN (leaf) ```plaintext interface nve1  no shutdown  host-reachability protocol bgp  source-interface loopback1  member vni 100    mcast-group  member vni 30010    mcast-group  member vni 30069    mcast-group  member vni 50000 associate-vrf ``` `host-reachability protocol bgp` - BGP EVPN tells NVE which VTEP owns each MAC/IP. --- ## Spine (route reflector) Spines are EVPN route reflectors. No host ports. L3 VNI only on NVE. ```plaintext interface nve1  no shutdown  host-reachability protocol bgp  source-interface loopback1  member vni 50000 associate-vrf router bgp  neighbor    remote-as    update-source loopback0    address-family l2vpn evpn      send-community extended      route-reflector-client    ! repeat per leaf ``` Inter-spine iBGP EVPN between spine loopback0s. Spines do not run OSPF to each other - they learn leaf loopbacks via leaf adjacencies. --- ## Tenant VRF + Anycast gateway (leaves) ```plaintext fabric forwarding anycast-gateway-mac 2020.0000.00aa vrf context  vni 50000  rd auto  address-family ipv4 unicast    route-target both auto evpn interface Vlan100  vrf member  mtu 9216  ip address / tag  fabric forwarding mode anycast-gateway  arp gratuitous accept route-map permit 10  match tag ``` EBox ARPs on VLAN 100. Leaf learns MAC+IP and originates EVPN Type-2. Same Anycast MAC on all leaves. --- ## iBGP EVPN (leaf) ```plaintext router bgp  router-id  neighbor    remote-as    update-source loopback0    address-family l2vpn evpn      send-community extended    ! repeat per spine  vrf    address-family ipv4 unicast      advertise l2vpn evpn      redistribute direct route-map      aggregate-address summary-only      maximum-paths ``` --- ## eBGP spine to client ```plaintext interface Ethernet.150  mtu 9216  encapsulation dot1q 150  vrf member  ip address / ip prefix-list PL-VIP-EXPORT seq 10 permit route-map permit 10  match ip address prefix-list PL-VIP-EXPORT router bgp  vrf    address-family ipv4 unicast      aggregate-address summary-only      advertise l2vpn evpn      maximum-paths    neighbor      remote-as external      address-family ipv4 unicast        route-map out ``` Clients receive the aggregate VIP prefix. Individual /32 VIPs stay in EVPN inside the fabric. --- ## PIM anycast-RP (when NVE uses mcast-group) Required when L2 VNIs use `mcast-group` for BUM. Skip if using ingress replication. ```plaintext interface loopback254  ip address /32  ip router ospf UNDERLAY area 0.0.0.0  ip pim sparse-mode ip pim rp-address group-list ip pim anycast-rp ! repeat per spine ``` PIM sparse-mode on underlay /30s and loopbacks. --- ## RoCE / QoS / PFC Apply to storage trunks and spine uplinks. **Classification:** ```plaintext class-map type qos match-any ROCEv2  match dscp 26 class-map type qos match-any CNP  match dscp 48 policy-map type qos QOS_CLASSIFICATION  class ROCEv2    set qos-group 3  class CNP    set qos-group 7  class class-default    set qos-group 0 ``` **Egress queuing:** ```plaintext policy-map type queuing QOS_EGRESS_PORT  class type queuing c-out-8q-q3    bandwidth remaining percent 50    random-detect minimum-threshold 950 kbytes maximum-threshold 3000 kbytes \      drop-probability 7 weight 0 ecn  class type queuing c-out-8q-q7    priority level 1  class type queuing c-out-8q-q-default    bandwidth remaining percent 50 ``` **PFC (system-wide):** ```plaintext policy-map type network-qos qos_network  class type network-qos c-8q-nq3    mtu 4200    pause pfc-cos 3 system qos  service-policy type network-qos qos_network  service-policy type queuing output QOS_EGRESS_PORT ``` | Traffic | DSCP | QoS group | Egress | | --- | --- | --- | --- | | RoCEv2 | 26 | 3 | q3, ECN WRED | | CNP | 48 | 7 | strict priority | | Default | — | 0 | best effort | --- ## Host-facing ports (leaf) **VIP-POOL** (EBox client VIPs): ```plaintext interface Ethernet  description VIP-POOL  switchport mode trunk  switchport trunk native vlan 100  switchport trunk allowed vlan 100  mtu 9216 ``` **Internal-RoCEv2** (storage backend): ```plaintext interface Ethernet  description Internal-RoCEv2  switchport mode trunk  switchport trunk native vlan 10  switchport trunk allowed vlan 10,69  priority-flow-control mode on  priority-flow-control watch-dog-interval on  service-policy type qos input QOS_CLASSIFICATION  service-policy type queuing output QOS_EGRESS_PORT  mtu 9216 ``` No IP on host ports. Bridging only into VXLAN. --- ## Verification | **Check** | **Command** | **Expected result** | | --- | --- | --- | | OSPF adjacencies | ```plaintext show ip ospf neighbors ``` | All fabric neighbors **FULL**; one adjacency per configured uplink | | Remote VTEP ECMP | ```plaintext show ip route ``` | `/32` to remote VTEP with multiple best paths (ECMP) via underlay next-hops | | BFD | ```plaintext show bfd neighbors ``` | All BFD sessions **Up** on OSPF-enabled fabric links | | EVPN BGP | ```plaintext show bgp l2vpn evpn summary ``` | All L2VPN EVPN neighbors **Established**; AF active | | VIP VNI | ```plaintext show bgp l2vpn evpn vni-id 100 ``` | VNI **100** routes present; Type-2/3 from local and remote VTEPs | | RoCE VNI | ```plaintext show bgp l2vpn evpn vni-id 30069 ``` | VNI **30069** routes present; Type-2 MAC/IP for RoCE hosts | | NVE status | ```plaintext show nve vni ``` | Site VNIs **Up** on leaves (`100`, `30010`, `30069`, `50000`); source **Loopback1** | | Client eBGP | ```plaintext show bgp vrf ipv4 unicast summary ``` | External peer(s) **Established**; `<vip-prefix>` in RIB per design | | PFC | ```plaintext show interface priority-flow-control detail ``` | PFC **on**; pause enabled for configured priority (e.g. CoS **3**) | | QoS | ```plaintext show policy-map interface input ``` | Input policy **QOS_CLASSIFICATION** attached; RoCE/CNP class counters increment under load | # Templates **Spine** ```plaintext hostname feature nxapi nv overlay evpn feature ospf feature bgp feature pim feature interface-vlan feature vn-segment-vlan-based feature bfd feature nv overlay ! ----- QoS / PFC (fabric uplinks) ----- class-map type qos match-any ROCEv2  match dscp 26 class-map type qos match-any CNP  match dscp 48 policy-map type qos QOS_CLASSIFICATION  class ROCEv2    set qos-group 3  class CNP    set qos-group 7  class class-default    set qos-group 0 policy-map type queuing QOS_EGRESS_PORT  class type queuing c-out-8q-q3    bandwidth remaining percent 50    random-detect minimum-threshold 950 kbytes maximum-threshold 3000 kbytes \      drop-probability 7 weight 0 ecn  class type queuing c-out-8q-q7    priority level 1  class type queuing c-out-8q-q-default    bandwidth remaining percent 50 policy-map type network-qos qos_network  class type network-qos c-8q-nq3    mtu 4200    pause pfc-cos 3 system qos  service-policy type network-qos qos_network  service-policy type queuing output QOS_EGRESS_PORT ! ----- VLAN / VNI (L3 only) ----- vlan 2000  vn-segment 50000 ! ----- VRF ----- vrf context  vni 50000  rd auto  address-family ipv4 unicast    route-target both auto evpn ! ----- Loopbacks ----- interface loopback0  description Routing loopback  ip address /32  ip router ospf UNDERLAY area 0.0.0.0  ip pim sparse-mode interface loopback1  description VTEP  ip address /32  ip router ospf UNDERLAY area 0.0.0.0  ip pim sparse-mode interface loopback254  description PIM anycast-RP  ip address /32  ip router ospf UNDERLAY area 0.0.0.0  ip pim sparse-mode ! ----- OSPF underlay ----- router ospf UNDERLAY  bfd  router-id ! ----- PIM anycast-RP ----- ip pim rp-address group-list ip pim anycast-rp ! repeat per spine ! ----- Leaf uplinks (repeat per leaf link) ----- interface Ethernet  description leaf-uplink  mtu 9216  ip address /30  ip ospf network point-to-point  ip router ospf UNDERLAY area 0.0.0.0  ip pim sparse-mode  priority-flow-control mode on  service-policy type qos input QOS_CLASSIFICATION  service-policy type queuing output QOS_EGRESS_PORT  no shutdown ! ----- NVE (L3 VNI only) ----- interface nve1  no shutdown  host-reachability protocol bgp  source-interface loopback1  member vni 50000 associate-vrf ! ----- L3 VNI SVI ----- interface Vlan2000  mtu 9216  vrf member  no ip redirects  ip forward  no shutdown ! ----- Client eBGP (repeat per client-facing port) ----- interface Ethernet.150  mtu 9216  encapsulation dot1q 150  vrf member  ip address /  no shutdown ip prefix-list PL-VIP-EXPORT seq 10 permit route-map permit 10  match ip address prefix-list PL-VIP-EXPORT ! ----- BGP ----- router bgp  router-id  cluster-id  neighbor    bfd    remote-as    update-source loopback0    address-family l2vpn evpn      send-community extended      route-reflector-client    ! repeat per leaf  neighbor    remote-as    update-source loopback0    address-family l2vpn evpn      send-community extended    ! inter-spine iBGP EVPN  vrf    address-family ipv4 unicast      advertise l2vpn evpn      aggregate-address summary-only      maximum-paths      maximum-paths ibgp    neighbor      remote-as external      address-family ipv4 unicast        route-map out      ! repeat per client interface ``` **Leaf** ```markdown hostname feature nxapi nv overlay evpn feature ospf feature bgp feature pim feature interface-vlan feature vn-segment-vlan-based feature bfd feature nv overlay fabric forwarding anycast-gateway-mac 2020.0000.00aa ! ----- QoS / PFC ----- class-map type qos match-any ROCEv2  match dscp 26 class-map type qos match-any CNP  match dscp 48 policy-map type qos QOS_CLASSIFICATION  class ROCEv2    set qos-group 3  class CNP    set qos-group 7  class class-default    set qos-group 0 policy-map type queuing QOS_EGRESS_PORT  class type queuing c-out-8q-q3    bandwidth remaining percent 50    random-detect minimum-threshold 950 kbytes maximum-threshold 3000 kbytes \      drop-probability 7 weight 0 ecn  class type queuing c-out-8q-q7    priority level 1  class type queuing c-out-8q-q-default    bandwidth remaining percent 50 policy-map type network-qos qos_network  class type network-qos c-8q-nq3    mtu 4200    pause pfc-cos 3 system qos  service-policy type network-qos qos_network  service-policy type queuing output QOS_EGRESS_PORT ! ----- VLAN / VNI ----- vlan 10  name VAST-Storage_VLAN_10  vn-segment 30010 vlan 69  name VAST-Storage-Network  vn-segment 30069 vlan 100  name vip-pool  vn-segment 100 vlan 2000  vn-segment 50000 evpn  vni 100 l2    rd auto    route-target import auto    route-target export auto  vni 30010 l2    rd auto    route-target import auto    route-target export auto  vni 30069 l2    rd auto    route-target import auto    route-target export auto ! ----- VRF ----- vrf context  vni 50000  rd auto  address-family ipv4 unicast    route-target both auto evpn ! ----- Loopbacks ----- interface loopback0  description Routing loopback  ip address /32  ip router ospf UNDERLAY area 0.0.0.0  ip pim sparse-mode interface loopback1  description VTEP  ip address /32  ip router ospf UNDERLAY area 0.0.0.0  ip pim sparse-mode ! ----- OSPF underlay ----- router ospf UNDERLAY  bfd  router-id ! ----- Spine uplinks (repeat per link) ----- interface Ethernet  description spine-uplink  mtu 9216  ip address /30  ip ospf network point-to-point  ip router ospf UNDERLAY area 0.0.0.0  ip pim sparse-mode  priority-flow-control mode on  service-policy type qos input QOS_CLASSIFICATION  service-policy type queuing output QOS_EGRESS_PORT  no shutdown ! ----- NVE ----- interface nve1  no shutdown  host-reachability protocol bgp  source-interface loopback1  member vni 100    mcast-group  member vni 30010    mcast-group  member vni 30069    mcast-group  member vni 50000 associate-vrf ! ----- L3 VNI SVI ----- interface Vlan2000  mtu 9216  vrf member  no ip redirects  ip forward  no shutdown ! ----- VIP anycast gateway ----- interface Vlan100  mtu 9216  vrf member  ip address / tag  fabric forwarding mode anycast-gateway  arp gratuitous accept  no shutdown route-map permit 10  match tag ! ----- Host: VIP-POOL (repeat per EBox port) ----- interface Ethernet  description VIP-POOL  switchport mode trunk  switchport trunk native vlan 100  switchport trunk allowed vlan 100  spanning-tree port type edge trunk  mtu 9216  no shutdown ! ----- Host: Internal-RoCEv2 (repeat per EBox port) ----- interface Ethernet  description Internal-RoCEv2  switchport mode trunk  switchport trunk native vlan 10  switchport trunk allowed vlan 10,69  priority-flow-control mode on  priority-flow-control watch-dog-interval on  spanning-tree port type edge trunk  mtu 9216  service-policy type qos input QOS_CLASSIFICATION  service-policy type queuing output QOS_EGRESS_PORT  no shutdown ! ----- BGP ----- router bgp  router-id  neighbor    bfd    remote-as    update-source loopback0    address-family l2vpn evpn      send-community extended    ! repeat per spine  vrf    address-family ipv4 unicast      advertise l2vpn evpn      redistribute direct route-map      aggregate-address summary-only      maximum-paths      maximum-paths ibgp ```