Introduction
Part 2 ended on a warning. Our fabric runs eBGP everywhere with a different ASN on every switch, and both choices bend standard EVPN in places where the failure looks the same every time: every BGP session stays up, routes appear in the tables, and host-1 still can’t reach host-2.
This post builds the Layer 2 overlay on Part 1’s eBGP underlay and walks into those bends one at a time. host-1 on LS-1 and host-2 on LS-3 share a VLAN and a /24. By the end they share a broadcast domain across a routed fabric, with VXLAN carrying their frames and EVPN telling each leaf where they are. Every failure comes with the output that shows it, and every output is a capture in the lab repo.
Part 4 adds routing between subnets on top of this overlay.
What This Post Covers
- The build order for a Layer 2 EVPN overlay, and why each piece depends on the one before it
- What Part 1’s underlay needs before it can carry VXLAN: MTU and a reachable VTEP address
- EVPN on its own loopback sessions, and the missing extended communities that leave the overlay dead with every session up
- What eBGP and a unique ASN per switch change: the spine next hop, route targets, and leaves passing routes between spines
- What actually crosses the wire: the VXLAN packet, BUM replication and ARP
- Whether the fabric spreads VXLAN traffic across both spines, and what the lab image can and can’t show
Lab Topology
The fabric underneath is Part 1’s, unchanged: two spines, three leaves, eBGP over /31 links with the spine on .0, and a unique private ASN on every switch. Part 3 adds three things. host-1 sits on LS-1’s Ethernet3 and host-2 on LS-3’s Ethernet3, both in VLAN 10 and 172.16.10.0/24. Each leaf gets a Loopback1 from 10.255.1.0/24 as its VTEP address. VLAN 10 maps to VNI 10010 on all three leaves.
LS-2 joins the VNI with no host behind it. That turns out to be useful: it shows what a leaf advertises when it has nothing to report but its membership. The hosts use .11 and .12, which keeps .1 free for the gateway in Part 4.
The running example is the one from Part 2: host-1 talking to host-2. Every section comes back to that one conversation.
The lab runs on netlab with the containerlab provider, Arista cEOS 4.36.2F on the switches and netlab’s default Linux containers as hosts. Each cEOS node used about 1 GB of RAM and close to half a CPU core at idle; the two hosts together used less than 30 MB. The topology, templates, scripts and every capture quoted here are in the lab repo. netlab up builds the finished state. Remove the five Part 3 templates from topology.yml and it builds the underlay and hosts only, ready for the overlay to be configured by hand.
The Build, in Order
The overlay is built in seven steps, and the order is not arbitrary: each step depends on the one before it, and a broken lower step looks exactly like a broken overlay. The rest of the post follows this order.
- Underlay. Part 1’s fabric: every Loopback0 reachable, two paths from each leaf to every other leaf. The EVPN sessions in step 4 ride on Lo0 reachability.
- MTU. Every fabric link carries the largest host frame plus 50 bytes, on spines and leaves. Done first because a wrong MTU fails silently and later, on large packets only.
- VTEP address. Loopback1 on each leaf, advertised into the underlay, and Part 1’s prefix-lists opened for the VTEP block on spines and leaves. Every EVPN route will name this address as its next hop, so it has to be reachable before any route arrives.
- EVPN sessions. Loopback0 to Loopback0 eBGP from each leaf to both spines, EVPN address family only, extended communities sent;
next-hop-unchangedon the spines. - VXLAN and MAC-VRF. On each leaf:
Vxlan1sourced from Loopback1 with VLAN 10 mapped to VNI 10010, and thevlan 10block under BGP with an RD, the shared route target andredistribute learned. - No transit. An outbound policy on the leaves’ EVPN sessions, so a leaf only advertises routes it originated.
- Verify. host-1 pings host-2; LS-3 lists host-1’s MAC on
Vx1and both route types carry the route target.
In short: underlay, MTU and VTEP reachability come before any EVPN config. Prove each step with its check before starting the next.
MTU Before Anything Else
VXLAN adds 50 bytes to every host frame, and a VTEP must not fragment it. If the fabric links can’t carry a full 1500-byte host packet plus those 50 bytes, small pings work and large ones vanish, which looks like an EVPN fault and isn’t one. So the first check happens before any overlay config exists.
netlab had left every fabric link at 1500:
LS-1#show ip interface brief
Interface IP Address Status Protocol MTU
-------------- ---------------------- ---------- ------------ ---------
Ethernet1 10.1.11.1/31 up up 1500
Ethernet2 10.2.11.1/31 up up 1500
Loopback0 10.11.11.11/32 up up 65535
Management0 192.168.121.103/24 up up 1500Arista’s EVPN sample configuration uses 9214 on fabric links. netlab wouldn’t build it:
[ERRORS] Errors found in topology.yml
[VALUE] nodes: Node MTU 9214 on node SP-1 is higher than the maximum MTU for eos (9194)
(the same error follows for SP-2, LS-1, LS-2 and LS-3)netlab caps EOS at 9194, which still leaves room for any host frame plus VXLAN. The value goes on the two switch groups only, not as a lab-wide default. A global default would also reach the Linux hosts, and a host at 9194 can send a frame that no longer fits once VXLAN adds its 50 bytes. The hosts stay at 1500.
groups:
spines:
members: [ SP-1, SP-2 ]
mtu: 9194
leaves:
members: [ LS-1, LS-2, LS-3 ]
mtu: 9194After a rebuild, the links show 9194 and a 9000-byte ping with the DF bit set crosses the fabric from LS-1’s loopback to LS-3’s:
LS-1#ping 10.13.13.13 source 10.11.11.11 size 9000 df-bit repeat 2
PING 10.13.13.13 (10.13.13.13) from 10.11.11.11 : 8972(9000) bytes of data.
8980 bytes from 10.13.13.13: icmp_seq=1 ttl=63 time=1.42 ms
8980 bytes from 10.13.13.13: icmp_seq=2 ttl=63 time=0.640 ms
--- 10.13.13.13 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 3ms
! output trimmedThe TTL of 63 means one router in between, a spine. One ping proves one of the two paths, since ECMP sends a flow one way only; the same setting is on every switch, so the other path matches.
In short: check MTU before any overlay config. The fabric links need host MTU plus 50 bytes; on this lab that’s 9194 on the switches and 1500 on the hosts, proven with a DF ping.
Giving Each Leaf a VTEP
Every EVPN route a leaf sends will name its Loopback1 as the next hop, and every VXLAN packet it builds will use Loopback1 as the outer source. So each leaf’s VTEP address has to be reachable from every other leaf before any EVPN route exists. Part 1’s underlay policy was written to carry Loopback0 and nothing else, and it does exactly that.
On LS-1, Loopback1 and a network statement put the route into BGP:
interface Loopback1
description VTEP
ip address 10.255.1.11/32
router bgp 65101
address-family ipv4
network 10.255.1.11/32The route is in LS-1’s own table, valid and best, and goes nowhere:
LS-1#show ip bgp 10.255.1.11/32
BGP routing table entry for 10.255.1.11/32
Paths: 1 available
Local
- from - (10.11.11.11)
Origin IGP, metric -, localpref -, IGP metric -, weight 0, tag 0
Received 00:00:34 ago, valid, local, best, redistributed (Connected)
! output trimmed
LS-1#show ip bgp neighbors 10.1.11.0 advertised-routes
Network Next Hop Metric AIGP LocPref Weight Path
* > 10.11.11.11/32 10.1.11.1 - - - - 65101 i
! output trimmedThere are four filters on the way to LS-3, one at each end of each session, and each of them drops the route without a log message or a session reset:
Opening only the first one shows where the next one sits. With seq 20 permit 10.255.1.11/32 added to LS-1’s outbound list, LS-1 advertises the route and SP-1 throws it away on arrival: two prefixes received, one accepted.
SP-1#show ip bgp summary
Description Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc PfxAdv
LS-1 10.1.11.1 4 65101 25 22 0 0 00:13:01 Estab 2 1 3
LS-2 10.1.12.1 4 65102 27 26 0 0 00:13:01 Estab 1 1 3
LS-3 10.1.13.1 4 65103 27 26 0 0 00:13:01 Estab 1 1 3
! output trimmedPart 1 used exact /32 entries and named their cost: every new leaf means editing lists on every switch. For the VTEP block I took a different route. Each leaf advertises its own VTEP as an exact /32 on the way out, and every other filter accepts one range, 10.255.1.0/24 ge 32, any /32 inside the block. A new leaf touches no other switch’s policy, and a leaf still can’t advertise anything outside its own address.
With Loopback1 on all three leaves and the range on all four filter points, every leaf reaches every VTEP over both spines, and the jumbo ping from the MTU check now works between VTEP addresses too:
LS-1#show ip route 10.255.1.13
B E 10.255.1.13/32 [200/0]
via 10.1.11.0, Ethernet1
via 10.2.11.0, Ethernet2
! output trimmed
LS-1#ping 10.255.1.13 source 10.255.1.11 size 9000 df-bit repeat 2
8980 bytes from 10.255.1.13: icmp_seq=1 ttl=63 time=1.04 ms
8980 bytes from 10.255.1.13: icmp_seq=2 ttl=63 time=0.675 ms
! output trimmedIn short: the VTEP address must be reachable before EVPN starts. Part 1’s exact-/32 policy blocks it at four points without a word; one range entry for the VTEP block opens all four and scales with the fabric.
EVPN on Its Own Sessions
EVPN routes need BGP sessions to travel on, and Part 1 already has six of them, one per spine-leaf link across its /31. EVPN could ride those. It doesn’t here.
Opinion: I’m putting the overlay on its own loopback sessions. Part 1’s underlay neighbors carry exact-/32 policy and next-hop-self, all tuned for IPv4. Keeping EVPN off those neighbors means none of that can leak into the overlay, and it follows Arista’s documented design, so the docs and my lab describe the same thing. The cost is real: six more BGP sessions, and an overlay that depends on loopback reachability through the underlay it sits on. iBGP over the eBGP underlay is out for this series; it would mean reworking the ASN plan Part 1 just justified.
Not everyone agrees. Ivan Pepelnjak argues against this exact design: eBGP EVPN sessions between loopbacks, over an eBGP underlay, keep the drawbacks of eBGP-only EVPN and add a loopback reachability problem. His recommendation is iBGP over an IGP. Several of the problems in the rest of this post come straight from that combination of eBGP and a different ASN on every switch.
Each leaf peers from its Loopback0 with both spines’ Loopback0, and each spine with all three leaves. Loopback0, not Loopback1: the session is control plane, the VTEP address is data plane. A session between loopbacks isn’t between directly connected interfaces, so it needs ebgp-multihop. On LS-1:
router bgp 65101
neighbor EVPN peer group
neighbor EVPN update-source Loopback0
neighbor EVPN ebgp-multihop 3
neighbor 10.1.1.1 peer group EVPN
neighbor 10.1.1.1 remote-as 65001
neighbor 10.1.1.1 description SP-1-EVPN
neighbor 10.2.2.2 peer group EVPN
neighbor 10.2.2.2 remote-as 65002
neighbor 10.2.2.2 description SP-2-EVPN
address-family evpn
neighbor EVPN activateThe spines mirror it, one neighbor per leaf. Nothing activates these neighbors for IPv4 unicast, and nothing has to turn it off: netlab’s generated config already contains no bgp default ipv4-unicast, so the new sessions carry EVPN only and can’t become a side door around Part 1’s IPv4 policy. EVPN on EOS also needs the multi-agent routing model; show running-config all | include service routing protocols returned service routing protocols model multi-agent on cEOS 4.36.2F, so no reload.
This is the lab’s first version of the sessions, and it leaves out one line Arista’s sample configuration has: send-community extended. The next section is about that line.
SP-1#show bgp evpn summary
Description Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc PfxAdv
LS-1-EVPN 10.11.11.11 4 65101 5 5 0 0 00:01:16 Estab 0 0 0
LS-2-EVPN 10.12.12.12 4 65102 5 5 0 0 00:01:06 Estab 0 0 0
LS-3-EVPN 10.13.13.13 4 65103 4 4 0 0 00:00:48 Estab 0 0 0
! output trimmedAll Established, all empty. There’s no VNI yet, so no leaf has anything to advertise.
In short: EVPN runs on separate eBGP sessions between Loopback0 addresses, EVPN address family only. They come up empty until the leaves have a VNI to advertise.
Routes Arrive, Nothing Works
Two pieces turn a leaf into a VTEP for VLAN 10. Vxlan1 is the data plane: it sources tunnels from Loopback1 and maps VLAN 10 to VNI 10010. The vlan 10 block under BGP is the MAC-VRF from Part 2: an RD, a route target, and redistribute learned, which turns every MAC the leaf learns locally into a Type 2 route. On LS-1:
interface Vxlan1
vxlan source-interface Loopback1
vxlan udp-port 4789
vxlan vlan 10 vni 10010
!
router bgp 65101
vlan 10
rd auto
route-target both 10010:10010
redistribute learnedLS-3 gets the same with its own ASN. LS-2 also needs vlan 10 created first: netlab only builds the VLAN where a host is attached.
The route target is 10010:10010 on every leaf, as Part 2 promised: one RT per VNI, shared by all. Each leaf now advertises a Type 3 route for VNI 10010, and host-1’s first frame gives LS-1 a Type 2. Both reach LS-3 through both spines, valid and best. They’re also useless. Compare LS-3’s own Type 3 with the one it received from LS-1:
LS-3#show bgp evpn route-type imet detail
BGP routing table entry for imet 10.255.1.11, Route Distinguisher: 10.11.11.11:10
Paths: 2 available
65001 65101
10.255.1.11 from 10.1.1.1 (10.1.1.1)
Origin IGP, metric -, localpref 100, weight 0, tag 0, valid, external, ECMP head, ECMP, best, ECMP contributor
VNI: 10010
PMSI Tunnel: Ingress Replication, VNI: 10010, Leaf Information Required: false, Tunnel ID: 10.255.1.11
! output trimmed
BGP routing table entry for imet 10.255.1.13, Route Distinguisher: 10.13.13.13:10
Paths: 1 available
Local
- from - (0.0.0.0)
Origin IGP, metric -, localpref -, weight 0, tag 0, valid, local, best
Extended Community: Route-Target-AS:10010:10010 TunnelEncap:tunnelTypeVxlan
VNI: 10010
PMSI Tunnel: Ingress Replication, VNI: 10010, Leaf Information Required: false, Tunnel ID: 10.255.1.13LS-3’s local route carries Route-Target-AS:10010:10010 TunnelEncap:tunnelTypeVxlan. The received one has no Extended Community line at all. The same goes for host-1’s Type 2: on LS-1 it carries both communities, on LS-3 it carries none. Part 2 warned about this: without the route target there is nothing to import the route with, and without the encapsulation community there is no VXLAN tunnel to build. EOS doesn’t send extended communities to a BGP neighbor unless told to, and none of the five switches was told.
The route target config itself is correct, which rules out the obvious suspect:
LS-3#show bgp evpn instance
EVPN instance: VLAN 10
Route distinguisher: 10.13.13.13:10
Route target import: Route-Target-AS:10010:10010
Route target export: Route-Target-AS:10010:10010
! output trimmed
LS-3#show vxlan vtep
Remote VTEPS for Vxlan1:
VTEP Tunnel Type(s)
---------- --------------
Total number of remote VTEPS: 0No remote VTEPs, an empty flood list, and host-1’s ping to host-2 fails with 100% loss, while every BGP session in the fabric is up and nothing has logged an error. This is the failure Part 2 promised, in its plainest form.
The fix is one line on all five switches. The spines need it too: they forward the leaves’ routes, and a spine without it strips the communities on the way through.
router bgp 65001
neighbor EVPN send-community extendedThe received routes now carry Route-Target-AS:10010:10010 TunnelEncap:tunnelTypeVxlan, LS-3 lists two remote VTEPs, and the ping works.
In short: EVPN routes without extended communities arrive valid and best, and nothing imports them. send-community extended belongs on every EVPN neighbor, spines included.
Following host-1’s MAC
With the communities in place, the two route types from Part 2 do exactly what Part 2 said they would, and LS-3’s tables show each one’s job.
The Type 3 routes build the flood list. Every leaf in VNI 10010 advertises one, and the set a leaf receives is where it sends broadcast, unknown unicast and multicast for that VNI:
LS-3#show vxlan flood vtep
VLANS Ip Address
----------------------------- ------------------------------------------------
10 10.255.1.11 10.255.1.12
! output trimmedLS-2 is on the list. It has no host, but it is in the VNI, so it gets a copy of every flooded frame. Nobody configured that list; LS-2 joined it by advertising one route.
The Type 2 routes carry the MACs. host-1’s MAC arrives twice, once through each spine, with the same next hop: LS-1’s VTEP.
LS-3#show bgp evpn route-type mac-ip detail
BGP routing table entry for mac-ip aac1.ab47.1dfb, Route Distinguisher: 10.11.11.11:10
Paths: 2 available
65002 65101
10.255.1.11 from 10.2.2.2 (10.2.2.2)
Origin IGP, metric -, localpref 100, weight 0, tag 0, valid, external, ECMP head, ECMP, best, ECMP contributor
Extended Community: Route-Target-AS:10010:10010 TunnelEncap:tunnelTypeVxlan
VNI: 10010 ESI: 0000:0000:0000:0000:0000
65001 65101
10.255.1.11 from 10.1.1.1 (10.1.1.1)
Origin IGP, metric -, localpref 100, weight 0, tag 0, valid, external, ECMP, ECMP contributor
Extended Community: Route-Target-AS:10010:10010 TunnelEncap:tunnelTypeVxlan
VNI: 10010 ESI: 0000:0000:0000:0000:0000
! output trimmed
LS-3#show mac address-table vlan 10
Vlan Mac Address Type Ports Moves Last Move
---- ----------- ---- ----- ----- ---------
10 aac1.ab47.1dfb DYNAMIC Vx1 1 0:01:16 ago
10 aac1.abbe.04fa DYNAMIC Et3 1 0:00:45 ago
! output trimmedhost-1’s MAC sits on Vx1, installed from BGP rather than learned from traffic; host-2’s sits on the access port. The route is mac-ip by name but carries a MAC only. With no gateway address on the leaves, there is no MAC-to-IP binding to advertise. That detail comes back in the ARP section.
One table shows both route types at a glance:
LS-3#show vxlan vtep
Remote VTEPS for Vxlan1:
VTEP Tunnel Type(s)
----------------- --------------
10.255.1.11 flood, unicast
10.255.1.12 flood
Total number of remote VTEPS: 210.255.1.11 is flood, unicast: LS-1 sent a Type 3 and a Type 2. 10.255.1.12 is flood only: LS-2 sent a Type 3 and has no MACs to report.
And host-1 reaches host-2:
ping -c 3 172.16.10.12
PING 172.16.10.12 (172.16.10.12): 56 data bytes
64 bytes from 172.16.10.12: seq=0 ttl=64 time=14.545 ms
64 bytes from 172.16.10.12: seq=1 ttl=64 time=2.623 ms
64 bytes from 172.16.10.12: seq=2 ttl=64 time=4.929 ms
! output trimmedThe TTL is 64, untouched: host-1’s frame was bridged at both leaves and never routed; the spine routed only the outer VXLAN packet. In Part 4, routed traffic between subnets should show two decrements instead.
In short: Type 3 routes build each leaf’s flood list, Type 2 routes install remote MACs on the VXLAN interface. show vxlan vtep shows both: flood for VNI membership, unicast for MACs behind a VTEP.
The Spine Next Hop
RFC 8365 requires the BGP next hop of an EVPN route to be the advertising leaf’s VTEP, because the receiving leaf uses it as the outer destination of every VXLAN packet. eBGP has the opposite habit: a router advertising a route to an eBGP neighbor normally puts its own address in the next hop. Part 1 even sets next-hop-self on the underlay sessions. If a spine did that to an EVPN route, LS-3 would build its tunnel to SP-1, and SP-1 isn’t a VTEP.
Arista’s sample configuration handles this with next-hop-unchanged on the spines’ EVPN neighbors. The first build left it out on purpose, and my prediction going in was that the spines would rewrite the next hop to their own Loopback0.
They didn’t. The capture from the communities section already showed it, before the communities were even fixed:
LS-3#show bgp evpn route-type imet detail
BGP routing table entry for imet 10.255.1.11, Route Distinguisher: 10.11.11.11:10
Paths: 2 available
65001 65101
10.255.1.11 from 10.1.1.1 (10.1.1.1)
! output trimmed10.255.1.11 from 10.1.1.1: learned from SP-1, next hop still LS-1’s VTEP. And SP-1’s configuration at that point had no next-hop command on the EVPN peer group at all:
SP-1#show running-config section router bgp
router bgp 65001
router-id 10.1.1.1
no bgp default ipv4-unicast
bgp advertise-inactive
neighbor EVPN peer group
neighbor EVPN update-source Loopback0
neighbor EVPN ebgp-multihop 3
! output trimmed
address-family evpn
neighbor EVPN activate
! output trimmedOn cEOS 4.36.2F, the spines passed the leaf’s VTEP through unchanged on eBGP EVPN sessions without being told to. To make sure the command really is redundant here and not doing something subtle, I added it to both spines and compared:
router bgp 65001
neighbor EVPN next-hop-unchangedNothing changed. LS-3’s Type 3 and Type 2 routes showed the same next hops, the same communities and the same two paths as before, and the ping kept working.
Opinion: the command stays in the final configuration anyway. It’s what Arista documents, it states the spines’ job in the config instead of leaving it to a default, and a default that holds on one release is a weak thing to build on. If a future EOS version changed this behavior, the overlay would break in the quietest possible way: sessions up, routes present, tunnels pointed at a spine.
In short: an EVPN next hop must stay the originating leaf’s VTEP. On cEOS 4.36.2F the spines kept it without next-hop-unchanged; it goes in the config anyway, so the design doesn’t depend on a default.
Route Targets With a Different ASN on Every Leaf
Part 2 said a shared route target “sounds trivial. With a different ASN on every switch it isn’t.” The manual 10010:10010 works. The interesting question is what happens when you let EOS pick the route target.
EOS offers it under import only, and the help output already tells most of the story:
LS-3(config-macvrf-10)#route-target both ?
ASN(asplain):nn or ASN(asdot):nn or IP-address:nn Route target
LS-3(config-macvrf-10)#route-target import ?
ASN(asplain):nn or ASN(asdot):nn or IP-address:nn Route target
any-as:nn Wildcard 2-byte AS Route Target
auto Auto Generate Route Target
evpn Configure route target for EVPN address family
export Route Export
LS-3(config-macvrf-10)#route-target import auto ?
<1-4294967295> Autonomous system number
<1-65535>.<0-65535> AS Number in asdot notationroute-target export auto is accepted as is. route-target import auto wants an ASN: the automatic RT of which switch. To test it, LS-3 got the setup a single-ASN fabric would use, exporting its automatic RT and importing the automatic RT of its own ASN:
router bgp 65103
vlan 10
route-target export auto
route-target import auto 65103LS-3#show bgp evpn instance
EVPN instance: VLAN 10
Route distinguisher: 10.13.13.13:10
Route target import: Route-Target-AS:65103:268445466 (auto, VNI 10010)
Route target export: Route-Target-AS:65103:268445466 (auto, VNI 10010)
! output trimmedThe value isn’t 65103:10010. 268445466 is 0x1000271A: the top four bits are 1, the type field RFC 8365 defines for automatically derived RTs, meaning VXLAN, and the low 24 bits are 0x271A, which is 10010, the VNI. EOS follows RFC 8365’s encoding exactly. The ASN half comes from the switch’s own ASN, so on this fabric no two leaves can ever produce the same value.
LS-1 received LS-3’s Type 3 carrying Route-Target-AS:65103:268445466, valid and best, and didn’t import it. LS-3 dropped to zero remote VTEPs, host-1’s MAC disappeared from its table and the ping failed. Every BGP session stayed up.
Automatic RTs can still work here. The wildcard ignores the ASN half, so every leaf can export its automatic RT and import one line that matches all of them:
router bgp 65101
vlan 10
route-target export auto
route-target import any-as:268445466With that on all three leaves, LS-1 imported routes carrying 65102:268445466 and 65103:268445466, LS-3 listed both remote VTEPs again and the ping worked. The wildcard has to name the encoded value, 268445466, not the VNI, so read the automatic RT off show bgp evpn instance before writing it.
| Route target setup | Result on this fabric |
|---|---|
route-target both 10010:10010 on every leaf | Works |
export auto + import auto <own ASN> | Fails silently: each leaf’s RT carries its own ASN |
export auto + import auto <ASN> per remote leaf | Not tested; every leaf would list every other leaf’s ASN |
export auto + import any-as:268445466 | Works |
The third row is Part 1’s scaling problem again: with a unique ASN per leaf, every new leaf means editing every other leaf, just like the exact-/32 prefix-lists. In a fabric where all leaves share an ASN, it would be one line.
Opinion: I’m keeping the manual shared RT. It’s what Part 2 committed to, every leaf’s config reads the same, and the value means something when you see it in a route. The wildcard is a fair alternative, but it hides a magic number that only makes sense after decoding RFC 8365.
In short: EOS builds automatic RTs from the local ASN and RFC 8365’s encoding, so with a different ASN on every leaf they never match. Use one shared manual RT, or export auto with an any-as wildcard import.
Leaves as Transit, Again
Part 1 made sure a leaf never passes one spine’s routes to the other: RM-UNDERLAY-OUT lets a leaf advertise only its own Loopback0. That policy lives in the IPv4 address family. The EVPN sessions have no policy at all, and with a different ASN on every spine, nothing else stops it either.
The first sign was in LS-1’s EVPN summary, which didn’t add up. LS-1 originates one route, its Type 3, yet it advertised three to SP-1 and one to SP-2:
LS-1#show bgp evpn summary
Description Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc PfxAdv
SP-1-EVPN 10.1.1.1 4 65001 12 13 0 0 00:05:56 Estab 2 2 3
SP-2-EVPN 10.2.2.2 4 65002 12 11 0 0 00:05:56 Estab 2 2 1
! output trimmedLS-1 had picked SP-2’s paths as best for LS-2’s and LS-3’s routes, and a BGP speaker advertises its best paths to its other neighbors. So LS-1 handed SP-2’s routes to SP-1. SP-1 then held paths through a leaf, including one for LS-1’s own route that had traveled LS-1, SP-2, LS-2, SP-1:
SP-1#show bgp evpn route-type imet detail
BGP routing table entry for imet 10.255.1.11, Route Distinguisher: 10.11.11.11:10
Paths: 2 available
65101
10.255.1.11 from 10.11.11.11 (10.11.11.11)
Origin IGP, metric -, localpref 100, weight 0, tag 0, valid, external, best
65102 65002 65101
10.255.1.11 from 10.12.12.12 (10.12.12.12)
Origin IGP, metric -, localpref 100, weight 0, tag 0, valid, external
! output trimmedIn a fabric where both spines share one ASN, that second path would carry the spines’ own ASN, and SP-1 would drop it on arrival as a loop. Here the spines are 65001 and 65002, so 65102 65002 65101 looks perfectly valid to SP-1. It isn’t harmful today. The direct path wins on AS_PATH length, and the next hop is still the VTEP, so the data plane is unaffected. It’s extra state on every spine, and the same failure mode Part 1 designed out, back in a new address family.
It also moves around. After the route-target experiments, LS-1’s best paths flipped to SP-1, and the extra copies went to SP-2 instead: LS-1 advertised 2 routes to SP-1 and 5 to SP-2, while originating 2 (its Type 3 and host-1’s Type 2). Which spine collects the transit copies depends on the leaf’s tiebreaks at that moment.
The fix is the EVPN version of Part 1’s rule: a leaf advertises only what it originated. A locally originated route has an empty AS_PATH, so one AS-path list and one route-map on each leaf’s EVPN sessions does it:
ip as-path access-list AS-LOCAL permit ^$ any
!
route-map RM-EVPN-OUT permit 10
match as-path AS-LOCAL
!
router bgp 65101
address-family evpn
neighbor EVPN route-map RM-EVPN-OUT outAfter it, LS-1 advertised 2 routes to each spine, and each spine received exactly what each leaf originates:
SP-2#show bgp evpn summary
Description Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc PfxAdv
LS-1-EVPN 10.11.11.11 4 65101 70 79 0 0 00:43:49 Estab 2 2 3
LS-2-EVPN 10.12.12.12 4 65102 80 81 0 0 00:43:38 Estab 1 1 4
LS-3-EVPN 10.13.13.13 4 65103 75 77 0 0 00:43:20 Estab 2 2 3
! output trimmedLS-1 and LS-3 each originate a Type 3 and their host’s Type 2; LS-2 only its Type 3. Both spines now hold one path per route, from the leaf that originated it, and the leaves still see two paths for every remote route, one through each spine. The filter removed the transit copies, not the redundancy.
In short: without the underlay’s outbound policy, leaves pass EVPN routes between spines, and unique spine ASNs mean AS_PATH doesn’t stop it. An ^$ AS-path filter on the leaves’ EVPN sessions restores Part 1’s no-transit rule.
On the Wire
Everything so far came from BGP tables and MAC tables. The packets themselves are one capture away. tcpdump runs from the lab host inside LS-1’s container namespace, on its two uplinks, while host-1 clears its ARP cache and pings host-2 three times. Inside a cEOS container the ports are et1 and et2, not eth1; the capture script is in the repo as scripts/capture-vxlan.sh.
The Packet
Here is host-1’s first echo request as it left LS-1 toward SP-1:
11:14:04.818295 ca:f0:00:03:00:01 > ca:f0:00:01:00:01, ethertype IPv4 (0x0800), length 148: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 134)
10.255.1.11.7928 > 10.255.1.13.4789: VXLAN, flags [I] (0x08), vni 10010
aa:c1:ab:47:1d:fb > aa:c1:ab:be:04:fa, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 13466, offset 0, flags [DF], proto ICMP (1), length 84)
172.16.10.11 > 172.16.10.12: ICMP echo request, id 184, seq 0, length 64
The first two lines are LS-1’s work, the last two are host-1’s frame. The overhead is exactly Part 2’s arithmetic: a 98-byte frame went in, a 148-byte packet went out. The outer source and destination are the two VTEP addresses, the destination port is 4789, the I flag marks the VNI as valid, and the DF bit is set, which is why the MTU check came first. The outer TTL leaves LS-1 at 64 and arrives at LS-3 at 63, one spine later; the inner TTL never changes.
The First ARP
host-1 had just flushed its ARP cache, so the ping started with a broadcast. LS-1 doesn’t know who has 172.16.10.12; it floods, and on an EVPN fabric flooding means one unicast copy to every VTEP on the flood list:
11:14:04.812810 ca:f0:00:03:00:01 > ca:f0:00:01:00:01, ethertype IPv4 (0x0800), length 110: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 96)
10.255.1.11.58242 > 10.255.1.13.4789: VXLAN, flags [I] (0x08), vni 10010
aa:c1:ab:47:1d:fb > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Ethernet (len 6), IPv4 (len 4), Request who-has 172.16.10.12 tell 172.16.10.11, length 46
11:14:04.812838 ca:f0:00:03:00:01 > ca:f0:00:01:00:01, ethertype IPv4 (0x0800), length 110: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 96)
10.255.1.11.58242 > 10.255.1.12.4789: VXLAN, flags [I] (0x08), vni 10010
aa:c1:ab:47:1d:fb > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Ethernet (len 6), IPv4 (len 4), Request who-has 172.16.10.12 tell 172.16.10.11, length 46
11:14:04.817064 ca:f0:00:01:00:01 > ca:f0:00:03:00:01, ethertype IPv4 (0x0800), length 110: (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto UDP (17), length 96)
10.255.1.13.6648 > 10.255.1.11.4789: VXLAN, flags [I] (0x08), vni 10010
aa:c1:ab:be:04:fa > aa:c1:ab:47:1d:fb, ethertype ARP (0x0806), length 60: Ethernet (len 6), IPv4 (len 4), Reply 172.16.10.12 is-at aa:c1:ab:be:04:fa, length 46
One broadcast left LS-1 as two packets, 28 microseconds apart: this is ingress replication, Part 2’s head-end copy per remote VTEP, built from the Type 3 routes. LS-2 got its copy even though nobody behind it could answer. The reply came back once, unicast, because LS-3 already had host-1’s MAC from a Type 2 route.
The Second ARP
Part 2 left an open question: once a host’s MAC and IP arrive as a Type 2 route, the next ARP for it can be answered by the local leaf, if cEOS does that by default. Here the answer was decided earlier, by show bgp evpn route-type mac-ip: the Type 2 routes carry a MAC and no IP, so there is no binding to answer from.
To check, host-1 flushed its ARP cache twice, five seconds apart, pinging host-2 after each flush, while tcpdump listened on host-2’s own interface:
11:01:57.890945 aa:c1:ab:47:1d:fb > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 172.16.10.12 tell 172.16.10.11, length 46
11:01:57.890954 aa:c1:ab:be:04:fa > aa:c1:ab:47:1d:fb, ethertype ARP (0x0806), length 42: Reply 172.16.10.12 is-at aa:c1:ab:be:04:fa, length 28
11:02:02.431873 aa:c1:ab:47:1d:fb > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 172.16.10.12 tell 172.16.10.11, length 46
11:02:02.431881 aa:c1:ab:be:04:fa > aa:c1:ab:47:1d:fb, ethertype ARP (0x0806), length 42: Reply 172.16.10.12 is-at aa:c1:ab:be:04:fa, length 28
! output trimmedBoth requests reached host-2, and host-2 answered both. No leaf answered on its behalf. In a pure Layer 2 segment, every ARP floods. Part 4 puts gateway addresses on the leaves, and with them MAC-and-IP Type 2 routes, which is where ARP suppression can appear.
Unknown Unicast After a Quiet Spell
The last kind of flooding showed up by accident. After nine minutes without traffic, during the ECMP test in the next section, the first echo request from host-1 (sent from one of its test addresses, 172.16.10.21) left LS-1 twice, to both VTEPs, even though it was addressed to host-2’s MAC:
11:23:16.853452 IP 10.255.1.11.248 > 10.255.1.13.4789: VXLAN, flags [I] (0x08), vni 10010
IP 172.16.10.21 > 172.16.10.12: ICMP echo request, id 244, seq 0, length 64
11:23:16.853485 IP 10.255.1.11.248 > 10.255.1.12.4789: VXLAN, flags [I] (0x08), vni 10010
IP 172.16.10.21 > 172.16.10.12: ICMP echo request, id 244, seq 0, length 64
! output trimmedhost-2’s MAC had aged out on LS-3, and its Type 2 route was withdrawn with it. LS-1 learns remote MACs from EVPN only (show interfaces Vxlan1 says “Remote MAC learning via EVPN”), so until LS-3 relearned the MAC and advertised a new Type 2, host-2 was an unknown unicast destination, the U in BUM, and LS-1 flooded. From the next packet on, the frames went to 10.255.1.13 only. The same aging explains why a show bgp evpn route-type mac-ip taken after a few idle minutes can come back empty.
In short: VXLAN added exactly 50 bytes. Broadcasts and unknown unicast leave the ingress leaf once per VTEP on the flood list; known unicast goes to one VTEP. With MAC-only Type 2 routes, every ARP floods.
Does Traffic Use Both Spines?
Between LS-1 and LS-3, every VXLAN packet has the same outer source and destination IP. Part 2 explained why that doesn’t defeat ECMP: the VTEP sets the outer UDP source port from a hash of the inner headers, so different host conversations get different ports and spread across both spines. Part 2 also left the question open for cEOS.
The test gives host-1 eight extra addresses, 172.16.10.21 to .28, and pings host-2 twice from each, while tcpdump watches both of LS-1’s uplinks. Each address is a separate inner flow. The script is scripts/ecmp-test.sh in the repo.
The first half of the answer is yes. Every flow got its own source port, kept it for both pings, and used the same port in both directions:
| host-1 source | Outer UDP source port |
|---|---|
| 172.16.10.21 | 248 |
| 172.16.10.22 | 1016 |
| 172.16.10.23 | 760 |
| 172.16.10.24 | 3576 |
| 172.16.10.25 | 3320 |
| 172.16.10.26 | 4088 |
| 172.16.10.27 | 3832 |
| 172.16.10.28 | 2552 |
Most of these sit well below the 49152 to 65535 range RFC 7348 recommends. That’s a recommendation, not a requirement, and the receiving VTEP only cares about the destination port.
The second half is no. All eight flows, in both directions, crossed SP-1. The capture on Ethernet2, toward SP-2, stayed empty:
### LS-1 et2 (Ethernet2) udp port 4789
reading from file outputs/20261015-1123_LS-1_et2_ecmp.pcap, link-type EN10MB (Ethernet), snapshot length 262144EOS has both paths. The Linux routing table inside LS-1’s container has one:
LS-1#show ip route 10.255.1.13
B E 10.255.1.13/32 [200/0]
via 10.1.11.0, Ethernet1
via 10.2.11.0, Ethernet2
! output trimmedsudo ip netns exec clab-l2-overlay-LS-1 ip route show 10.255.1.13
10.255.1.13 via 10.1.11.0 dev et1 proto gatedThat looked like the whole explanation, and it isn’t. Ivan Pepelnjak tested ECMP on cEOS 4.35.2F and saw the same single next hop in the Linux table, yet traffic forwarded through the switch was spread across both paths. Traffic the switch originated itself used one path. A VTEP builds the outer VXLAN packet itself, so from cEOS’s point of view encapsulated traffic may count as locally originated. That fits everything captured here, including the detail that both leaves always picked Ethernet1, the first and only next hop in their kernel tables. It isn’t proven, and this fabric has no plain transit-ECMP case to compare against: every packet a leaf sends into the fabric is VXLAN, and each spine has a single path to each leaf.
The same single kernel path exists for Part 1’s Loopback0 routes. Part 1 showed ECMP in the routing table, which is accurate as far as it goes; whether the lab image would spread forwarded traffic across both spines is a separate question.
In short: the VXLAN source port did its job, one port per flow. cEOS in containerlab still sent every VXLAN flow over one uplink. That’s a property of the lab image, not of EVPN; hardware VTEPs hash in the forwarding chip.
Lab Summary
Each step had a prediction written down before it ran. Two were wrong, and one finding wasn’t predicted at all; all three are marked.
| Step | Expected | Observed |
|---|---|---|
| MTU | Fabric links at 1500, no room for VXLAN | 1500. netlab rejects 9214 for EOS; 9194 set on switches only, 9000-byte DF ping works |
| Loopback1, Part 1 policy | Route in LS-1’s table, not advertised | As expected; then dropped by SP-1 on arrival (2 received, 1 accepted) until all four filters allowed the VTEP range |
| EVPN sessions | Established, no prefixes | As expected; EVPN only, thanks to netlab’s no bgp default ipv4-unicast |
| No extended communities | Routes arrive without RT or encapsulation, nothing imported | As expected: valid and best, 0 remote VTEPs, ping fails, sessions up |
Spine next hop without next-hop-unchanged | Rewritten to the spine’s Loopback0 | Wrong: next hop stayed the leaf’s VTEP; adding the command changed nothing |
| With extended communities | Flood list built, MACs on Vx1, ping works | As expected; ping TTL 64, Type 2 routes MAC only |
| Automatic RT, own ASN only | Fails silently | As expected: 65103:268445466 (RFC 8365 encoding), not imported elsewhere; any-as:268445466 fixes it |
| EVPN transit through leaves | Not predicted | Leaves passed spine-learned routes to the other spine; ^$ filter stops it |
| ARP | First ARP floods, second floods too (no IP in Type 2) | As expected: both requests reached host-2 |
| ECMP across spines | Different flows on different spines | Wrong: one source port per flow, but every flow used Ethernet1 to SP-1 |
| Rebuild from templates | Full overlay from netlab up alone | As expected; scripts/verify-overlay.sh passes |
What’s Next
host-1 and host-2 now share a broadcast domain across a routed fabric, and nothing on the leaves knows an IP address. Part 4 changes that. Each leaf becomes the anycast gateway for its subnets, a second subnet appears on LS-3, and routing between them uses symmetric IRB with a per-tenant L3 VNI. The Type 2 routes start carrying IP addresses, Type 5 routes carry subnets, and the ARP suppression this post couldn’t show gets its chance. The traceroute should show two routing hops where this post’s ping showed none.
References
- RFC 7348, Virtual eXtensible Local Area Network (VXLAN), RFC Editor, 2014 (link)
- RFC 7432, BGP MPLS-Based Ethernet VPN, IETF, 2015 (link)
- RFC 8365, A Network Virtualization Overlay Solution Using Ethernet VPN (EVPN), IETF, 2018 (link)
- Arista EOS 4.36.2F User Manual: Configuring EVPN, Sample Configurations, Arista Networks
- Ivan Pepelnjak, EVPN Designs: EVPN EBGP over IPv4 EBGP, ipSpace.net, 2024
- Ivan Pepelnjak, ECMP on Arista cEOS, ipSpace.net, 2026
- Lab sources: dc-fabric-03-l2-overlay on GitHub