I want to create a mesh network with 10GBit Ethernet links between servers and a switch stack capable of distributed trunking.
In this scenario for a number n>2 (but typically 2<n<6) imagine:
1 Switch stack with at least n physical members, capable of distributed trunking on 10GB links.
n hosts, each with
- n+1 10GB interfaces, of those using
- n-1 for connecting directly to all other n-1 hosts to create the physical full mesh between the servers. Example pictures below
- 2 connections in one
bond0to the switch stack, to different members.
- n+1 10GB interfaces, of those using
k number of VLANs
GraphViz PNG rendering(1) of fully meshed server node constellation with n=5:
/*_5_Server_Nodes_Network_Mesh_°°°*/ graph n5 {
node[shape=plaintext,height=0,width=0,margin=0]
edge[len=0.6]
a-- b-- c-- d-- e-- a-- c-- e-- b-- d-- a
}
Note: To keep the graphs small, I chose n=3 and k=2 for the rest of this question, because 3 Nodes in a triangle formation are fully meshed, and the same solution should apply.
/*_3_Server_Nodes_Network_Mesh_°°°*/ graph n3 {
node[shape=plaintext,height=0,width=0,margin=0]
edge[len=0.4]
a -- b -- c -- a }

Lets say, the cards used for these connection are eth0ð1 ({0..(n-2)})
I also need to be establish k zones, VLANs(blue). Say VLANs M, and N:
/*_3_Server_Mesh_with_VLAN__°°°°°°*/ graph n3 {
node[shape=plaintext,height=0,width=0,margin=0]
edge[len=0.4]
a -- b -- c -- a
edge[color=blue]
a--{aM aN}; b--{bM bN}; c--{cM cN};
}
2 more of the 10GBit Ethernet Ports (here eth2ð3) on each hosts are bonded(brown) for upstream connection to outside world:
/*_3_Server_Mesh_w/_VLAN_and_bond0__*/ graph n3 {
node[shape=plaintext,height=0,width=0,margin=0]
edge[len=0.3]; a -- b -- c -- a
edge[color=blue]
a--{aM aN}; b--{bM bN}; c--{cM cN};
edge[color=brown,len=0.4,penwidth=2]
a -- au; b -- bu; c -- cu;}

Both ports of the bond0 interface to the different members of the upstream switch stack, that itself uses a ring of 10GBit connections(green) for stacking:
/*_3_Servers_w/VLAN+Stack*/ graph n3 {
node[shape=plaintext,margin=0,
height=0,width=0]
edge[len=0.3]
a -- b -- c -- a
edge[color=blue,len=0.3]
a--{aM aX}; b--{bM bN}; c--{cM cN};
edge[color=brown,len=0.4]
a -- au; b -- bu; c -- cu;
edge[color=brown,len=1.1]
{cu[shape=point] au} -- S1
{au[shape=point] bu} -- S2
{bu[shape=point] cu} -- S3
edge[color=green,len=2.9]
S1 -- S2 -- S3 --S1 }

And all the users would would be connected to (networks on) the "outside" of S1..S$n
Logically, the Switch stack S1..S$n behaves like one big Switching Unit. So, the servers a,b,..,n see the following structure, with everything else connected to S
/*_3_Server_Mesh_w/_VLAN_and_Stack__*/ graph n3 {
node[shape=plaintext,height=0,width=0,margin=0]
edge[len=1]; a -- b -- c -- a
edge[color=brown,len=0.3,penwidth=2]
S -- {a b c}
edge[color=blue,len=0.3]
a--{aM aN}; b--{bM bN}; c--{cM cN};
}

Now we connect some Virtual Machines(red) to these VLANs:
/*_3_Server_Mesh_with_VLAN_clients__*/ graph n3 {
node[shape=plaintext,height=0,width=0,margin=0]
edge[len=1]; a -- b -- c -- a
edge[color=brown,len=0.3,penwidth=2]
S -- {a b c}
edge[color=blue,len=0.3,penwidth=1]
a--{aM aN}; b--{bM bN}; c--{cM cN};
edge[color=red,len=0.3,penwidth=1]
v1--{aM aN}; {v2 v7}--bM; v3--{cM cN};
v4--aN; {v5 v6 v8}--bN; v9--cM;
}

This is the sort of functionality often seen in context with SDN, SPB, TRILL, VxLAN, all of which seem relevant after checking them out. Of those Shortest Path Bridging(802.1aq) sounds like the best fit. But where are the implementations for linux, and how do I use them on Debian to build a fully meshed brouter?
- SPB - Shortest Path Bridging(802.1aq) sounds like a winner: Official replacement for Spanning Tree, since 2012. There seems to be ongoing project on Github, not sure, if complete/functional
- TRILL - Transparent Interconnection of Lots of Links: Major competitor to SPB. Github Project, also does not seem available/functional
- OpenMesh batman-adv. It's available in Debian. But everywhere I read about it is about wireless. If it would work for 10GBit links, wouldn't everybody use that? (And there'd be online chat about it)
- VDE Virtual Switch. Also available in Debian. Entirely in userspace, therefore too slow for 10GBit/s?
- @JuliePelletier in a comment suggested BGP as the means to achieve this brouted,fault-tolerant Layer2 setup, but I cannot yet see how exactly she meant that.
So what could I install/configure on each node to achieve the desired constellation?
(1) Graphviz renderings were created with
neato -Tpng -O file.dot && browser file.dot.png