Skip to main content

02 · Cluster layout

Level 0 — Harvester · bare metal · hosts two separate Level-1 RKE2 clusters as VMs. 3 × master (control-plane/etcd) · 15 × worker (compute+storage). · interactive original ↗

Rancher MCMHA · 3 replicas · Harvester node driverLEVEL 0 · HARVESTER (RKE2) · BARE METAL · 18 NODES3 × master (control-plane/etcd) · 15 × worker (compute + storage)hosts two separate Level-1 RKE2 clusters as VMsevery L1 VM on its own host · 3 zones (racks)LEVEL 1 · APPLICATION RKE2 CLUSTER3 × master VM + 3 × worker VMown VMs · separate cluster · provisioned by RancherApp podsCanal/Cilium · 10.42/16Ceph CSI (RBD/FS)PVCs → Ceph · VLAN 210Harvester CSIPVCs → Longhorn volumesMetalLBL2/BGP · VIPs on VLANs 300–399Gateway APIEnvoy/Traefik · HTTPRoute/TLSPLAIN VMSPlain VMsDB · appliances · legacyRoot disks on Longhorn1 replica · localNICs on guest VLANs300–399 · bridgeLive migrationmgmt / storage netLEVEL 1 · STORAGE RKE2 CLUSTER — ROOK / CEPH3 × master VM + 6 × OSD worker VMown VMs · separate cluster · 2 OSD hosts per zone3 × MON + 2 × MGRanti-affinity across hostsOSDs on raw NVMepassthrough · not LonghornRBD · CephFS · RGWCeph public VLAN 210OSD VM gets ALL host NVMe via passthrough4 NVMe = 4 OSD daemons

Level 1 · Application RKE2 cluster​

Own VMs · separate cluster. 3 × master VM + 3 × worker VM (1 per zone, each on its own host, scale out evenly) · provisioned by Rancher.

Workloads and storage classes​

  • App pods — CNI: Canal/Cilium · pod CIDR 10.42.0.0/16
  • Ceph CSI (RBD/FS) — PVCs → storage cluster over VLAN 210
  • Harvester CSI — PVCs → Longhorn volumes (hot-plug disks)

Ingress​

  • MetalLB — L2/BGP · LoadBalancer VIPs from a pool on guest VLANs 300–399
  • Gateway API — Envoy/Traefik GatewayClass · HTTPRoute/TLS behind MetalLB VIP

Plain VMs​

Databases, appliances, legacy services.

  • Root disks on Longhorn (1 replica, local — no network I/O)
  • NICs on guest VLANs 300–399 (bridge)
  • Live migration over mgmt/storage net

Level 1 · Storage RKE2 cluster — Rook / Ceph​

Own VMs · separate cluster. Separate RKE2 · 3 × master VM + 6 × OSD worker VM — each on its own host, 2 OSD hosts per zone · OSD VM gets ALL host NVMe via passthrough (4 NVMe = 4 OSD daemons).

Components​

  • 3 × MON + 2 × MGR — anti-affinity across hosts
  • OSDs on raw NVMe — passthrough disks, not Longhorn
  • RBD · CephFS · RGW — served on Ceph public VLAN 210

Node anatomy​

Each bare-metal host:

InterfaceCarries
bond0 2×25GMgmt VLAN 100 — RKE2, Harvester API, live migration
bond0 (tagged)Guest VLANs 300–399 — VM traffic via bridge/mgmt-bo
bond1 2×25GLonghorn storage VLAN 200 — replica sync (dedicated)
bond1 (tagged)Ceph public 210 + cluster 211 — storage VM NICs
eth-bmc 1GOOB / IPMI VLAN 90 — isolated
disks2×SSD (OS+Longhorn) · 2–4×NVMe reserved for Ceph OSD passthrough

18-node layout​

18 nodes · each row = L0 bare-metal host (Harvester's own RKE2) + its L1 guest VM · 3 zones (racks).

ZONE A · RACK 1L0hv-01HV masterL1no guest VML0hv-02HV workerL1app-master-1L0hv-03HV workerL1app-worker-1L0hv-04HV workerL1stor-master-1L0hv-05HV workerL1stor-osd-1L0hv-06HV workerL1stor-osd-2ZONE B · RACK 2L0hv-07HV masterL1no guest VML0hv-08HV workerL1app-master-2L0hv-09HV workerL1app-worker-2L0hv-10HV workerL1stor-master-2L0hv-11HV workerL1stor-osd-3L0hv-12HV workerL1stor-osd-4ZONE C · RACK 3L0hv-13HV masterL1no guest VML0hv-14HV workerL1app-master-3L0hv-15HV workerL1app-worker-3L0hv-16HV workerL1stor-master-3L0hv-17HV workerL1stor-osd-5L0hv-18HV workerL1stor-osd-6Harvester master hosthost for app-RKE2 VMhost for storage-RKE2 VM (OSD hosts pass NVMe through)

Placement rules​

  • Harvester master hosts (hv-01, hv-07, hv-13) carry no guest VM — one per zone.
  • Every application-cluster VM and every storage-cluster VM sits alone on its host.
  • OSD hosts pass all their NVMe through to the OSD VM; masters live on separate hosts.