Purplelinks = Thunderbolt 5 RDMA · Indigo dashed = Tailscale mesh
How distributed inference works
When a model is too large for a single machine, r1o splits it across nodes using tensor parallelism. Each node holds a shard of the model weights and runs its portion of every forward pass. After each layer, an all_reduce operation combines partial results across the RDMA fabric before the next layer begins.
The collective communication library powering this is JACCL — Jack and Angelos' Collective Communication Library, built directly into Apple's MLX framework. JACCL uses InfiniBand Verbs over Thunderbolt 5, delivering 3.5–6 GB/s sustained throughput per link with zero CPU copy overhead. No NVIDIA GPUs, no InfiniBand switches, no datacenter hardware — just Thunderbolt cables between Macs on your desk.
1. Shard
Model split across nodes
2. Compute
Each node runs its shard
3. Reduce
RDMA all_reduce combines results
What we contributed
Backend Selection Fix
Patched MLX to correctly select the JACCL backend over Ring when launching distributed inference. Without this, ring::init() silently claims the singleton and JACCL never initializes.
Group.split()
Implemented sub-group communication for mixed tensor parallelism and pipeline parallelism. Uses TCP fallback to avoid ibv_context deadlocks on Apple TB5 hardware.
4-Node Ring
Enabled the jaccl-ring backend for non-mesh topologies where not every node has a direct TB5 cable to every other node. Only left/right RDMA neighbors required.
ARP Filtering
3-layer filter in asmi for clean RDMA peer discovery: active RDMA device, link-local 169.254 IP on local interface, and not self. Eliminates phantom peers.
TB5 class
Per-link bandwidth
+38%
2-node speedup
Measured
Generation (27B class)
Your nodes
Unified cluster RAM