Topics covering distributed ML training, parallelism strategies, and communication efficiency.
View All TagsBefore building BlockZero, we surveyed the literature on distributed Mixture-of-Experts training. This is a practical synthesis of what we found — what methods exist, what problems they solve, where they fall short, and what gap BlockZero fills.
This post introduces TEFT (Targeted Expert Fine-Tuning) — the protocol at the core of BlockZero — and explains how it achieves communication-efficient, quality-gated distributed MoE fine-tuning over a permissionless network. This is the research paper translated into blog form.
The single most important architectural decision in BlockZero's design was the choice of expert parallelism over data parallel or pipeline parallel training. This post explains why the other approaches would have failed — and why MoE expert parallelism is uniquely suited to decentralized networks.