Deep dives into Mixture-of-Experts architecture, expert routing, and specialization.
View All TagsBefore building BlockZero, we surveyed the literature on distributed Mixture-of-Experts training. This is a practical synthesis of what we found — what methods exist, what problems they solve, where they fall short, and what gap BlockZero fills.
Catastrophic forgetting is the central unsolved problem of continual learning. You train a model to be better at task B, and it forgets how to do task A. The more you specialize, the more you destroy. BlockZero's MoE architecture makes this tradeoff avoidable — by construction.
Most AI customization is one-off work. A consulting firm spends six months fine-tuning a model for a client, and when the engagement ends, all that accumulated knowledge walks out the door. The next client starts from scratch. There is no compounding.
BlockZero is built around a fundamentally different model: every training job produces a reusable expert module that compounds in value with every subsequent use.
This post introduces TEFT (Targeted Expert Fine-Tuning) — the protocol at the core of BlockZero — and explains how it achieves communication-efficient, quality-gated distributed MoE fine-tuning over a permissionless network. This is the research paper translated into blog form.
The single most important architectural decision in BlockZero's design was the choice of expert parallelism over data parallel or pipeline parallel training. This post explains why the other approaches would have failed — and why MoE expert parallelism is uniquely suited to decentralized networks.