Skip to main content

SN Owner Design Decision

Peer-to-peer. This is what we learned.

Building BlockZero required solving problems that many AI training systems have not yet encountered — because most are not coordinating distributed contributors around real-world quality standards with a paying customer at the end of the pipeline.

It’s one thing to run experiments or optimize benchmarks. It’s another to deliver consistent, production-grade results that a business depends on. When real customers are involved, reliability, evaluation rigor, and operational discipline become non-negotiable.

What follows is not a claim that there is only one way to design distributed training systems. It is a record of the challenges we faced, the tradeoffs we encountered, and the architectural decisions we made in response. We’re sharing what we learned — not as a prescription, but as a transparent explanation of why the system looks the way it does.

Four Design Principles​

These aren't abstract values. They came directly from watching what goes wrong when you don't follow them.

1. Solve Overfitting - Align Training and Evaluation​

The first thing that happens on any incentivized network is that miners optimize for whatever you measure. This is not a bug — it is the rational behavior of economically motivated participants. If your evaluation is narrow, miners will train to it exclusively, because they have no incentive to do more.

The implication is that your validation distribution must match your training distribution. If miners are training on a 200B token math dataset, validating them on a 2B token public benchmark creates a misalignment: miners will learn to overfit the benchmark rather than genuinely improve on the training task.

In BlockZero, validators evaluate on held-out data drawn from the same distribution miners train on. The evaluation signal stays honest because it can't be gamed independently of actual training quality.

2. Keep Validation Cheap​

If validators must reproduce or redo the miners' work to verify it, your subnet does not scale. Validation becomes the bottleneck, and you end up with a system where the "decentralized training" is actually verified by a small number of validators running near-equivalent compute.

BlockZero uses Proof-of-Loss validation: the validator receives the submitted expert artifact and evaluates it on held-out data. No re-training. No replication of the forward-pass computation at training scale. The validator runs one inference pass on a small held-out set and scores the loss reduction. This is orders of magnitude cheaper than re-training would be, and it stays honest because loss on a genuinely held-out set cannot be faked without actually training well.

The principle: validators should be lightweight verifiers, not redundant trainers. Miners are the intelligence in the system.

3. Run Kaggle or Compute Arbitrage? Reward and Integrate Contribution, Not Competition​

Winner-takes-all architectures waste compute and create fragile participation economics. When only the top miner's result is used, all the compute spent by second, third, and fourth place is discarded. Over time, this discourages participation by miners who aren't certain they can always win — which reduces network diversity and resilience.

BlockZero merges the top-N miner updates. A miner that finishes second still contributes real signal to the merged expert. The merged result is typically better than any single submission.

The reward structure is "top-N take-most": contributions are weighted by quality, not winner-takes-all. The earnings floor for honest participants is viable. The incentive to improve is strong. Both conditions are necessary for a healthy, growing miner pool.

4. Build for Economic Sustainability​

A subnet that requires miners to spend more than they earn will eventually lose its miners. This sounds obvious, but it's easy to underweight when you're focused on model quality.

The specific risk: if validation requires miners to hold the full model and run expensive evaluations on every submission, participation costs balloon as the network grows. We've seen other subnets run into this — the top miners have large GPU clusters and can afford it, but everyone else gets priced out.

BlockZero addresses this by:

  • Keeping per-miner hardware requirements low via expert parallelism (each miner holds a fraction of the model)
  • Keeping communication costs low via DILoCo async sync (hourly, not per-step)
  • Using a "top-N take-most" reward curve that creates a viable floor for honest participants

The goal is a network where mid-tier hardware can participate meaningfully, earn reliably, and improve over time.

Where to Go Next​