Skip to main content

Inter-Validator Merging

Multiple validators can run simultaneously on the BlockZero subnet. Inter-validator merging ensures that all active validators converge on the same model update before calling set_weights() — preventing conflicting writes to the blockchain and improving robustness against individual validator failures.

Why Merging Is Needed​

Different validators may:

  • Evaluate slightly different subsets of miners (if some checkpoints download successfully for one validator but not another)
  • Have different timing relative to the submit phase
  • Experience network issues that cause them to miss a submission or two

Without merging, different validators would apply different gradient updates and diverge over time. Bittensor's Yuma Consensus algorithm handles weight agreement at the protocol level, but aligned validators who explicitly synchronize their gradients produce more consistent and predictable model updates.

The Merging Protocol​

1. Shared Random Seed​

Before each evaluation cycle, validators use get_combined_validator_seed() to agree on a shared random seed:

seed = get_combined_validator_seed(validator_hotkeys, block_number)

This seed is derived from all active validator hotkeys and the current block number — it is deterministic and observable by all validators, requiring no communication. The seed is used to determine the evaluation batch for Proof-of-Loss — ensuring all validators evaluate on exactly the same data and produce the same loss measurements for honest miners.

2. Local Scoring​

Each validator runs Proof-of-Loss independently on all submitted checkpoints. Because they use the same random seed (same evaluation batch) and the same base model, honest validators produce the same w_i scores for each miner.

3. Gradient Synchronization​

After local scoring and gradient aggregation, validators synchronize their locally aggregated gradients across all active validators using a decentralized allreduce. Each validator averages its local gradients with those of peer validators per expert group, ensuring all validators converge on the same global update before running the outer optimizer.

4. Byzantine-Fault Tolerant Aggregation​

Once all validators' gradients are collected (or the consensus_timeout is reached), the final merged gradient is computed:

# Simple version: average of all validators' local gradients
merged_gradients = average_gradients(all_validator_gradients, min_validators=2)

In practice, outlier gradients are detected and down-weighted. A validator whose gradients deviate significantly from the group (e.g., by more than 3σ) has reduced influence in the final aggregation. This provides Byzantine-fault tolerance: a single malicious or malfunctioning validator cannot unilaterally manipulate the model update.

5. Calling set_weights()​

Once merging is complete, all validators call set_weights() with the agreed-upon weight vector:

subtensor.set_weights(
    wallet=validator_wallet,
    netuid=netuid,
    uids=miner_uids,
    weights=final_weights,
    version_key=version_key,
)

Bittensor's Yuma Consensus algorithm on the blockchain further aggregates across all validators — so even if one validator calls with slightly different weights, the blockchain-level aggregation produces consistent emissions.

Stake-Weighted Influence​

Each validator's contribution to the merging is weighted by their stake. A validator with more TAO staked has more influence in the final aggregated gradient. This aligns incentives: validators with more at stake have stronger economic motivation to be honest.

Merging Timeout​

If not all validators respond within consensus_timeout seconds (default: 60), the merging proceeds with the validators that did respond. A single offline validator does not block the process.

If fewer than the minimum required validators (default: 2) respond, the merging is considered failed for that cycle — no set_weights() is called, and the previous weights remain in effect.

Figure: inter-validator-consensus Figure: Inter-validator merging flow. Each validator scores independently, synchronizes gradients via allreduce, aggregates with BFT tolerance, and calls set_weights() with the agreed vector.

note

The inter-validator communication is separate from the miner-validator communication. Miners never participate in inter-validator merging directly — they only communicate with the validator's HTTP server.