deepseek-ai/DeepSeek-V3 · critical · AssertionError

Vocabulary size must be divisible by world size (world_size=

Error message

Vocabulary size must be divisible by world size (world_size=${world_size})

What it means

Thrown when constructing ParallelEmbedding in inference/model.py:101, which shards the token embedding table across distributed ranks. The vocabulary dimension must split evenly across all processes in the torch.distributed world, because each rank owns a contiguous slice vocab_size // world_size. This assert fires at model construction time, before any weights are loaded. The literal '${world_size}' in the message indicates an f-string placeholder bug — the actual world size may not render in the printed message.

Source

Thrown at inference/model.py:101

    rope_factor: float = 40
    beta_fast: int = 32
    beta_slow: int = 1
    mscale: float = 1.


class ParallelEmbedding(nn.Module):
    """
    Embedding layer with parallelism support across distributed processes.

    Args:
        vocab_size (int): Vocabulary size.
        dim (int): Embedding dimension.
    """
    def __init__(self, vocab_size: int, dim: int):
        super().__init__()
        self.vocab_size = vocab_size
        self.dim = dim
        assert vocab_size % world_size == 0, f"Vocabulary size must be divisible by world size (world_size={world_size})"
        self.part_vocab_size = (vocab_size // world_size)
        self.vocab_start_idx = rank * self.part_vocab_size
        self.vocab_end_idx = self.vocab_start_idx + self.part_vocab_size
        self.weight = nn.Parameter(torch.empty(self.part_vocab_size, self.dim))

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        """
        Forward pass for parallel embedding layer.

        Args:
            x (torch.Tensor): Input tensor containing token indices.

        Returns:
            torch.Tensor: Embedded representations.

        Raises:
            ValueError: If `world_size` is not defined.
        """

View on GitHub (pinned to 9b4e9788e4)

Solutions

Launch with a GPU count that divides the vocab size — powers of two (2, 4, 8, 16) work for DeepSeek-V3's 129280 vocab
Check the config's vocab_size vs your nproc_per_node: python -c "print(129280 % 8)" before launching
If you truly need an odd world size, pad vocab_size to the next multiple of world_size in your config and adjust the tokenizer/remap the lm_head rows
Verify world_size is what you expect: print(dist.get_world_size()) right before model construction

Example fix

# before: 3 GPUs, 129280 % 3 != 0
torchrun --nproc_per_nodes 3 generate.py ...

# after: use a divisor of vocab_size
torchrun --nproc_per_node 8 generate.py --ckpt-path ... --config configs/config_671b.json

Defensive patterns

Strategy: validation

Validate before calling

import torch.distributed as dist

world = dist.get_world_size() if dist.is_initialized() else 1
VOCAB = 129280  # from your config
assert VOCAB % world == 0, f"vocab {VOCAB} not divisible by world_size {world}; use 2/4/8/16 GPUs"

Type guard

def divisible(dim: int, world: int) -> bool:
    return dim % world == 0

Prevention

Launch only power-of-two GPU counts for DeepSeek models
Assert divisibility of vocab_size, hidden dim, expert count against world_size in a startup check before building the model
Print dist.get_world_size() at startup to catch stale distributed env vars

When it happens

Trigger: Instantiating Transformer/ParallelEmbedding with world_size > 1 where vocab_size (e.g. 129280) is not divisible by the number of launched processes. Concretely: launching with torchrun --nproc_per_node=N where N does not divide the config's vocab_size, or running with world_size=1 after dist init failed and vocab was padded/changed in a custom config.

Common situations: Odd GPU counts (e.g. 3, 5, 6, 7 GPUs) against DeepSeek-V3's vocab of 129280 (divisible by 2,4,8,16,64... but not by 3/5/6/7); editing config.json vocab_size; running the single-GPU code path on a node where dist.init_process_group already set a larger world_size.

Related errors

AI-assisted analysis of deepseek-ai/DeepSeek-V3@9b4e9788e4 (2026-08-14). Data as JSON: /api/errors/06e860a252c9d9a3. Report an issue: GitHub.