{"record":{"id":"06e860a252c9d9a3","repo":"deepseek-ai/DeepSeek-V3","slug":"vocabulary-size-must-be-divisible-by-world-size-w","errorCode":null,"errorMessage":"Vocabulary size must be divisible by world size (world_size=${world_size})","messagePattern":"Vocabulary size must be divisible by world size \\(world_size=(.+?)\\)","errorType":"exception","errorClass":"AssertionError","httpStatus":null,"severity":"critical","filePath":"inference/model.py","lineNumber":101,"sourceCode":"    rope_factor: float = 40\n    beta_fast: int = 32\n    beta_slow: int = 1\n    mscale: float = 1.\n\n\nclass ParallelEmbedding(nn.Module):\n    \"\"\"\n    Embedding layer with parallelism support across distributed processes.\n\n    Args:\n        vocab_size (int): Vocabulary size.\n        dim (int): Embedding dimension.\n    \"\"\"\n    def __init__(self, vocab_size: int, dim: int):\n        super().__init__()\n        self.vocab_size = vocab_size\n        self.dim = dim\n        assert vocab_size % world_size == 0, f\"Vocabulary size must be divisible by world size (world_size={world_size})\"\n        self.part_vocab_size = (vocab_size // world_size)\n        self.vocab_start_idx = rank * self.part_vocab_size\n        self.vocab_end_idx = self.vocab_start_idx + self.part_vocab_size\n        self.weight = nn.Parameter(torch.empty(self.part_vocab_size, self.dim))\n\n    def forward(self, x: torch.Tensor) -> torch.Tensor:\n        \"\"\"\n        Forward pass for parallel embedding layer.\n\n        Args:\n            x (torch.Tensor): Input tensor containing token indices.\n\n        Returns:\n            torch.Tensor: Embedded representations.\n\n        Raises:\n            ValueError: If `world_size` is not defined.\n        \"\"\"","sourceCodeStart":83,"sourceCodeEnd":119,"githubUrl":"https://github.com/deepseek-ai/DeepSeek-V3/blob/9b4e9788e4a3a731f7567338ed15d3ec549ce03b/inference/model.py#L83-L119","documentation":"Thrown when constructing ParallelEmbedding in inference/model.py:101, which shards the token embedding table across distributed ranks. The vocabulary dimension must split evenly across all processes in the torch.distributed world, because each rank owns a contiguous slice vocab_size // world_size. This assert fires at model construction time, before any weights are loaded. The literal '${world_size}' in the message indicates an f-string placeholder bug — the actual world size may not render in the printed message.","triggerScenarios":"Instantiating Transformer/ParallelEmbedding with world_size > 1 where vocab_size (e.g. 129280) is not divisible by the number of launched processes. Concretely: launching with torchrun --nproc_per_node=N where N does not divide the config's vocab_size, or running with world_size=1 after dist init failed and vocab was padded/changed in a custom config.","commonSituations":"Odd GPU counts (e.g. 3, 5, 6, 7 GPUs) against DeepSeek-V3's vocab of 129280 (divisible by 2,4,8,16,64... but not by 3/5/6/7); editing config.json vocab_size; running the single-GPU code path on a node where dist.init_process_group already set a larger world_size.","solutions":["Launch with a GPU count that divides the vocab size — powers of two (2, 4, 8, 16) work for DeepSeek-V3's 129280 vocab","Check the config's vocab_size vs your nproc_per_node: python -c \"print(129280 % 8)\" before launching","If you truly need an odd world size, pad vocab_size to the next multiple of world_size in your config and adjust the tokenizer/remap the lm_head rows","Verify world_size is what you expect: print(dist.get_world_size()) right before model construction"],"exampleFix":"# before: 3 GPUs, 129280 % 3 != 0\ntorchrun --nproc_per_nodes 3 generate.py ...\n\n# after: use a divisor of vocab_size\ntorchrun --nproc_per_node 8 generate.py --ckpt-path ... --config configs/config_671b.json","handlingStrategy":"validation","validationCode":"import torch.distributed as dist\n\nworld = dist.get_world_size() if dist.is_initialized() else 1\nVOCAB = 129280  # from your config\nassert VOCAB % world == 0, f\"vocab {VOCAB} not divisible by world_size {world}; use 2/4/8/16 GPUs\"","typeGuard":"def divisible(dim: int, world: int) -> bool:\n    return dim % world == 0","tryCatchPattern":null,"preventionTips":["Launch only power-of-two GPU counts for DeepSeek models","Assert divisibility of vocab_size, hidden dim, expert count against world_size in a startup check before building the model","Print dist.get_world_size() at startup to catch stale distributed env vars"],"tags":["distributed","tensor-parallelism","embedding","model-construction","deepseek"],"backgroundTag":null,"analyzedSha":"9b4e9788e4a3a731f7567338ed15d3ec549ce03b","analyzedAt":"2026-08-14T19:02:32.748Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}