{"record":{"id":"dd116ac5fd174593","repo":"deepseek-ai/DeepSeek-V3","slug":"input-features-must-be-divisible-by-world-size-wo","errorCode":null,"errorMessage":"Input features must be divisible by world size (world_size=${world_size})","messagePattern":"Input features must be divisible by world size \\(world_size=(.+?)\\)","errorType":"exception","errorClass":"AssertionError","httpStatus":null,"severity":"critical","filePath":"inference/model.py","lineNumber":248,"sourceCode":"        Returns:\n            torch.Tensor: Transformed tensor with column-parallel computation.\n        \"\"\"\n        y = linear(x, self.weight, self.bias)\n        return y\n\n\nclass RowParallelLinear(Linear):\n    \"\"\"\n    Linear layer with row parallelism, splitting input features across distributed processes.\n\n    Args:\n        in_features (int): Total number of input features.\n        out_features (int): Number of output features.\n        bias (bool): Whether to include a bias term. Defaults to False.\n        dtype (optional): Data type for the layer. Defaults to `torch.bfloat16`.\n    \"\"\"\n    def __init__(self, in_features: int, out_features: int, bias: bool = False, dtype = None):\n        assert in_features % world_size == 0, f\"Input features must be divisible by world size (world_size={world_size})\"\n        self.part_in_features = in_features // world_size\n        super().__init__(self.part_in_features, out_features, bias, dtype)\n\n    def forward(self, x: torch.Tensor) -> torch.Tensor:\n        \"\"\"\n        Forward pass for row parallel linear layer.\n\n        Args:\n            x (torch.Tensor): Input tensor.\n\n        Returns:\n            torch.Tensor: Transformed tensor with row-parallel computation.\n        \"\"\"\n        y = linear(x, self.weight)\n        if world_size > 1:\n            dist.all_reduce(y)\n        if self.bias is not None:\n            y += self.bias","sourceCodeStart":230,"sourceCodeEnd":266,"githubUrl":"https://github.com/deepseek-ai/DeepSeek-V3/blob/9b4e9788e4a3a731f7567338ed15d3ec549ce03b/inference/model.py#L230-L266","documentation":"Thrown when constructing RowParallelLinear (inference/model.py:248), which shards a linear layer's INPUT features across ranks (e.g. wo, w2 projections take partial inputs and all-reduce the result). in_features must divide evenly so each rank holds in_features // world_size rows of the weight. Assert fires at model build time. The ${world_size} placeholder rendering is broken in this build.","triggerScenarios":"Building the model with world_size > 1 where an input dimension (attention out dim = n_heads * v_head_dim, or n_shared_experts * moe_inter_dim for shared expert down-proj) is not divisible by the process count. Triggered by odd GPU counts or edited ModelArgs.","commonSituations":"Non-power-of-two GPU launches against DeepSeek-V3 (7168 hidden, 128*128 attn out dim); reducing n_heads or n_shared_experts in a test config; mismatched world_size inherited from environment.","solutions":["Use 2, 4, 8, or 16 GPUs — DeepSeek-V3 dims are divisible by these","If editing dims in ModelArgs, keep every dim consumed by RowParallelLinear divisible by world_size (hidden 7168, shared-expert inter dim 2048*n_shared_experts)","Confirm actual world_size via dist.get_world_size() before building the model"],"exampleFix":"# before: 5-way parallel, 7168 % 5 != 0\ntorchrun --nproc_per_node 5 ...\n\n# after\ntorchrun --nproc_per_node 8 ...","handlingStrategy":"validation","validationCode":"world = dist.get_world_size() if dist.is_initialized() else 1\n# RowParallelLinear inputs: attn out dim and shared-expert inter dim\nassert (args.n_heads * args.v_head_dim) % world == 0\nassert (args.n_shared_experts * args.moe_inter_dim) % world == 0","typeGuard":"def row_parallel_ok(in_features: int, world_size: int) -> bool:\n    return in_features % world_size == 0","tryCatchPattern":null,"preventionTips":["Use divisor-friendly GPU counts","When editing dims (n_heads, n_shared_experts, moe_inter_dim), keep every product divisible by world_size","Preflight-validate the config before torchrun launches 8 processes that all crash"],"tags":["distributed","tensor-parallelism","linear-layer","model-construction","deepseek"],"backgroundTag":null,"analyzedSha":"9b4e9788e4a3a731f7567338ed15d3ec549ce03b","analyzedAt":"2026-08-14T19:02:32.748Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}