deepseek-ai/DeepSeek-V3 · critical · AssertionError

Input features must be divisible by world size (world_size=$

Error message

Input features must be divisible by world size (world_size=${world_size})

What it means

Thrown when constructing RowParallelLinear (inference/model.py:248), which shards a linear layer's INPUT features across ranks (e.g. wo, w2 projections take partial inputs and all-reduce the result). in_features must divide evenly so each rank holds in_features // world_size rows of the weight. Assert fires at model build time. The ${world_size} placeholder rendering is broken in this build.

Source

Thrown at inference/model.py:248

        Returns:
            torch.Tensor: Transformed tensor with column-parallel computation.
        """
        y = linear(x, self.weight, self.bias)
        return y


class RowParallelLinear(Linear):
    """
    Linear layer with row parallelism, splitting input features across distributed processes.

    Args:
        in_features (int): Total number of input features.
        out_features (int): Number of output features.
        bias (bool): Whether to include a bias term. Defaults to False.
        dtype (optional): Data type for the layer. Defaults to `torch.bfloat16`.
    """
    def __init__(self, in_features: int, out_features: int, bias: bool = False, dtype = None):
        assert in_features % world_size == 0, f"Input features must be divisible by world size (world_size={world_size})"
        self.part_in_features = in_features // world_size
        super().__init__(self.part_in_features, out_features, bias, dtype)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        """
        Forward pass for row parallel linear layer.

        Args:
            x (torch.Tensor): Input tensor.

        Returns:
            torch.Tensor: Transformed tensor with row-parallel computation.
        """
        y = linear(x, self.weight)
        if world_size > 1:
            dist.all_reduce(y)
        if self.bias is not None:
            y += self.bias

View on GitHub (pinned to 9b4e9788e4)

Solutions

Use 2, 4, 8, or 16 GPUs — DeepSeek-V3 dims are divisible by these
If editing dims in ModelArgs, keep every dim consumed by RowParallelLinear divisible by world_size (hidden 7168, shared-expert inter dim 2048*n_shared_experts)
Confirm actual world_size via dist.get_world_size() before building the model

Example fix

# before: 5-way parallel, 7168 % 5 != 0
torchrun --nproc_per_node 5 ...

# after
torchrun --nproc_per_node 8 ...

Defensive patterns

Strategy: validation

Validate before calling

world = dist.get_world_size() if dist.is_initialized() else 1
# RowParallelLinear inputs: attn out dim and shared-expert inter dim
assert (args.n_heads * args.v_head_dim) % world == 0
assert (args.n_shared_experts * args.moe_inter_dim) % world == 0

Type guard

def row_parallel_ok(in_features: int, world_size: int) -> bool:
    return in_features % world_size == 0

Prevention

Use divisor-friendly GPU counts
When editing dims (n_heads, n_shared_experts, moe_inter_dim), keep every product divisible by world_size
Preflight-validate the config before torchrun launches 8 processes that all crash

When it happens

Trigger: Building the model with world_size > 1 where an input dimension (attention out dim = n_heads * v_head_dim, or n_shared_experts * moe_inter_dim for shared expert down-proj) is not divisible by the process count. Triggered by odd GPU counts or edited ModelArgs.

Common situations: Non-power-of-two GPU launches against DeepSeek-V3 (7168 hidden, 128*128 attn out dim); reducing n_heads or n_shared_experts in a test config; mismatched world_size inherited from environment.

Related errors

AI-assisted analysis of deepseek-ai/DeepSeek-V3@9b4e9788e4 (2026-08-14). Data as JSON: /api/errors/dd116ac5fd174593. Report an issue: GitHub.