deepseek-ai/DeepSeek-V3 · error · AssertionError

Input tensors must have 2 dimensions

Error message

Input tensors must have 2 dimensions

What it means

Thrown by weight_dequant (inference/kernel.py:105): the kernel treats x as an (M, N) matrix with s of shape (M//block, N//block) and launches a 2-D tile grid, so both tensors must be exactly 2-D. Higher- or lower-rank tensors would make the block-to-scale indexing ambiguous, hence the assert.

Source

Thrown at inference/kernel.py:105

def weight_dequant(x: torch.Tensor, s: torch.Tensor, block_size: int = 128) -> torch.Tensor:
    """
    Dequantizes the given weight tensor using the provided scale tensor.

    Args:
        x (torch.Tensor): The quantized weight tensor of shape (M, N).
        s (torch.Tensor): The scale tensor of shape (M//block_size, N//block_size).
        block_size (int, optional): The block size to use for dequantization. Defaults to 128.

    Returns:
        torch.Tensor: The dequantized weight tensor of the same shape as `x`.

    Raises:
        AssertionError: If `x` or `s` are not contiguous or if their dimensions are not 2.
    """
    assert x.is_contiguous() and s.is_contiguous(), 'Input tensors must be contiguous'
    assert x.dim() == 2 and s.dim() == 2, 'Input tensors must have 2 dimensions'
    M, N = x.size()
    y = torch.empty_like(x, dtype=torch.get_default_dtype())
    grid = lambda meta: (triton.cdiv(M, meta['BLOCK_SIZE']), triton.cdiv(N, meta['BLOCK_SIZE']))
    weight_dequant_kernel[grid](x, s, y, M, N, BLOCK_SIZE=block_size)
    return y


fp8_gemm_configs = [
    Config({'BLOCK_SIZE_M': block_m, 'BLOCK_SIZE_N': block_n, 'BLOCK_SIZE_K': 128}, num_stages=num_stages, num_warps=8)
    for block_m in [16, 32, 64] for block_n in [32, 64, 128] for num_stages in [3, 4, 5, 6]
]

@triton.autotune(configs=fp8_gemm_configs, key=['N', 'K'])
@triton.jit
def fp8_gemm_kernel(a_ptr, b_ptr, c_ptr,
                    a_s_ptr, b_s_ptr,
                    M, N: tl.constexpr, K: tl.constexpr,
                    BLOCK_SIZE_M: tl.constexpr,

View on GitHub (pinned to 9b4e9788e4)

Solutions

1. Only dequantize 2-D FP8 weights: guard with x.dim() == 2 and weight.element_size() == 1
2. Flatten/reshape deliberately if you truly have a higher-rank weight: x = x.reshape(-1, x.size(-1)) with matching scale reshape
3. Skip 1-D tensors (norms, biases) — they are stored in bf16 already

Example fix

# before — dequant everything in the dict
for name, t in state_dict.items():
    out[name] = weight_dequant(t, scales[name])

# after
for name, t in state_dict.items():
    if t.dim() == 2 and t.element_size() == 1:
        out[name] = weight_dequant(t, scales[f'{name}_scale_inv'])
    else:
        out[name] = t

assert x.dim() == 2 and s.dim() == 2, (
    f"weight_dequant needs 2-D tensors, got x.dim()={x.dim()}, s.dim()={s.dim()}; "
    f"skip 1-D norms/biases or reshape batched weights"
)

def is_dequantizable_weight(t: torch.Tensor) -> bool:
    return t.dim() == 2 and t.element_size() == 1  # 2-D FP8 weight

When it happens

Trigger: Calling weight_dequant on a 1-D bias/scale vector or a 3-D+ tensor (e.g. a batched weight or an unsqueezed tensor). Also fires if x and s come from mismatched checkpoints with different rank.

Common situations: Custom scripts iterating over ALL tensors in a state dict (including 1-D norms/biases) and calling weight_dequant unconditionally instead of only on element_size()==1 2-D weights as fp8_cast_bf16.py does.

Related errors

• Last dimension size must be divisible by block_size (block_s
• Input tensors must be contiguous
• Input tensor must be contiguous
• Scaling factor tensors must be contiguous
• Warning: Missing scale_inv tensor for ${weight_name}, skippi

AI-assisted analysis of deepseek-ai/DeepSeek-V3@9b4e9788e4 (2026-08-14). Data as JSON: /api/errors/6957d27a33dbb3b9. Report an issue: GitHub.