sgl-project/sglang · error · ValueError

GGUF tensor {tensor.name} has inner dimension {inner_dim}, w

Error message

GGUF tensor {tensor.name} has inner dimension {inner_dim}, which is not a multiple of block size {block_size}

What it means

For a quantized GGUF tensor, the inner (last) dimension must be a multiple of the quantization block size (e.g. 32 for Q8_0, 256 for some FP8/Q4 variants). When the tensor has no explicit original-shape field, the loader cannot rebuild a dequantization-friendly layout, so it rejects the tensor.

Source

Thrown at python/sglang/multimodal_gen/runtime/loader/gguf_weights.py:114

            raise ValueError(
                f"GGUF tensor {tensor.name} declares original shape "
                f"{logical_shape}, which contains {math.prod(logical_shape)} "
                f"elements instead of {tensor.n_elements}"
            )
        is_quantized = int(weight_type) not in _UNQUANTIZED_TYPES
        dequantize_on_load = False
        if is_quantized:
            if len(logical_shape) != 2 or not tensor.name.endswith(".weight"):
                raise ValueError(
                    f"GGUF tensor {tensor.name} is quantized, but diffusion GGUF "
                    "currently supports packed data only for 2D linear .weight "
                    "tensors"
                )
            block_size, type_size = gguf.GGML_QUANT_SIZES[weight_type]
            inner_dim = logical_shape[-1]
            if inner_dim % block_size:
                if shape_field is None:
                    raise ValueError(
                        f"GGUF tensor {tensor.name} has inner dimension {inner_dim}, "
                        f"which is not a multiple of block size {block_size}"
                    )
                dequantize_on_load = True
                stored_shape = logical_shape
            else:
                stored_shape = (
                    *logical_shape[:-1],
                    inner_dim // block_size * type_size,
                )
            if (
                int(weight_type) in _SUPER_BLOCK_DEQUANT_TYPES
                and math.prod(logical_shape) % _GGML_SUPER_BLOCK
            ):
                raise ValueError(
                    f"GGUF tensor {tensor.name} is not aligned to "
                    f"{_GGML_SUPER_BLOCK}-element super blocks"
                )

View on GitHub (pinned to 0132848349)

Solutions

1. Use a GGUF export that includes the original-shape field so the loader can dequantize on load instead of failing
2. Re-quantize with a block size that divides the inner dimension (or pad the inner dimension to a block multiple before export)
3. If you control the writer, add the shape annotation field for each quantized tensor

Example fix

# before: Q8_0 tensor inner_dim=320, block_size=... -> 320 % block != 0, no shape field -> raise
# after: writer adds shape field, loader dequantizes on load
tensor.add_text(f'{name}.shape', '[320, 320]')  # ComfyUI-style annotation

from gguf import GGML_QUANT_SIZES
for t in reader.tensors:
    if t.tensor_type not in UNQUANTIZED:
        bs, _ = GGML_QUANT_SIZES[t.tensor_type]
        has_shape_field = reader.get_field(f'{t.name}.shape') is not None
        inner = t.shape[-1]
        if inner % bs and not has_shape_field:
            raise MisalignedQuantTensor(t.name, inner, bs)

def is_block_aligned_or_annotated(reader, t, block_size: int) -> bool:
    return t.shape[-1] % block_size == 0 or reader.get_field(f'{t.name}.shape') is not None

When it happens

Trigger: read_gguf_tensor_meta on a quantized tensor where shape_field is None and logical_shape[-1] % block_size != 0. Note: when a shape field IS present, this situation instead sets dequantize_on_load=True and does not raise.

Common situations: Quantized GGUF checkpoints whose inner dimensions (e.g. 320, 640 typical of diffusion models) are not multiples of the block size; GGUFs written without ComfyUI-style shape annotations; switching block-size-sensitive quant formats on an existing checkpoint.

Related errors

• GGUFConfig must be constructed from a GGUF checkpoint
• A GGUF encoder checkpoint cannot be combined with a second q
• Cannot parse checkpoint quantization for {component_name!r}:
• GGUF tensor {tensor.name} declares original shape {logical_s
• GGUF tensor {tensor.name} is quantized, but diffusion GGUF c

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/207546133c7def2b. Report an issue: GitHub.