sgl-project/sglang · error · ValueError

Only per-tensor FP8 scales are supported for diffusion…

Error message

Only per-tensor FP8 scales are supported for diffusion checkpoints, got shape {tuple(scale.shape)}.

What it means

Raised when quantization scale tensors in a diffusion checkpoint have a non-scalar shape; only per-tensor (scalar) FP8 scales are supported by this tool.

Solutions

Re-export from ModelOpt with per-tensor (scalar) quantization for weights
If conversion happens upstream in your pipeline, squeeze/reduce scales to per-tensor before invoking the tool
Verify the export's quantization_config specifies per-tensor scales

Defensive patterns

Strategy: validation

Validate before calling

if weight_scale.numel() != 1:
    raise SystemExit(f"non-scalar scale {tuple(weight_scale.shape)}; re-export per-tensor")

Try / catch

try:
    q = quantize_fp8_weight(w, s)
except ValueError as e:
    logger.error("%s", e)
    raise

Prevention

Export ModelOpt checkpoints with per-tensor weight quantization
Validate scale shapes in any pre-processing/conversion step

When it happens

Trigger: Feeding a ModelOpt export whose weight_scale tensors are per-channel (e.g. shape [N]) or per-block instead of scalar.

Common situations: Exporting with a newer ModelOpt version that defaults to per-channel quantization; hand-modified checkpoints converting scales to vectors.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/b60927ce173608ea. Report an issue: GitHub.

Appendix: source

Thrown at python/sglang/multimodal_gen/tools/build_modelopt_fp8_transformer.py:506

            )

    return {
        weight_name: scale_tensors
        for weight_name, scale_tensors in scale_map.items()
        if {"weight_scale", "input_scale"} <= set(scale_tensors)
    }


def quantize_fp8_weight(
    weight: torch.Tensor,
    weight_scale: torch.Tensor,
) -> torch.Tensor:
    if weight.dtype == torch.float8_e4m3fn:
        return weight.contiguous()

    scale = weight_scale.to(weight.device, dtype=torch.float32)
    if scale.numel() != 1:
        raise ValueError(
            "Only per-tensor FP8 scales are supported for diffusion checkpoints, "
            f"got shape {tuple(scale.shape)}."
        )

    quantized = (weight.to(torch.float32) / scale.reshape(1)).to(torch.float8_e4m3fn)
    return quantized.cpu().contiguous()


def _copy_non_shard_files(source_dir: str, output_dir: str) -> None:
    ignored = set(INDEX_FILENAMES)
    for entry in os.listdir(source_dir):
        if entry.endswith(".safetensors") or entry in ignored:
            continue
        source_path = os.path.join(source_dir, entry)
        output_path = os.path.join(output_dir, entry)
        if os.path.isdir(source_path):
            shutil.copytree(source_path, output_path, dirs_exist_ok=True)
        else:

View on GitHub (pinned to 0132848349)