sgl-project/sglang · error · ValueError

Comfy NVFP4 layer {prefix!r} needs a 2D packed weight, got {

Error message

Comfy NVFP4 layer {prefix!r} needs a 2D packed weight, got {weight_shape}

What it means

The NVFP4 code path assumes a 2D packed weight matrix (out_features, in_features/2 packed bytes). If `{prefix}.weight` has any other rank — e.g. a 4D conv kernel or a 1D vector — the subsequent shape arithmetic (logical_input_size = weight_shape[1]*2, expected scale shape) would be meaningless, so the checker rejects it up front.

Source

Thrown at python/sglang/multimodal_gen/runtime/utils/quantization_utils.py:265

                    f"and convrot_groupsize={convrot_group_size}"
                )
            continue
        if marker_format == "nvfp4":
            weight_dtype, weight_shape = checkpoint_meta[f"{prefix}.weight"]
            scale_dtype, scale_shape = checkpoint_meta[f"{prefix}.weight_scale"]
            scale_2_dtype, scale_2_shape = checkpoint_meta[f"{prefix}.weight_scale_2"]
            if weight_dtype != "U8" or scale_dtype != "F8_E4M3":
                raise ValueError(
                    f"Comfy NVFP4 layer {prefix!r} needs U8 packed weights and "
                    f"FP8 block scales, got {weight_dtype} and {scale_dtype}"
                )
            if scale_2_dtype != "F32" or scale_2_shape not in ((), (1,)):
                raise ValueError(
                    f"Comfy NVFP4 layer {prefix!r} needs a scalar F32 "
                    f"weight_scale_2, got {scale_2_dtype}{scale_2_shape}"
                )
            if len(weight_shape) != 2:
                raise ValueError(
                    f"Comfy NVFP4 layer {prefix!r} needs a 2D packed weight, "
                    f"got {weight_shape}"
                )
            logical_input_size = weight_shape[1] * 2
            expected_scale_shape = (weight_shape[0], logical_input_size // 16)
            if logical_input_size % 16 or scale_shape != expected_scale_shape:
                raise ValueError(
                    f"Comfy NVFP4 layer {prefix!r} has incompatible weight/scale "
                    f"shapes: {weight_shape} and {scale_shape}"
                )
            pre_quant_scale_key = f"{prefix}.pre_quant_scale"
            marker["_has_pre_quant_scale"] = pre_quant_scale_key in checkpoint_meta
            if marker["_has_pre_quant_scale"]:
                pre_scale_dtype, pre_scale_shape = checkpoint_meta[pre_quant_scale_key]
                if pre_scale_dtype not in ("BF16", "F16", "F32") or (
                    pre_scale_shape != (logical_input_size,)
                ):
                    raise ValueError(

View on GitHub (pinned to 0132848349)

Solutions

Inspect weight_shape for the failing prefix to confirm its rank
Re-export with the conv weight flattened to (out_channels, in_channels*kh*kw/2) if the kernel supports matrix form
If the layer genuinely can't be 2D, exclude it from nvfp4 quantization (leave it higher precision) in the export recipe
Verify the marker's 'format' field matches the actual tensor layout

Example fix

# before: weight shape (256, 128, 3, 3)
# after (flattened, packed): weight shape (256, 2304)  # 4608/2 bytes

Defensive patterns

Strategy: validation

Validate before calling

_, w_shape = checkpoint_meta[f"{prefix}.weight"]
if marker.get("format") == "nvfp4":
    assert len(w_shape) == 2, f"{prefix}: expected 2D packed weight, got {w_shape}"

Type guard

def is_2d_weight(meta: dict, prefix: str) -> bool:
    return len(meta[f"{prefix}.weight"][1]) == 2

Prevention

Flatten conv weights to 2D in the export pipeline or exclude convs from nvfp4
Add a rank assertion in your export script for every quantized weight
Tag only matrix-multiply layers with the nvfp4 format marker

When it happens

Trigger: `inspect_comfy_quant_markers` on an nvfp4 marker whose weight tensor is not 2D — typically a conv layer exported without being reshaped into a 2D matrix, or a mistakenly tagged layer.

Common situations: Quantizing convolutional/patch-embed layers as nvfp4 without flattening spatial dims to a matrix; exporter bugs that leave conv weights 4D; format marker applied to the wrong module prefix.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/592ee17ba9d2d430. Report an issue: GitHub.