sgl-project/sglang · error · ValueError

Humming expected DeepEP FP8 hidden states and group-128 scal

Error message

Humming expected DeepEP FP8 hidden states and group-128 scales.

What it means

The mirror of error 4058: the humming layer expects DeepEP FP8 dispatch (expects_fp8 True) but hidden states are not float8_e4m3fn or hidden_states_scale is None, so the FP8 group-quantized path cannot proceed.

Source

Thrown at python/sglang/srt/layers/moe/moe_runner/humming.py:822

def _validate_deepep_dispatch_input(
    hidden_states: torch.Tensor,
    hidden_states_scale: torch.Tensor | None,
    layer: torch.nn.Module,
) -> None:
    expects_fp8 = layer._humming_uses_deepep_fp8_dispatch
    is_fp8 = hidden_states.dtype == torch.float8_e4m3fn
    if not expects_fp8:
        if is_fp8 or hidden_states_scale is not None:
            raise ValueError(
                "DeepEP returned FP8 input while Humming is configured for BF16 "
                "dispatch."
            )
        return

    if not is_fp8 or hidden_states_scale is None:
        raise ValueError(
            "Humming expected DeepEP FP8 hidden states and group-128 scales."
        )

    expected_groups, remainder = divmod(hidden_states.size(-1), 128)
    if (
        remainder != 0
        or hidden_states_scale.dtype != torch.float32
        or hidden_states_scale.size(-1) != expected_groups
        or hidden_states_scale.numel() != hidden_states.numel() // 128
    ):
        raise ValueError(
            "Humming requires row-major FP32 DeepEP scales with group size 128."
        )
    meta = layer.humming_metas["w13"]
    if meta.a_dtype != dtypes.float8e4m3 or meta.input_scale_group_size != 128:
        raise ValueError("Humming w13 must use FP8 group-128 input metadata.")

View on GitHub (pinned to 0132848349)

Solutions

1. Enable the DeepEP FP8 dispatch options so dispatch returns fp8 tensors plus group-128 scales
2. Or turn off the humming fp8-dispatch expectation so BF16 dispatch is used consistently
3. Verify hidden size is a multiple of 128 (required for group-128 scales)

assert hidden_states.dtype == torch.float8_e4m3fn and hidden_states_scale is not None, \
    'expected FP8 hidden states + group-128 scales from DeepEP dispatch'

def is_valid_fp8_dispatch(h: torch.Tensor, s) -> bool:
    return h.dtype == torch.float8_e4m3fn and s is not None and h.size(-1) % 128 == 0

When it happens

Trigger: Layer configured with _humming_uses_deepep_fp8_dispatch=True while the DeepEP dispatcher returned BF16 tensors (no scales), reaching _validate_deepep_dispatch_input via pre_permute_deepep_ll_to_humming or pre_permute_deepep_normal_to_humming.

Common situations: Turning on humming FP8 dispatch config without enabling the deepep FP8 dispatch flags (or vice versa); DP/EP setups where only some ranks enable fp8 dispatch; flag drift across sglang versions.

Related errors

• DeepEP returned FP8 input while Humming is configured for BF
• The hpc_ops MoE runner backend only supports FP8-quantized M
• Unknown gemm type: {gemm_type}
• Unsupported activation: {self.activation}
• cannot found moe_block_size for shape {valid_shape_m}

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/44690bab1b49db5b. Report an issue: GitHub.