sgl-project/sglang · error · ValueError

DeepEP returned FP8 input while Humming is configured for BF

Error message

DeepEP returned FP8 input while Humming is configured for BF16 dispatch.

What it means

In DeepEP low-latency/normal pre-permute to Humming, if the layer is configured for BF16 dispatch (layer._humming_uses_deepep_fp8_dispatch is False) but DeepEP returns float8_e4m3fn hidden states or a non-None scale, the mismatched FP8 payload is rejected.

Source

Thrown at python/sglang/srt/layers/moe/moe_runner/humming.py:815

        gemm_type=get_standard_humming_moe_gemm_type(),
    )

    runner_core = HummingRunnerCore(runner_config)
    runner_output = runner_core.run(runner_input, quant_info, {})

    return StandardCombineInput(hidden_states=runner_output.hidden_states)


def _validate_deepep_dispatch_input(
    hidden_states: torch.Tensor,
    hidden_states_scale: torch.Tensor | None,
    layer: torch.nn.Module,
) -> None:
    expects_fp8 = layer._humming_uses_deepep_fp8_dispatch
    is_fp8 = hidden_states.dtype == torch.float8_e4m3fn
    if not expects_fp8:
        if is_fp8 or hidden_states_scale is not None:
            raise ValueError(
                "DeepEP returned FP8 input while Humming is configured for BF16 "
                "dispatch."
            )
        return

    if not is_fp8 or hidden_states_scale is None:
        raise ValueError(
            "Humming expected DeepEP FP8 hidden states and group-128 scales."
        )

    expected_groups, remainder = divmod(hidden_states.size(-1), 128)
    if (
        remainder != 0
        or hidden_states_scale.dtype != torch.float32
        or hidden_states_scale.size(-1) != expected_groups
        or hidden_states_scale.numel() != hidden_states.numel() // 128
    ):
        raise ValueError(

View on GitHub (pinned to 0132848349)

Solutions

1. Make dispatch dtype consistent: either enable FP8 dispatch on both sides (humming uses deepep fp8 + deepep fp8 dispatch flags) or disable FP8 dispatch entirely
2. Check the flags controlling deepep fp8 dispatch (e.g. --deepep-enc-fp8-format / sm-scale related options and humming fp8 config) and align them
3. Upgrade sglang so humming auto-detects deepep fp8 dispatch mode

fp8_dispatch_enabled = getattr(layer, '_humming_uses_deepep_fp8_dispatch', False)
is_fp8 = hidden_states.dtype == torch.float8_e4m3fn
assert fp8_dispatch_enabled == is_fp8, 'dispatch dtype disagrees with humming config'

def dispatch_dtype_ok(hidden_states, scale, expects_fp8: bool) -> bool:
    is_fp8 = hidden_states.dtype == torch.float8_e4m3fn
    return (is_fp8 and scale is not None) if expects_fp8 else (not is_fp8 and scale is None)

When it happens

Trigger: Running Humming + DeepEP where the layer was configured for BF16 dispatch but the dispatcher was actually run in FP8 mode (config divergence between the humming runner and the deepep token dispatcher).

Common situations: Enabling --deepep-mode auto/low_latency with fp8 dispatch flags while the humming layer defaults disagree; version mismatch after flags were renamed; partial config where only one side got the FP8 dispatch option.

Related errors

• Humming expected DeepEP FP8 hidden states and group-128 scal
• The hpc_ops MoE runner backend only supports FP8-quantized M
• Unknown gemm type: {gemm_type}
• Unsupported activation: {self.activation}
• cannot found moe_block_size for shape {valid_shape_m}

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/91c3572935a4a929. Report an issue: GitHub.