{"record":{"id":"91c3572935a4a929","repo":"sgl-project/sglang","slug":"deepep-returned-fp8-input-while-humming-is-configu","errorCode":null,"errorMessage":"DeepEP returned FP8 input while Humming is configured for BF16 dispatch.","messagePattern":"DeepEP returned FP8 input while Humming is configured for BF16 dispatch\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/layers/moe/moe_runner/humming.py","lineNumber":815,"sourceCode":"        gemm_type=get_standard_humming_moe_gemm_type(),\n    )\n\n    runner_core = HummingRunnerCore(runner_config)\n    runner_output = runner_core.run(runner_input, quant_info, {})\n\n    return StandardCombineInput(hidden_states=runner_output.hidden_states)\n\n\ndef _validate_deepep_dispatch_input(\n    hidden_states: torch.Tensor,\n    hidden_states_scale: torch.Tensor | None,\n    layer: torch.nn.Module,\n) -> None:\n    expects_fp8 = layer._humming_uses_deepep_fp8_dispatch\n    is_fp8 = hidden_states.dtype == torch.float8_e4m3fn\n    if not expects_fp8:\n        if is_fp8 or hidden_states_scale is not None:\n            raise ValueError(\n                \"DeepEP returned FP8 input while Humming is configured for BF16 \"\n                \"dispatch.\"\n            )\n        return\n\n    if not is_fp8 or hidden_states_scale is None:\n        raise ValueError(\n            \"Humming expected DeepEP FP8 hidden states and group-128 scales.\"\n        )\n\n    expected_groups, remainder = divmod(hidden_states.size(-1), 128)\n    if (\n        remainder != 0\n        or hidden_states_scale.dtype != torch.float32\n        or hidden_states_scale.size(-1) != expected_groups\n        or hidden_states_scale.numel() != hidden_states.numel() // 128\n    ):\n        raise ValueError(","sourceCodeStart":797,"sourceCodeEnd":833,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/layers/moe/moe_runner/humming.py#L797-L833","documentation":"In DeepEP low-latency/normal pre-permute to Humming, if the layer is configured for BF16 dispatch (layer._humming_uses_deepep_fp8_dispatch is False) but DeepEP returns float8_e4m3fn hidden states or a non-None scale, the mismatched FP8 payload is rejected.","triggerScenarios":"Running Humming + DeepEP where the layer was configured for BF16 dispatch but the dispatcher was actually run in FP8 mode (config divergence between the humming runner and the deepep token dispatcher).","commonSituations":"Enabling --deepep-mode auto/low_latency with fp8 dispatch flags while the humming layer defaults disagree; version mismatch after flags were renamed; partial config where only one side got the FP8 dispatch option.","solutions":["Make dispatch dtype consistent: either enable FP8 dispatch on both sides (humming uses deepep fp8 + deepep fp8 dispatch flags) or disable FP8 dispatch entirely","Check the flags controlling deepep fp8 dispatch (e.g. --deepep-enc-fp8-format / sm-scale related options and humming fp8 config) and align them","Upgrade sglang so humming auto-detects deepep fp8 dispatch mode"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"fp8_dispatch_enabled = getattr(layer, '_humming_uses_deepep_fp8_dispatch', False)\nis_fp8 = hidden_states.dtype == torch.float8_e4m3fn\nassert fp8_dispatch_enabled == is_fp8, 'dispatch dtype disagrees with humming config'","typeGuard":"def dispatch_dtype_ok(hidden_states, scale, expects_fp8: bool) -> bool:\n    is_fp8 = hidden_states.dtype == torch.float8_e4m3fn\n    return (is_fp8 and scale is not None) if expects_fp8 else (not is_fp8 and scale is None)","tryCatchPattern":null,"preventionTips":["Set deepep fp8 dispatch flags and humming fp8 config from one shared variable","Smoke-test one forward pass at startup to catch dtype mismatches early"],"tags":["sglang","moe","humming","deepep","fp8","dtype-mismatch","distributed"],"backgroundTag":"dtype-mismatch-between-components","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}