{"record":{"id":"44690bab1b49db5b","repo":"sgl-project/sglang","slug":"humming-expected-deepep-fp8-hidden-states-and-grou","errorCode":null,"errorMessage":"Humming expected DeepEP FP8 hidden states and group-128 scales.","messagePattern":"Humming expected DeepEP FP8 hidden states and group-128 scales\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/layers/moe/moe_runner/humming.py","lineNumber":822,"sourceCode":"\n\ndef _validate_deepep_dispatch_input(\n    hidden_states: torch.Tensor,\n    hidden_states_scale: torch.Tensor | None,\n    layer: torch.nn.Module,\n) -> None:\n    expects_fp8 = layer._humming_uses_deepep_fp8_dispatch\n    is_fp8 = hidden_states.dtype == torch.float8_e4m3fn\n    if not expects_fp8:\n        if is_fp8 or hidden_states_scale is not None:\n            raise ValueError(\n                \"DeepEP returned FP8 input while Humming is configured for BF16 \"\n                \"dispatch.\"\n            )\n        return\n\n    if not is_fp8 or hidden_states_scale is None:\n        raise ValueError(\n            \"Humming expected DeepEP FP8 hidden states and group-128 scales.\"\n        )\n\n    expected_groups, remainder = divmod(hidden_states.size(-1), 128)\n    if (\n        remainder != 0\n        or hidden_states_scale.dtype != torch.float32\n        or hidden_states_scale.size(-1) != expected_groups\n        or hidden_states_scale.numel() != hidden_states.numel() // 128\n    ):\n        raise ValueError(\n            \"Humming requires row-major FP32 DeepEP scales with group size 128.\"\n        )\n    meta = layer.humming_metas[\"w13\"]\n    if meta.a_dtype != dtypes.float8e4m3 or meta.input_scale_group_size != 128:\n        raise ValueError(\"Humming w13 must use FP8 group-128 input metadata.\")\n\n","sourceCodeStart":804,"sourceCodeEnd":840,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/layers/moe/moe_runner/humming.py#L804-L840","documentation":"The mirror of error 4058: the humming layer expects DeepEP FP8 dispatch (expects_fp8 True) but hidden states are not float8_e4m3fn or hidden_states_scale is None, so the FP8 group-quantized path cannot proceed.","triggerScenarios":"Layer configured with _humming_uses_deepep_fp8_dispatch=True while the DeepEP dispatcher returned BF16 tensors (no scales), reaching _validate_deepep_dispatch_input via pre_permute_deepep_ll_to_humming or pre_permute_deepep_normal_to_humming.","commonSituations":"Turning on humming FP8 dispatch config without enabling the deepep FP8 dispatch flags (or vice versa); DP/EP setups where only some ranks enable fp8 dispatch; flag drift across sglang versions.","solutions":["Enable the DeepEP FP8 dispatch options so dispatch returns fp8 tensors plus group-128 scales","Or turn off the humming fp8-dispatch expectation so BF16 dispatch is used consistently","Verify hidden size is a multiple of 128 (required for group-128 scales)"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"assert hidden_states.dtype == torch.float8_e4m3fn and hidden_states_scale is not None, \\\n    'expected FP8 hidden states + group-128 scales from DeepEP dispatch'","typeGuard":"def is_valid_fp8_dispatch(h: torch.Tensor, s) -> bool:\n    return h.dtype == torch.float8_e4m3fn and s is not None and h.size(-1) % 128 == 0","tryCatchPattern":null,"preventionTips":["Enable DeepEP FP8 dispatch flags whenever humming fp8 dispatch is configured","Ensure hidden_size % 128 == 0 for group-128 scaling","Run a single-rank sanity forward before production traffic"],"tags":["sglang","moe","humming","deepep","fp8","dtype-mismatch","distributed"],"backgroundTag":"dtype-mismatch-between-components","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}