{"record":{"id":"80d59f60535a68fe","repo":"sgl-project/sglang","slug":"humming-fp8-dispatch-requires-sublayer-name-k-s","errorCode":null,"errorMessage":"Humming FP8 dispatch requires {sublayer_name} K={shape_k} to be divisible by 128.","messagePattern":"Humming FP8 dispatch requires (.+?) K=(.+?) to be divisible by 128\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/layers/quantization/humming_utils.py","lineNumber":77,"sourceCode":"    if output_dtype == \"auto\":\n        output_dtype = \"bf16\" if envs.SGLANG_DEEPEP_BF16_DISPATCH.get() else \"fp8\"\n    if output_dtype not in (\"bf16\", \"fp8\"):\n        raise ValueError(\n            f\"Humming does not support DeepEP {output_dtype} dispatch; \"\n            \"use --deepep-dispatcher-output-dtype=bf16 or fp8.\"\n        )\n\n    _set_humming_dispatcher_output_dtype(layer, output_dtype)\n    use_fp8 = output_dtype == \"fp8\"\n    layer._humming_uses_deepep_fp8_dispatch = use_fp8\n    return use_fp8\n\n\ndef make_humming_deepep_input_schema(\n    sublayer_name: str, shape_k: int\n) -> HummingInputSchema:\n    if shape_k % 128 != 0:\n        raise ValueError(\n            f\"Humming FP8 dispatch requires {sublayer_name} K={shape_k} \"\n            \"to be divisible by 128.\"\n        )\n    return HummingInputSchema(a_dtype=\"float8e4m3\", input_scale_group_size=128)\n\n\ndef prepare_humming_layer(layer: LinearBase, quant_config: dict):\n    weight_schema = BaseWeightSchema.from_config(quant_config)\n    input_schema = HummingInputSchema()\n\n    shape_k_stacks = [layer.input_size_per_partition]\n    shape_n_stacks = layer.output_partition_sizes\n\n    # Step 1: convert weight to humming standard format\n    weight_schema, tensors = weight_schema.convert_humming(\n        tensors=layer.named_parameters(),\n        shape_n_stacks=shape_n_stacks,\n        shape_k_stacks=shape_k_stacks,","sourceCodeStart":59,"sourceCodeEnd":95,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/layers/quantization/humming_utils.py#L59-L95","documentation":"When Humming MoE uses FP8 DeepEP dispatch, activations are quantized per 128-element group, so each sublayer's local K dimension must be a multiple of 128. make_humming_deepep_input_schema raises this during process_weights_after_loading or prepare_humming_moe_layer when shape_k % 128 != 0, which is a tensor-parallel sharding artifact (global K is almost always divisible by 128).","triggerScenarios":"Running a Humming FP8-dispatch model with a TP degree (e.g. 3, 6, 5, 7) that leaves a rank-local K not divisible by 128; expert-interleaved sharding producing odd local K sizes.","commonSituations":"Choosing non-power-of-two TP sizes to fit an odd GPU count; mixing a DeepEP-enabled config with a TP layout validated only for BF16 dispatch (which has no 128 constraint).","solutions":["Use a tensor_parallel_size that keeps local K divisible by 128 (typically 1, 2, 4, 8)","Switch dispatch to BF16: --deepep-dispatcher-output-dtype bf16 (no 128-divisibility requirement)","Verify expert sharding counts divide the expert hidden dims evenly"],"exampleFix":"# before\n--tensor-parallel-size 6 --deepep-dispatcher-output-dtype fp8\n# after\n--tensor-parallel-size 4 --deepep-dispatcher-output-dtype fp8","handlingStrategy":"validation","validationCode":"if dispatch_dtype == \"fp8\":\n    assert shape_k % 128 == 0, f\"local K {shape_k} not divisible by 128 for FP8 dispatch\"","typeGuard":null,"tryCatchPattern":null,"preventionTips":["With FP8 DeepEP dispatch keep TP power-of-two","Fall back to bf16 dispatch when K alignment can't be guaranteed"],"tags":["deepep","humming","fp8","tensor-parallel","shape-validation"],"backgroundTag":"quantization-shape-divisibility","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}