sgl-project/sglang · error · ValueError

Humming FP8 dispatch requires {sublayer_name} K={shape_k} to

Error message

Humming FP8 dispatch requires {sublayer_name} K={shape_k} to be divisible by 128.

What it means

When Humming MoE uses FP8 DeepEP dispatch, activations are quantized per 128-element group, so each sublayer's local K dimension must be a multiple of 128. make_humming_deepep_input_schema raises this during process_weights_after_loading or prepare_humming_moe_layer when shape_k % 128 != 0, which is a tensor-parallel sharding artifact (global K is almost always divisible by 128).

Source

Thrown at python/sglang/srt/layers/quantization/humming_utils.py:77

    if output_dtype == "auto":
        output_dtype = "bf16" if envs.SGLANG_DEEPEP_BF16_DISPATCH.get() else "fp8"
    if output_dtype not in ("bf16", "fp8"):
        raise ValueError(
            f"Humming does not support DeepEP {output_dtype} dispatch; "
            "use --deepep-dispatcher-output-dtype=bf16 or fp8."
        )

    _set_humming_dispatcher_output_dtype(layer, output_dtype)
    use_fp8 = output_dtype == "fp8"
    layer._humming_uses_deepep_fp8_dispatch = use_fp8
    return use_fp8


def make_humming_deepep_input_schema(
    sublayer_name: str, shape_k: int
) -> HummingInputSchema:
    if shape_k % 128 != 0:
        raise ValueError(
            f"Humming FP8 dispatch requires {sublayer_name} K={shape_k} "
            "to be divisible by 128."
        )
    return HummingInputSchema(a_dtype="float8e4m3", input_scale_group_size=128)


def prepare_humming_layer(layer: LinearBase, quant_config: dict):
    weight_schema = BaseWeightSchema.from_config(quant_config)
    input_schema = HummingInputSchema()

    shape_k_stacks = [layer.input_size_per_partition]
    shape_n_stacks = layer.output_partition_sizes

    # Step 1: convert weight to humming standard format
    weight_schema, tensors = weight_schema.convert_humming(
        tensors=layer.named_parameters(),
        shape_n_stacks=shape_n_stacks,
        shape_k_stacks=shape_k_stacks,

View on GitHub (pinned to 0132848349)

Solutions

Use a tensor_parallel_size that keeps local K divisible by 128 (typically 1, 2, 4, 8)
Switch dispatch to BF16: --deepep-dispatcher-output-dtype bf16 (no 128-divisibility requirement)
Verify expert sharding counts divide the expert hidden dims evenly

Example fix

# before
--tensor-parallel-size 6 --deepep-dispatcher-output-dtype fp8
# after
--tensor-parallel-size 4 --deepep-dispatcher-output-dtype fp8

Defensive patterns

Strategy: validation

Validate before calling

if dispatch_dtype == "fp8":
    assert shape_k % 128 == 0, f"local K {shape_k} not divisible by 128 for FP8 dispatch"

Prevention

With FP8 DeepEP dispatch keep TP power-of-two
Fall back to bf16 dispatch when K alignment can't be guaranteed

When it happens

Trigger: Running a Humming FP8-dispatch model with a TP degree (e.g. 3, 6, 5, 7) that leaves a rank-local K not divisible by 128; expert-interleaved sharding producing odd local K sizes.

Common situations: Choosing non-power-of-two TP sizes to fit an odd GPU count; mixing a DeepEP-enabled config with a TP layout validated only for BF16 dispatch (which has no 128 constraint).

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/80d59f60535a68fe. Report an issue: GitHub.