sgl-project/sglang · error · ValueError

cannot found moe_block_size for shape {valid_shape_m}

Error message

cannot found moe_block_size for shape {valid_shape_m}

What it means

_prepare_indexed_gemm_kwargs picks the moe_block_size by finding the first w13_tuning_config entry whose (min_shape_m, max_shape_m] range contains valid_shape_m. If valid_shape_m exceeds all configured ranges, the for/else raises because no block size applies.

Source

Thrown at python/sglang/srt/layers/moe/moe_runner/humming.py:511

            apply_routed_scaling_factor=runner_input.apply_routed_scaling_factor,
        )

        return HummingRunnerOutput(hidden_states=output)

    def _prepare_indexed_gemm_kwargs(
        self, topk_ids: torch.Tensor
    ) -> tuple[dict[str, Any], dict[str, Any]]:
        from sglang.srt.layers.moe.fused_moe_triton import moe_align_block_size

        configs = self.get_humming_gemm_configs(HummingGemmType.INDEXED)
        valid_shape_m = self.estimate_local_valid_shape_m(topk_ids)

        for min_shape_m, max_shape_m, config in configs["w13_tuning_config"]:
            if valid_shape_m > min_shape_m and valid_shape_m <= max_shape_m:
                moe_block_size = config["block_shape"][0]
                break
        else:
            raise ValueError(f"cannot found moe_block_size for shape {valid_shape_m}")

        sorted_ids, expert_ids, num_tokens_padded = moe_align_block_size(
            topk_ids=topk_ids,
            block_size=moe_block_size,
            num_experts=self.num_experts,
            ignore_invalid_expert=True,
        )

        moe_common_kwargs = {
            "sorted_ids": sorted_ids,
            "expert_ids": expert_ids,
            "num_tokens_padded": num_tokens_padded,
            "compute_config": configs["compute_config_str"],
            "valid_shape_m": valid_shape_m,
        }

        top_k = topk_ids.size(1)
        moe_kwargs1 = {

View on GitHub (pinned to 0132848349)

Solutions

1. Reduce batch size / chunked prefill size so shape_m falls inside the tuned range
2. Update/extend the w13_tuning_config table with a range covering larger M
3. Add a default block-size fallback entry with max_shape_m = inf in the tuning config

Example fix

# tuning config addition (conceptual)
# before
{"w13_tuning_config": [{"shape_range": [1, 4096], "block_shape": [128, 128]}]}
# after
{"w13_tuning_config": [{"shape_range": [1, 4096], "block_shape": [128, 128]},
                         {"shape_range": [4096, float("inf")], "block_shape": [64, 128]}]}

max_m = max(cfg[1] for cfg in runner.tuning['w13_tuning_config'])
assert estimated_shape_m <= max_m, f'shape_m {estimated_shape_m} exceeds tuned max {max_m}'

When it happens

Trigger: Running the indexed-GEMM humming path with an M (token count) larger than the largest max_shape_m in the tuned w13_tuning_config table — e.g. very large batch/prefill or many tokens per expert after alignment.

Common situations: Huge prefill batches or high concurrency exceeding the tuning table shipped with the humming backend; a stale tuning config after a batch-size increase or --max-running-requests raise.

Related errors

• Unknown gemm type: {gemm_type}
• Unsupported activation: {self.activation}
• DeepEP returned FP8 input while Humming is configured for BF
• Humming expected DeepEP FP8 hidden states and group-128 scal
• Cannot duplicate reference image of batch size {latent_condi

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/36677aad2583f008. Report an issue: GitHub.