sgl-project/sglang · error · ValueError

Unknown MLX async mode: {launch.mode}

Error message

Unknown MLX async mode: {launch.mode}

What it means

finalize_mlx_result consumes the MlxLaunch produced earlier by the async runner; it switches on launch.mode and, for any mode it does not recognize (outside decode/extend/prefill result maps), raises this ValueError. It indicates an internal contract break between the launcher and finalizer, or a new async mode added on one side only.

Source

Thrown at python/sglang/srt/hardware_backend/mlx/tp_worker.py:658

                mixed_tokens = self._mlx_runner.decode_batch_finalize(decode)
                decode_map = {
                    rid: tok for rid, tok in zip(decode.req_ids, mixed_tokens)
                }
                self._collect_step_logprobs(
                    step_logprob_rows, decode.lazy_logprobs, decode.req_ids
                )

            next_tokens_list = []
            for req in reqs:
                if req.rid in decode_map:
                    next_tokens_list.append(decode_map[req.rid])
                elif req.rid in extend_map:
                    next_tokens_list.append(extend_map[req.rid])
                else:
                    next_tokens_list.append(prefill_map[req.rid])

        else:
            raise ValueError(f"Unknown MLX async mode: {launch.mode}")

        next_token_ids = torch.tensor(next_tokens_list, dtype=torch.long, device="cpu")
        logits_output = (
            self._assemble_logprob_output(step_logprob_rows, reqs)
            if step_logprob_rows
            else LogitsProcessorOutput(next_token_logits=None)
        )
        return GenerationBatchResult(
            logits_output=logits_output,
            next_token_ids=next_token_ids,
            can_run_cuda_graph=False,
        )

View on GitHub (pinned to 0132848349)

Solutions

1. If hit without custom code, capture launch.mode from logs and report an SGLang issue; disable the feature triggering the exotic mode
2. If developing a new mode, add a matching branch in finalize_mlx_result that fills next_tokens_list/step_logprob_rows for that mode
3. Ensure launcher and finalizer come from the same SGLang version (no partial upgrade / mixed install)
4. Retry with the sync MLX path or another backend to confirm it is mode-specific

Example fix

# before: new mode launched but not finalized
launch = MlxLaunch(mode="speculative", ...)
result = worker.finalize_mlx_result(launch)  # ValueError
# after: handle the mode
elif launch.mode == "speculative":
    next_tokens_list.append(spec_map[req.rid])

KNOWN_MODES = {"decode", "extend", "prefill"}  # match finalize_mlx_result branches
assert launch.mode in KNOWN_MODES, f"finalize cannot handle mode {launch.mode}"

def is_finalizable_mlx_launch(launch) -> bool:
    return getattr(launch, "mode", None) in {"decode", "extend", "prefill"}

try:
    out = finalize_mlx_result(launch)
except ValueError as e:
    if "Unknown MLX async mode" in str(e):
        raise RuntimeError(f"bug: launcher produced unsupported mode {launch.mode}") from e
    raise

When it happens

Trigger: An MlxLaunch reaching finalize_mlx_result with launch.mode outside the handled set — typically the same unsupported forward mode that slipped past (or was deliberately routed through) the launcher, or custom code constructing an MlxLaunch with an unregistered mode string.

Common situations: Extending the MLX worker with a new async mode but forgetting the finalize branch; race where a code change updated only the launch half; bugs from cherry-picks between versions.

Related errors

• MLX async runner does not support forward mode: {forward_mod
• SGLANG_USE_MLX requires stable Torch 2.13.x and MLX >= 0.32.
• SGLANG_USE_MLX requires stable Torch 2.13.x and MLX >= 0.32.
• SGLANG_USE_MLX requires an available MLX Metal device
• The MLX tensor bridge requires MLX >= 0.32.0

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/2f61984c034089bc. Report an issue: GitHub.