sgl-project/sglang · error · ValueError

MLX async runner does not support forward mode: {forward_mod

Error message

MLX async runner does not support forward mode: {forward_mode}

What it means

The MLX async forward dispatcher in tp_worker handles only a fixed set of forward modes (decode/extend paths that end in _async_extend_batch or flush+extend). Any other ForwardMode reaching async_forward_batch_generation_mlx raises this ValueError, guarding against unimplemented async paths.

Source

Thrown at python/sglang/srt/hardware_backend/mlx/tp_worker.py:452

                pending_decode.lazy_tokens,
                *lazy_logprob_arrays(pending_decode.lazy_logprobs),
            )
            return MlxLaunch(
                lazy_tokens=pending_decode.lazy_tokens,
                prefills=[],
                extends=[],
                decode=pending_decode,
                mode="decode",
            )

        if forward_mode.is_extend():
            # TODO (changminbark): Implement per-batch flushing using prefix_slot_ids
            # Ensure the pool is up-to-date before pool-backed attention
            # reads it for prefix-cached prefills. Mirror the sync path.
            self._mlx_runner.flush_all_decode_kv()
            return self._async_extend_batch(batch)

        raise ValueError(
            f"MLX async runner does not support forward mode: {forward_mode}"
        )

    def _async_extend_batch(self, batch: ScheduleBatch) -> MlxLaunch:
        """Launch each request in an EXTEND batch lazily and kick GPU work."""
        reqs = batch.reqs
        input_ids_cpu = batch.input_ids.cpu().tolist()
        out_cache_loc_cpu = batch.out_cache_loc.cpu().tolist()
        extend_seq_lens = batch.extend_lens
        edit_rows = self._build_logit_edit_rows(batch)
        logprob_rows = self._logprob_rows(batch)

        offset = 0
        slot_offset = 0
        pending_prefills: list[MlxPendingPrefill] = []
        pending_extends: list[MlxPendingExtend] = []
        mixed_decode_rids: list[str] = []
        # Genuine decode steps mixed into this extend batch; see

View on GitHub (pinned to 0132848349)

Solutions

  1. Disable features that produce unsupported forward modes (e.g. --disable-mixed-chunk, turn off speculative decoding / idle mode) when using the MLX backend
  2. Check server args for async/overlap scheduler interactions and fall back to the synchronous path
  3. Report/file an issue with the ForwardMode name printed in the message so the branch can be added
  4. As a stopgap, unset SGLANG_USE_MLX and run a supported backend

Example fix

# before
python -m sglang.launch_server ... --enable-mixed-chunk  # mixed mode hits MLX async runner
# after
python -m sglang.launch_server ...  # mixed-chunk disabled for MLX backend
Defensive patterns

Strategy: fallback

Validate before calling

SUPPORTED_MLX_MODES = {ForwardMode.DECODE, ForwardMode.EXTEND}
assert batch.forward_mode.is_decode or batch.forward_mode.is_extend, (
    f"MLX async runner cannot handle {batch.forward_mode}")

Try / catch

try:
    launch = self.async_forward_batch_generation_mlx(batch)
except ValueError as e:
    if "forward mode" in str(e):
        launch = self._forward_batch_generation_mlx(batch)  # sync fallback
    else:
        raise

Prevention

When it happens

Trigger: Calling the MLX async runner (via _launch_fresh/_forward_batch_generation_mlx) with a ForwardMode other than the supported decode/extend ones — e.g. idle mode, mixed chunk batches routed to the async path, or a new ForwardMode enum value added without an MLX branch.

Common situations: Enabling speculative decoding, mixed-chunk, or other scheduler features that emit exotic forward modes while SGLANG_USE_MLX is on; upgrading SGLang where a new ForwardMode was introduced but MLX support lags.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/760f4194daa96b62. Report an issue: GitHub.