sgl-project/sglang · error · ValueError

Invalid forward mode: {forward_mode=}

Error message

Invalid forward mode: {forward_mode=}

What it means

During CUDA graph capture, HybridLinearAttnBackend._capture_metadata pre-populates static buffers (query_start_loc, state_indices) per batch size for each supported forward mode. An unrecognized forward_mode at capture time raises this ValueError, preventing a graph from being captured with metadata layouts the backend cannot reproduce at replay time.

Source

Thrown at python/sglang/srt/layers/attention/hybrid_linear_attn_backend.py:524

        spec_info: Optional[Union[EagleDraftInput, EagleVerifyInput]],
    ):
        if forward_mode.is_decode_or_idle():
            self.query_start_loc_list[bs - 1].copy_(
                self.cached_cuda_graph_decode_query_start_loc[: bs + 1]
            )
        elif forward_mode.is_target_verify():
            ragged_layout = (
                spec_info.ragged_verify_layout if spec_info is not None else None
            )
            if ragged_layout is not None:
                # Ragged capture: qsl from the runner's synthetic layout.
                self.query_start_loc_list[bs - 1].copy_(ragged_layout.qo_indptr_device)
            else:
                self.query_start_loc_list[bs - 1].copy_(
                    self.cached_cuda_graph_verify_query_start_loc[: bs + 1]
                )
        else:
            raise ValueError(f"Invalid forward mode: {forward_mode=}")
        mamba_indices = self.req_to_token_pool.get_mamba_indices(req_pool_indices)
        # Captured Mamba kernels read state_indices_list as PHYSICAL ids; translate
        # before copying (no-op for non-unified pool).
        mamba_indices = self._translate_mamba_indices(mamba_indices)
        self.state_indices_list[bs - 1][: len(mamba_indices)].copy_(mamba_indices)

        # Capture records the pointer to the static per-bs buffers; their zeros are
        # overwritten in-place by _replay_metadata before each replay. None when off.
        replayssm_write_pos = (
            self.replayssm_write_pos_list[bs - 1]
            if self.replayssm_write_pos_list is not None
            else None
        )
        replayssm_force_flush = (
            self.replayssm_force_flush_list[bs - 1]
            if self.replayssm_force_flush_list is not None
            else None
        )

View on GitHub (pinned to 0132848349)

Solutions

  1. Disable CUDA graph (--disable-cuda-graph) for the unsupported mode on hybrid models
  2. Update _capture_metadata's mode chain to handle the new forward_mode (mirror the replay-side layout)
  3. Use a sglang version where this decoding mode and the hybrid backend are compatible

Example fix

# before
python -m sglang.launch_server --model hybridenabled --speculative-algorithm NEW --cuda-graph-max-bs ...
# after
python -m sglang.launch_server --model hybridenabled --speculative-algorithm NEW --disable-cuda-graph
Defensive patterns

Strategy: validation

Validate before calling

# before launching with cuda graphs
assert backend_handles_mode(mode), f'cannot capture graph for {mode}'
# or just: server_args.disable_cuda_graph = True when using unsupported mode

Try / catch

try:
    backend.init_forward_metadata_capture_cpu_graph(...)
except ValueError as e:
    if 'Invalid forward mode' in str(e):
        disable_cuda_graphs_and_rerun()
    else:
        raise

Prevention

When it happens

Trigger: init_forward_metadata_capture_cpu_graph is invoked with a forward_mode outside the supported set (e.g. a new speculative mode enabled with --cuda-graph on a hybrid/mamba model), hitting the final else branch.

Common situations: Enabling cuda graph with a hybrid linear attention model plus a new or experimental decoding mode that the backend's capture path was not updated for; sglang version skew between core modes and the mamba backend.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/b24e5af7781b6bd2. Report an issue: GitHub.