sgl-project/sglang · critical · RuntimeError

draft sampler set but the draft forward has no hidden_states

Error message

draft sampler set but the draft forward has no hidden_states to capture into the graph.

What it means

During CUDA graph capture, the draft sampler capture hook requires the draft forward output to be a LogitsProcessorOutput carrying hidden_states, because the sampler feeds hidden_states and input_ids into the captured graph. If the forward returned something else or hidden_states is None (model not configured to return hidden states), the graph would silently capture a bad sampler, so the hook raises a RuntimeError.

Source

Thrown at python/sglang/srt/speculative/draft_worker_common.py:143

    *,
    draft_token_num: int,
    device: torch.device,
) -> DFlashVerifyInput:
    return DFlashVerifyInput(
        draft_token=torch.empty((0,), dtype=torch.long, device=device),
        positions=torch.empty((0,), dtype=torch.int64, device=device),
        draft_token_num=int(draft_token_num),
        custom_mask=None,
        capture_hidden_mode=CaptureHiddenMode.NULL,
    )


def make_draft_sampler_capture_hook(draft_sampler):

    def capture_hook(runner, out, forward_batch, num_tokens):
        del runner, num_tokens
        if not isinstance(out, LogitsProcessorOutput) or out.hidden_states is None:
            raise RuntimeError(
                "draft sampler set but the draft forward has no "
                "hidden_states to capture into the graph."
            )
        draft_sampler(out.hidden_states, forward_batch.input_ids)

    return capture_hook


def build_block_pos_offsets(*, length: int, device: torch.device) -> torch.Tensor:
    return torch.arange(int(length), device=device, dtype=torch.int64)

View on GitHub (pinned to 0132848349)

Solutions

1. Ensure the draft model/runner is configured to return hidden_states (enable the return-hidden-states option on the draft model config)
2. Verify the draft forward path produces LogitsProcessorOutput with hidden_states before enabling the draft sampler
3. Update the custom draft model to attach hidden_states to LogitsProcessorOutput

Example fix

# before
draft_out = runner.forward(batch)  # hidden_states=None
# after
runner.model_runner.model.config.return_hidden_states = True
draft_out = runner.forward(batch)  # hidden_states populated

out = runner.forward(small_batch)
assert isinstance(out, LogitsProcessorOutput) and out.hidden_states is not None, "draft must return hidden_states"

def has_hidden_states(out) -> bool:
    return isinstance(out, LogitsProcessorOutput) and out.hidden_states is not None

When it happens

Trigger: Enabling a draft sampler that installs make_draft_sampler_capture_hook while the draft model's forward does not populate out.hidden_states — e.g. hidden-state return disabled in the model config or a custom draft model that skips hidden_states.

Common situations: New DSpark-style draft heads that must expose hidden states for in-graph sampling; a model patch that stopped returning hidden_states; draft runner misconfiguration where return_hidden_states is off.

Related errors

• Unsupported text encoder output: expected `hidden_states`.
• Kimi-K3 DCP + DSPARK currently requires SGLANG_RAGGED_VERIFY
• --disaggregation-decode-enable-radix-cache is incompatible w
• Gemma4AssistantForCausalLM draft requires --speculative-algo
• --speculative-draft-window-size must be positive, got {}.

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/95a7f575c2cef59d. Report an issue: GitHub.