{"record":{"id":"95a7f575c2cef59d","repo":"sgl-project/sglang","slug":"draft-sampler-set-but-the-draft-forward-has-no-hid","errorCode":null,"errorMessage":"draft sampler set but the draft forward has no hidden_states to capture into the graph.","messagePattern":"draft sampler set but the draft forward has no hidden_states to capture into the graph\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"python/sglang/srt/speculative/draft_worker_common.py","lineNumber":143,"sourceCode":"    *,\n    draft_token_num: int,\n    device: torch.device,\n) -> DFlashVerifyInput:\n    return DFlashVerifyInput(\n        draft_token=torch.empty((0,), dtype=torch.long, device=device),\n        positions=torch.empty((0,), dtype=torch.int64, device=device),\n        draft_token_num=int(draft_token_num),\n        custom_mask=None,\n        capture_hidden_mode=CaptureHiddenMode.NULL,\n    )\n\n\ndef make_draft_sampler_capture_hook(draft_sampler):\n\n    def capture_hook(runner, out, forward_batch, num_tokens):\n        del runner, num_tokens\n        if not isinstance(out, LogitsProcessorOutput) or out.hidden_states is None:\n            raise RuntimeError(\n                \"draft sampler set but the draft forward has no \"\n                \"hidden_states to capture into the graph.\"\n            )\n        draft_sampler(out.hidden_states, forward_batch.input_ids)\n\n    return capture_hook\n\n\ndef build_block_pos_offsets(*, length: int, device: torch.device) -> torch.Tensor:\n    return torch.arange(int(length), device=device, dtype=torch.int64)\n","sourceCodeStart":125,"sourceCodeEnd":154,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/speculative/draft_worker_common.py#L125-L154","documentation":"During CUDA graph capture, the draft sampler capture hook requires the draft forward output to be a LogitsProcessorOutput carrying hidden_states, because the sampler feeds hidden_states and input_ids into the captured graph. If the forward returned something else or hidden_states is None (model not configured to return hidden states), the graph would silently capture a bad sampler, so the hook raises a RuntimeError.","triggerScenarios":"Enabling a draft sampler that installs make_draft_sampler_capture_hook while the draft model's forward does not populate out.hidden_states — e.g. hidden-state return disabled in the model config or a custom draft model that skips hidden_states.","commonSituations":"New DSpark-style draft heads that must expose hidden states for in-graph sampling; a model patch that stopped returning hidden_states; draft runner misconfiguration where return_hidden_states is off.","solutions":["Ensure the draft model/runner is configured to return hidden_states (enable the return-hidden-states option on the draft model config)","Verify the draft forward path produces LogitsProcessorOutput with hidden_states before enabling the draft sampler","Update the custom draft model to attach hidden_states to LogitsProcessorOutput"],"exampleFix":"# before\ndraft_out = runner.forward(batch)  # hidden_states=None\n# after\nrunner.model_runner.model.config.return_hidden_states = True\ndraft_out = runner.forward(batch)  # hidden_states populated","handlingStrategy":"validation","validationCode":"out = runner.forward(small_batch)\nassert isinstance(out, LogitsProcessorOutput) and out.hidden_states is not None, \"draft must return hidden_states\"","typeGuard":"def has_hidden_states(out) -> bool:\n    return isinstance(out, LogitsProcessorOutput) and out.hidden_states is not None","tryCatchPattern":null,"preventionTips":["Enable return_hidden_states on draft models used with draft samplers","Smoke-test one forward before CUDA graph capture","Keep draft model patches in sync with sampler requirements"],"tags":["sglang","speculative-decoding","cuda-graph","draft-sampler","hidden-states","runtime"],"backgroundTag":"cuda-graph-capture-precondition-failed","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}