sgl-project/sglang · error · RuntimeError

spec_info is unset in TARGET_VERIFY mode; the extend_* metad

Error message

spec_info is unset in TARGET_VERIFY mode; the extend_* metadata can only be derived from spec_info for speculative verify batches.

What it means

The Intel AMX attention backend derives TARGET_VERIFY (speculative decoding verify) extend metadata — extend_seq_lens, tree/chain masks — from forward_batch.spec_info (e.g. EagleVerifyInput). If spec_info is None in TARGET_VERIFY mode, the metadata cannot be built and this RuntimeError is raised from _build_extend_metadata during init_forward_metadata.

Source

Thrown at python/sglang/srt/layers/attention/intel_amx_backend.py:83

        self.num_draft_tokens = get_spec().speculative_num_draft_tokens

    def _build_extend_metadata(self, forward_batch: ForwardBatch):
        """Resolve (seq_lens, extend_seq_lens, extend_start_loc, tree_mask) for
        forward_extend, once per forward pass.

        In TARGET_VERIFY mode the batch carries no extend_* fields, so they are
        derived from spec_info (mirrors the CUDA unified path in
        triton_backend.py); each request extends by exactly num_draft_tokens
        tokens. Outside spec decoding the fields are passed through.
        """
        bs = forward_batch.batch_size
        seq_lens = forward_batch.seq_lens
        tree_mask = None

        if forward_batch.forward_mode.is_target_verify():
            spec_info = forward_batch.spec_info
            if spec_info is None:
                raise RuntimeError(
                    "spec_info is unset in TARGET_VERIFY mode; the extend_* "
                    "metadata can only be derived from spec_info for "
                    "speculative verify batches."
                )
            num_draft_tokens = spec_info.draft_token_num
            extend_seq_lens = torch.full(
                (bs,), num_draft_tokens, dtype=torch.int32, device=self.device
            )
            # Uniform extend lengths: start locations form a plain range.
            extend_start_loc = torch.arange(
                0,
                bs * num_draft_tokens,
                num_draft_tokens,
                dtype=torch.int32,
                device=self.device,
            )
            seq_lens = forward_batch.seq_lens + num_draft_tokens
            # Speculative verify with a token tree: each draft token may only

View on GitHub (pinned to 0132848349)

Solutions

1. Ensure the code building TARGET_VERIFY batches always sets forward_batch.spec_info (EagleVerifyInput etc.) before attention metadata init
2. Debug upstream: log where the verify ForwardBatch is created and why spec_info is missing (often a partial refactoring or new spec algorithm)
3. If using a custom speculative path, port the spec_info attachment logic from the eagle scheduler path

Example fix

# before
forward_batch = ForwardBatch(..., forward_mode=ForwardMode.TARGET_VERIFY)  # spec_info None
# after
forward_batch = ForwardBatch(..., forward_mode=ForwardMode.TARGET_VERIFY,
                             spec_info=eagle_verify_input)

if forward_batch.forward_mode.is_target_verify():
    assert forward_batch.spec_info is not None, (
        'TARGET_VERIFY batch missing spec_info (EagleVerifyInput/TLVerifyInput)')
backend.init_forward_metadata(forward_batch)

def verify_batch_is_valid(forward_batch) -> bool:
    if forward_batch.forward_mode.is_target_verify():
        return forward_batch.spec_info is not None
    return True

try:
    backend.init_forward_metadata(forward_batch)
except RuntimeError as e:
    if 'spec_info is unset' in str(e):
        log.error('verify batch built without spec_info at %s', batch_origin)
        raise
    raise

When it happens

Trigger: A ForwardBatch arrives with forward_mode=TARGET_VERIFY but forward_batch.spec_info unset (None), while running the intel_amx backend's init_forward_metadata.

Common situations: Scheduler/speculative-decoding wiring bugs where the verify batch is constructed without attaching EagleVerifyInput/TLVerifyInput; a new speculative algorithm that forgets to populate spec_info; or mixed-version code paths constructing verify batches manually.

Related errors

• Kimi-K3 DCP + DSPARK currently requires SGLANG_RAGGED_VERIFY
• --disaggregation-decode-enable-radix-cache is incompatible w
• Gemma4AssistantForCausalLM draft requires --speculative-algo
• --speculative-draft-window-size must be positive, got {}.
• DFLASH speculative decoding only supports CUDA and NPU devic

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/018e342afa37fa9e. Report an issue: GitHub.