sgl-project/sglang · critical · ValueError

DFLASH mask_token_id is outside the target vocab size. mask_

Error message

DFLASH mask_token_id is outside the target vocab size. mask_token_id={resolved_id}, vocab_size={vocab_size}. This likely means mask_token={mask_token!r} requires vocab expansion beyond the model's embedding size. SGLang does not support resizing target embeddings for DFLASH yet.

What it means

Raised during DFLASH worker init when the resolved mask token id is >= the target model's vocab size. DFLASH drafts like Nemotron 3.5 add a mask token whose id may lie beyond the original vocab; sglang cannot resize the target model's embedding matrix to accommodate it, so it refuses to start. The message explicitly calls out that embedding expansion for DFLASH is unsupported.

Source

Thrown at python/sglang/srt/speculative/dflash_worker_v2.py:884

                draft_prefix_lens,
                block_end,
                verify_out_cache_loc_2d.reshape(-1),
                bs,
            )

    def _resolve_mask_token_id(
        self, *, mask_token: str, mask_token_id: Optional[int] = None
    ) -> int:
        if not isinstance(mask_token, str) or not mask_token:
            raise ValueError(
                f"DFLASH mask_token must be a non-empty string, got {mask_token!r}."
            )

        vocab_size = int(self.target_worker.model_runner.model_config.vocab_size)
        if mask_token_id is not None:
            resolved_id = int(mask_token_id)
            if resolved_id >= vocab_size:
                raise ValueError(
                    "DFLASH mask_token_id is outside the target vocab size. "
                    f"mask_token_id={resolved_id}, vocab_size={vocab_size}. "
                    f"This likely means mask_token={mask_token!r} requires vocab expansion beyond the model's embedding size. "
                    "SGLang does not support resizing target embeddings for DFLASH yet."
                )

            tokenizer = getattr(self.target_worker, "tokenizer", None)
            if tokenizer is not None:
                token_id_from_vocab = tokenizer.get_vocab().get(mask_token, None)
                if (
                    token_id_from_vocab is not None
                    and int(token_id_from_vocab) != resolved_id
                ):
                    raise ValueError(
                        "DFLASH config mismatch: dflash_config.mask_token_id conflicts with tokenizer vocab id "
                        f"for dflash_config.mask_token. mask_token={mask_token!r}, "
                        f"mask_token_id={resolved_id}, tokenizer_vocab_id={int(token_id_from_vocab)}."
                    )

View on GitHub (pinned to 0132848349)

Solutions

Use matching target + draft + tokenizer revisions released together (e.g. the official Nemotron 3.5 bundle) so the mask token is within the target vocab
Check tokenizer.convert_tokens_to_ids(mask_token) vs model_config.vocab_size; if the id is out of range, your tokenizer/target mismatch is the root cause
If you control the checkpoints, remap or retrain so the mask token id sits inside the target vocab — sglang cannot resize target embeddings for DFLASH today
Do not hand-override mask_token_id to an out-of-vocab value; remove the override and let resolution use the tokenizer

Defensive patterns

Strategy: validation

Validate before calling

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained(draft_path)
mask_id = tok.convert_tokens_to_ids(mask_token)
vocab = target_model_config.vocab_size
assert mask_id is not None and mask_id < vocab, (
    f"mask token id {mask_id} outside target vocab {vocab}; "
    "use matching target/draft/tokenizer revisions"
)

Prevention

Always load target model, draft model, and tokenizer from the same official release bundle
Check mask token id against vocab_size before launching a DFLASH server
Never hand-set mask_token_id beyond the target embedding size

When it happens

Trigger: Loading a DFLASH draft config whose mask_token_id (or tokenizer-resolved mask token) falls outside target vocab_size — typically a draft checkpoint with an extended vocab (extra special tokens) paired with a target model whose embeddings were never expanded.

Common situations: Using a tokenizer/draft checkpoint revision that added the mask token past the vocab boundary while the target model weights keep the original vocab; mixing model components from incompatible revisions; explicitly overriding mask_token_id with a value beyond vocab_size.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/028f0ed0c513ef6e. Report an issue: GitHub.