sgl-project/sglang · error · ValueError

A tokenizer is required to load an external ngram corpus.

Error message

A tokenizer is required to load an external ngram corpus.

What it means

The external ngram corpus loader requires a tokenizer to encode each JSONL document into token ids before inserting chunks into the ngram matcher. Passing tokenizer=None raises this ValueError immediately after the path check.

Source

Thrown at python/sglang/srt/speculative/cpp_ngram/external_corpus.py:20

from collections.abc import Iterator
from pathlib import Path

# Must match SuffixAutomaton::kSeparatorToken in suffix_automaton.h.
SEPARATOR_TOKEN = -(2**31)

# Default chunk size for streaming tokenized documents into the SAM.
DEFAULT_CHUNK_SIZE = 4096


def iter_external_corpus_chunks(
    path: str, tokenizer, max_tokens: int, chunk_size: int = DEFAULT_CHUNK_SIZE
) -> Iterator[list[int]]:
    """Chunk documents and yield fixed-size token chunks from a JSONL corpus file."""
    corpus_path = Path(path)
    if not corpus_path.is_file():
        raise ValueError(f"External ngram corpus path does not exist: {path}")
    if tokenizer is None:
        raise ValueError("A tokenizer is required to load an external ngram corpus.")
    if max_tokens <= 0:
        raise ValueError("External ngram corpus max tokens must be positive.")

    total_tokens = 0
    has_previous_doc = False
    with corpus_path.open("r", encoding="utf-8") as f:
        for line_no, line in enumerate(f, start=1):
            if not line.strip():
                continue

            try:
                record = json.loads(line)
            except json.JSONDecodeError as e:
                raise ValueError(
                    f"Invalid JSON in external ngram corpus at line {line_no}: {e.msg}"
                ) from e

            if not isinstance(record, str):

View on GitHub (pinned to 0132848349)

Solutions

1. Pass a real tokenizer (e.g. the model's AutoTokenizer or the TokenizerManager's tokenizer) to add_external_corpus
2. Reorder initialization so the corpus is added after tokenizer load

Example fix

# before
add_external_corpus(path, tokenizer=None, max_tokens=N)
# after
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained(model_path)
add_external_corpus(path, tokenizer=tok, max_tokens=N)

assert tokenizer is not None, "load tokenizer before adding external corpus"
add_external_corpus(path, tokenizer=tokenizer, max_tokens=n)

def has_tokenizer(t) -> bool:
    return t is not None and callable(getattr(t, "encode", None))

When it happens

Trigger: Calling add_external_corpus or iter_external_corpus_chunks with tokenizer=None, e.g. building the corpus before the tokenizer manager is initialized.

Common situations: Initializing the ngram corpus component during server startup before the tokenizer is available, or a test/script that constructs the corpus manager standalone without loading a tokenizer.

Understand the failure class

Background: "missing required argument" and "the following required arguments were not provided": what required-argument errors mean and how to fix them — this error's family across 20 libraries.

Related errors

• External ngram corpus path does not exist: {path}
• External ngram corpus max tokens must be positive.
• v_cache must be provided
• q can only be None when only_qv=True
• q must be provided unless qv is provided with only_qv=True

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/aa4c68488ec9829e. Report an issue: GitHub.