sgl-project/sglang · error · ValueError

External ngram corpus exceeds the configured token limit ({m

Error message

External ngram corpus exceeds the configured token limit ({max_tokens}) at line {line_no} after loading {total_tokens} tokens.

What it means

The external ngram corpus loader caps total tokens at max_tokens (plus one separator token between documents). When adding the current line's tokens would exceed the cap, it raises with the line number and how many tokens were already loaded, so the caller knows the corpus does not fit the budget.

Source

Thrown at python/sglang/srt/speculative/cpp_ngram/external_corpus.py:51

            except json.JSONDecodeError as e:
                raise ValueError(
                    f"Invalid JSON in external ngram corpus at line {line_no}: {e.msg}"
                ) from e

            if not isinstance(record, str):
                raise ValueError(
                    "Invalid external ngram corpus record at line "
                    f"{line_no}: expected a JSON string."
                )

            token_ids = list(tokenizer.encode(record, add_special_tokens=False))
            if not token_ids:
                continue

            separator_cost = 1 if has_previous_doc else 0
            next_total_tokens = total_tokens + separator_cost + len(token_ids)
            if next_total_tokens > max_tokens:
                raise ValueError(
                    "External ngram corpus exceeds the configured token limit "
                    f"({max_tokens}) at line {line_no} after loading "
                    f"{total_tokens} tokens."
                )
            total_tokens = next_total_tokens

            if has_previous_doc:
                token_ids = [SEPARATOR_TOKEN] + token_ids
            for i in range(0, len(token_ids), chunk_size):
                yield token_ids[i : i + chunk_size]
            has_previous_doc = True

View on GitHub (pinned to 0132848349)

Solutions

1. Increase max_tokens (or free budget by removing another external corpus first)
2. Trim the corpus file to fewer/shorter documents
3. Estimate tokens beforehand with the same tokenizer and pre-check the sum

Example fix

# before
add_external_corpus(path, tok, max_tokens=50_000)  # corpus is ~120k tokens
# after
add_external_corpus(path, tok, max_tokens=200_000)

estimated = sum(len(tok.encode(json.loads(l))) + 1 for l in open(path) if l.strip())
assert estimated <= max_tokens, f"corpus {estimated} tokens > budget {max_tokens}"

When it happens

Trigger: Calling add_external_corpus with a max_tokens smaller than the tokenized size of the JSONL corpus; the check fires mid-stream at the first line that would overflow.

Common situations: Remaining global ngram budget is smaller than the corpus; corpus grew after a fixed limit was chosen; tokenizer produces more tokens than estimated.

Related errors

• External ngram corpus path does not exist: {path}
• A tokenizer is required to load an external ngram corpus.
• External ngram corpus max tokens must be positive.
• Invalid JSON in external ngram corpus at line {line_no}: {e.
• Invalid external ngram corpus record at line {line_no}: expe

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/1f3066665c5374e3. Report an issue: GitHub.