{"record":{"id":"1f3066665c5374e3","repo":"sgl-project/sglang","slug":"external-ngram-corpus-exceeds-the-configured-token","errorCode":null,"errorMessage":"External ngram corpus exceeds the configured token limit ({max_tokens}) at line {line_no} after loading {total_tokens} tokens.","messagePattern":"External ngram corpus exceeds the configured token limit \\((.+?)\\) at line (.+?) after loading (.+?) tokens\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/speculative/cpp_ngram/external_corpus.py","lineNumber":51,"sourceCode":"            except json.JSONDecodeError as e:\n                raise ValueError(\n                    f\"Invalid JSON in external ngram corpus at line {line_no}: {e.msg}\"\n                ) from e\n\n            if not isinstance(record, str):\n                raise ValueError(\n                    \"Invalid external ngram corpus record at line \"\n                    f\"{line_no}: expected a JSON string.\"\n                )\n\n            token_ids = list(tokenizer.encode(record, add_special_tokens=False))\n            if not token_ids:\n                continue\n\n            separator_cost = 1 if has_previous_doc else 0\n            next_total_tokens = total_tokens + separator_cost + len(token_ids)\n            if next_total_tokens > max_tokens:\n                raise ValueError(\n                    \"External ngram corpus exceeds the configured token limit \"\n                    f\"({max_tokens}) at line {line_no} after loading \"\n                    f\"{total_tokens} tokens.\"\n                )\n            total_tokens = next_total_tokens\n\n            if has_previous_doc:\n                token_ids = [SEPARATOR_TOKEN] + token_ids\n            for i in range(0, len(token_ids), chunk_size):\n                yield token_ids[i : i + chunk_size]\n            has_previous_doc = True\n","sourceCodeStart":33,"sourceCodeEnd":63,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/speculative/cpp_ngram/external_corpus.py#L33-L63","documentation":"The external ngram corpus loader caps total tokens at max_tokens (plus one separator token between documents). When adding the current line's tokens would exceed the cap, it raises with the line number and how many tokens were already loaded, so the caller knows the corpus does not fit the budget.","triggerScenarios":"Calling add_external_corpus with a max_tokens smaller than the tokenized size of the JSONL corpus; the check fires mid-stream at the first line that would overflow.","commonSituations":"Remaining global ngram budget is smaller than the corpus; corpus grew after a fixed limit was chosen; tokenizer produces more tokens than estimated.","solutions":["Increase max_tokens (or free budget by removing another external corpus first)","Trim the corpus file to fewer/shorter documents","Estimate tokens beforehand with the same tokenizer and pre-check the sum"],"exampleFix":"# before\nadd_external_corpus(path, tok, max_tokens=50_000)  # corpus is ~120k tokens\n# after\nadd_external_corpus(path, tok, max_tokens=200_000)","handlingStrategy":"validation","validationCode":"estimated = sum(len(tok.encode(json.loads(l))) + 1 for l in open(path) if l.strip())\nassert estimated <= max_tokens, f\"corpus {estimated} tokens > budget {max_tokens}\"","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Pre-count tokens with the same tokenizer before loading","Free budget by removing stale corpora before adding new ones"],"tags":["sglang","ngram","token-limit","budget-exceeded"],"backgroundTag":"resource-limit-exceeded","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}