{"record":{"id":"aa4c68488ec9829e","repo":"sgl-project/sglang","slug":"a-tokenizer-is-required-to-load-an-external-ngram","errorCode":null,"errorMessage":"A tokenizer is required to load an external ngram corpus.","messagePattern":"A tokenizer is required to load an external ngram corpus\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/speculative/cpp_ngram/external_corpus.py","lineNumber":20,"sourceCode":"from collections.abc import Iterator\nfrom pathlib import Path\n\n# Must match SuffixAutomaton::kSeparatorToken in suffix_automaton.h.\nSEPARATOR_TOKEN = -(2**31)\n\n# Default chunk size for streaming tokenized documents into the SAM.\nDEFAULT_CHUNK_SIZE = 4096\n\n\ndef iter_external_corpus_chunks(\n    path: str, tokenizer, max_tokens: int, chunk_size: int = DEFAULT_CHUNK_SIZE\n) -> Iterator[list[int]]:\n    \"\"\"Chunk documents and yield fixed-size token chunks from a JSONL corpus file.\"\"\"\n    corpus_path = Path(path)\n    if not corpus_path.is_file():\n        raise ValueError(f\"External ngram corpus path does not exist: {path}\")\n    if tokenizer is None:\n        raise ValueError(\"A tokenizer is required to load an external ngram corpus.\")\n    if max_tokens <= 0:\n        raise ValueError(\"External ngram corpus max tokens must be positive.\")\n\n    total_tokens = 0\n    has_previous_doc = False\n    with corpus_path.open(\"r\", encoding=\"utf-8\") as f:\n        for line_no, line in enumerate(f, start=1):\n            if not line.strip():\n                continue\n\n            try:\n                record = json.loads(line)\n            except json.JSONDecodeError as e:\n                raise ValueError(\n                    f\"Invalid JSON in external ngram corpus at line {line_no}: {e.msg}\"\n                ) from e\n\n            if not isinstance(record, str):","sourceCodeStart":2,"sourceCodeEnd":38,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/speculative/cpp_ngram/external_corpus.py#L2-L38","documentation":"The external ngram corpus loader requires a tokenizer to encode each JSONL document into token ids before inserting chunks into the ngram matcher. Passing tokenizer=None raises this ValueError immediately after the path check.","triggerScenarios":"Calling add_external_corpus or iter_external_corpus_chunks with tokenizer=None, e.g. building the corpus before the tokenizer manager is initialized.","commonSituations":"Initializing the ngram corpus component during server startup before the tokenizer is available, or a test/script that constructs the corpus manager standalone without loading a tokenizer.","solutions":["Pass a real tokenizer (e.g. the model's AutoTokenizer or the TokenizerManager's tokenizer) to add_external_corpus","Reorder initialization so the corpus is added after tokenizer load"],"exampleFix":"# before\nadd_external_corpus(path, tokenizer=None, max_tokens=N)\n# after\nfrom transformers import AutoTokenizer\ntok = AutoTokenizer.from_pretrained(model_path)\nadd_external_corpus(path, tokenizer=tok, max_tokens=N)","handlingStrategy":"validation","validationCode":"assert tokenizer is not None, \"load tokenizer before adding external corpus\"\nadd_external_corpus(path, tokenizer=tokenizer, max_tokens=n)","typeGuard":"def has_tokenizer(t) -> bool:\n    return t is not None and callable(getattr(t, \"encode\", None))","tryCatchPattern":null,"preventionTips":["Initialize tokenizer before corpus loading in startup ordering","Fail fast on None dependencies in constructors"],"tags":["sglang","ngram","tokenizer","validation","null-argument"],"backgroundTag":"missing-required-argument","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}