{"record":{"id":"ee97b5bd05d71a3a","repo":"sgl-project/sglang","slug":"external-ngram-corpus-path-does-not-exist-path","errorCode":null,"errorMessage":"External ngram corpus path does not exist: {path}","messagePattern":"External ngram corpus path does not exist: (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/speculative/cpp_ngram/external_corpus.py","lineNumber":18,"sourceCode":"import json\nfrom collections.abc import Iterator\nfrom pathlib import Path\n\n# Must match SuffixAutomaton::kSeparatorToken in suffix_automaton.h.\nSEPARATOR_TOKEN = -(2**31)\n\n# Default chunk size for streaming tokenized documents into the SAM.\nDEFAULT_CHUNK_SIZE = 4096\n\n\ndef iter_external_corpus_chunks(\n    path: str, tokenizer, max_tokens: int, chunk_size: int = DEFAULT_CHUNK_SIZE\n) -> Iterator[list[int]]:\n    \"\"\"Chunk documents and yield fixed-size token chunks from a JSONL corpus file.\"\"\"\n    corpus_path = Path(path)\n    if not corpus_path.is_file():\n        raise ValueError(f\"External ngram corpus path does not exist: {path}\")\n    if tokenizer is None:\n        raise ValueError(\"A tokenizer is required to load an external ngram corpus.\")\n    if max_tokens <= 0:\n        raise ValueError(\"External ngram corpus max tokens must be positive.\")\n\n    total_tokens = 0\n    has_previous_doc = False\n    with corpus_path.open(\"r\", encoding=\"utf-8\") as f:\n        for line_no, line in enumerate(f, start=1):\n            if not line.strip():\n                continue\n\n            try:\n                record = json.loads(line)\n            except json.JSONDecodeError as e:\n                raise ValueError(\n                    f\"Invalid JSON in external ngram corpus at line {line_no}: {e.msg}\"\n                ) from e","sourceCodeStart":1,"sourceCodeEnd":36,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/speculative/cpp_ngram/external_corpus.py#L1-L36","documentation":"iter_external_corpus_chunks validates that the given path is an existing regular file before streaming a JSONL corpus for the C++ ngram speculative backend. If Path(path).is_file() is False it raises this ValueError, protecting the tokenizer-loading path from missing or mis-typed corpus files.","triggerScenarios":"Calling add_external_corpus or constructing the corpus loader with a path that does not exist, points to a directory, or is misspelled; also triggered directly in tests test_external_corpus_iterator_streams_documents / test_external_corpus_iterator_rejects_oversized_corpus.","commonSituations":"Relative path resolved against a different working directory, mount missing in a container, typo in the corpus file argument, or passing a directory instead of the .jsonl file.","solutions":["Verify the path exists and is a file: os.path.isfile(path)","Use an absolute path (os.path.abspath / Path.resolve()) so it is independent of CWD","If running in Docker/k8s, confirm the corpus file is actually mounted into the container"],"exampleFix":"# before\nmanager.add_external_corpus(\"data/corpus.jsonl\", tokenizer=tok, max_tokens=100_000)\n# after\nfrom pathlib import Path\np = Path(\"data/corpus.jsonl\").resolve()\nassert p.is_file(), f\"corpus missing: {p}\"\nmanager.add_external_corpus(str(p), tokenizer=tok, max_tokens=100_000)","handlingStrategy":"validation","validationCode":"from pathlib import Path\ncorpus = Path(path).resolve()\nif not corpus.is_file():\n    raise FileNotFoundError(f\"corpus not found: {corpus}\")\nadd_external_corpus(str(corpus), tokenizer=tok, max_tokens=n)","typeGuard":null,"tryCatchPattern":"try:\n    add_external_corpus(path, tok, n)\nexcept ValueError as e:\n    if \"does not exist\" in str(e):\n        log.warning(\"corpus missing, skipping: %s\", path)\n    else:\n        raise","preventionTips":["Always resolve corpus paths to absolute paths before passing them","In containers, assert mounted files exist before server start"],"tags":["sglang","ngram","speculative-decoding","file-not-found","validation"],"backgroundTag":"file-path-not-found","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}