{"record":{"id":"d7d1ecc4b8e48f46","repo":"sgl-project/sglang","slug":"invalid-json-in-external-ngram-corpus-at-line-lin","errorCode":null,"errorMessage":"Invalid JSON in external ngram corpus at line {line_no}: {e.msg}","messagePattern":"Invalid JSON in external ngram corpus at line (.+?): (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/speculative/cpp_ngram/external_corpus.py","lineNumber":34,"sourceCode":"    corpus_path = Path(path)\n    if not corpus_path.is_file():\n        raise ValueError(f\"External ngram corpus path does not exist: {path}\")\n    if tokenizer is None:\n        raise ValueError(\"A tokenizer is required to load an external ngram corpus.\")\n    if max_tokens <= 0:\n        raise ValueError(\"External ngram corpus max tokens must be positive.\")\n\n    total_tokens = 0\n    has_previous_doc = False\n    with corpus_path.open(\"r\", encoding=\"utf-8\") as f:\n        for line_no, line in enumerate(f, start=1):\n            if not line.strip():\n                continue\n\n            try:\n                record = json.loads(line)\n            except json.JSONDecodeError as e:\n                raise ValueError(\n                    f\"Invalid JSON in external ngram corpus at line {line_no}: {e.msg}\"\n                ) from e\n\n            if not isinstance(record, str):\n                raise ValueError(\n                    \"Invalid external ngram corpus record at line \"\n                    f\"{line_no}: expected a JSON string.\"\n                )\n\n            token_ids = list(tokenizer.encode(record, add_special_tokens=False))\n            if not token_ids:\n                continue\n\n            separator_cost = 1 if has_previous_doc else 0\n            next_total_tokens = total_tokens + separator_cost + len(token_ids)\n            if next_total_tokens > max_tokens:\n                raise ValueError(\n                    \"External ngram corpus exceeds the configured token limit \"","sourceCodeStart":16,"sourceCodeEnd":52,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/speculative/cpp_ngram/external_corpus.py#L16-L52","documentation":"While streaming the JSONL external corpus, a non-blank line failed json.loads. The original JSONDecodeError is chained (raise ... from e) and re-raised as a ValueError that includes the 1-based line number and the parser's message so the offending line can be located.","triggerScenarios":"A corpus .jsonl file containing malformed JSON on any non-empty line (trailing comma, unquoted string, truncated line from an interrupted write).","commonSituations":"Corpus generated by a script without proper json.dumps, file truncated mid-download, or mixed formats (plain text lines instead of JSON strings).","solutions":["Inspect the reported line number and fix or remove that line","Regenerate the corpus with json.dumps for every record, one per line","Validate the whole file up front: [json.loads(l) for l in open(p) if l.strip()]"],"exampleFix":"# before (bad corpus line)\nHello \"world\" \n# after\n\"Hello \\\"world\\\"\"","handlingStrategy":"validation","validationCode":"import json\nwith open(path) as f:\n    for i, line in enumerate(f, 1):\n        if line.strip():\n            json.loads(line)  # raises here with line number instead of mid-load","typeGuard":null,"tryCatchPattern":"try:\n    add_external_corpus(path, tok, n)\nexcept ValueError as e:\n    if \"Invalid JSON\" in str(e):\n        # e carries the offending line number; quarantine and continue\n        log.error(\"bad corpus: %s\", e)\n    else:\n        raise","preventionTips":["Generate corpora only via json.dumps per line","Validate JSONL files in a pre-deploy lint step"],"tags":["sglang","ngram","json","jsonl","corrupt-data"],"backgroundTag":"json-parse-error","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}