{"record":{"id":"fe8e34b02c3a417f","repo":"sgl-project/sglang","slug":"external-corpus-is-empty-no-tokens-were-loaded","errorCode":null,"errorMessage":"External corpus is empty — no tokens were loaded.","messagePattern":"External corpus is empty — no tokens were loaded\\.","errorType":"exception","errorClass":"std::runtime_error","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/jit/csrc/ngram_corpus/ngram.cpp","lineNumber":93,"sourceCode":"  }\n  staging_sam_ = std::make_unique<SuffixAutomaton>();\n}\n\nvoid Ngram::appendExternalCorpusTokens(const std::vector<int32_t>& tokens) {\n  if (!staging_sam_) {\n    throw std::runtime_error(\"appendExternalCorpusTokens called without startExternalCorpusLoad\");\n  }\n  staging_sam_->appendTokens(tokens);\n}\n\nvoid Ngram::finishExternalCorpusLoad(const std::string& corpus_id) {\n  if (!staging_sam_) {\n    throw std::runtime_error(\"finishExternalCorpusLoad called without startExternalCorpusLoad\");\n  }\n  staging_sam_->finalize();\n  if (staging_sam_->empty()) {\n    staging_sam_.reset();\n    throw std::runtime_error(\"External corpus is empty — no tokens were loaded.\");\n  }\n  // Only lock briefly to install the completed SAM.\n  std::unique_lock<std::mutex> lock(mutex_);\n  if (sams_.find(corpus_id) != sams_.end()) {\n    throw std::runtime_error(\n        \"External corpus '\" + corpus_id + \"' already exists. Remove it before adding a new corpus with the same id.\");\n  }\n  sams_.emplace(corpus_id, std::move(staging_sam_));\n}\n\nvoid Ngram::removeExternalCorpus(const std::string& corpus_id) {\n  std::unique_lock<std::mutex> lock(mutex_);\n  sams_.erase(corpus_id);\n}\n\nvoid Ngram::resetStagingSam() {\n  // staging_sam_ is only accessed from the loading thread — no lock needed.\n  staging_sam_.reset();","sourceCodeStart":75,"sourceCodeEnd":111,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/jit/csrc/ngram_corpus/ngram.cpp#L75-L111","documentation":"After finalization, if the staged suffix automaton contains zero tokens the load is considered failed and staging is reset. This catches corpora that were started but never received any tokens via appendExternalCorpusTokens.","triggerScenarios":"Calling startExternalCorpusLoad() followed by finishExternalCorpusLoad() with zero appendExternalCorpusTokens calls, or appends of only empty vectors.","commonSituations":"Empty corpus file / 0-byte download; a filtering step that dropped all tokens; chunked reader that read nothing (bad path, IO error swallowed); racing append thread not yet run.","solutions":["Verify the corpus source actually contains tokens before finishing (check file size / token count)","Fix the upstream read/filter pipeline so at least one non-empty token vector is appended","Log appended token counts during load and finish only after > 0"],"exampleFix":"// before\nngram.startExternalCorpusLoad();\n// (no appends)\nngram.finishExternalCorpusLoad(\"wiki\"); // throws\n// after\nngram.startExternalCorpusLoad();\nfor (auto& chunk : read_chunks(path)) if (!chunk.empty()) ngram.appendExternalCorpusTokens(chunk);\nngram.finishExternalCorpusLoad(\"wiki\");","handlingStrategy":"validation","validationCode":"size_t appended = 0;\nfor (auto& chunk : chunks) { if (!chunk.empty()) { ngram.appendExternalCorpusTokens(chunk); appended += chunk.size(); } }\nif (appended > 0) ngram.finishExternalCorpusLoad(id); else abort_load();","typeGuard":null,"tryCatchPattern":"try { ngram.finishExternalCorpusLoad(id); } catch (const std::runtime_error& e) { /* empty corpus: check source file and retry after fixing */ }","preventionTips":["Count tokens as you append; only finish when > 0","Sanity-check corpus file size > 0 before loading"],"tags":["ngram","corpus-loading","empty-input"],"backgroundTag":"empty-input-validation","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}