{"record":{"id":"f1347a64fcabbfef","repo":"sgl-project/sglang","slug":"out-tokens-buffer-too-small-out-tokens-size-0","errorCode":null,"errorMessage":"out_tokens buffer too small: ${out_tokens.size(0)} < ${result.token.size()}","messagePattern":"out_tokens buffer too small: (.+?) < (.+?)","errorType":"exception","errorClass":"std::runtime_error","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/jit/csrc/ngram_corpus/ngram_corpus_ffi.cpp","lineNumber":139,"sourceCode":"    }\n    return result;\n  }\n\n  void synchronize() {\n    ngram_->synchronize();\n  }\n\n  void reset() {\n    ngram_->reset();\n  }\n\n private:\n  void write_result_(\n      const ngram::Result& result, const tvm::ffi::TensorView& out_tokens, const tvm::ffi::TensorView& out_mask) {\n    auto* out_tok = static_cast<int32_t*>(out_tokens.data_ptr());\n    auto* out_msk = static_cast<uint8_t*>(out_mask.data_ptr());\n    if (result.token.size() > static_cast<size_t>(out_tokens.size(0))) {\n      throw std::runtime_error(\n          \"out_tokens buffer too small: \" + std::to_string(out_tokens.size(0)) + \" < \" +\n          std::to_string(result.token.size()));\n    }\n    if (result.mask.size() > static_cast<size_t>(out_mask.size(0))) {\n      throw std::runtime_error(\n          \"out_mask buffer too small: \" + std::to_string(out_mask.size(0)) + \" < \" +\n          std::to_string(result.mask.size()));\n    }\n    std::memcpy(out_tok, result.token.data(), result.token.size() * sizeof(int32_t));\n    std::memcpy(out_msk, result.mask.data(), result.mask.size() * sizeof(uint8_t));\n  }\n\n  std::unique_ptr<ngram::Ngram> ngram_;\n};\n\nvoid register_ngram_corpus() {\n  namespace refl = tvm::ffi::reflection;\n  refl::ObjectDef<NgramCorpusObj>()","sourceCodeStart":121,"sourceCodeEnd":157,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/jit/csrc/ngram_corpus/ngram_corpus_ffi.cpp#L121-L157","documentation":"write_result_ copies a match Result into caller-provided out_tokens/out_mask tensor views. If the matched draft token count exceeds out_tokens.size(0), it throws rather than overflowing the buffer. (A parallel check exists for out_mask.)","triggerScenarios":"batch_match_stateful returned more draft tokens (result.token.size()) than the capacity of the preallocated out_tokens tensor row — typically when the match length exceeds the allocated draft_token_num columns.","commonSituations":"Allocating output buffers with the wrong leading dimension (batch vs draft length swap); shrinking draft_token_num without resizing outputs; larger-than-expected match after raising max_match_length / batch_draft_token_num.","solutions":["Allocate out_tokens/out_mask with at least draft_token_num (and >= max possible result length) in dim 0 per request","Re-check tensor shapes passed to batch_match_stateful against current draft_token_num config","Cap result length in Param so results cannot exceed the buffer size"],"exampleFix":"// before\nauto out_tokens = torch::empty({batch, draft_num}, opts); // row of size draft_num too small\n// after\nauto out_tokens = torch::empty({batch, std::max(draft_num, max_match_len)}, opts);\nauto out_mask = torch::empty({batch, std::max(draft_num, max_match_len)}, opts);","handlingStrategy":"validation","validationCode":"size_t cap = std::max<size_t>(param.draft_token_num, max_match_len);\n TORCH_CHECK(out_tokens.size(0) >= cap && out_mask.size(0) >= cap);","typeGuard":"bool buffers_big_enough(const torch::Tensor& t, size_t need) { return (size_t)t.size(0) >= need; }","tryCatchPattern":"try { batch_match_stateful(...); } catch (const std::runtime_error& e) { if (std::string(e.what()).find(\"buffer too small\") != std::string::npos) { /* reallocate larger outputs and retry once */ } }","preventionTips":["Allocate outputs with the same draft_token_num used to configure Ngram","Re-audit output shapes whenever draft_token_num or max match length changes"],"tags":["ngram","buffer-overflow","ffi","tensor-shape"],"backgroundTag":"buffer-size-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}