sgl-project/sglang · error · std::runtime_error

out_tokens buffer too small: ${out_tokens.size(0)} < ${resul

Error message

out_tokens buffer too small: ${out_tokens.size(0)} < ${result.token.size()}

What it means

write_result_ copies a match Result into caller-provided out_tokens/out_mask tensor views. If the matched draft token count exceeds out_tokens.size(0), it throws rather than overflowing the buffer. (A parallel check exists for out_mask.)

Source

Thrown at python/sglang/kernels/jit/csrc/ngram_corpus/ngram_corpus_ffi.cpp:139

    }
    return result;
  }

  void synchronize() {
    ngram_->synchronize();
  }

  void reset() {
    ngram_->reset();
  }

 private:
  void write_result_(
      const ngram::Result& result, const tvm::ffi::TensorView& out_tokens, const tvm::ffi::TensorView& out_mask) {
    auto* out_tok = static_cast<int32_t*>(out_tokens.data_ptr());
    auto* out_msk = static_cast<uint8_t*>(out_mask.data_ptr());
    if (result.token.size() > static_cast<size_t>(out_tokens.size(0))) {
      throw std::runtime_error(
          "out_tokens buffer too small: " + std::to_string(out_tokens.size(0)) + " < " +
          std::to_string(result.token.size()));
    }
    if (result.mask.size() > static_cast<size_t>(out_mask.size(0))) {
      throw std::runtime_error(
          "out_mask buffer too small: " + std::to_string(out_mask.size(0)) + " < " +
          std::to_string(result.mask.size()));
    }
    std::memcpy(out_tok, result.token.data(), result.token.size() * sizeof(int32_t));
    std::memcpy(out_msk, result.mask.data(), result.mask.size() * sizeof(uint8_t));
  }

  std::unique_ptr<ngram::Ngram> ngram_;
};

void register_ngram_corpus() {
  namespace refl = tvm::ffi::reflection;
  refl::ObjectDef<NgramCorpusObj>()

View on GitHub (pinned to 0132848349)

Solutions

1. Allocate out_tokens/out_mask with at least draft_token_num (and >= max possible result length) in dim 0 per request
2. Re-check tensor shapes passed to batch_match_stateful against current draft_token_num config
3. Cap result length in Param so results cannot exceed the buffer size

Example fix

// before
auto out_tokens = torch::empty({batch, draft_num}, opts); // row of size draft_num too small
// after
auto out_tokens = torch::empty({batch, std::max(draft_num, max_match_len)}, opts);
auto out_mask = torch::empty({batch, std::max(draft_num, max_match_len)}, opts);

size_t cap = std::max<size_t>(param.draft_token_num, max_match_len);
 TORCH_CHECK(out_tokens.size(0) >= cap && out_mask.size(0) >= cap);

bool buffers_big_enough(const torch::Tensor& t, size_t need) { return (size_t)t.size(0) >= need; }

try { batch_match_stateful(...); } catch (const std::runtime_error& e) { if (std::string(e.what()).find("buffer too small") != std::string::npos) { /* reallocate larger outputs and retry once */ } }

When it happens

Trigger: batch_match_stateful returned more draft tokens (result.token.size()) than the capacity of the preallocated out_tokens tensor row — typically when the match length exceeds the allocated draft_token_num columns.

Common situations: Allocating output buffers with the wrong leading dimension (batch vs draft length swap); shrinking draft_token_num without resizing outputs; larger-than-expected match after raising max_match_length / batch_draft_token_num.

Related errors

• No frames were recorded
• This browser cannot encode H.264 MP4
• H.264 encoder did not return MP4 decoder config
• This browser does not support gzip stream decoding
• delta payload size mismatch: expected ${expectedSize}, got $

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/f1347a64fcabbfef. Report an issue: GitHub.