sgl-project/sglang · error · std::runtime_error

batchMatch expects state_ids, tokens, and total_lens to matc

Error message

batchMatch expects state_ids, tokens, and total_lens to match in size

What it means

batchMatch takes parallel arrays: one state id, one token-tail vector, and one total length per request. It throws at entry if the three containers differ in length, since indexing them together would be out-of-bounds.

Source

Thrown at python/sglang/kernels/jit/csrc/ngram_corpus/ngram.cpp:149

  for (;;) {
    std::vector<int32_t> data;
    if (!insert_queue_.dequeue(data)) {
      break;
    }
    std::unique_lock<std::mutex> lock(mutex_);
    trie_->insert(data.data(), data.size());
    --pending_count_;
    lock.unlock();
    sync_cv_.notify_all();
  }
}

Result Ngram::batchMatch(
    const std::vector<int64_t>& state_ids,
    const std::vector<std::vector<int32_t>>& tokens,
    const std::vector<size_t>& total_lens) {
  if (state_ids.size() != tokens.size() || state_ids.size() != total_lens.size()) {
    throw std::runtime_error("batchMatch expects state_ids, tokens, and total_lens to match in size");
  }

  std::unique_lock<std::mutex> lock(mutex_);

  using TrieResultBuildFn =
      Result (Trie::*)(const int32_t*, size_t, int32_t, size_t, const Param&, MatchState&, size_t) const;
  using SamResultBuildFn = Result (SuffixAutomaton::*)(const int32_t*, size_t, int32_t, size_t, const Param&) const;
  TrieResultBuildFn trie_result_build_fn;
  SamResultBuildFn sam_result_build_fn;
  if (param_.match_type == "BFS") {
    trie_result_build_fn = &Trie::buildRecency;
    sam_result_build_fn = &SuffixAutomaton::buildRecency;
  } else if (param_.match_type == "PROB") {
    trie_result_build_fn = &Trie::buildFrequency;
    sam_result_build_fn = &SuffixAutomaton::buildFrequency;
  } else {
    throw std::runtime_error("Unknown match_type: '" + param_.match_type + "'. Must be 'BFS' or 'PROB'.");
  }

View on GitHub (pinned to 0132848349)

Solutions

1. Build all three arrays from a single request list in one loop so they cannot diverge
2. Add an assert/equality check on the Python side before calling batch_match
3. If filtering requests, filter the struct/record, then unzip to the three arrays

Example fix

// before
auto res = ngram.batchMatch(ids, tails, lens); // lens stale
// after
assert(ids.size() == tails.size() && ids.size() == lens.size());
auto res = ngram.batchMatch(ids, tails, lens);

assert(state_ids.size() == tokens.size() && state_ids.size() == total_lens.size());

bool batch_valid(const std::vector<int64_t>& ids, const std::vector<std::vector<int32_t>>& toks, const std::vector<size_t>& lens) { return ids.size() == toks.size() && ids.size() == lens.size(); }

When it happens

Trigger: Calling Ngram::batchMatch(state_ids, tokens, total_lens) where state_ids.size() != tokens.size() or != total_lens.size() — e.g. batch got truncated on the Python/FFI side before reaching C++.

Common situations: Mismatched batching when building request lists across components; a filter applied to tokens but not total_lens; empty-vs-partial batch assembly in the FFI wrapper.

Related errors

• batchMatch received an empty token tail
• No frames were recorded
• This browser cannot encode H.264 MP4
• H.264 encoder did not return MP4 decoder config
• This browser does not support gzip stream decoding

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/427e2164b708e0a6. Report an issue: GitHub.