sgl-project/sglang · error · std::runtime_error

batchMatch received an empty token tail

Error message

batchMatch received an empty token tail

What it means

Each request in the batch must carry a non-empty token tail (the prompt/context suffix used for n-gram matching). An empty tail has no matches possible, so batchMatch rejects it rather than returning garbage.

Source

Thrown at python/sglang/kernels/jit/csrc/ngram_corpus/ngram.cpp:181

    trie_result_build_fn = &Trie::buildFrequency;
    sam_result_build_fn = &SuffixAutomaton::buildFrequency;
  } else {
    throw std::runtime_error("Unknown match_type: '" + param_.match_type + "'. Must be 'BFS' or 'PROB'.");
  }

  // All budget values are loop-invariant (mutex_ held, sams_ won't change).
  const size_t num_sams = sams_.size();
  const auto total_draft_token_num = param_.get_draft_token_num(tokens.size());
  const size_t total_sam_budget =
      num_sams > 0 ? std::min(param_.external_sam_budget, total_draft_token_num) : size_t{0};
  const size_t per_sam_budget = num_sams > 0 ? total_sam_budget / num_sams : size_t{0};
  const size_t trie_budget = total_draft_token_num - (per_sam_budget * num_sams);

  Result merged;
  for (size_t i = 0; i < state_ids.size(); ++i) {
    const auto& suffix = tokens[i];
    if (suffix.empty()) {
      throw std::runtime_error("batchMatch received an empty token tail");
    }

    auto& state = match_state_[state_ids[i]];

    if (total_sam_budget == 0 || per_sam_budget == 0) {
      auto res = (trie_.get()->*trie_result_build_fn)(
          suffix.data(), suffix.size(), suffix.back(), total_draft_token_num, param_, state, total_lens[i]);
      merged.token.insert(merged.token.end(), res.token.begin(), res.token.end());
      merged.mask.insert(merged.mask.end(), res.mask.begin(), res.mask.end());
      continue;
    }

    auto combined = (trie_.get()->*trie_result_build_fn)(
        suffix.data(), suffix.size(), suffix.back(), trie_budget, param_, state, total_lens[i]);

    for (const auto& [_, sam] : sams_) {
      auto sam_res =
          (sam.get()->*sam_result_build_fn)(suffix.data(), suffix.size(), suffix.back(), per_sam_budget, param_);

View on GitHub (pinned to 0132848349)

Solutions

1. Skip/defer requests whose token tail is empty instead of passing them to batchMatch
2. Fix the tail-construction logic (min length 1, check slice bounds)
3. Pad or wait until at least one context token exists before matching

Example fix

// before
for (auto& t : tokens) /* ... */ ngram.batchMatch(ids, tokens, lens); // some t empty
// after
std::vector<int64_t> ids2; std::vector<std::vector<int32_t>> toks2; std::vector<size_t> lens2;
for (size_t i = 0; i < ids.size(); ++i) if (!tokens[i].empty()) { ids2.push_back(ids[i]); toks2.push_back(tokens[i]); lens2.push_back(lens[i]); }
auto res = ngram.batchMatch(ids2, toks2, lens2);

for (auto& t : tokens) if (t.empty()) { /* drop or defer that request */ }

bool tails_nonempty(const std::vector<std::vector<int32_t>>& toks) { for (auto& t : toks) if (t.empty()) return false; return true; }

When it happens

Trigger: batchMatch where tokens[i] is an empty vector — e.g. a request whose context tail was truncated to zero, or a newly initialized request with no history.

Common situations: First decode step of a request with empty prompt; slicing the context with an off-by-one that yields an empty window; scheduler sending a placeholder request with no tokens yet.

Related errors

• External corpus is empty — no tokens were loaded.
• batchMatch expects state_ids, tokens, and total_lens to matc
• No frames were recorded
• This browser cannot encode H.264 MP4
• H.264 encoder did not return MP4 decoder config

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/357e475e91d6a100. Report an issue: GitHub.