sgl-project/sglang · error · std::runtime_error

batch_draft_token_num config value ${config} must be less th

Error message

batch_draft_token_num config value ${config} must be less than or equal to draft_token_num: ${param_.draft_token_num}

What it means

The Ngram constructor validates that every per-request value in batch_draft_token_num does not exceed the global draft_token_num. Each batch config controls how many draft tokens a single request may use, which can never exceed the engine-wide draft_token_num budget.

Source

Thrown at python/sglang/kernels/jit/csrc/ngram_corpus/ngram.cpp:34

  }
  if (!(param_.min_bfs_breadth > 0)) {
    throw std::runtime_error(
        "min_bfs_breadth must be greater than 0, current value: " + std::to_string(param_.min_bfs_breadth));
  }
  if (!(param_.min_bfs_breadth <= param_.max_bfs_breadth)) {
    throw std::runtime_error(
        "min_bfs_breadth must be less than or equal to max_bfs_breadth, "
        "current min_bfs_breadth: " +
        std::to_string(param_.min_bfs_breadth) + ", max_bfs_breadth: " + std::to_string(param_.max_bfs_breadth));
  }
  if (!(param_.draft_token_num > 0)) {
    throw std::runtime_error(
        "draft_token_num must be greater than 0, current value: " + std::to_string(param_.draft_token_num));
  }
  for (auto config : param_.batch_draft_token_num) {
    if (config != std::numeric_limits<decltype(config)>::max()) {
      if (!(config <= param_.draft_token_num)) {
        throw std::runtime_error(
            "batch_draft_token_num config value " + std::to_string(config) +
            " must be less than or equal to draft_token_num: " + std::to_string(param_.draft_token_num));
      }
    }
  }

  trie_ = std::make_unique<Trie>(capacity, param_);

  insert_worker_ = std::thread(&Ngram::insertWorker, this);
}

Ngram::~Ngram() {
  insert_queue_.close();
  if (insert_worker_.joinable()) {
    insert_worker_.join();
  }
}

View on GitHub (pinned to 0132848349)

Solutions

1. Raise draft_token_num to at least max(batch_draft_token_num) (excluding sentinel max())
2. Lower the offending batch_draft_token_num entries to <= draft_token_num
3. Use the sentinel value (numeric_limits<int>::max(), i.e. -1/INT_MAX as configured) for entries that should default to the global limit

Example fix

// before
param.draft_token_num = 4;
param.batch_draft_token_num = {4, 16};
// after
param.draft_token_num = 16;
param.batch_draft_token_num = {4, 16};

int dt = param.draft_token_num;
for (int v : param.batch_draft_token_num)
  if (v != std::numeric_limits<int>::max() && v > dt)
    throw std::invalid_argument("batch_draft_token_num entry too large");

bool params_consistent(const ngram::Param& p) {
  int dt = p.draft_token_num;
  if (dt <= 0) return false;
  for (int v : p.batch_draft_token_num)
    if (v != std::numeric_limits<int>::max() && v > dt) return false;
  return true;
}

try { Ngram n(param); } catch (const std::runtime_error& e) { /* fix config, log e.what() */ }

When it happens

Trigger: Constructing ngram::Ngram with a Param where some batch_draft_token_num[i] (other than the sentinel numeric_limits::max()) is greater than draft_token_num.

Common situations: Setting per-request speculative draft lengths (e.g. [8, 64]) while leaving --speculative-num-draft-tokens at a smaller default; tuning batch draft sizes for mixed workloads without raising the global cap.

Related errors

• This browser cannot encode H.264 MP4
• Invalid fused KV rotary/head dim pair: rotary_dim={rotary_di
• num_kv_heads mismatch across layers for fused KV path: expec
• Unknown match_type: '{match_type}'. Must be 'BFS' or 'PROB'.
• Ngram speculative decoding only supports CUDA or CPU devices

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/6d41a61a7aabef7c. Report an issue: GitHub.