sgl-project/sglang · error · RuntimeError

top_logprobs_num {top_logprobs_len} exceeds disaggregation m

Error message

top_logprobs_num {top_logprobs_len} exceeds disaggregation metadata capacity {max_top_logprobs_len}. Lower top_logprobs_num or increase the metadata buffer.

What it means

When a request with logprobs is handed off in disaggregated serving, its top-logprobs are copied into a fixed-size pre-allocated metadata buffer. If the request's top_logprobs_num (length of output_top_logprobs_val entries) exceeds the buffer's second dimension, the copy would silently truncate so the transfer layer raises instead.

Source

Thrown at python/sglang/srt/disaggregation/utils.py:485

            image_t = audio_t = video_t = 0
        self.cached_tokens[req.metadata_buffer_index][4] = image_t
        self.cached_tokens[req.metadata_buffer_index][5] = audio_t
        self.cached_tokens[req.metadata_buffer_index][6] = video_t
        if req.return_logprob:
            if req.logprob.output_token_logprobs_val:  # not none or empty list
                self.output_token_logprobs_val[req.metadata_buffer_index][0] = (
                    req.logprob.output_token_logprobs_val[0]
                )
            if req.logprob.output_token_logprobs_idx:  # not none or empty list
                self.output_token_logprobs_idx[req.metadata_buffer_index][0] = (
                    req.logprob.output_token_logprobs_idx[0]
                )

            if req.logprob.output_top_logprobs_val:  # not none or empty list
                top_logprobs_len = len(req.logprob.output_top_logprobs_val[0])
                max_top_logprobs_len = self.output_top_logprobs_val.shape[1]
                if top_logprobs_len > max_top_logprobs_len:
                    raise RuntimeError(
                        f"top_logprobs_num {top_logprobs_len} exceeds "
                        f"disaggregation metadata capacity {max_top_logprobs_len}. "
                        "Lower top_logprobs_num or increase the metadata buffer."
                    )
                self.output_top_logprobs_val[req.metadata_buffer_index][
                    : len(req.logprob.output_top_logprobs_val[0])
                ] = torch.tensor(
                    req.logprob.output_top_logprobs_val[0],
                    dtype=torch.float32,
                    device="cpu",
                )
            if req.logprob.output_top_logprobs_idx:  # not none or empty list
                self.output_top_logprobs_idx[req.metadata_buffer_index][
                    : len(req.logprob.output_top_logprobs_idx[0])
                ] = torch.tensor(
                    req.logprob.output_top_logprobs_idx[0],
                    dtype=torch.int32,
                    device="cpu",

View on GitHub (pinned to 0132848349)

Solutions

Lower the client's top_logprobs_num to fit the server buffer capacity.
Or increase the metadata buffer capacity: configure the env/startup arg that sizes top-logprobs metadata (see SGLANG_DISAGGREGATION_* metadata sizing env vars) before server start.
If you don't need logprobs on handoff, drop logprobs from the request.

Example fix

# before
outputs = llm.generate(prompts, SamplingParams(top_logprobs_num=50))
# after
outputs = llm.generate(prompts, SamplingParams(top_logprobs_num=20))  # <= server metadata capacity

Defensive patterns

Strategy: validation

Validate before calling

max_cap = metadata_buffer.output_top_logprobs_val.shape[1]
if sampling_params.top_logprobs_num > max_cap:
    sampling_params.top_logprobs_num = max_cap  # or reject the request

Try / catch

try:
    controller.send_kv_chunk(req, ...)
except RuntimeError as e:
    if 'exceeds disaggregation metadata capacity' in str(e):
        req.reject_with_4xx('top_logprobs_num too large for PD metadata buffer')
    raise

Prevention

Surface the server's max top_logprobs capacity in model metadata so clients can clamp.
Size the metadata buffer from the max top_logprobs_num you intend to allow.
Clamp client-side top_logprobs_num before submitting to disaggregated endpoints.

When it happens

Trigger: Calling set_buf() (via send_kv_chunk) for a request whose req.logprob.output_top_logprobs_val[0] is longer than output_top_logprobs_val.shape[1]; the capacity was fixed when the metadata buffer was allocated, typically from server defaults or SGLANG env sizing.

Common situations: Client passes a large top_logprobs_num (e.g. 50) in sampling params to a PD-disaggregated server whose metadata buffer was sized for a smaller default (e.g. 20), often after enabling logprobs mid-deployment or raising the client-side parameter.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/83d0cfac0791d468. Report an issue: GitHub.