sgl-project/sglang · error · RuntimeError

return_sampling_mask with disaggregation requires SGLANG_DIS

Error message

return_sampling_mask with disaggregation requires SGLANG_DISAGGREGATION_SAMPLING_MASK_MAX_TOKENS > 0.

What it means

A request with return_sampling_mask=True is being handed off via disaggregation, but the server did not enable sampling-mask transfer (SGLANG_DISAGGREGATION_SAMPLING_MASK_MAX_TOKENS <= 0), so there is no buffer to carry the mask and the copy raises.

Source

Thrown at python/sglang/srt/disaggregation/utils.py:507

                    )
                self.output_top_logprobs_val[req.metadata_buffer_index][
                    : len(req.logprob.output_top_logprobs_val[0])
                ] = torch.tensor(
                    req.logprob.output_top_logprobs_val[0],
                    dtype=torch.float32,
                    device="cpu",
                )
            if req.logprob.output_top_logprobs_idx:  # not none or empty list
                self.output_top_logprobs_idx[req.metadata_buffer_index][
                    : len(req.logprob.output_top_logprobs_idx[0])
                ] = torch.tensor(
                    req.logprob.output_top_logprobs_idx[0],
                    dtype=torch.int32,
                    device="cpu",
                )
        if req.return_sampling_mask:
            if not self.enable_sampling_mask:
                raise RuntimeError(
                    "return_sampling_mask with disaggregation requires "
                    "SGLANG_DISAGGREGATION_SAMPLING_MASK_MAX_TOKENS > 0."
                )
            # Sentinel -1: the decode side records None for this handoff token.
            self.output_token_sampling_mask_len[req.metadata_buffer_index][0] = -1
            sampling_masks = req.output_token_sampling_mask
            sampling_logprobs = req.output_token_sampling_logprobs
            if sampling_masks:
                sampling_mask = sampling_masks[0]
                sampling_logprob = sampling_logprobs[0] if sampling_logprobs else None
                if sampling_mask is not None and sampling_logprob is not None:
                    mask_len = len(sampling_mask)
                    max_mask_len = self.output_token_sampling_mask_idx.shape[1]
                    if mask_len > max_mask_len:
                        raise RuntimeError(
                            f"Sampling mask length {mask_len} exceeds disaggregation "
                            f"metadata capacity {max_mask_len}. Increase "
                            "SGLANG_DISAGGREGATION_SAMPLING_MASK_MAX_TOKENS."

View on GitHub (pinned to 0132848349)

Solutions

1. Set SGLANG_DISAGGREGATION_SAMPLING_MASK_MAX_TOKENS to a positive value >= max expected mask length and restart the server.
2. Or stop sending return_sampling_mask=True on requests served by this disaggregated deployment.

Example fix

# before
export SGLANG_DISAGGREGATION_SAMPLING_MASK_MAX_TOKENS=0  # or unset
# after
export SGLANG_DISAGGREGATION_SAMPLING_MASK_MAX_TOKENS=1024
python -m sglang.launch_server --disaggregation-prefill ...

import os
need_mask = any(r.return_sampling_mask for r in pending_requests)
if need_mask and int(os.environ.get('SGLANG_DISAGGREGATION_SAMPLING_MASK_MAX_TOKENS', '0')) <= 0:
    raise SystemExit('enable SGLANG_DISAGGREGATION_SAMPLING_MASK_MAX_TOKENS before handoff')

try:
    controller.send_kv_chunk(req, ...)
except RuntimeError as e:
    if 'SGLANG_DISAGGREGATION_SAMPLING_MASK_MAX_TOKENS' in str(e):
        drop_sampling_mask_and_retry(req)
    raise

When it happens

Trigger: set_buf() (from send_kv_chunk) sees req.return_sampling_mask true while self.enable_sampling_mask is false, i.e. the env var was unset or set to 0 at scheduler startup.

Common situations: Enabling return_sampling_mask in the API request (e.g. for speculative-decoding replay or constrained decoding) against a disaggregated server started without SGLANG_DISAGGREGATION_SAMPLING_MASK_MAX_TOKENS; common after adding the request flag without redeploying the server with the env var.

Understand the failure class

Background: "environment variable is not set" and "Missing keys in environment" errors: what missing required env var messages mean and how to fix them — this error's family across 28 libraries.

Related errors

• SGLANG_DISAGG_STAGING_BUFFER is designed for non-MLA models
• SGLANG_DISAGG_STAGING_BUFFER requires a positive chunked_pre
• SGLANG_DISAGG_STAGING_BUFFER with pp_size > 1 is only suppor
• SGLANG_DISAGG_STAGING_BUFFER does not support prefill contex
• num_instances must be >= 1, got {num_instances}

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/0e4cc0ef854d000f. Report an issue: GitHub.