sgl-project/sglang · error · RuntimeError

Sampling mask length {mask_len} exceeds disaggregation metad

Error message

Sampling mask length {mask_len} exceeds disaggregation metadata capacity {max_mask_len}. Increase SGLANG_DISAGGREGATION_SAMPLING_MASK_MAX_TOKENS.

What it means

During PD (prefill-decode) disaggregation, the per-request sampling mask sent in the metadata buffer is longer than the preallocated second dimension of output_token_sampling_mask_idx. The buffer is sized once from SGLANG_DISAGGREGATION_SAMPLING_MASK_MAX_TOKENS, so any request whose sampled-token mask exceeds that capacity is rejected in set_buf before transfer.

Source

Thrown at python/sglang/srt/disaggregation/utils.py:522

                )
        if req.return_sampling_mask:
            if not self.enable_sampling_mask:
                raise RuntimeError(
                    "return_sampling_mask with disaggregation requires "
                    "SGLANG_DISAGGREGATION_SAMPLING_MASK_MAX_TOKENS > 0."
                )
            # Sentinel -1: the decode side records None for this handoff token.
            self.output_token_sampling_mask_len[req.metadata_buffer_index][0] = -1
            sampling_masks = req.output_token_sampling_mask
            sampling_logprobs = req.output_token_sampling_logprobs
            if sampling_masks:
                sampling_mask = sampling_masks[0]
                sampling_logprob = sampling_logprobs[0] if sampling_logprobs else None
                if sampling_mask is not None and sampling_logprob is not None:
                    mask_len = len(sampling_mask)
                    max_mask_len = self.output_token_sampling_mask_idx.shape[1]
                    if mask_len > max_mask_len:
                        raise RuntimeError(
                            f"Sampling mask length {mask_len} exceeds disaggregation "
                            f"metadata capacity {max_mask_len}. Increase "
                            "SGLANG_DISAGGREGATION_SAMPLING_MASK_MAX_TOKENS."
                        )
                    self.output_token_sampling_mask_len[req.metadata_buffer_index][
                        0
                    ] = mask_len
                    if mask_len:
                        self.output_token_sampling_mask_idx[
                            req.metadata_buffer_index, :mask_len
                        ].copy_(
                            torch.tensor(
                                sampling_mask,
                                dtype=torch.int32,
                                device=self.output_token_sampling_mask_idx.device,
                            )
                        )
                    self.output_token_sampling_logprobs[req.metadata_buffer_index][

View on GitHub (pinned to 0132848349)

Solutions

1. Increase SGLANG_DISAGGREGATION_SAMPLING_MASK_MAX_TOKENS (e.g. export SGLANG_DISAGGREGATION_SAMPLING_MASK_MAX_TOKENS=4096) to at least your maximum expected mask length on both prefill and decode nodes
2. Reduce the number of sampled tokens / spec-decoding draft tokens per request so the mask fits the default capacity
3. Verify len(sampling_masks[0]) at request build time and reject/trim oversized requests before send_kv_chunk

Example fix

# before
SGLANG_DISAGGREGATION_SAMPLING_MASK_MAX_TOKENS=512 python -m sglang.launch_server --disaggregation-prefill ...
# after
SGLANG_DISAGGREGATION_SAMPLING_MASK_MAX_TOKENS=8192 python -m sglang.launch_server --disaggregation-prefill ...

max_len = bootstrap_out.output_token_sampling_mask_idx.shape[1]
assert all(m is None or len(m) <= max_len for m in sampling_masks), f"mask exceeds {max_len}"

When it happens

Trigger: Calling set_buf (via send_kv_chunk) with sampling_masks[0] whose len exceeds output_token_sampling_mask_idx.shape[1], e.g. large speculative-decoding lookahead, long branch sampling, or many sampled tokens per request while the env var was left at default.

Common situations: Running PD disaggregation with speculative decoding or wide sampling beams after upgrading; capacity env var not raised when max_new_tokens/branch width grew; smaller value set in deployment scripts than tests use.

Related errors

• NIXL PD transfer does not support HiSparse combined with dec
• NIXL KV transfer has no KV memory segments
• NIXL heterogeneous-TP direct-to-host KV transfer is not impl
• Invalid disaggregation_mode={cfg.disaggregation_mode!r}
• Input probs contains NaN.

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/d1cfbc43ded5dbec. Report an issue: GitHub.