{"record":{"id":"d1cfbc43ded5dbec","repo":"sgl-project/sglang","slug":"sampling-mask-length-mask-len-exceeds-disaggrega","errorCode":null,"errorMessage":"Sampling mask length {mask_len} exceeds disaggregation metadata capacity {max_mask_len}. Increase SGLANG_DISAGGREGATION_SAMPLING_MASK_MAX_TOKENS.","messagePattern":"Sampling mask length (.+?) exceeds disaggregation metadata capacity (.+?)\\. Increase SGLANG_DISAGGREGATION_SAMPLING_MASK_MAX_TOKENS\\.","errorType":"validation","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/disaggregation/utils.py","lineNumber":522,"sourceCode":"                )\n        if req.return_sampling_mask:\n            if not self.enable_sampling_mask:\n                raise RuntimeError(\n                    \"return_sampling_mask with disaggregation requires \"\n                    \"SGLANG_DISAGGREGATION_SAMPLING_MASK_MAX_TOKENS > 0.\"\n                )\n            # Sentinel -1: the decode side records None for this handoff token.\n            self.output_token_sampling_mask_len[req.metadata_buffer_index][0] = -1\n            sampling_masks = req.output_token_sampling_mask\n            sampling_logprobs = req.output_token_sampling_logprobs\n            if sampling_masks:\n                sampling_mask = sampling_masks[0]\n                sampling_logprob = sampling_logprobs[0] if sampling_logprobs else None\n                if sampling_mask is not None and sampling_logprob is not None:\n                    mask_len = len(sampling_mask)\n                    max_mask_len = self.output_token_sampling_mask_idx.shape[1]\n                    if mask_len > max_mask_len:\n                        raise RuntimeError(\n                            f\"Sampling mask length {mask_len} exceeds disaggregation \"\n                            f\"metadata capacity {max_mask_len}. Increase \"\n                            \"SGLANG_DISAGGREGATION_SAMPLING_MASK_MAX_TOKENS.\"\n                        )\n                    self.output_token_sampling_mask_len[req.metadata_buffer_index][\n                        0\n                    ] = mask_len\n                    if mask_len:\n                        self.output_token_sampling_mask_idx[\n                            req.metadata_buffer_index, :mask_len\n                        ].copy_(\n                            torch.tensor(\n                                sampling_mask,\n                                dtype=torch.int32,\n                                device=self.output_token_sampling_mask_idx.device,\n                            )\n                        )\n                    self.output_token_sampling_logprobs[req.metadata_buffer_index][","sourceCodeStart":504,"sourceCodeEnd":540,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/disaggregation/utils.py#L504-L540","documentation":"During PD (prefill-decode) disaggregation, the per-request sampling mask sent in the metadata buffer is longer than the preallocated second dimension of output_token_sampling_mask_idx. The buffer is sized once from SGLANG_DISAGGREGATION_SAMPLING_MASK_MAX_TOKENS, so any request whose sampled-token mask exceeds that capacity is rejected in set_buf before transfer.","triggerScenarios":"Calling set_buf (via send_kv_chunk) with sampling_masks[0] whose len exceeds output_token_sampling_mask_idx.shape[1], e.g. large speculative-decoding lookahead, long branch sampling, or many sampled tokens per request while the env var was left at default.","commonSituations":"Running PD disaggregation with speculative decoding or wide sampling beams after upgrading; capacity env var not raised when max_new_tokens/branch width grew; smaller value set in deployment scripts than tests use.","solutions":["Increase SGLANG_DISAGGREGATION_SAMPLING_MASK_MAX_TOKENS (e.g. export SGLANG_DISAGGREGATION_SAMPLING_MASK_MAX_TOKENS=4096) to at least your maximum expected mask length on both prefill and decode nodes","Reduce the number of sampled tokens / spec-decoding draft tokens per request so the mask fits the default capacity","Verify len(sampling_masks[0]) at request build time and reject/trim oversized requests before send_kv_chunk"],"exampleFix":"# before\nSGLANG_DISAGGREGATION_SAMPLING_MASK_MAX_TOKENS=512 python -m sglang.launch_server --disaggregation-prefill ...\n# after\nSGLANG_DISAGGREGATION_SAMPLING_MASK_MAX_TOKENS=8192 python -m sglang.launch_server --disaggregation-prefill ...","handlingStrategy":"validation","validationCode":"max_len = bootstrap_out.output_token_sampling_mask_idx.shape[1]\nassert all(m is None or len(m) <= max_len for m in sampling_masks), f\"mask exceeds {max_len}\"","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Set SGLANG_DISAGGREGATION_SAMPLING_MASK_MAX_TOKENS to your worst-case sampled-token count on all PD nodes","Log len(sampling_masks[0]) per request during rollout testing to size the buffer"],"tags":["disaggregation","sampling","buffer-capacity","pd-disaggregation"],"backgroundTag":"buffer-capacity-exceeded","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}