{"record":{"id":"0e4cc0ef854d000f","repo":"sgl-project/sglang","slug":"return-sampling-mask-with-disaggregation-requires","errorCode":null,"errorMessage":"return_sampling_mask with disaggregation requires SGLANG_DISAGGREGATION_SAMPLING_MASK_MAX_TOKENS > 0.","messagePattern":"return_sampling_mask with disaggregation requires SGLANG_DISAGGREGATION_SAMPLING_MASK_MAX_TOKENS > 0\\.","errorType":"validation","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/disaggregation/utils.py","lineNumber":507,"sourceCode":"                    )\n                self.output_top_logprobs_val[req.metadata_buffer_index][\n                    : len(req.logprob.output_top_logprobs_val[0])\n                ] = torch.tensor(\n                    req.logprob.output_top_logprobs_val[0],\n                    dtype=torch.float32,\n                    device=\"cpu\",\n                )\n            if req.logprob.output_top_logprobs_idx:  # not none or empty list\n                self.output_top_logprobs_idx[req.metadata_buffer_index][\n                    : len(req.logprob.output_top_logprobs_idx[0])\n                ] = torch.tensor(\n                    req.logprob.output_top_logprobs_idx[0],\n                    dtype=torch.int32,\n                    device=\"cpu\",\n                )\n        if req.return_sampling_mask:\n            if not self.enable_sampling_mask:\n                raise RuntimeError(\n                    \"return_sampling_mask with disaggregation requires \"\n                    \"SGLANG_DISAGGREGATION_SAMPLING_MASK_MAX_TOKENS > 0.\"\n                )\n            # Sentinel -1: the decode side records None for this handoff token.\n            self.output_token_sampling_mask_len[req.metadata_buffer_index][0] = -1\n            sampling_masks = req.output_token_sampling_mask\n            sampling_logprobs = req.output_token_sampling_logprobs\n            if sampling_masks:\n                sampling_mask = sampling_masks[0]\n                sampling_logprob = sampling_logprobs[0] if sampling_logprobs else None\n                if sampling_mask is not None and sampling_logprob is not None:\n                    mask_len = len(sampling_mask)\n                    max_mask_len = self.output_token_sampling_mask_idx.shape[1]\n                    if mask_len > max_mask_len:\n                        raise RuntimeError(\n                            f\"Sampling mask length {mask_len} exceeds disaggregation \"\n                            f\"metadata capacity {max_mask_len}. Increase \"\n                            \"SGLANG_DISAGGREGATION_SAMPLING_MASK_MAX_TOKENS.\"","sourceCodeStart":489,"sourceCodeEnd":525,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/disaggregation/utils.py#L489-L525","documentation":"A request with return_sampling_mask=True is being handed off via disaggregation, but the server did not enable sampling-mask transfer (SGLANG_DISAGGREGATION_SAMPLING_MASK_MAX_TOKENS <= 0), so there is no buffer to carry the mask and the copy raises.","triggerScenarios":"set_buf() (from send_kv_chunk) sees req.return_sampling_mask true while self.enable_sampling_mask is false, i.e. the env var was unset or set to 0 at scheduler startup.","commonSituations":"Enabling return_sampling_mask in the API request (e.g. for speculative-decoding replay or constrained decoding) against a disaggregated server started without SGLANG_DISAGGREGATION_SAMPLING_MASK_MAX_TOKENS; common after adding the request flag without redeploying the server with the env var.","solutions":["Set SGLANG_DISAGGREGATION_SAMPLING_MASK_MAX_TOKENS to a positive value >= max expected mask length and restart the server.","Or stop sending return_sampling_mask=True on requests served by this disaggregated deployment."],"exampleFix":"# before\nexport SGLANG_DISAGGREGATION_SAMPLING_MASK_MAX_TOKENS=0  # or unset\n# after\nexport SGLANG_DISAGGREGATION_SAMPLING_MASK_MAX_TOKENS=1024\npython -m sglang.launch_server --disaggregation-prefill ...","handlingStrategy":"validation","validationCode":"import os\nneed_mask = any(r.return_sampling_mask for r in pending_requests)\nif need_mask and int(os.environ.get('SGLANG_DISAGGREGATION_SAMPLING_MASK_MAX_TOKENS', '0')) <= 0:\n    raise SystemExit('enable SGLANG_DISAGGREGATION_SAMPLING_MASK_MAX_TOKENS before handoff')","typeGuard":null,"tryCatchPattern":"try:\n    controller.send_kv_chunk(req, ...)\nexcept RuntimeError as e:\n    if 'SGLANG_DISAGGREGATION_SAMPLING_MASK_MAX_TOKENS' in str(e):\n        drop_sampling_mask_and_retry(req)\n    raise","preventionTips":["Set SGLANG_DISAGGREGATION_SAMPLING_MASK_MAX_TOKENS at deploy time if any client uses return_sampling_mask.","Gate return_sampling_mask requests at the router/frontend against server capability flags."],"tags":["sglang","disaggregation","sampling-mask","env-var","config-validation"],"backgroundTag":"missing-env-var","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}