{"record":{"id":"83d0cfac0791d468","repo":"sgl-project/sglang","slug":"top-logprobs-num-top-logprobs-len-exceeds-disagg","errorCode":null,"errorMessage":"top_logprobs_num {top_logprobs_len} exceeds disaggregation metadata capacity {max_top_logprobs_len}. Lower top_logprobs_num or increase the metadata buffer.","messagePattern":"top_logprobs_num (.+?) exceeds disaggregation metadata capacity (.+?)\\. Lower top_logprobs_num or increase the metadata buffer\\.","errorType":"validation","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/disaggregation/utils.py","lineNumber":485,"sourceCode":"            image_t = audio_t = video_t = 0\n        self.cached_tokens[req.metadata_buffer_index][4] = image_t\n        self.cached_tokens[req.metadata_buffer_index][5] = audio_t\n        self.cached_tokens[req.metadata_buffer_index][6] = video_t\n        if req.return_logprob:\n            if req.logprob.output_token_logprobs_val:  # not none or empty list\n                self.output_token_logprobs_val[req.metadata_buffer_index][0] = (\n                    req.logprob.output_token_logprobs_val[0]\n                )\n            if req.logprob.output_token_logprobs_idx:  # not none or empty list\n                self.output_token_logprobs_idx[req.metadata_buffer_index][0] = (\n                    req.logprob.output_token_logprobs_idx[0]\n                )\n\n            if req.logprob.output_top_logprobs_val:  # not none or empty list\n                top_logprobs_len = len(req.logprob.output_top_logprobs_val[0])\n                max_top_logprobs_len = self.output_top_logprobs_val.shape[1]\n                if top_logprobs_len > max_top_logprobs_len:\n                    raise RuntimeError(\n                        f\"top_logprobs_num {top_logprobs_len} exceeds \"\n                        f\"disaggregation metadata capacity {max_top_logprobs_len}. \"\n                        \"Lower top_logprobs_num or increase the metadata buffer.\"\n                    )\n                self.output_top_logprobs_val[req.metadata_buffer_index][\n                    : len(req.logprob.output_top_logprobs_val[0])\n                ] = torch.tensor(\n                    req.logprob.output_top_logprobs_val[0],\n                    dtype=torch.float32,\n                    device=\"cpu\",\n                )\n            if req.logprob.output_top_logprobs_idx:  # not none or empty list\n                self.output_top_logprobs_idx[req.metadata_buffer_index][\n                    : len(req.logprob.output_top_logprobs_idx[0])\n                ] = torch.tensor(\n                    req.logprob.output_top_logprobs_idx[0],\n                    dtype=torch.int32,\n                    device=\"cpu\",","sourceCodeStart":467,"sourceCodeEnd":503,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/disaggregation/utils.py#L467-L503","documentation":"When a request with logprobs is handed off in disaggregated serving, its top-logprobs are copied into a fixed-size pre-allocated metadata buffer. If the request's top_logprobs_num (length of output_top_logprobs_val entries) exceeds the buffer's second dimension, the copy would silently truncate so the transfer layer raises instead.","triggerScenarios":"Calling set_buf() (via send_kv_chunk) for a request whose req.logprob.output_top_logprobs_val[0] is longer than output_top_logprobs_val.shape[1]; the capacity was fixed when the metadata buffer was allocated, typically from server defaults or SGLANG env sizing.","commonSituations":"Client passes a large top_logprobs_num (e.g. 50) in sampling params to a PD-disaggregated server whose metadata buffer was sized for a smaller default (e.g. 20), often after enabling logprobs mid-deployment or raising the client-side parameter.","solutions":["Lower the client's top_logprobs_num to fit the server buffer capacity.","Or increase the metadata buffer capacity: configure the env/startup arg that sizes top-logprobs metadata (see SGLANG_DISAGGREGATION_* metadata sizing env vars) before server start.","If you don't need logprobs on handoff, drop logprobs from the request."],"exampleFix":"# before\noutputs = llm.generate(prompts, SamplingParams(top_logprobs_num=50))\n# after\noutputs = llm.generate(prompts, SamplingParams(top_logprobs_num=20))  # <= server metadata capacity","handlingStrategy":"validation","validationCode":"max_cap = metadata_buffer.output_top_logprobs_val.shape[1]\nif sampling_params.top_logprobs_num > max_cap:\n    sampling_params.top_logprobs_num = max_cap  # or reject the request","typeGuard":null,"tryCatchPattern":"try:\n    controller.send_kv_chunk(req, ...)\nexcept RuntimeError as e:\n    if 'exceeds disaggregation metadata capacity' in str(e):\n        req.reject_with_4xx('top_logprobs_num too large for PD metadata buffer')\n    raise","preventionTips":["Surface the server's max top_logprobs capacity in model metadata so clients can clamp.","Size the metadata buffer from the max top_logprobs_num you intend to allow.","Clamp client-side top_logprobs_num before submitting to disaggregated endpoints."],"tags":["sglang","disaggregation","logprobs","metadata-buffer","capacity"],"backgroundTag":"buffer-capacity-exceeded","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}