{"record":{"id":"50f15bd6db1f5302","repo":"sgl-project/sglang","slug":"received-a-non-tensor-auxiliary-pp-output","errorCode":null,"errorMessage":"received a non-tensor auxiliary PP output","messagePattern":"received a non-tensor auxiliary PP output","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/sampling/sampling_observer_pp.py","lineNumber":79,"sourceCode":"def pop_auxiliary_output_from_pp_tensors(\n    tensors: MutableMapping[str, Any],\n    observer: Optional[SamplingObserver],\n) -> Optional[DeviceAuxiliaryOutput]:\n    output_tensors = {\n        key.removeprefix(_OUTPUT_PREFIX): value\n        for key, value in tensors.items()\n        if key.startswith(_OUTPUT_PREFIX)\n    }\n    if not output_tensors:\n        return None\n    if observer is None:\n        raise RuntimeError(\"received auxiliary PP output without a sampling observer\")\n    if not isinstance(observer, PipelineParallelSamplingObserver):\n        raise RuntimeError(\n            \"sampling observer does not support pipeline-parallel transport\"\n        )\n    if any(not torch.is_tensor(tensor) for tensor in output_tensors.values()):\n        raise RuntimeError(\"received a non-tensor auxiliary PP output\")\n\n    output = observer.from_pp_tensors(output_tensors)\n    if output is None:\n        raise RuntimeError(\"sampling observer did not reconstruct its PP output\")\n    for name in output_tensors:\n        del tensors[f\"{_OUTPUT_PREFIX}{name}\"]\n    return output\n","sourceCodeStart":61,"sourceCodeEnd":87,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/sampling/sampling_observer_pp.py#L61-L87","documentation":"Defensive check on the receiving side: at least one value in the received auxiliary PP tensors is not a torch.Tensor, which shouldn't happen if the sender-side validation ran.","triggerScenarios":"Tensors mutated/replaced with non-tensors between add and pop — e.g. custom transport code, pickling artifacts, or a tensor dict polluted by other code paths.","commonSituations":"Custom inter-process tensor transfer converting values; monkeypatched or older-version sender missing validation (version skew between stages).","solutions":["Ensure all stages run the same sglang version with sender-side validation","Avoid mutating the tensor dict between add_ and pop_ calls","Tensorize any custom injected values"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"import torch\nassert all(torch.is_tensor(v) for v in output_tensors.values()), \"non-tensor leaked into PP tensors\"","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Run identical sglang versions on all PP stages","Don't mutate the tensor dict between add_ and pop_ calls"],"tags":["pipeline-parallel","sampling","torch","validation"],"backgroundTag":"payload-schema-validation","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}