{"record":{"id":"0f6c2e5d6c51fec0","repo":"sgl-project/sglang","slug":"received-auxiliary-pp-output-without-a-sampling-ob","errorCode":null,"errorMessage":"received auxiliary PP output without a sampling observer","messagePattern":"received auxiliary PP output without a sampling observer","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/sampling/sampling_observer_pp.py","lineNumber":73,"sourceCode":"        key = f\"{_OUTPUT_PREFIX}{name}\"\n        if key in tensors:\n            raise RuntimeError(f\"duplicate auxiliary PP tensor {name!r}\")\n        tensors[key] = tensor\n\n\ndef pop_auxiliary_output_from_pp_tensors(\n    tensors: MutableMapping[str, Any],\n    observer: Optional[SamplingObserver],\n) -> Optional[DeviceAuxiliaryOutput]:\n    output_tensors = {\n        key.removeprefix(_OUTPUT_PREFIX): value\n        for key, value in tensors.items()\n        if key.startswith(_OUTPUT_PREFIX)\n    }\n    if not output_tensors:\n        return None\n    if observer is None:\n        raise RuntimeError(\"received auxiliary PP output without a sampling observer\")\n    if not isinstance(observer, PipelineParallelSamplingObserver):\n        raise RuntimeError(\n            \"sampling observer does not support pipeline-parallel transport\"\n        )\n    if any(not torch.is_tensor(tensor) for tensor in output_tensors.values()):\n        raise RuntimeError(\"received a non-tensor auxiliary PP output\")\n\n    output = observer.from_pp_tensors(output_tensors)\n    if output is None:\n        raise RuntimeError(\"sampling observer did not reconstruct its PP output\")\n    for name in output_tensors:\n        del tensors[f\"{_OUTPUT_PREFIX}{name}\"]\n    return output\n","sourceCodeStart":55,"sourceCodeEnd":87,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/sampling/sampling_observer_pp.py#L55-L87","documentation":"On the receiving pipeline stage, auxiliary PP tensors arrived in the transported dict but no SamplingObserver was provided to reconstruct them.","triggerScenarios":"pop_auxiliary_output_from_pp_tensors finding prefix-keys while observer is None — last PP stage lacking a sampling observer configuration while earlier stages emitted auxiliary outputs.","commonSituations":"PP deployment where the final stage has sampling observers disabled (e.g. no sampling params requiring observation) but the model still emits auxiliary outputs.","solutions":["Construct/pass a PipelineParallelSamplingObserver on the stage that pops the outputs","Disable auxiliary output emission on earlier stages when no observer is configured","Ensure observer wiring is propagated with the batch result across stages"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"from sglang.srt.sampling.sampling_observer_pp import pop_auxiliary_output_from_pp_tensors, _OUTPUT_PREFIX\nhas_aux = any(k.startswith(_OUTPUT_PREFIX) for k in tensors)\nif has_aux:\n    assert observer is not None, \"configure a sampling observer on the final PP stage\"","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Always configure a PipelineParallelSamplingObserver on stages that receive batch results","Disable auxiliary output emission when no observer is set"],"tags":["pipeline-parallel","sampling","observer","configuration"],"backgroundTag":"missing-handler-on-receive","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}