{"record":{"id":"e1c93dccdc3bb582","repo":"sgl-project/sglang","slug":"minimax-h3-text-payload-broadcast-failed","errorCode":null,"errorMessage":"MiniMax H3 text payload broadcast failed","messagePattern":"MiniMax H3 text payload broadcast failed","errorType":"error_code","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"python/sglang/multimodal_gen/runtime/pipelines_core/stages/model_specific_stages/minimax_h3/stages/text_encoding.py","lineNumber":184,"sourceCode":"                    )\n                    if not isinstance(payload, dict):\n                        raise ValueError(\n                            \"MiniMax H3 text encode produced no native payload\"\n                        )\n                except Exception as exc:\n                    owner_exception = exc\n                    owner_error = f\"{type(exc).__name__}: {exc}\"\n\n            owner_error = dp_group.broadcast_object(owner_error, src=owner)\n            if owner_error is not None:\n                if owner_exception is not None:\n                    raise owner_exception\n                raise RuntimeError(\n                    f\"MiniMax H3 text encode failed on rank {owner}: {owner_error}\"\n                )\n            payload = dp_group.broadcast_tensor_dict(payload, src=owner)\n            if not isinstance(payload, dict):\n                raise RuntimeError(\"MiniMax H3 text payload broadcast failed\")\n\n            if dp_group.rank_in_group != owner:\n                first_batch.extra[MINIMAX_H3_TEXT_EMBEDDINGS_EXTRA_KEY] = payload\n                self._publish_native_text_conditioning(first_batch)\n                first_result = first_batch\n            results[first_index] = first_result\n\n            for index, batch in equivalent[1:]:\n                self.copy_deduplicated_outputs(first_result, batch)\n                results[index] = batch\n\n        return [result for result in results if result is not None]\n\n    def _log_dp_choice(self, batch_size: int, world_size: int) -> None:\n        if self._dp_choice_logged:\n            return\n        self._dp_choice_logged = True\n        logger.info(","sourceCodeStart":166,"sourceCodeEnd":202,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/multimodal_gen/runtime/pipelines_core/stages/model_specific_stages/minimax_h3/stages/text_encoding.py#L166-L202","documentation":"After a successful owner encode, the payload dict is broadcast with broadcast_tensor_dict; if the received object is not a dict, the group cannot sync text embeddings and this RuntimeError fires. It signals a collective/broadcast corruption rather than an encode failure.","triggerScenarios":"dp_group.broadcast_tensor_dict returns a non-dict (None, list, or truncated object) for MINIMAX_H3_TEXT_EMBEDDINGS_EXTRA_KEY — typically from a NCCL/collective desync, mismatched group membership, or tensor types that cannot traverse the tensor-dict protocol.","commonSituations":"DP ranks disagreeing on whether encode runs (divergent control flow), mismatched versions of the collective helper across ranks, or NCCL timeouts silently degrading the broadcast.","solutions":["Verify all DP ranks take the same grouped-encode path (same batch grouping decisions) so collectives line up","Check NCCL health / ranks' GPU state; rerun the job if the collective layer glitched","Upgrade sglang so all ranks run identical broadcast_tensor_dict implementations"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"assert all(r.will_group_encode for r in dp_ranks), \"divergent control flow across DP ranks\"","typeGuard":null,"tryCatchPattern":"try:\n    stage.run_grouped_requests(batches)\nexcept RuntimeError as e:\n    if \"payload broadcast failed\" in str(e):\n        return retry_after_nccl_check()  # collective desync, often transient\n    raise","preventionTips":["Keep grouping decisions deterministic across DP ranks","Run identical sglang versions on every rank"],"tags":["minimax-h3","broadcast","tensor-dict","collective","distributed-communication"],"backgroundTag":"distributed-broadcast-failure","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}