{"record":{"id":"d827fff1fb5b9d22","repo":"sgl-project/sglang","slug":"minimax-h3-text-encode-failed-on-rank-owner-ow","errorCode":null,"errorMessage":"MiniMax H3 text encode failed on rank {owner}: {owner_error}","messagePattern":"MiniMax H3 text encode failed on rank (.+?): (.+?)","errorType":"error_code","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/multimodal_gen/runtime/pipelines_core/stages/model_specific_stages/minimax_h3/stages/text_encoding.py","lineNumber":179,"sourceCode":"                        first_batch.extra.pop(\n                            _MINIMAX_H3_SINGLE_COPY_TEXT_ENCODE_EXTRA_KEY, None\n                        )\n                    payload = first_result.extra.get(\n                        MINIMAX_H3_TEXT_EMBEDDINGS_EXTRA_KEY\n                    )\n                    if not isinstance(payload, dict):\n                        raise ValueError(\n                            \"MiniMax H3 text encode produced no native payload\"\n                        )\n                except Exception as exc:\n                    owner_exception = exc\n                    owner_error = f\"{type(exc).__name__}: {exc}\"\n\n            owner_error = dp_group.broadcast_object(owner_error, src=owner)\n            if owner_error is not None:\n                if owner_exception is not None:\n                    raise owner_exception\n                raise RuntimeError(\n                    f\"MiniMax H3 text encode failed on rank {owner}: {owner_error}\"\n                )\n            payload = dp_group.broadcast_tensor_dict(payload, src=owner)\n            if not isinstance(payload, dict):\n                raise RuntimeError(\"MiniMax H3 text payload broadcast failed\")\n\n            if dp_group.rank_in_group != owner:\n                first_batch.extra[MINIMAX_H3_TEXT_EMBEDDINGS_EXTRA_KEY] = payload\n                self._publish_native_text_conditioning(first_batch)\n                first_result = first_batch\n            results[first_index] = first_result\n\n            for index, batch in equivalent[1:]:\n                self.copy_deduplicated_outputs(first_result, batch)\n                results[index] = batch\n\n        return [result for result in results if result is not None]\n","sourceCodeStart":161,"sourceCodeEnd":197,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/multimodal_gen/runtime/pipelines_core/stages/model_specific_stages/minimax_h3/stages/text_encoding.py#L161-L197","documentation":"When any rank in the data-parallel group fails text encode, the error string is broadcast to all ranks; ranks without the original exception raise this RuntimeError, attributing the failure to the owner rank. It is a propagation wrapper — the root cause is the owner_error text from the failing rank.","triggerScenarios":"The owner rank's encode raised (owner_error non-None after broadcast_object) and non-owner ranks hit this raise in run_grouped_requests, since owner_exception only exists on the failing rank.","commonSituations":"OOM or CUDA error on one DP rank, shape mismatch in the encoder on the owner, or any exception inside the owner's forward that the group then synchronizes on.","solutions":["Read the message text: the real exception is the owner_error suffix from the owner rank — debug that rank's log","Fix the underlying owner-rank failure (OOM, input shape, component mismatch)","If flaky/infra-related, retry the request after the owner rank recovers"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try:\n    stage.run_grouped_requests(batches)\nexcept RuntimeError as e:\n    if \"text encode failed on rank\" in str(e):\n        log_owner_rank_failure(str(e)); return retryable_error(e)\n    raise","preventionTips":["Treat this as propagation: always inspect the owner rank's log for the root cause","Monitor per-rank OOM/CUDA health before grouped encode"],"tags":["minimax-h3","data-parallel","error-propagation","text-encoding"],"backgroundTag":"distributed-worker-failure","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}