{"record":{"id":"e9a046fb6dca6058","repo":"sgl-project/sglang","slug":"minimax-h3-audio-decode-failed-on-rank-0-owner-e","errorCode":null,"errorMessage":"MiniMax H3 audio decode failed on rank 0: {owner_error}","messagePattern":"MiniMax H3 audio decode failed on rank 0: (.+?)","errorType":"error_code","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"python/sglang/multimodal_gen/runtime/pipelines_core/stages/model_specific_stages/minimax_h3/stages/decoding.py","lineNumber":446,"sourceCode":"        # Audio VAE weights are replicated. Decode on replica rank 0 and broadcast\n        # only within the request's replica, excluding independent DP replicas.\n        replica_group = get_replica_group() if model_parallel_is_initialized() else None\n        is_audio_owner = replica_group is None or replica_group.rank_in_group == 0\n        owner_exception = None\n        owner_error = None\n        audio_payload = None\n        if is_audio_owner:\n            try:\n                audio_payload = self._decode_audio(audio_latent, server_args)\n            except Exception as exc:\n                owner_exception = exc\n                owner_error = f\"{type(exc).__name__}: {exc}\"\n        if replica_group is not None:\n            owner_error = replica_group.broadcast_object(owner_error, src=0)\n        if owner_error is not None:\n            if owner_exception is not None:\n                raise owner_exception\n            raise RuntimeError(\n                f\"MiniMax H3 audio decode failed on rank 0: {owner_error}\"\n            )\n        if replica_group is not None:\n            audio_payload = replica_group.broadcast_tensor_dict(audio_payload, src=0)\n        if not isinstance(audio_payload, dict):\n            raise RuntimeError(\"MiniMax H3 audio decode produced no output payload\")\n        audio_waveform = _required_tensor(\n            audio_payload.get(\"waveform\"), \"audio_vae.decode\"\n        )\n        audio_sample_rate = int(audio_payload[\"sample_rate\"])\n\n        visual_frames = server_args.pipeline_config.post_decoding(\n            visual_frames, server_args\n        )\n        output_audio_waveform = _canonical_output_audio_waveform(\n            audio_waveform, batch_size=int(visual_frames.shape[0])\n        )\n        return OutputBatch(","sourceCodeStart":428,"sourceCodeEnd":464,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/multimodal_gen/runtime/pipelines_core/stages/model_specific_stages/minimax_h3/stages/decoding.py#L428-L464","documentation":"Audio VAE decode runs on rank 0 and its error is broadcast through the replica group. If rank 0 failed and the original exception object is not available on this rank, a RuntimeError 'MiniMax H3 audio decode failed on rank 0: <error>' is raised so every rank fails consistently.","triggerScenarios":"Any rank-0 exception inside the audio VAE decode (bad latent shapes, OOM, missing audio_vae weights) re-surfaced on non-owner ranks of the replica group.","commonSituations":"Multi-replica serving where the audio decoder OOMs or receives malformed latents on one replica; root-cause details only present in rank-0 logs.","solutions":["Read the rank-0 log for the underlying error and fix it (memory, weights, latent shape)","Validate batch.audio_latents shape [audio_channel, latent_dim, T] before decode","Free GPU memory / reduce concurrency if the root cause is OOM"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"assert batch.audio_latents.ndim == 3","typeGuard":"def valid_audio_latents(t) -> bool:\n    return t is not None and t.ndim == 3","tryCatchPattern":"try:\n    out = stage.forward(batch)\nexcept RuntimeError as e:\n    if \"audio decode failed on rank 0\" in str(e):\n        collect_rank0_traceback(); raise","preventionTips":["Validate latent shapes pre-decode","Watch rank-0 memory; root cause only appears in rank-0 logs"],"tags":["minimax-h3","audio-decode","distributed","rank0"],"backgroundTag":"distributed-rank0-failure","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}