sgl-project/sglang · critical · RuntimeError

MiniMax H3 audio decode failed on rank 0: {owner_error}

Error message

MiniMax H3 audio decode failed on rank 0: {owner_error}

What it means

Audio VAE decode runs on rank 0 and its error is broadcast through the replica group. If rank 0 failed and the original exception object is not available on this rank, a RuntimeError 'MiniMax H3 audio decode failed on rank 0: <error>' is raised so every rank fails consistently.

Source

Thrown at python/sglang/multimodal_gen/runtime/pipelines_core/stages/model_specific_stages/minimax_h3/stages/decoding.py:446

        # Audio VAE weights are replicated. Decode on replica rank 0 and broadcast
        # only within the request's replica, excluding independent DP replicas.
        replica_group = get_replica_group() if model_parallel_is_initialized() else None
        is_audio_owner = replica_group is None or replica_group.rank_in_group == 0
        owner_exception = None
        owner_error = None
        audio_payload = None
        if is_audio_owner:
            try:
                audio_payload = self._decode_audio(audio_latent, server_args)
            except Exception as exc:
                owner_exception = exc
                owner_error = f"{type(exc).__name__}: {exc}"
        if replica_group is not None:
            owner_error = replica_group.broadcast_object(owner_error, src=0)
        if owner_error is not None:
            if owner_exception is not None:
                raise owner_exception
            raise RuntimeError(
                f"MiniMax H3 audio decode failed on rank 0: {owner_error}"
            )
        if replica_group is not None:
            audio_payload = replica_group.broadcast_tensor_dict(audio_payload, src=0)
        if not isinstance(audio_payload, dict):
            raise RuntimeError("MiniMax H3 audio decode produced no output payload")
        audio_waveform = _required_tensor(
            audio_payload.get("waveform"), "audio_vae.decode"
        )
        audio_sample_rate = int(audio_payload["sample_rate"])

        visual_frames = server_args.pipeline_config.post_decoding(
            visual_frames, server_args
        )
        output_audio_waveform = _canonical_output_audio_waveform(
            audio_waveform, batch_size=int(visual_frames.shape[0])
        )
        return OutputBatch(

View on GitHub (pinned to 0132848349)

Solutions

1. Read the rank-0 log for the underlying error and fix it (memory, weights, latent shape)
2. Validate batch.audio_latents shape [audio_channel, latent_dim, T] before decode
3. Free GPU memory / reduce concurrency if the root cause is OOM

assert batch.audio_latents.ndim == 3

def valid_audio_latents(t) -> bool:
    return t is not None and t.ndim == 3

try:
    out = stage.forward(batch)
except RuntimeError as e:
    if "audio decode failed on rank 0" in str(e):
        collect_rank0_traceback(); raise

When it happens

Trigger: Any rank-0 exception inside the audio VAE decode (bad latent shapes, OOM, missing audio_vae weights) re-surfaced on non-owner ranks of the replica group.

Common situations: Multi-replica serving where the audio decoder OOMs or receives malformed latents on one replica; root-cause details only present in rank-0 logs.

Related errors

• MiniMax H3 audio encode failed on rank 0: {owner_error}
• MiniMax H3 audio decode produced no output payload
• replica broadcast of batch.extra[{key!r}] got None
• world_size must be positive and divide global_heads
• fl2va requires first_frame, last_frame, or both

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/e9a046fb6dca6058. Report an issue: GitHub.