sgl-project/sglang · critical · RuntimeError

MiniMax H3 text payload broadcast failed

Error message

MiniMax H3 text payload broadcast failed

What it means

After a successful owner encode, the payload dict is broadcast with broadcast_tensor_dict; if the received object is not a dict, the group cannot sync text embeddings and this RuntimeError fires. It signals a collective/broadcast corruption rather than an encode failure.

Source

Thrown at python/sglang/multimodal_gen/runtime/pipelines_core/stages/model_specific_stages/minimax_h3/stages/text_encoding.py:184

                    )
                    if not isinstance(payload, dict):
                        raise ValueError(
                            "MiniMax H3 text encode produced no native payload"
                        )
                except Exception as exc:
                    owner_exception = exc
                    owner_error = f"{type(exc).__name__}: {exc}"

            owner_error = dp_group.broadcast_object(owner_error, src=owner)
            if owner_error is not None:
                if owner_exception is not None:
                    raise owner_exception
                raise RuntimeError(
                    f"MiniMax H3 text encode failed on rank {owner}: {owner_error}"
                )
            payload = dp_group.broadcast_tensor_dict(payload, src=owner)
            if not isinstance(payload, dict):
                raise RuntimeError("MiniMax H3 text payload broadcast failed")

            if dp_group.rank_in_group != owner:
                first_batch.extra[MINIMAX_H3_TEXT_EMBEDDINGS_EXTRA_KEY] = payload
                self._publish_native_text_conditioning(first_batch)
                first_result = first_batch
            results[first_index] = first_result

            for index, batch in equivalent[1:]:
                self.copy_deduplicated_outputs(first_result, batch)
                results[index] = batch

        return [result for result in results if result is not None]

    def _log_dp_choice(self, batch_size: int, world_size: int) -> None:
        if self._dp_choice_logged:
            return
        self._dp_choice_logged = True
        logger.info(

View on GitHub (pinned to 0132848349)

Solutions

  1. Verify all DP ranks take the same grouped-encode path (same batch grouping decisions) so collectives line up
  2. Check NCCL health / ranks' GPU state; rerun the job if the collective layer glitched
  3. Upgrade sglang so all ranks run identical broadcast_tensor_dict implementations
Defensive patterns

Strategy: retry

Validate before calling

assert all(r.will_group_encode for r in dp_ranks), "divergent control flow across DP ranks"

Try / catch

try:
    stage.run_grouped_requests(batches)
except RuntimeError as e:
    if "payload broadcast failed" in str(e):
        return retry_after_nccl_check()  # collective desync, often transient
    raise

Prevention

When it happens

Trigger: dp_group.broadcast_tensor_dict returns a non-dict (None, list, or truncated object) for MINIMAX_H3_TEXT_EMBEDDINGS_EXTRA_KEY — typically from a NCCL/collective desync, mismatched group membership, or tensor types that cannot traverse the tensor-dict protocol.

Common situations: DP ranks disagreeing on whether encode runs (divergent control flow), mismatched versions of the collective helper across ranks, or NCCL timeouts silently degrading the broadcast.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/e1c93dccdc3bb582. Report an issue: GitHub.