sgl-project/sglang · error · RuntimeError

MiniMax H3 text encode failed on rank {owner}: {owner_error}

Error message

MiniMax H3 text encode failed on rank {owner}: {owner_error}

What it means

When any rank in the data-parallel group fails text encode, the error string is broadcast to all ranks; ranks without the original exception raise this RuntimeError, attributing the failure to the owner rank. It is a propagation wrapper — the root cause is the owner_error text from the failing rank.

Source

Thrown at python/sglang/multimodal_gen/runtime/pipelines_core/stages/model_specific_stages/minimax_h3/stages/text_encoding.py:179

                        first_batch.extra.pop(
                            _MINIMAX_H3_SINGLE_COPY_TEXT_ENCODE_EXTRA_KEY, None
                        )
                    payload = first_result.extra.get(
                        MINIMAX_H3_TEXT_EMBEDDINGS_EXTRA_KEY
                    )
                    if not isinstance(payload, dict):
                        raise ValueError(
                            "MiniMax H3 text encode produced no native payload"
                        )
                except Exception as exc:
                    owner_exception = exc
                    owner_error = f"{type(exc).__name__}: {exc}"

            owner_error = dp_group.broadcast_object(owner_error, src=owner)
            if owner_error is not None:
                if owner_exception is not None:
                    raise owner_exception
                raise RuntimeError(
                    f"MiniMax H3 text encode failed on rank {owner}: {owner_error}"
                )
            payload = dp_group.broadcast_tensor_dict(payload, src=owner)
            if not isinstance(payload, dict):
                raise RuntimeError("MiniMax H3 text payload broadcast failed")

            if dp_group.rank_in_group != owner:
                first_batch.extra[MINIMAX_H3_TEXT_EMBEDDINGS_EXTRA_KEY] = payload
                self._publish_native_text_conditioning(first_batch)
                first_result = first_batch
            results[first_index] = first_result

            for index, batch in equivalent[1:]:
                self.copy_deduplicated_outputs(first_result, batch)
                results[index] = batch

        return [result for result in results if result is not None]

View on GitHub (pinned to 0132848349)

Solutions

1. Read the message text: the real exception is the owner_error suffix from the owner rank — debug that rank's log
2. Fix the underlying owner-rank failure (OOM, input shape, component mismatch)
3. If flaky/infra-related, retry the request after the owner rank recovers

try:
    stage.run_grouped_requests(batches)
except RuntimeError as e:
    if "text encode failed on rank" in str(e):
        log_owner_rank_failure(str(e)); return retryable_error(e)
    raise

When it happens

Trigger: The owner rank's encode raised (owner_error non-None after broadcast_object) and non-owner ranks hit this raise in run_grouped_requests, since owner_exception only exists on the failing rank.

Common situations: OOM or CUDA error on one DP rank, shape mismatch in the encoder on the owner, or any exception inside the owner's forward that the group then synchronizes on.

Related errors

• MiniMax H3 text encode produced no native payload
• prompt must be non-empty
• MiniMax H3 text payload must contain positive.hidden_states
• MiniMax H3 text payload positive.text_len must match the hid
• MiniMaxH3TextEncodingStage direct encode requires a text_enc

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/d827fff1fb5b9d22. Report an issue: GitHub.