sgl-project/sglang · error · RuntimeError

Cannot split GLM-Image AR output for sequential inference: e

Error message

Cannot split GLM-Image AR output for sequential inference: expected {output_count} token rows, got {actual_count}.

What it means

For sequential (non-batched) inference, the GLM-Image stage must split the AR output into output_count requests, one per original prompt. The returned prior_token_ids tensor did not have exactly output_count rows (e.g. the external AR server collapsed or dropped rows), so the split is impossible.

Source

Thrown at python/sglang/multimodal_gen/runtime/pipelines_core/stages/model_specific_stages/glm_image.py:664

        self, batch: Req, server_args: ServerArgs
    ) -> Iterator[Req]:
        if not server_args.pipeline_config.supports_sequential_multi_output_inference():
            return iter((batch,))

        output_count = _num_outputs_per_prompt(batch)
        if output_count == 1:
            return iter((batch,))

        prior_token_ids = batch.prior_token_id
        if not isinstance(prior_token_ids, torch.Tensor) or (
            prior_token_ids.shape[0] != output_count
        ):
            actual_count = (
                prior_token_ids.shape[0]
                if isinstance(prior_token_ids, torch.Tensor)
                else type(prior_token_ids).__name__
            )
            raise RuntimeError(
                "Cannot split GLM-Image AR output for sequential inference: "
                f"expected {output_count} token rows, got {actual_count}."
            )

        return map(
            lambda output_index: self._make_sequential_request(
                batch, prior_token_ids, output_index
            ),
            range(output_count),
        )

    @staticmethod
    def _make_sequential_request(
        batch: Req, prior_token_ids: torch.Tensor, output_index: int
    ) -> Req:
        output_req = copy(batch)
        output_req.sampling_params = copy(batch.sampling_params)
        output_req.extra = dict(batch.extra)

View on GitHub (pinned to 0132848349)

Solutions

Verify prior_token_ids.shape[0] equals the number of prompts fed to the AR batch call
Inspect the external AR server response for dropped/merged outputs and check its logs
Re-run with batch size 1 to confirm the pipeline works, then bisect the batch size that breaks it
Align server/client SGLang versions so batch output ordering and count are preserved

Example fix

# before
outputs = list(stage.iter_sequential_requests(prior_token_ids, output_count=4, ...))

# after
assert prior_token_ids.shape[0] == output_count, prior_token_ids.shape
outputs = list(stage.iter_sequential_requests(prior_token_ids, output_count=4, ...))

Defensive patterns

Strategy: validation

Validate before calling

assert isinstance(prior_token_ids, torch.Tensor)
assert prior_token_ids.shape[0] == output_count, (
    prior_token_ids.shape, output_count
)

Type guard

def is_valid_prior_batch(t, output_count) -> bool:
    return isinstance(t, torch.Tensor) and t.ndim >= 1 and t.shape[0] == output_count

Try / catch

except RuntimeError as e:
    if "Cannot split" in str(e):
        regenerate_batch_with_size_1()  # fall back to sequential single calls
    else:
        raise

Prevention

Always assert row count equals request count before splitting
Run batch size 1 as a smoke test when the external server changes
Log shapes at every handoff between batch and sequential stages

When it happens

Trigger: Calling iter_sequential_requests after a batched AR call where prior_token_ids.shape[0] != number of requests; the external server returned merged or fewer token rows than prompts; prior_token_ids is not a tensor at all (then the type name is reported).

Common situations: External AR server returning a concatenated or truncated batch; a retry path feeding a partially-consumed tensor back in; mismatch between the number of prompts used to build the batch and the count passed to iter_sequential_requests.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/c039bd8cc76cd2aa. Report an issue: GitHub.