sgl-project/sglang · error · RuntimeError

GLM-Image AR batch returned an unexpected response: expected

Error message

GLM-Image AR batch returned an unexpected response: expected {len(prompts)} outputs, got {len(data) if isinstance(data, list) else type(data).__name__}.

What it means

When batching prior-token generation against the external SGLang AR endpoint, the response payload was expected to be a JSON list with exactly one output entry per input prompt. The response was either not a list or had a different length than the batch, indicating a malformed, truncated, or misrouted server response.

Source

Thrown at python/sglang/multimodal_gen/runtime/pipelines_core/stages/model_specific_stages/glm_image.py:563

                    image_grid_thw=image_grid_thw,
                    is_text_to_image=True,
                )
            )
            input_ids.append(inputs["input_ids"][0].tolist())
            image_data.append([{"image_grid_thw": image_grid_thw.tolist()}])
            sampling_params.append(
                self._external_ar_sampling_params(max_new_tokens, seed)
            )
            generation_shapes.append((large_image_offset, token_h, token_w))

        payload = {
            "input_ids": input_ids,
            "image_data": image_data,
            "sampling_params": sampling_params,
        }
        data = self._request_external_ar(payload, server_args)
        if not isinstance(data, list) or len(data) != len(prompts):
            raise RuntimeError(
                "GLM-Image AR batch returned an unexpected response: "
                f"expected {len(prompts)} outputs, got "
                f"{len(data) if isinstance(data, list) else type(data).__name__}."
            )

        prior_token_ids = []
        usages = []
        for item, generation_shape in zip(data, generation_shapes, strict=True):
            prior_token_ids.append(
                self._extract_prior_token_ids(
                    item.get("output_ids"), generation_shape, device
                )
            )
            usages.append(_extract_srt_usage(item.get("meta_info")))
        return prior_token_ids, usages

    def run_grouped_requests(
        self,

View on GitHub (pinned to 0132848349)

Solutions

1. Log/inspect the raw response body from _request_external_ar to see whether it is an error dict or partial list
2. Check the external AR server logs for the failing batch (OOM, context-length, dropped requests)
3. Verify the server runs a compatible SGLang version whose batch AR endpoint returns one entry per prompt
4. Retry with a smaller batch size to rule out server-side truncation
5. Ensure len(image_data) and len(input_ids) match len(prompts) before sending

Example fix

# before
priors = stage.generate_prior_tokens_batch(prompts=8 * [p], ...)

# after
data = stage._request_external_ar(payload, server_args)
assert isinstance(data, list) and len(data) == len(prompts), data  # surface server error early
priors = stage.generate_prior_tokens_batch(prompts=8 * [p], ...)

assert len(prompts) == len(input_ids) == len(image_data), "batch inputs misaligned"

for attempt in range(3):
    try:
        return stage.generate_prior_tokens_batch(prompts, ...)
    except RuntimeError as e:
        if "unexpected response" in str(e) and attempt < 2:
            time.sleep(2 ** attempt)
            continue
        raise

When it happens

Trigger: Calling generate_prior_tokens_batch with N prompts while server_args.srt_encoder_url points at an AR server; the endpoint returns a dict (e.g. an error object like {"error": ...}) or a list whose length differs from the number of prompts sent.

Common situations: The external server hit an OOM/limit and returned a partial batch or an error JSON; version mismatch between client payload schema and server response schema; a proxy/load balancer returning an HTML or dict error body; accidental misalignment between prompts and image_data lists built by the caller.

Related errors

• GLM-Image AR returned too few output_ids: got {actual_output
• Cannot split GLM-Image AR output for sequential inference: e
• The number of initial states is expected to be equal to the
• Mismatched batch sizes: mixed_qkv.shape[0]={B}, a.shape[0]={
• Mismatched batch sizes: mixed_qkv.shape[0]={B}, a.shape[0]={

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/1b814e733c1c53c6. Report an issue: GitHub.