{"record":{"id":"c039bd8cc76cd2aa","repo":"sgl-project/sglang","slug":"cannot-split-glm-image-ar-output-for-sequential-in","errorCode":null,"errorMessage":"Cannot split GLM-Image AR output for sequential inference: expected {output_count} token rows, got {actual_count}.","messagePattern":"Cannot split GLM-Image AR output for sequential inference: expected (.+?) token rows, got (.+?)\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/multimodal_gen/runtime/pipelines_core/stages/model_specific_stages/glm_image.py","lineNumber":664,"sourceCode":"        self, batch: Req, server_args: ServerArgs\n    ) -> Iterator[Req]:\n        if not server_args.pipeline_config.supports_sequential_multi_output_inference():\n            return iter((batch,))\n\n        output_count = _num_outputs_per_prompt(batch)\n        if output_count == 1:\n            return iter((batch,))\n\n        prior_token_ids = batch.prior_token_id\n        if not isinstance(prior_token_ids, torch.Tensor) or (\n            prior_token_ids.shape[0] != output_count\n        ):\n            actual_count = (\n                prior_token_ids.shape[0]\n                if isinstance(prior_token_ids, torch.Tensor)\n                else type(prior_token_ids).__name__\n            )\n            raise RuntimeError(\n                \"Cannot split GLM-Image AR output for sequential inference: \"\n                f\"expected {output_count} token rows, got {actual_count}.\"\n            )\n\n        return map(\n            lambda output_index: self._make_sequential_request(\n                batch, prior_token_ids, output_index\n            ),\n            range(output_count),\n        )\n\n    @staticmethod\n    def _make_sequential_request(\n        batch: Req, prior_token_ids: torch.Tensor, output_index: int\n    ) -> Req:\n        output_req = copy(batch)\n        output_req.sampling_params = copy(batch.sampling_params)\n        output_req.extra = dict(batch.extra)","sourceCodeStart":646,"sourceCodeEnd":682,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/multimodal_gen/runtime/pipelines_core/stages/model_specific_stages/glm_image.py#L646-L682","documentation":"For sequential (non-batched) inference, the GLM-Image stage must split the AR output into output_count requests, one per original prompt. The returned prior_token_ids tensor did not have exactly output_count rows (e.g. the external AR server collapsed or dropped rows), so the split is impossible.","triggerScenarios":"Calling iter_sequential_requests after a batched AR call where prior_token_ids.shape[0] != number of requests; the external server returned merged or fewer token rows than prompts; prior_token_ids is not a tensor at all (then the type name is reported).","commonSituations":"External AR server returning a concatenated or truncated batch; a retry path feeding a partially-consumed tensor back in; mismatch between the number of prompts used to build the batch and the count passed to iter_sequential_requests.","solutions":["Verify prior_token_ids.shape[0] equals the number of prompts fed to the AR batch call","Inspect the external AR server response for dropped/merged outputs and check its logs","Re-run with batch size 1 to confirm the pipeline works, then bisect the batch size that breaks it","Align server/client SGLang versions so batch output ordering and count are preserved"],"exampleFix":"# before\noutputs = list(stage.iter_sequential_requests(prior_token_ids, output_count=4, ...))\n\n# after\nassert prior_token_ids.shape[0] == output_count, prior_token_ids.shape\noutputs = list(stage.iter_sequential_requests(prior_token_ids, output_count=4, ...))","handlingStrategy":"validation","validationCode":"assert isinstance(prior_token_ids, torch.Tensor)\nassert prior_token_ids.shape[0] == output_count, (\n    prior_token_ids.shape, output_count\n)","typeGuard":"def is_valid_prior_batch(t, output_count) -> bool:\n    return isinstance(t, torch.Tensor) and t.ndim >= 1 and t.shape[0] == output_count","tryCatchPattern":"except RuntimeError as e:\n    if \"Cannot split\" in str(e):\n        regenerate_batch_with_size_1()  # fall back to sequential single calls\n    else:\n        raise","preventionTips":["Always assert row count equals request count before splitting","Run batch size 1 as a smoke test when the external server changes","Log shapes at every handoff between batch and sequential stages"],"tags":["glm-image","sequential-inference","batch-split","shape-mismatch","runtimeerror"],"backgroundTag":"tensor-shape-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}