{"record":{"id":"8343d4ddce4a56b6","repo":"docling-project/docling","slug":"kserve-v2-output-batch-size-mismatch-for-labels-e","errorCode":null,"errorMessage":"KServe v2 output batch size mismatch for labels: expected {len(input_batch)}, got {len(labels_batch)}","messagePattern":"KServe v2 output batch size mismatch for labels: expected (.+?), got (.+?)","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"docling/models/inference_engines/object_detection/api_kserve_v2_engine.py","lineNumber":229,"sourceCode":"                self._output_boxes_name,\n                self._output_scores_name,\n            ],\n            request_parameters=self.options.request_parameters,\n        )\n        try:\n            labels_batch = outputs[self._output_labels_name]\n            boxes_batch = outputs[self._output_boxes_name]\n            scores_batch = outputs[self._output_scores_name]\n        except KeyError as exc:\n            raise RuntimeError(\n                \"Missing one or more expected KServe v2 outputs: \"\n                f\"{self._output_labels_name}, \"\n                f\"{self._output_boxes_name}, \"\n                f\"{self._output_scores_name}\"\n            ) from exc\n\n        if len(labels_batch) != len(input_batch):\n            raise RuntimeError(\n                \"KServe v2 output batch size mismatch for labels: \"\n                f\"expected {len(input_batch)}, got {len(labels_batch)}\"\n            )\n\n        batch_outputs: List[ObjectDetectionEngineOutput] = []\n        for idx, input_item in enumerate(input_batch):\n            batch_outputs.append(\n                self._build_output(\n                    input_item=input_item,\n                    labels=labels_batch[idx],\n                    scores=scores_batch[idx],\n                    boxes=boxes_batch[idx],\n                    apply_score_threshold=True,\n                )\n            )\n\n        return batch_outputs\n","sourceCodeStart":211,"sourceCodeEnd":247,"githubUrl":"https://github.com/docling-project/docling/blob/61d76f1ff3f8428065465889f7b4577da7df704c/docling/models/inference_engines/object_detection/api_kserve_v2_engine.py#L211-L247","documentation":"The KServe v2 engine validates that the labels output batch has exactly one entry per input image. If len(labels_batch) != len(input_batch), it raises RuntimeError, because it cannot map detections back to inputs and silently mis-assigning boxes would corrupt results.","triggerScenarios":"Sending a batch of N images to a KServe endpoint that returns fewer/more label entries — e.g. a server configured with max batch size 1 that processes only the first image, or a dynamic-batching layer that aggregates requests.","commonSituations":"Batch size exceeding the server's configured max_batch_size; KServe transformer components that change batch shape; version differences in how the served RT-DETR export handles batching.","solutions":["Reduce the batch size (send images one at a time or in smaller batches) to stay within the server's batching capability.","Raise max_batch_size / max_queue_delay on the KServe/Triton server so full batches are returned.","Verify with a direct KServe client that sending N inputs yields N label arrays; if not, the server config is the problem."],"exampleFix":"# before\noutputs = engine.predict_batch(all_images)  # e.g. 32 images, server caps at 8\n\n# after\noutputs = []\nfor chunk in chunks(all_images, 8):\n    outputs.extend(engine.predict_batch(chunk))","handlingStrategy":"fallback","validationCode":"MAX_BATCH = 8  # match server's configured max batch size\nchunks = [input_batch[i:i + MAX_BATCH] for i in range(0, len(input_batch), MAX_BATCH)]","typeGuard":null,"tryCatchPattern":"try:\n    outputs = engine.predict_batch(input_batch)\nexcept RuntimeError as e:\n    if \"batch size mismatch\" in str(e):\n        outputs = [o for b in chunks(input_batch, 1) for o in engine.predict_batch(b)]  # fall back to batch=1\n    else:\n        raise","preventionTips":["Know the KServe/Triton server's max_batch_size and chunk client batches below it.","Include multi-image batches in integration tests against the real endpoint.","Verify each infer response has one entry per request before shipping config."],"tags":["kserve","batching","inference","validation"],"backgroundTag":null,"analyzedSha":"61d76f1ff3f8428065465889f7b4577da7df704c","analyzedAt":"2026-08-14T23:53:18.727Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}