docling-project/docling · error · RuntimeError

KServe v2 output batch size mismatch for labels: expected {l

Error message

KServe v2 output batch size mismatch for labels: expected {len(input_batch)}, got {len(labels_batch)}

What it means

The KServe v2 engine validates that the labels output batch has exactly one entry per input image. If len(labels_batch) != len(input_batch), it raises RuntimeError, because it cannot map detections back to inputs and silently mis-assigning boxes would corrupt results.

Source

Thrown at docling/models/inference_engines/object_detection/api_kserve_v2_engine.py:229

                self._output_boxes_name,
                self._output_scores_name,
            ],
            request_parameters=self.options.request_parameters,
        )
        try:
            labels_batch = outputs[self._output_labels_name]
            boxes_batch = outputs[self._output_boxes_name]
            scores_batch = outputs[self._output_scores_name]
        except KeyError as exc:
            raise RuntimeError(
                "Missing one or more expected KServe v2 outputs: "
                f"{self._output_labels_name}, "
                f"{self._output_boxes_name}, "
                f"{self._output_scores_name}"
            ) from exc

        if len(labels_batch) != len(input_batch):
            raise RuntimeError(
                "KServe v2 output batch size mismatch for labels: "
                f"expected {len(input_batch)}, got {len(labels_batch)}"
            )

        batch_outputs: List[ObjectDetectionEngineOutput] = []
        for idx, input_item in enumerate(input_batch):
            batch_outputs.append(
                self._build_output(
                    input_item=input_item,
                    labels=labels_batch[idx],
                    scores=scores_batch[idx],
                    boxes=boxes_batch[idx],
                    apply_score_threshold=True,
                )
            )

        return batch_outputs

View on GitHub (pinned to 61d76f1ff3)

Solutions

1. Reduce the batch size (send images one at a time or in smaller batches) to stay within the server's batching capability.
2. Raise max_batch_size / max_queue_delay on the KServe/Triton server so full batches are returned.
3. Verify with a direct KServe client that sending N inputs yields N label arrays; if not, the server config is the problem.

Example fix

# before
outputs = engine.predict_batch(all_images)  # e.g. 32 images, server caps at 8

# after
outputs = []
for chunk in chunks(all_images, 8):
    outputs.extend(engine.predict_batch(chunk))

MAX_BATCH = 8  # match server's configured max batch size
chunks = [input_batch[i:i + MAX_BATCH] for i in range(0, len(input_batch), MAX_BATCH)]

try:
    outputs = engine.predict_batch(input_batch)
except RuntimeError as e:
    if "batch size mismatch" in str(e):
        outputs = [o for b in chunks(input_batch, 1) for o in engine.predict_batch(b)]  # fall back to batch=1
    else:
        raise

When it happens

Trigger: Sending a batch of N images to a KServe endpoint that returns fewer/more label entries — e.g. a server configured with max batch size 1 that processes only the first image, or a dynamic-batching layer that aggregates requests.

Common situations: Batch size exceeding the server's configured max_batch_size; KServe transformer components that change batch shape; version differences in how the served RT-DETR export handles batching.

Related errors

• Examples batch length must match messages batch length
• Number of templates ({len(prompt)}) must match number of ima
• Examples batch length must match messages batch length
• Number of prompts ({len(prompt)}) must match number of image
• Unsupported KServe request parameter integer range for gRPC:

AI-assisted analysis of docling-project/docling@61d76f1ff3 (2026-08-14). Data as JSON: /api/errors/8343d4ddce4a56b6. Report an issue: GitHub.