What causes "Invalid inference response from {self.infer_url}: {exc}" in docling-project/docling?

Inspect exc.__cause__ - it carries the specific decode/validation failure; fix that first

docling-project/docling · error · RuntimeError

Invalid inference response from {self.infer_url}: {exc}

Error message

Invalid inference response from {self.infer_url}: {exc}

What it means

Top-level guard in the HTTP infer path: whatever the response was (binary decode via _decode_binary_response or plain JSON validation) could not be turned into a KserveV2InferResponse, and the client wraps that failure with the infer URL. The original error is chained; this message tells you the problem is the response payload, not the request encoding.

Source

Thrown at docling/models/inference_engines/common/kserve_v2_http.py:402

        if _log.isEnabledFor(logging.DEBUG):
            _log.debug(
                "PIPELINE_PROFILING KServe infer http round-trip: batch_size=%d start=%.3f end=%.3f duration=%.3fs",
                _batch_size,
                _t_http_start,
                time.time(),
                time.monotonic() - _t_http_mono,
            )
            _t_deser_start = time.time()
            _t_deser_mono = time.monotonic()

        try:
            body = (
                _decode_binary_response(response)
                if self.use_binary_data
                else KserveV2InferResponse.model_validate(response.json())
            )
        except Exception as exc:
            raise RuntimeError(
                f"Invalid inference response from {self.infer_url}: {exc}"
            ) from exc

        decoded_outputs: Dict[str, np.ndarray] = {}
        header_len_text = response.headers.get(_INFERENCE_HEADER_CONTENT_LENGTH)
        raw_body = b""
        if self.use_binary_data and header_len_text is not None:
            raw_body = response.content[int(header_len_text) :]
        raw_offset = 0
        for output in body.outputs:
            binary_data_size = _parse_binary_data_size(output.parameters)
            if binary_data_size is None:
                decoded_outputs[output.name] = _decode_output_tensor(output)
                continue

            raw_end = raw_offset + binary_data_size
            if raw_end > len(raw_body):
                raise RuntimeError(

View on GitHub (pinned to 61d76f1ff3)

Solutions

Inspect exc.__cause__ - it carries the specific decode/validation failure; fix that first
Reproduce the POST with curl and examine status and body
Verify the infer URL and that the model is actually loaded and healthy on the server
If a gateway rewrites status codes or bodies, exclude the infer route from it

Defensive patterns

Strategy: try-catch

Try / catch

try:
    outputs = client.infer(inputs=inputs, output_names=[...])
except RuntimeError as e:
    if "Invalid inference response" in str(e):
        cause = e.__cause__  # real decode/validation failure
        raise RuntimeError(f"infer response unusable: {cause}") from cause
    raise

Prevention

Always inspect the chained cause; it pinpoints the actual response problem
Smoke-test one inference after any server or gateway change
Do not route model traffic through components that rewrite bodies or status codes

When it happens

Trigger: Server returned a JSON error object on 200; body is HTML from a gateway; binary response with a broken Inference-Header-Content-Length (see the chained _decode_binary_response errors); pydantic rejecting the outputs structure; empty body.

Common situations: Gateways that map upstream 5xx to 200 with an error JSON; wrong infer URL path; model crash mid-request returning a stack-trace body; version-skewed response schema.

Related errors

AI-assisted analysis of docling-project/docling@61d76f1ff3 (2026-08-14). Data as JSON: /api/errors/f90833c72aacbe43. Report an issue: GitHub.