docling-project/docling · error · RuntimeError

Unsupported KServe v2 gRPC output datatype: {output_tensor.d

Error message

Unsupported KServe v2 gRPC output datatype: {output_tensor.datatype}. Supported types: {list(KSERVE_V2_NUMPY_DATATYPES.keys())}

What it means

While decoding a binary gRPC response, an output tensor's datatype string (Triton-style, e.g. FP32, INT64, BYTES) is not present in KSERVE_V2_NUMPY_DATATYPES, so the client cannot pick a numpy dtype to np.frombuffer the raw payload. Supported names: BOOL, UINT8-64, INT8-64, FP16, FP32, FP64, BYTES.

Source

Thrown at docling/models/inference_engines/common/kserve_v2_grpc.py:375

                time.monotonic() - _t_grpc_mono,
            )
            _t_deser_start = time.time()
            _t_deser_mono = time.monotonic()

        decoded_outputs: Dict[str, np.ndarray] = {}

        if self.use_binary_data:
            if len(response.raw_output_contents) != len(response.outputs):
                raise RuntimeError(
                    "KServe v2 gRPC response did not include binary output payloads for all tensors. "
                    "Set use_binary_data=False or ensure server supports binary_data outputs."
                )
            for output_tensor, raw_output in zip(
                response.outputs, response.raw_output_contents
            ):
                np_dtype = KSERVE_V2_NUMPY_DATATYPES.get(output_tensor.datatype)
                if np_dtype is None:
                    raise RuntimeError(
                        f"Unsupported KServe v2 gRPC output datatype: {output_tensor.datatype}. "
                        f"Supported types: {list(KSERVE_V2_NUMPY_DATATYPES.keys())}"
                    )
                shape = tuple(int(dim) for dim in output_tensor.shape)

                # Bytes decoding
                # Special handling for BYTES datatype (variable-length strings)
                if output_tensor.datatype == "BYTES":
                    decoded_outputs[output_tensor.name] = decode_bytes_tensor(
                        raw_output, shape
                    )
                else:
                    array = np.frombuffer(raw_output, dtype=np_dtype)
                    decoded_outputs[output_tensor.name] = array.reshape(shape)
        else:
            for output_tensor in response.outputs:
                np_dtype = KSERVE_V2_NUMPY_DATATYPES.get(output_tensor.datatype)
                if np_dtype is None:

View on GitHub (pinned to 61d76f1ff3)

Solutions

Reconfigure/re-export the model so its outputs use one of the supported datatypes (most commonly cast outputs to FP32 or FP64)
Check the server's model config / metadata endpoint to see the exact datatype string it advertises
If BF16 support is genuinely needed, add it to KSERVE_V2_NUMPY_DATATYPES (numpy has no native bfloat16, so map to a custom decode) via a PR
As a stopgap, disable use_binary_data - though the same datatype will still fail in the non-binary path with the same message

Defensive patterns

Strategy: validation

Validate before calling

from docling.models.inference_engines.common.kserve_v2_types import KSERVE_V2_NUMPY_DATATYPES

def server_output_types_supported(metadata) -> bool:
    return all(
        spec.datatype in KSERVE_V2_NUMPY_DATATYPES
        for spec in metadata.outputs
    )

Try / catch

try:
    outputs = engine.infer(inputs=inputs)
except RuntimeError as e:
    if "Unsupported KServe v2 gRPC output datatype" in str(e):
        # fix model config to emit FP32, then retry; do not ignore
        raise
    raise

Prevention

Check the model metadata endpoint's output datatypes before deploying a model
Cast model outputs to FP32/FP64 in the model repo config when in doubt
Avoid serving BF16/FP16-output models to clients with fixed datatype tables

When it happens

Trigger: A model exported with an output datatype the mapping lacks - typically BF16, FP8, or a custom string; a server bug putting the shape or name into the datatype field; a non-Triton-compatible v2 server using different datatype names.

Common situations: Newer transformer/LLM models with BF16 outputs served by a Triton version that reports BF16; custom KServe predictors that invent datatype labels; server upgrades introducing new datatypes the client predates.

Related errors

AI-assisted analysis of docling-project/docling@61d76f1ff3 (2026-08-14). Data as JSON: /api/errors/97bad6a1b2232d12. Report an issue: GitHub.