{"record":{"id":"97bad6a1b2232d12","repo":"docling-project/docling","slug":"unsupported-kserve-v2-grpc-output-datatype-outpu","errorCode":null,"errorMessage":"Unsupported KServe v2 gRPC output datatype: {output_tensor.datatype}. Supported types: {list(KSERVE_V2_NUMPY_DATATYPES.keys())}","messagePattern":"Unsupported KServe v2 gRPC output datatype: (.+?)\\. Supported types: (.+?)","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"docling/models/inference_engines/common/kserve_v2_grpc.py","lineNumber":375,"sourceCode":"                time.monotonic() - _t_grpc_mono,\n            )\n            _t_deser_start = time.time()\n            _t_deser_mono = time.monotonic()\n\n        decoded_outputs: Dict[str, np.ndarray] = {}\n\n        if self.use_binary_data:\n            if len(response.raw_output_contents) != len(response.outputs):\n                raise RuntimeError(\n                    \"KServe v2 gRPC response did not include binary output payloads for all tensors. \"\n                    \"Set use_binary_data=False or ensure server supports binary_data outputs.\"\n                )\n            for output_tensor, raw_output in zip(\n                response.outputs, response.raw_output_contents\n            ):\n                np_dtype = KSERVE_V2_NUMPY_DATATYPES.get(output_tensor.datatype)\n                if np_dtype is None:\n                    raise RuntimeError(\n                        f\"Unsupported KServe v2 gRPC output datatype: {output_tensor.datatype}. \"\n                        f\"Supported types: {list(KSERVE_V2_NUMPY_DATATYPES.keys())}\"\n                    )\n                shape = tuple(int(dim) for dim in output_tensor.shape)\n\n                # Bytes decoding\n                # Special handling for BYTES datatype (variable-length strings)\n                if output_tensor.datatype == \"BYTES\":\n                    decoded_outputs[output_tensor.name] = decode_bytes_tensor(\n                        raw_output, shape\n                    )\n                else:\n                    array = np.frombuffer(raw_output, dtype=np_dtype)\n                    decoded_outputs[output_tensor.name] = array.reshape(shape)\n        else:\n            for output_tensor in response.outputs:\n                np_dtype = KSERVE_V2_NUMPY_DATATYPES.get(output_tensor.datatype)\n                if np_dtype is None:","sourceCodeStart":357,"sourceCodeEnd":393,"githubUrl":"https://github.com/docling-project/docling/blob/61d76f1ff3f8428065465889f7b4577da7df704c/docling/models/inference_engines/common/kserve_v2_grpc.py#L357-L393","documentation":"While decoding a binary gRPC response, an output tensor's datatype string (Triton-style, e.g. FP32, INT64, BYTES) is not present in KSERVE_V2_NUMPY_DATATYPES, so the client cannot pick a numpy dtype to np.frombuffer the raw payload. Supported names: BOOL, UINT8-64, INT8-64, FP16, FP32, FP64, BYTES.","triggerScenarios":"A model exported with an output datatype the mapping lacks - typically BF16, FP8, or a custom string; a server bug putting the shape or name into the datatype field; a non-Triton-compatible v2 server using different datatype names.","commonSituations":"Newer transformer/LLM models with BF16 outputs served by a Triton version that reports BF16; custom KServe predictors that invent datatype labels; server upgrades introducing new datatypes the client predates.","solutions":["Reconfigure/re-export the model so its outputs use one of the supported datatypes (most commonly cast outputs to FP32 or FP64)","Check the server's model config / metadata endpoint to see the exact datatype string it advertises","If BF16 support is genuinely needed, add it to KSERVE_V2_NUMPY_DATATYPES (numpy has no native bfloat16, so map to a custom decode) via a PR","As a stopgap, disable use_binary_data - though the same datatype will still fail in the non-binary path with the same message"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"from docling.models.inference_engines.common.kserve_v2_types import KSERVE_V2_NUMPY_DATATYPES\n\ndef server_output_types_supported(metadata) -> bool:\n    return all(\n        spec.datatype in KSERVE_V2_NUMPY_DATATYPES\n        for spec in metadata.outputs\n    )","typeGuard":null,"tryCatchPattern":"try:\n    outputs = engine.infer(inputs=inputs)\nexcept RuntimeError as e:\n    if \"Unsupported KServe v2 gRPC output datatype\" in str(e):\n        # fix model config to emit FP32, then retry; do not ignore\n        raise\n    raise","preventionTips":["Check the model metadata endpoint's output datatypes before deploying a model","Cast model outputs to FP32/FP64 in the model repo config when in doubt","Avoid serving BF16/FP16-output models to clients with fixed datatype tables"],"tags":["grpc","kserve","dtype","binary-data","model-config"],"backgroundTag":null,"analyzedSha":"61d76f1ff3f8428065465889f7b4577da7df704c","analyzedAt":"2026-08-14T23:53:18.727Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}