docling-project/docling · error · RuntimeError

gRPC infer call failed for model {self.model_name}: {exc}

Error message

gRPC infer call failed for model {self.model_name}: {exc}

What it means

The gRPC ModelInfer RPC raised grpc.RpcError and this RuntimeError is the chained wrapper that adds the model name. The underlying gRPC status code and details are in `exc` and the original error is accessible via __cause__. Typical codes: UNAVAILABLE (server down/unreachable), DEADLINE_EXCEEDED (timeout), NOT_FOUND (wrong model_name), PERMISSION_DENIED / UNAUTHENTICATED (missing credentials), RESOURCE_EXHAUSTED (server OOM or flow control).

Source

Thrown at docling/models/inference_engines/common/kserve_v2_grpc.py:347

        if _log.isEnabledFor(logging.DEBUG):
            _log.debug(
                "PIPELINE_PROFILING KServe gRPC infer serialization: batch_size=%d start=%.3f end=%.3f duration=%.3fs",
                _batch_size,
                _t_ser_start,
                time.time(),
                time.monotonic() - _t_ser_mono,
            )
            _t_grpc_start = time.time()
            _t_grpc_mono = time.monotonic()

        try:
            response = self._stub.ModelInfer(
                request,
                timeout=self.timeout,
                metadata=self._grpc_metadata,
            )
        except grpc.RpcError as exc:
            raise RuntimeError(
                f"gRPC infer call failed for model {self.model_name}: {exc}"
            ) from exc

        if _log.isEnabledFor(logging.DEBUG):
            _log.debug(
                "PIPELINE_PROFILING KServe gRPC infer round-trip: batch_size=%d start=%.3f end=%.3f duration=%.3fs",
                _batch_size,
                _t_grpc_start,
                time.time(),
                time.monotonic() - _t_grpc_mono,
            )
            _t_deser_start = time.time()
            _t_deser_mono = time.monotonic()

        decoded_outputs: Dict[str, np.ndarray] = {}

        if self.use_binary_data:
            if len(response.raw_output_contents) != len(response.outputs):

View on GitHub (pinned to 61d76f1ff3)

Solutions

1. Read the nested grpc status: catch RuntimeError and inspect exc.__cause__.code() and .details() to get the real cause
2. For DEADLINE_EXCEEDED, raise the timeout value configured on the gRPC engine options
3. For UNAVAILABLE, verify the gRPC endpoint URL/port with grpcurl and check the server pod/process is up
4. For NOT_FOUND, confirm the model is loaded and that model_name/model_version match the server's model registry
5. For UNAUTHENTICATED/PERMISSION_DENIED, refresh or correct the metadata used for auth

Example fix

// before
outputs = engine.infer(...)  # RuntimeError: gRPC infer call failed for model X: <AuroraError...>

// after
try:
    outputs = engine.infer(...)
except RuntimeError as e:
    rpc = e.__cause__
    if rpc is not None and rpc.code() == grpc.StatusCode.DEADLINE_EXCEEDED:
        ...  # retry with smaller batch

import grpc

try:
    outputs = engine.infer(inputs=inputs)
except RuntimeError as e:
    cause = e.__cause__
    if isinstance(cause, grpc.RpcError):
        code = cause.code()
        if code in (grpc.StatusCode.UNAVAILABLE, grpc.StatusCode.DEADLINE_EXCEEDED):
            outputs = engine.infer(inputs=inputs)  # or backoff-retry via tenacity
        else:
            raise
    else:
        raise

When it happens

Trigger: Wrong service URL/port for the gRPC endpoint; model_name not loaded on the KServe/Triton server; self.timeout too small for a big batch; TLS mismatch (client plaintext hitting TLS port); missing or expired auth metadata in _grpc_metadata; server restarting when the call lands.

Common situations: Cluster service renamed or port changed after redeploy; model not yet loaded when the first inference arrives; large batch inference exceeding a 5s default timeout; Istio/service-mesh sidecar rejecting the long-lived HTTP/2 stream; token in metadata expired.

Related errors

• gRPC metadata call failed for model {self.model_name}: {exc}
• Artifact download failed: {exc}
• URL must contain a valid hostname
• Cannot resolve hostname: {hostname}
• Access to restricted IP address not allowed: {ip}

AI-assisted analysis of docling-project/docling@61d76f1ff3 (2026-08-14). Data as JSON: /api/errors/05ad4550f6cb45ec. Report an issue: GitHub.