immich-app/immich · error · RuntimeError

RKNN inference failed!

Error message

RKNN inference failed!

What it means

Raised by RknnSession.run() when rknnpool.get() returns None. In practice this means run_inference returned None — i.e. RKNNLite.inference() produced no output list — because get() only returns None for an empty queue, and run() always puts before it gets. It signals that the NPU executed but yielded no usable output (silent runtime failure), as opposed to the load/init failures in init_rknn which raise earlier.

Source

Thrown at machine-learning/immich_ml/sessions/rknn/__init__.py:67

        log.info(f"Loaded RKNN model from {model_path} with {self.tpe} threads.")

    def get_inputs(self) -> list[SessionNode]:
        return [RknnNode(name=k, shape=v) for k, v in input_output_mapping[self.model_type]["input"].items()]

    def get_outputs(self) -> list[SessionNode]:
        return [RknnNode(name=k, shape=v) for k, v in input_output_mapping[self.model_type]["output"].items()]

    def run(
        self,
        output_names: list[str] | None,
        input_feed: dict[str, NDArray[np.float32]] | dict[str, NDArray[np.int32]],
        run_options: Any = None,
    ) -> list[NDArray[np.float32]]:
        input_data: list[NDArray[np.float32]] = [np.ascontiguousarray(v) for v in input_feed.values()]
        self.rknnpool.put(input_data)
        res = self.rknnpool.get()
        if res is None:
            raise RuntimeError("RKNN inference failed!")
        return res


class RknnNode(NamedTuple):
    name: str
    shape: tuple[int, ...]


__all__ = ["RknnSession", "RknnNode", "is_available", "soc_name", "model_prefix"]

View on GitHub (pinned to 199723261c)

Solutions

1. Validate that every value in input_feed matches the shape in input_output_mapping[self.model_type]['input'] and is float32, contiguous, NCHW.
2. Lower settings.rknn_threads (the pool size) to reduce NPU contention and retry.
3. Check SoC temperature (`cat /sys/class/thermal/thermal_zone*/temp`) and cool down if throttling.
4. Confirm the RKNNLite runtime version matches the toolchain that produced the .rknn model; regenerate the model if unsure.
5. Wrap the failing call and inspect dmesg / NPU driver logs for runtime errors during the inference.

Example fix

# before
session.run(None, {'norm_tensor:0': img_uint8})   # wrong dtype/shape -> inference returns None
# ValueError-adjacent: 'RKNN inference failed!'

# after
import numpy as np
img = np.ascontiguousarray(img, dtype=np.float32)   # NCHW, 1x3x640x640 for detection
img = img.reshape(1, 3, 640, 640)
session.run(None, {'norm_tensor:0': img})

import numpy as np

def validate_rknn_inputs(
    input_feed: dict, expected: dict[str, tuple[int, ...]]
) -> None:
    for name, shape in expected.items():
        if name not in input_feed:
            raise ValueError(f"Missing RKNN input tensor {name!r}")
        arr = input_feed[name]
        if not isinstance(arr, np.ndarray):
            raise ValueError(f"{name}: expected ndarray, got {type(arr)}")
        if arr.dtype != np.float32:
            raise ValueError(f"{name}: expected float32, got {arr.dtype}")
        if tuple(arr.shape) != shape:
            raise ValueError(f"{name}: shape {arr.shape} != expected {shape}")

# call before session.run():
from immich_ml.sessions.rknn import input_output_mapping
validate_rknn_inputs(input_feed, input_output_mapping[session.model_type]['input'])

import numpy as np

def is_valid_rknn_feed(input_feed: dict, expected: dict[str, tuple[int, ...]]) -> bool:
    return all(
        name in input_feed
        and isinstance(input_feed[name], np.ndarray)
        and input_feed[name].dtype == np.float32
        and tuple(input_feed[name].shape) == shape
        for name, shape in expected.items()
    )

try:
    out = session.run(output_names, input_feed)
except RuntimeError as e:
    if 'RKNN inference failed' in str(e):
        log.warning("RKNN inference returned None for %s; reducing load and retrying once")
        validate_rknn_inputs(input_feed, expected_shapes)  # raises with a precise message
        out = session.run(output_names, input_feed)        # single retry
    else:
        raise

When it happens

Trigger: Calling RknnSession.run() after the pool was constructed successfully but the NPU fails mid-inference: input tensor shape/dtype mismatch with the model's expected input, NPU thermal throttling or driver reset, RKNNLite.inference returning None on internal error, or the underlying future producing None because inference() swallowed an exception.

Common situations: Feeding input_feed whose tensors do not match input_output_mapping shapes (e.g. wrong image size for detection: not 640x640); NPU overloaded by too many concurrent threads (rknn_threads too high); thermal throttling on the Rockchip SoC; RKNNLite runtime version mismatch with the .rknn model; memory pressure causing inference to fail silently.

Related errors

• rknn is not available!
• Failed to load RKNN model
• Failed to initialize RKNN runtime environment
• Unable to infer workflow event type from steps

AI-assisted analysis of immich-app/immich@199723261c (2026-08-12). Data as JSON: /api/errors/f5a28df1ce8510b5. Report an issue: GitHub.