{"record":{"id":"f5a28df1ce8510b5","repo":"immich-app/immich","slug":"rknn-inference-failed","errorCode":null,"errorMessage":"RKNN inference failed!","messagePattern":"RKNN inference failed!","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"machine-learning/immich_ml/sessions/rknn/__init__.py","lineNumber":67,"sourceCode":"        log.info(f\"Loaded RKNN model from {model_path} with {self.tpe} threads.\")\n\n    def get_inputs(self) -> list[SessionNode]:\n        return [RknnNode(name=k, shape=v) for k, v in input_output_mapping[self.model_type][\"input\"].items()]\n\n    def get_outputs(self) -> list[SessionNode]:\n        return [RknnNode(name=k, shape=v) for k, v in input_output_mapping[self.model_type][\"output\"].items()]\n\n    def run(\n        self,\n        output_names: list[str] | None,\n        input_feed: dict[str, NDArray[np.float32]] | dict[str, NDArray[np.int32]],\n        run_options: Any = None,\n    ) -> list[NDArray[np.float32]]:\n        input_data: list[NDArray[np.float32]] = [np.ascontiguousarray(v) for v in input_feed.values()]\n        self.rknnpool.put(input_data)\n        res = self.rknnpool.get()\n        if res is None:\n            raise RuntimeError(\"RKNN inference failed!\")\n        return res\n\n\nclass RknnNode(NamedTuple):\n    name: str\n    shape: tuple[int, ...]\n\n\n__all__ = [\"RknnSession\", \"RknnNode\", \"is_available\", \"soc_name\", \"model_prefix\"]\n","sourceCodeStart":49,"sourceCodeEnd":77,"githubUrl":"https://github.com/immich-app/immich/blob/199723261c6ffa897fec8ccdaea6359e39c37cc3/machine-learning/immich_ml/sessions/rknn/__init__.py#L49-L77","documentation":"Raised by RknnSession.run() when rknnpool.get() returns None. In practice this means run_inference returned None — i.e. RKNNLite.inference() produced no output list — because get() only returns None for an empty queue, and run() always puts before it gets. It signals that the NPU executed but yielded no usable output (silent runtime failure), as opposed to the load/init failures in init_rknn which raise earlier.","triggerScenarios":"Calling RknnSession.run() after the pool was constructed successfully but the NPU fails mid-inference: input tensor shape/dtype mismatch with the model's expected input, NPU thermal throttling or driver reset, RKNNLite.inference returning None on internal error, or the underlying future producing None because inference() swallowed an exception.","commonSituations":"Feeding input_feed whose tensors do not match input_output_mapping shapes (e.g. wrong image size for detection: not 640x640); NPU overloaded by too many concurrent threads (rknn_threads too high); thermal throttling on the Rockchip SoC; RKNNLite runtime version mismatch with the .rknn model; memory pressure causing inference to fail silently.","solutions":["Validate that every value in input_feed matches the shape in input_output_mapping[self.model_type]['input'] and is float32, contiguous, NCHW.","Lower settings.rknn_threads (the pool size) to reduce NPU contention and retry.","Check SoC temperature (`cat /sys/class/thermal/thermal_zone*/temp`) and cool down if throttling.","Confirm the RKNNLite runtime version matches the toolchain that produced the .rknn model; regenerate the model if unsure.","Wrap the failing call and inspect dmesg / NPU driver logs for runtime errors during the inference."],"exampleFix":"# before\nsession.run(None, {'norm_tensor:0': img_uint8})   # wrong dtype/shape -> inference returns None\n# ValueError-adjacent: 'RKNN inference failed!'\n\n# after\nimport numpy as np\nimg = np.ascontiguousarray(img, dtype=np.float32)   # NCHW, 1x3x640x640 for detection\nimg = img.reshape(1, 3, 640, 640)\nsession.run(None, {'norm_tensor:0': img})","handlingStrategy":"validation","validationCode":"import numpy as np\n\ndef validate_rknn_inputs(\n    input_feed: dict, expected: dict[str, tuple[int, ...]]\n) -> None:\n    for name, shape in expected.items():\n        if name not in input_feed:\n            raise ValueError(f\"Missing RKNN input tensor {name!r}\")\n        arr = input_feed[name]\n        if not isinstance(arr, np.ndarray):\n            raise ValueError(f\"{name}: expected ndarray, got {type(arr)}\")\n        if arr.dtype != np.float32:\n            raise ValueError(f\"{name}: expected float32, got {arr.dtype}\")\n        if tuple(arr.shape) != shape:\n            raise ValueError(f\"{name}: shape {arr.shape} != expected {shape}\")\n\n# call before session.run():\nfrom immich_ml.sessions.rknn import input_output_mapping\nvalidate_rknn_inputs(input_feed, input_output_mapping[session.model_type]['input'])","typeGuard":"import numpy as np\n\ndef is_valid_rknn_feed(input_feed: dict, expected: dict[str, tuple[int, ...]]) -> bool:\n    return all(\n        name in input_feed\n        and isinstance(input_feed[name], np.ndarray)\n        and input_feed[name].dtype == np.float32\n        and tuple(input_feed[name].shape) == shape\n        for name, shape in expected.items()\n    )","tryCatchPattern":"try:\n    out = session.run(output_names, input_feed)\nexcept RuntimeError as e:\n    if 'RKNN inference failed' in str(e):\n        log.warning(\"RKNN inference returned None for %s; reducing load and retrying once\")\n        validate_rknn_inputs(input_feed, expected_shapes)  # raises with a precise message\n        out = session.run(output_names, input_feed)        # single retry\n    else:\n        raise","preventionTips":["Always reshape and cast input tensors to float32 NCHW matching input_output_mapping before run().","Cap settings.rknn_threads to the number of physical NPU cores to avoid oversubscription.","Monitor SoC temperature and back off when the NPU throttles.","Keep RKNNLite runtime and the .rknn model toolchain versions in lockstep."],"tags":["rknn","npu","inference","rockchip"],"backgroundTag":null,"analyzedSha":"199723261c6ffa897fec8ccdaea6359e39c37cc3","analyzedAt":"2026-08-12T04:54:27.085Z","schemaVersion":2},"datasetVersion":"2026-08-12T13:17:24.610Z"}