{"record":{"id":"1ff01403780f6c91","repo":"blakeblackshear/frigate","slug":"hailort-inference-thread-has-stopped-restart-requ","errorCode":null,"errorMessage":"HailoRT inference thread has stopped, restart required.","messagePattern":"HailoRT inference thread has stopped, restart required\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"frigate/detectors/plugins/hailo8l.py","lineNumber":350,"sourceCode":"        return cached_model_path\n\n    def detect_raw(self, tensor_input):\n        tensor_input = self.preprocess(tensor_input)\n\n        if isinstance(tensor_input, np.ndarray) and len(tensor_input.shape) == 3:\n            tensor_input = np.expand_dims(tensor_input, axis=0)\n\n        request_id = self.input_store.put(tensor_input)\n\n        try:\n            _, infer_results = self.response_store.get(request_id, timeout=1.0)\n        except TimeoutError:\n            logger.error(\n                f\"Timeout waiting for inference results for request {request_id}\"\n            )\n\n            if not self.inference_thread.is_alive():\n                raise RuntimeError(\n                    \"HailoRT inference thread has stopped, restart required.\"\n                ) from None\n\n            return np.zeros((20, 6), dtype=np.float32)\n\n        if isinstance(infer_results, list) and len(infer_results) == 1:\n            infer_results = infer_results[0]\n\n        threshold = 0.4\n        all_detections = []\n        for class_id, detection_set in enumerate(infer_results):\n            if not isinstance(detection_set, np.ndarray) or detection_set.size == 0:\n                continue\n            for det in detection_set:\n                if det.shape[0] < 5:\n                    continue\n                score = float(det[4])\n                if score < threshold:","sourceCodeStart":332,"sourceCodeEnd":368,"githubUrl":"https://github.com/blakeblackshear/frigate/blob/ca18b8dc131ccb3e086f13a780853b775ef13426/frigate/detectors/plugins/hailo8l.py#L332-L368","documentation":"A HailoRT inference request timed out and, on checking, the background HailoRT inference thread is no longer alive. The plugin can recover from mere timeouts by returning zero detections, but a dead worker thread means the accelerator/session is wedged, so it raises RuntimeError signalling the detector process must restart.","triggerScenarios":"detect_raw waits on the inference result queue past the timeout; infer_results never arrive; self.inference_thread.is_alive() is False (thread crashed due to HailoRT error, device disconnect, or shutdown race).","commonSituations":"HailoRT library/driver crash mid-run; USB/PCIe Hailo device reset or removed; version mismatch between hailoRT and firmware; heavy load causing thread exception then subsequent timeouts.","solutions":["Restart the Frigate detector process/container (the error explicitly says restart is required)","Update HailoRT (hailort and hailort driver/firmware) to a matched, current version","Check device health: hailortcli scan, dmesg for device resets, power/thermal issues on the H8L board","If it recurs, inspect Frigate logs for the earlier exception that killed the inference thread and address that root cause"],"exampleFix":null,"handlingStrategy":"fallback","validationCode":"def hailo_thread_alive(detector) -> bool:\n    return detector.inference_thread is not None and detector.inference_thread.is_alive()","typeGuard":null,"tryCatchPattern":"try:\n    dets = detector.detect_raw(tensor)\nexcept RuntimeError as e:\n    if 'restart required' in str(e):\n        logger.error('HailoRT thread dead; scheduling container/detector restart')\n        request_restart()  # e.g. supervisor/docker autoheal\n    raise","preventionTips":["Enable container auto-restart/supervision for detector processes","Keep hailort library and device firmware versions matched","Monitor dmesg for Hailo device resets and thermal issues"],"tags":["hailo","inference-thread","timeout","accelerator","restart-required"],"backgroundTag":"inference-worker-thread-dead","analyzedSha":"ca18b8dc131ccb3e086f13a780853b775ef13426","analyzedAt":"2026-08-27T11:28:05.618Z","schemaVersion":2},"datasetVersion":"2026-08-27T13:17:12.746Z"}