blakeblackshear/frigate · critical · RuntimeError

HailoRT inference thread has stopped, restart required.

Error message

HailoRT inference thread has stopped, restart required.

What it means

A HailoRT inference request timed out and, on checking, the background HailoRT inference thread is no longer alive. The plugin can recover from mere timeouts by returning zero detections, but a dead worker thread means the accelerator/session is wedged, so it raises RuntimeError signalling the detector process must restart.

Source

Thrown at frigate/detectors/plugins/hailo8l.py:350

        return cached_model_path

    def detect_raw(self, tensor_input):
        tensor_input = self.preprocess(tensor_input)

        if isinstance(tensor_input, np.ndarray) and len(tensor_input.shape) == 3:
            tensor_input = np.expand_dims(tensor_input, axis=0)

        request_id = self.input_store.put(tensor_input)

        try:
            _, infer_results = self.response_store.get(request_id, timeout=1.0)
        except TimeoutError:
            logger.error(
                f"Timeout waiting for inference results for request {request_id}"
            )

            if not self.inference_thread.is_alive():
                raise RuntimeError(
                    "HailoRT inference thread has stopped, restart required."
                ) from None

            return np.zeros((20, 6), dtype=np.float32)

        if isinstance(infer_results, list) and len(infer_results) == 1:
            infer_results = infer_results[0]

        threshold = 0.4
        all_detections = []
        for class_id, detection_set in enumerate(infer_results):
            if not isinstance(detection_set, np.ndarray) or detection_set.size == 0:
                continue
            for det in detection_set:
                if det.shape[0] < 5:
                    continue
                score = float(det[4])
                if score < threshold:

View on GitHub (pinned to ca18b8dc13)

Solutions

1. Restart the Frigate detector process/container (the error explicitly says restart is required)
2. Update HailoRT (hailort and hailort driver/firmware) to a matched, current version
3. Check device health: hailortcli scan, dmesg for device resets, power/thermal issues on the H8L board
4. If it recurs, inspect Frigate logs for the earlier exception that killed the inference thread and address that root cause

def hailo_thread_alive(detector) -> bool:
    return detector.inference_thread is not None and detector.inference_thread.is_alive()

try:
    dets = detector.detect_raw(tensor)
except RuntimeError as e:
    if 'restart required' in str(e):
        logger.error('HailoRT thread dead; scheduling container/detector restart')
        request_restart()  # e.g. supervisor/docker autoheal
    raise

When it happens

Trigger: detect_raw waits on the inference result queue past the timeout; infer_results never arrive; self.inference_thread.is_alive() is False (thread crashed due to HailoRT error, device disconnect, or shutdown race).

Common situations: HailoRT library/driver crash mid-run; USB/PCIe Hailo device reset or removed; version mismatch between hailoRT and firmware; heavy load causing thread exception then subsequent timeouts.

Related errors

• Invalid model URL. Only .hef files are supported.
• Failed to download model from {url}: {str(e)}
• Model file not found at: {self.model_path}

AI-assisted analysis of blakeblackshear/frigate@ca18b8dc13 (2026-08-27). Data as JSON: /api/errors/1ff01403780f6c91. Report an issue: GitHub.