redis/redis-py · error · RuntimeError

Health check event loop failed to start within {timeout} sec

Error message

Health check event loop failed to start within {timeout} seconds

What it means

Raised as RuntimeError by BackgroundScheduler._ensure_health_check_loop (redis/background.py:254) when the dedicated health-check event loop does not signal readiness within the timeout (default 5s). The background thread either failed to start, failed to run the loop, or the system is too loaded for call_soon to fire promptly.

Source

Thrown at redis/background.py:254

            )
            self._health_check_thread.start()

            # Wait for loop to be running INSIDE the lock with a timeout.
            # This prevents other threads from trying to create another loop
            # before this one is fully started, while avoiding permanent deadlock
            # if the background thread fails to start the loop.
            if not self._health_check_loop_ready.wait(timeout=timeout):
                # Timeout expired - the loop failed to start
                # Clean up the failed loop to allow retry
                failed_loop = self._health_check_loop
                self._health_check_loop = None
                if failed_loop in self._event_loops:
                    self._event_loops.remove(failed_loop)
                try:
                    failed_loop.close()
                except Exception:
                    pass
                raise RuntimeError(
                    f"Health check event loop failed to start within {timeout} seconds"
                )

    def _run_health_check_loop(self):
        """Run the shared health check event loop."""
        asyncio.set_event_loop(self._health_check_loop)

        # Signal that the loop is ready before running
        # Use call_soon to signal after run_forever starts processing
        self._health_check_loop.call_soon(self._health_check_loop_ready.set)

        try:
            self._health_check_loop.run_forever()
        finally:
            try:
                pending = asyncio.all_tasks(self._health_check_loop)
                for task in pending:
                    task.cancel()

View on GitHub (pinned to da03cdc7e8)

Solutions

1. Raise the timeout passed to _ensure_health_check_loop (call site) on loaded hosts.
2. Increase thread/process limits for the container/process (pids.max, RLIMIT_NPROC).
3. Reduce background CPU pressure or co-locate fewer busy loops on the same process.
4. Investigate why the daemon thread is not reaching run_forever (check for early thread death / exceptions in _run_health_check_loop).

Example fix

# before
scheduler._ensure_health_check_loop()  # default timeout=5.0 may be too short
# after
scheduler._ensure_health_check_loop(timeout=15.0)

import resource
soft, hard = resource.getrlimit(resource.RLIMIT_NPROC)
if soft != resource.RLIM_INFINITY and soft < 50:
    logger.warning('thread limit low; health-check loop may not start')

try:
    scheduler._ensure_health_check_loop(timeout=15.0)
except RuntimeError as e:
    logger.error('health-check loop unavailable: %s', e)
    raise

When it happens

Trigger: run_coro_sync / run_recurring_coro / run_coro_fire_and_forget triggering _ensure_health_check_loop on a thread-starved or overloaded host; thread creation blocked by resource limits; the loop thread died before signaling readiness.

Common situations: Container/process at thread limit (RLIMIT_NPROC, cgroup pids.max); very high CPU load so the daemon thread never gets scheduled; thread creation blocked by seccomp/AppArmor; intermittent under heavy load spikes.

Related errors

• Scheduler is stopped
• health_check_probes must be greater than 0
• Database health check url is not set. Please check DatabaseC
• Could not find a matching bdb
• Channel: '{channel}' has no handler registered

AI-assisted analysis of redis/redis-py@da03cdc7e8 (2026-08-04). Data as JSON: /data/errors/510feee8de97ce34.json. Report an issue: GitHub.