tirth8205/code-review-graph · critical · RuntimeError

watch observer stopped: dead thread(s) {names}

Error message

watch observer stopped: dead thread(s) {names}

What it means

The watch loop detects that one or more watchdog observer threads have died, and exits with RuntimeError instead of continuing with a silently-stale graph. The message names the dead thread(s) so the daemon supervisor can restart the watcher.

Source

Thrown at code_review_graph/incremental.py:2462

                # nothing else would ever catch the stale side.
                handler.dispatch(DirDeletedEvent(path))
                handler.dispatch(DirCreatedEvent(path))
            if dead:
                names = ", ".join(dead)
                supervisor.report_health(
                    observer_alive=False,
                    last_event_at=handler.last_event_at,
                    events_seen=handler.events_seen,
                    dead_threads=tuple(dead),
                    force=True,
                )
                logger.error(
                    "Filesystem watcher thread(s) died (%s); %s would stop updating "
                    "silently, so this watcher is exiting for the daemon to restart it",
                    names,
                    repo_root,
                )
                raise RuntimeError(f"watch observer stopped: dead thread(s) {names}")
            supervisor.report_health(
                observer_alive=True,
                last_event_at=handler.last_event_at,
                events_seen=handler.events_seen,
            )
        supervisor.clear_health()
    except KeyboardInterrupt:
        supervisor.clear_health()
        _run_time_boxed(observer.stop, "observer stop")
    finally:
        restore_sigterm()
        _run_time_boxed(observer.stop, "observer stop")
        observer.join(timeout=_WATCH_STOP_TIMEOUT)
        handler.stop()
    logger.info("Watch stopped.")


def start_watch_thread(

View on GitHub (pinned to b58668751a)

Solutions

1. Raise the inotify limit: 'sudo sysctl fs.inotify.max_user_watches=524288' (persist in /etc/sysctl.conf) — the most common cause of dead observer threads on Linux.
2. Check the logged error just above the raise; the watcher logs why the thread(s) died before exiting.
3. Ensure only one watcher instance runs per repository to avoid resource exhaustion.
4. Restart the daemon; the watcher exits precisely so the supervisor can bring it back with fresh threads.

Example fix

# before
watchdog start /path/to/repo   # observer threads keep dying
# after
sudo sysctl fs.inotify.max_user_watches=524288
sudo sysctl -p
watchdog start /path/to/repo

import os
if not os.path.exists('/proc/sys/fs/inotify/max_user_watches') is False:
    pass
# Linux: check watch budget before starting
import subprocess
limit = int(subprocess.run(['sysctl','-n','fs.inotify.max_user_watches'],capture_output=True,text=True).stdout.strip() or 0)
if limit and limit < len(list(repo_root.rglob('*'))):
    raise SystemExit('raise fs.inotify.max_user_watches before watching this repo')

try:
    updater.watch(repo_root)
except RuntimeError as exc:
    if str(exc).startswith('watch observer stopped'):
        restart_watcher_with_backoff(updater, repo_root)

When it happens

Trigger: Running watch() (or the daemon 'start'/'run' flow) when the underlying watchdog Observer thread crashes or exits — commonly caused by inotify watch limits being exhausted on Linux, or the observer being stopped externally.

Common situations: Linux systems where fs.inotify.max_user_watches is exceeded on large repos; multiple watchers on the same tree; observer threads killed by OOM or errors inside watchdog.

Related errors

• watch update failed

AI-assisted analysis of tirth8205/code-review-graph@b58668751a (2026-08-28). Data as JSON: /api/errors/9fc6af266dfa85e3. Report an issue: GitHub.