{"record":{"id":"9fc6af266dfa85e3","repo":"tirth8205/code-review-graph","slug":"watch-observer-stopped-dead-thread-s-names","errorCode":null,"errorMessage":"watch observer stopped: dead thread(s) {names}","messagePattern":"watch observer stopped: dead thread\\(s\\) (.+?)","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"code_review_graph/incremental.py","lineNumber":2462,"sourceCode":"                # nothing else would ever catch the stale side.\n                handler.dispatch(DirDeletedEvent(path))\n                handler.dispatch(DirCreatedEvent(path))\n            if dead:\n                names = \", \".join(dead)\n                supervisor.report_health(\n                    observer_alive=False,\n                    last_event_at=handler.last_event_at,\n                    events_seen=handler.events_seen,\n                    dead_threads=tuple(dead),\n                    force=True,\n                )\n                logger.error(\n                    \"Filesystem watcher thread(s) died (%s); %s would stop updating \"\n                    \"silently, so this watcher is exiting for the daemon to restart it\",\n                    names,\n                    repo_root,\n                )\n                raise RuntimeError(f\"watch observer stopped: dead thread(s) {names}\")\n            supervisor.report_health(\n                observer_alive=True,\n                last_event_at=handler.last_event_at,\n                events_seen=handler.events_seen,\n            )\n        supervisor.clear_health()\n    except KeyboardInterrupt:\n        supervisor.clear_health()\n        _run_time_boxed(observer.stop, \"observer stop\")\n    finally:\n        restore_sigterm()\n        _run_time_boxed(observer.stop, \"observer stop\")\n        observer.join(timeout=_WATCH_STOP_TIMEOUT)\n        handler.stop()\n    logger.info(\"Watch stopped.\")\n\n\ndef start_watch_thread(","sourceCodeStart":2444,"sourceCodeEnd":2480,"githubUrl":"https://github.com/tirth8205/code-review-graph/blob/b58668751ab0c7670c078cf7cbd4d1f5b8e54f81/code_review_graph/incremental.py#L2444-L2480","documentation":"The watch loop detects that one or more watchdog observer threads have died, and exits with RuntimeError instead of continuing with a silently-stale graph. The message names the dead thread(s) so the daemon supervisor can restart the watcher.","triggerScenarios":"Running watch() (or the daemon 'start'/'run' flow) when the underlying watchdog Observer thread crashes or exits — commonly caused by inotify watch limits being exhausted on Linux, or the observer being stopped externally.","commonSituations":"Linux systems where fs.inotify.max_user_watches is exceeded on large repos; multiple watchers on the same tree; observer threads killed by OOM or errors inside watchdog.","solutions":["Raise the inotify limit: 'sudo sysctl fs.inotify.max_user_watches=524288' (persist in /etc/sysctl.conf) — the most common cause of dead observer threads on Linux.","Check the logged error just above the raise; the watcher logs why the thread(s) died before exiting.","Ensure only one watcher instance runs per repository to avoid resource exhaustion.","Restart the daemon; the watcher exits precisely so the supervisor can bring it back with fresh threads."],"exampleFix":"# before\nwatchdog start /path/to/repo   # observer threads keep dying\n# after\nsudo sysctl fs.inotify.max_user_watches=524288\nsudo sysctl -p\nwatchdog start /path/to/repo","handlingStrategy":"retry","validationCode":"import os\nif not os.path.exists('/proc/sys/fs/inotify/max_user_watches') is False:\n    pass\n# Linux: check watch budget before starting\nimport subprocess\nlimit = int(subprocess.run(['sysctl','-n','fs.inotify.max_user_watches'],capture_output=True,text=True).stdout.strip() or 0)\nif limit and limit < len(list(repo_root.rglob('*'))):\n    raise SystemExit('raise fs.inotify.max_user_watches before watching this repo')","typeGuard":null,"tryCatchPattern":"try:\n    updater.watch(repo_root)\nexcept RuntimeError as exc:\n    if str(exc).startswith('watch observer stopped'):\n        restart_watcher_with_backoff(updater, repo_root)","preventionTips":["Raise fs.inotify.max_user_watches on Linux for large repos.","Run the watcher under a supervisor that restarts it with backoff.","Avoid multiple concurrent watchers over the same tree."],"tags":["watcher","watchdog","inotify","linux","thread-death"],"backgroundTag":"inotify-watch-limit-exceeded","analyzedSha":"b58668751ab0c7670c078cf7cbd4d1f5b8e54f81","analyzedAt":"2026-08-28T13:19:08.966Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}