{"record":{"id":"287bfe2fd4b17871","repo":"thanos-io/thanos","slug":"hashring-config-watcher-stopped-unexpectedly","errorCode":null,"errorMessage":"hashring config watcher stopped unexpectedly","messagePattern":"hashring config watcher stopped unexpectedly","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/receive/config.go","lineNumber":377,"sourceCode":"\n\tlevel.Debug(cw.logger).Log(\"msg\", \"refreshed hashring config\")\n\tselect {\n\tcase <-ctx.Done():\n\t\treturn\n\tcase cw.ch <- config:\n\t\treturn\n\t}\n}\n\nfunc ConfigFromWatcher(ctx context.Context, updates chan<- []HashringConfig, cw *ConfigWatcher) error {\n\tdefer close(updates)\n\tgo cw.Run(ctx)\n\n\tfor {\n\t\tselect {\n\t\tcase cfg, ok := <-cw.C():\n\t\t\tif !ok {\n\t\t\t\treturn errors.New(\"hashring config watcher stopped unexpectedly\")\n\t\t\t}\n\t\t\tupdates <- cfg\n\t\tcase <-ctx.Done():\n\t\t\treturn ctx.Err()\n\t\t}\n\t}\n}\n\n// ParseConfig parses the raw configuration content and returns a HashringConfig.\nfunc ParseConfig(content []byte) ([]HashringConfig, error) {\n\tvar config []HashringConfig\n\terr := json.Unmarshal(content, &config)\n\treturn config, err\n}\n\n// loadConfig loads raw configuration content and returns a configuration.\nfunc loadConfig(logger log.Logger, path string) ([]HashringConfig, float64, error) {\n\tcfgContent, err := readFile(logger, path)","sourceCodeStart":359,"sourceCodeEnd":395,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/receive/config.go#L359-L395","documentation":"ConfigFromWatcher returns this error when the ConfigWatcher's update channel C() is closed while it is still forwarding config updates. Channel closure means the watcher's Run loop has exited (fsnotify watcher closed, internal error, or watcher stopped) outside of context cancellation, so config updates can no longer be delivered. Callers treat this as an unexpected loss of the hashring update stream.","triggerScenarios":"The underlying fsnotify watcher stops and ConfigWatcher.Run exits without ctx cancellation while a consumer is blocked in ConfigFromWatcher's select loop.","commonSituations":"inotify queue overflow or watcher teardown on the host; a bug or panic inside the watcher goroutine; the config file being removed in a way that halts the watch loop.","solutions":["Check logs from the ConfigWatcher.Run goroutine for the root cause of the shutdown","Ensure the consumer's context is the only lifecycle owner and is not cancelled early","Recreate the ConfigWatcher (call NewConfigWatcher + ConfigFromWatcher again) to re-establish the stream","Verify the config file still exists on disk and the watch is still valid (Kubernetes symlink swaps can invalidate watches)"],"exampleFix":"// before\nerr := cw.ConfigFromWatcher(ctx, updates) // aborts everything on watcher exit\n// after\nfor {\n\tif err := cw.ConfigFromWatcher(ctx, updates); ctx.Err() != nil {\n\t\treturn ctx.Err()\n\t}\n\tlogger.Error(err, \"watcher stopped, reconnecting\")\n\tcw = recreateWatcher()\n}","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"for {\n\terr := receive.ConfigFromWatcher(logger, cw, ctx, updates)\n\tif ctx.Err() != nil { return ctx.Err() }\n\tlog.Error(err, \"hashring watcher stopped; recreating\")\n\tcw = recreateWatcher(ctx) // backoff, then rebuild\n}","preventionTips":["Wrap ConfigFromWatcher in a supervisor loop that recreates the watcher on unexpected exit","Keep the config file present for the watcher's whole lifetime (atomic rename, never delete)","Watch for inotify overflow warnings on the host"],"tags":["watcher","lifecycle","hashring"],"backgroundTag":"file-watcher-stopped","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}