thanos-io/thanos · error

hashring config watcher stopped unexpectedly

Error message

hashring config watcher stopped unexpectedly

What it means

ConfigFromWatcher returns this error when the ConfigWatcher's update channel C() is closed while it is still forwarding config updates. Channel closure means the watcher's Run loop has exited (fsnotify watcher closed, internal error, or watcher stopped) outside of context cancellation, so config updates can no longer be delivered. Callers treat this as an unexpected loss of the hashring update stream.

Solutions

1. Check logs from the ConfigWatcher.Run goroutine for the root cause of the shutdown
2. Ensure the consumer's context is the only lifecycle owner and is not cancelled early
3. Recreate the ConfigWatcher (call NewConfigWatcher + ConfigFromWatcher again) to re-establish the stream
4. Verify the config file still exists on disk and the watch is still valid (Kubernetes symlink swaps can invalidate watches)

Example fix

// before
err := cw.ConfigFromWatcher(ctx, updates) // aborts everything on watcher exit
// after
for {
	if err := cw.ConfigFromWatcher(ctx, updates); ctx.Err() != nil {
		return ctx.Err()
	}
	logger.Error(err, "watcher stopped, reconnecting")
	cw = recreateWatcher()
}

for {
	err := receive.ConfigFromWatcher(logger, cw, ctx, updates)
	if ctx.Err() != nil { return ctx.Err() }
	log.Error(err, "hashring watcher stopped; recreating")
	cw = recreateWatcher(ctx) // backoff, then rebuild
}

When it happens

Trigger: The underlying fsnotify watcher stops and ConfigWatcher.Run exits without ctx cancellation while a consumer is blocked in ConfigFromWatcher's select loop.

Common situations: inotify queue overflow or watcher teardown on the host; a bug or panic inside the watcher goroutine; the config file being removed in a way that halts the watch loop.

Related errors

• invalid service state
• invalid service state
• failed to run pre compaction callback for plan
• configuration file is not parsable
• configuration file is empty

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/287bfe2fd4b17871. Report an issue: GitHub.

Appendix: source

Thrown at pkg/receive/config.go:377

	level.Debug(cw.logger).Log("msg", "refreshed hashring config")
	select {
	case <-ctx.Done():
		return
	case cw.ch <- config:
		return
	}
}

func ConfigFromWatcher(ctx context.Context, updates chan<- []HashringConfig, cw *ConfigWatcher) error {
	defer close(updates)
	go cw.Run(ctx)

	for {
		select {
		case cfg, ok := <-cw.C():
			if !ok {
				return errors.New("hashring config watcher stopped unexpectedly")
			}
			updates <- cfg
		case <-ctx.Done():
			return ctx.Err()
		}
	}
}

// ParseConfig parses the raw configuration content and returns a HashringConfig.
func ParseConfig(content []byte) ([]HashringConfig, error) {
	var config []HashringConfig
	err := json.Unmarshal(content, &config)
	return config, err
}

// loadConfig loads raw configuration content and returns a configuration.
func loadConfig(logger log.Logger, path string) ([]HashringConfig, float64, error) {
	cfgContent, err := readFile(logger, path)

View on GitHub (pinned to 35b8b99117)