netbirdio/netbird · error

watcher closed unexpectedly

Error message

watcher closed unexpectedly

What it means

Returned by the installer ResultHandler read loop (client/internal/updater/installer/result.go:138) when fsnotify's Events channel closes unexpectedly (the ok flag on receive is false). A closed Events channel means the watcher was shut down (Close called elsewhere or the watcher died), so the handler can no longer observe creation/write events for the installer result file and aborts instead of hanging.

Source

Thrown at client/internal/updater/installer/result.go:138

	if err := watcher.Add(dir); err != nil {
		return Result{}, fmt.Errorf("failed to watch directory: %v", err)
	}

	// Check again after setting up watcher to avoid race condition
	// (file could have been created between initial check and watcher setup)
	if result, err := rh.tryReadResult(); err == nil {
		log.Infof("installer result: %v", result)
		return result, nil
	}

	for {
		select {
		case <-ctx.Done():
			return Result{}, ctx.Err()
		case event, ok := <-watcher.Events:
			if !ok {
				return Result{}, errors.New("watcher closed unexpectedly")
			}

			if result, done := rh.handleWatchEvent(event); done {
				return result, nil
			}
		case err, ok := <-watcher.Errors:
			if !ok {
				return Result{}, errors.New("watcher closed unexpectedly")
			}
			return Result{}, fmt.Errorf("watcher error: %w", err)
		}
	}
}

func (rh *ResultHandler) handleWatchEvent(event fsnotify.Event) (Result, bool) {
	if event.Name != rh.resultFile {
		return Result{}, false
	}

View on GitHub (pinned to 93e97f4bf1)

Solutions

1. Give the ResultHandler its own fsnotify watcher so nothing else can close it mid-read.
2. Retry once: recreate the watcher and re-enter the read loop (the source already rechecks the result file after setup, so a retry converges).
3. If inotify limits are the cause, raise fs.inotify.max_user_watches / max_user_instances on the host.
4. As a last resort, poll the result file with a timeout instead of watching.

Example fix

// before: shared watcher closed by another component -> "watcher closed unexpectedly"

// after: own watcher + bounded retry inside the handler
for attempt := 0; attempt < 2; attempt++ {
    result, err := rh.readWithFreshWatcher(ctx)
    if err == nil || !isWatcherClosed(err) {
        return result, err
    }
    log.Warnf("watcher closed, retrying (attempt %d)", attempt+1)
}

func isWatcherClosed(err error) bool {
    return err != nil && strings.Contains(err.Error(), "watcher closed unexpectedly")
}

result, err := rh.waitForResult(ctx)
if isWatcherClosed(err) {
    // watcher died mid-wait: rebuild it and retry once; the initial
    // tryReadResult recheck makes the retry converge if the file appeared
    result, err = rh.waitForResultWithFreshWatcher(ctx)
}
if err != nil {
    return Result{}, fmt.Errorf("read installer result: %w", err)
}

When it happens

Trigger: Another goroutine calls watcher.Close() while ResultHandler waits for the installer to write its result file; the watcher object is shared and torn down during updater shutdown; fsnotify runs out of kernel watch resources (inotify limits) and the watcher is removed.

Common situations: Updater restart or cancellation racing an in-flight installer run; two components sharing one fsnotify watcher with different lifetimes; constrained environments (containers) with low fs.inotify.max_user_watches causing watcher failure.

Related errors

• management client is not initialised
• profile not found
• failed to verify signature of artifact keys
• no keys found in bundle
• failed to decode PEM data

AI-assisted analysis of netbirdio/netbird@93e97f4bf1 (2026-08-16). Data as JSON: /api/errors/c9b6ed273cdc5b9e. Report an issue: GitHub.