vitessio/vitess · warning

watch on %v was closed

Error message

watch on %v was closed

What it means

The zk2 topo watcher delivers WatchData records over a channel. When the underlying ZooKeeper watch channel is closed by the server or connection (ok == false on receive), the watcher forwards a WatchData carrying this error to the consumer, signaling that the watch can no longer be trusted and the consumer must re-establish it.

Source

Thrown at go/vt/topo/zk2topo/watch.go:59

	if stats == nil {
		// No stats --> node doesn't exist.
		return nil, nil, topo.NewError(topo.NoNode, zkPath)
	}
	wd := &topo.WatchData{
		Contents: data,
		Version:  ZKVersion(stats.Version),
	}

	c := make(chan *topo.WatchData, 10)
	go func() {
		defer close(c)

		for {
			// Act on the watch, or on 'stop' close.
			select {
			case event, ok := <-watch:
				if !ok {
					c <- &topo.WatchData{Err: fmt.Errorf("watch on %v was closed", zkPath)}
					return
				}

				if event.Err != nil {
					c <- &topo.WatchData{Err: vterrors.Wrapf(event.Err, "received a non-OK event for %v", zkPath)}
					return
				}

			case <-ctx.Done():
				// user is not interested any more
				c <- &topo.WatchData{Err: topo.NewError(topo.Interrupted, "watch")}
				return
			}

			// Get the value again, and send it, or error.
			data, stats, watch, err = zs.conn.GetW(ctx, zkPath)
			if err != nil {
				c <- &topo.WatchData{Err: convertError(err, zkPath)}

View on GitHub (pinned to 01a25a7d17)

Solutions

1. Re-establish the watch: consume the error WatchData, then loop and re-issue the watch call until a valid event arrives (clients that loop on watch results handle this transparently).
2. Check ZooKeeper client/server logs for session expiration or connection close around the failure time and fix the underlying connectivity.
3. If this happens during planned ZK maintenance, restart or let the affected vttablet/vtctld reconnect after the ensemble is stable.

Example fix

// before: treat any WatchData.Err as fatal
wd := <-watchChan
if wd.Err != nil {
    return wd.Err
}
// after: reconnect on watch-closed errors, fail on real errors
for {
    wd := <-watchChan
    if wd.Err == nil {
        return nil
    }
    if strings.Contains(wd.Err.Error(), "was closed") {
        continue // re-watch
    }
    return wd.Err
}

exists, _, err := conn.Exists(ctx, zkPath)
if err != nil {
    return fmt.Errorf("path %v not watchable right now: %w", zkPath, err)
}

func isWatchClosedErr(err error) bool { return err != nil && strings.Contains(err.Error(), "was closed") }

for wd := range watchChan {
    if wd.Err != nil {
        if isWatchClosedErr(wd.Err) {
            watchChan = reEstablishWatch(ctx, zkPath) // re-watch and continue
            continue
        }
        return wd.Err
    }
    handle(wd)
}

When it happens

Trigger: Any topo watch API (WatchString/WaitForFile or derived code in watch.go) whose zk watch channel is closed — server-side connection close, ZK session end, or shutdown of the connection the ExistsW/GetW watch was registered on.

Common situations: ZooKeeper ensemble restart or leader failover while a vttablet/vtctld holds watches; network drops causing the ZK client to close the connection and its watches; rolling upgrades of the zk cluster.

Related errors

• cannot watch directory %v in cell %v
• failed reading existing tablet %v: %v
• DeleteRecursive: nodes getting recreated underneath delete (
• obtainQueueLock: empty queue node: %v
• obtainQueueLock: no previous queue node found: %v

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/6bf36dddef905281. Report an issue: GitHub.