{"record":{"id":"6bf36dddef905281","repo":"vitessio/vitess","slug":"watch-on-v-was-closed","errorCode":null,"errorMessage":"watch on %v was closed","messagePattern":"watch on (.+?) was closed","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"go/vt/topo/zk2topo/watch.go","lineNumber":59,"sourceCode":"\tif stats == nil {\n\t\t// No stats --> node doesn't exist.\n\t\treturn nil, nil, topo.NewError(topo.NoNode, zkPath)\n\t}\n\twd := &topo.WatchData{\n\t\tContents: data,\n\t\tVersion:  ZKVersion(stats.Version),\n\t}\n\n\tc := make(chan *topo.WatchData, 10)\n\tgo func() {\n\t\tdefer close(c)\n\n\t\tfor {\n\t\t\t// Act on the watch, or on 'stop' close.\n\t\t\tselect {\n\t\t\tcase event, ok := <-watch:\n\t\t\t\tif !ok {\n\t\t\t\t\tc <- &topo.WatchData{Err: fmt.Errorf(\"watch on %v was closed\", zkPath)}\n\t\t\t\t\treturn\n\t\t\t\t}\n\n\t\t\t\tif event.Err != nil {\n\t\t\t\t\tc <- &topo.WatchData{Err: vterrors.Wrapf(event.Err, \"received a non-OK event for %v\", zkPath)}\n\t\t\t\t\treturn\n\t\t\t\t}\n\n\t\t\tcase <-ctx.Done():\n\t\t\t\t// user is not interested any more\n\t\t\t\tc <- &topo.WatchData{Err: topo.NewError(topo.Interrupted, \"watch\")}\n\t\t\t\treturn\n\t\t\t}\n\n\t\t\t// Get the value again, and send it, or error.\n\t\t\tdata, stats, watch, err = zs.conn.GetW(ctx, zkPath)\n\t\t\tif err != nil {\n\t\t\t\tc <- &topo.WatchData{Err: convertError(err, zkPath)}","sourceCodeStart":41,"sourceCodeEnd":77,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/topo/zk2topo/watch.go#L41-L77","documentation":"The zk2 topo watcher delivers WatchData records over a channel. When the underlying ZooKeeper watch channel is closed by the server or connection (ok == false on receive), the watcher forwards a WatchData carrying this error to the consumer, signaling that the watch can no longer be trusted and the consumer must re-establish it.","triggerScenarios":"Any topo watch API (WatchString/WaitForFile or derived code in watch.go) whose zk watch channel is closed — server-side connection close, ZK session end, or shutdown of the connection the ExistsW/GetW watch was registered on.","commonSituations":"ZooKeeper ensemble restart or leader failover while a vttablet/vtctld holds watches; network drops causing the ZK client to close the connection and its watches; rolling upgrades of the zk cluster.","solutions":["Re-establish the watch: consume the error WatchData, then loop and re-issue the watch call until a valid event arrives (clients that loop on watch results handle this transparently).","Check ZooKeeper client/server logs for session expiration or connection close around the failure time and fix the underlying connectivity.","If this happens during planned ZK maintenance, restart or let the affected vttablet/vtctld reconnect after the ensemble is stable."],"exampleFix":"// before: treat any WatchData.Err as fatal\nwd := <-watchChan\nif wd.Err != nil {\n    return wd.Err\n}\n// after: reconnect on watch-closed errors, fail on real errors\nfor {\n    wd := <-watchChan\n    if wd.Err == nil {\n        return nil\n    }\n    if strings.Contains(wd.Err.Error(), \"was closed\") {\n        continue // re-watch\n    }\n    return wd.Err\n}","handlingStrategy":"retry","validationCode":"exists, _, err := conn.Exists(ctx, zkPath)\nif err != nil {\n    return fmt.Errorf(\"path %v not watchable right now: %w\", zkPath, err)\n}","typeGuard":"func isWatchClosedErr(err error) bool { return err != nil && strings.Contains(err.Error(), \"was closed\") }","tryCatchPattern":"for wd := range watchChan {\n    if wd.Err != nil {\n        if isWatchClosedErr(wd.Err) {\n            watchChan = reEstablishWatch(ctx, zkPath) // re-watch and continue\n            continue\n        }\n        return wd.Err\n    }\n    handle(wd)\n}","preventionTips":["Monitor ZooKeeper connection/session state and alert on disconnects so watch closures are expected","Always re-establish watches in a loop rather than treating one closure as fatal","Avoid ZK rolling restarts during critical replication/election activity"],"tags":["zookeeper","topo","watch","connection"],"backgroundTag":"zk-watch-closed","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}