{"record":{"id":"7d882ca74045a7cb","repo":"t8y2/dbx","slug":"etcd-watch-limit-7d882c","errorCode":"ETCD_WATCH_LIMIT","errorMessage":"ETCD_WATCH_LIMIT: at most %d watches are allowed per connection","messagePattern":"ETCD_WATCH_LIMIT: at most (.+?) watches are allowed per connection","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"error","filePath":"agents/drivers/etcd2-go/watch.go","lineNumber":214,"sourceCode":"\tdefer s.watchesMu.Unlock()\n\tif _, exists := s.watches[id]; exists {\n\t\treturn false\n\t}\n\ts.watches[id] = state\n\treturn true\n}\n\nfunc (s *etcd2Session) removeWatch(id string) *watchState {\n\ts.watchesMu.Lock()\n\tdefer s.watchesMu.Unlock()\n\tstate := s.watches[id]\n\tdelete(s.watches, id)\n\treturn state\n}\n\nfunc (s *etcd2Session) watchStart(params map[string]json.RawMessage) (any, error) {\n\tif s.watchCount() >= maxWatches {\n\t\treturn nil, fmt.Errorf(\"ETCD_WATCH_LIMIT: at most %d watches are allowed per connection\", maxWatches)\n\t}\n\tkey, err := keyBytesParam(params)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\tscope := stringOrDefault(params, \"scope\", \"key\")\n\tif scope != \"key\" && scope != \"prefix\" {\n\t\treturn nil, errors.New(\"ETCD_WATCH_SCOPE_INVALID: scope must be key or prefix\")\n\t}\n\tclient, err := s.activeClient()\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\t// startRevision maps to the v2 waitIndex: default to the current index+1.\n\trequestedRevision := longOrNull(params, \"startRevision\")\n\tvar waitIndex int64\n\tif requestedRevision != nil && *requestedRevision > 0 {","sourceCodeStart":196,"sourceCodeEnd":232,"githubUrl":"https://github.com/t8y2/dbx/blob/c0390bff16418b651f4728520d99adf8ce48829a/agents/drivers/etcd2-go/watch.go#L196-L232","documentation":"The etcd2 agent enforces a hard cap (maxWatches) on the number of concurrent watches per session connection. When watchStart is called and the session already has maxWatches active watches, it refuses to start another and returns this error instead of silently degrading or leaking connections. It is a protocol-level resource limit mirroring how real etcd servers bound watch resources per client.","triggerScenarios":"Calling watchStart (via the watch JSON-RPC handle) on an etcd2-go session that already has maxWatches watches registered; tests TestWatchLimits and TestLiveEtcd2Agent exercise this path. Watches are only freed when watchStop deletes them from s.watches.","commonSituations":"An agent creates a watch per table/record and never stops old watches; a long-lived session accumulates watches over many queries until the cap is reached; a reconnect loop that re-issues watches without cancelling the previous ones.","solutions":["Stop unused watches with watchStop before starting new ones so s.watches entries are deleted","Reuse an existing watch on the same key/scope instead of starting a duplicate","Create a new session/connection if you genuinely need more concurrent watches","Raise the maxWatches constant if your workload legitimately needs more watches per connection"],"exampleFix":"// before\nsession.watchStart(map[string]json.RawMessage{\"key\": raw(\"/cfg/a\")}) // ... repeatedly, never stopping\n// after\nfor id := range session.watches {\n    if isStale(id) { session.watchStop(id) }\n}\nsession.watchStart(map[string]json.RawMessage{\"key\": raw(\"/cfg/a\")})","handlingStrategy":"try-catch","validationCode":"if session.watchCount() >= maxWatches {\n    // stop stale watches or open a new session before calling watchStart\n}","typeGuard":null,"tryCatchPattern":"id, err := session.watchStart(params)\nif err != nil {\n    var limitErr bool\n    limitErr = strings.Contains(err.Error(), \"ETCD_WATCH_LIMIT\")\n    if limitErr {\n        session.watchStop(oldestWatchID)\n        id, err = session.watchStart(params)\n    }\n}","preventionTips":["Track watch IDs and always pair watchStart with watchStop","Audit long-lived sessions for leaked watches","Stay well below maxWatches (alert at ~80% usage)","Reuse watches for repeated queries on the same key"],"tags":["etcd","watch-limit","resource-exhaustion"],"backgroundTag":"watch-limit-exceeded","analyzedSha":"c0390bff16418b651f4728520d99adf8ce48829a","analyzedAt":"2026-09-05T23:05:10.900Z","contentChangedAt":"2026-09-05T23:05:10.900Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}