{"record":{"id":"e420761f907cdb78","repo":"t8y2/dbx","slug":"etcd-history-timeout","errorCode":"ETCD_HISTORY_TIMEOUT","errorMessage":"ETCD_HISTORY_TIMEOUT: watcher was not created","messagePattern":"ETCD_HISTORY_TIMEOUT: watcher was not created","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"error","filePath":"agents/drivers/etcd-go/history.go","lineNumber":177,"sourceCode":"\t\t\t\t\trow[\"metadata\"] = metadataMap(item)\n\t\t\t\t}\n\t\t\t\tcollector.append(row)\n\t\t\t\tif revision >= targetKeyRevision {\n\t\t\t\t\tcompletedOnce.Do(func() { close(completed) })\n\t\t\t\t}\n\t\t\t}\n\t\t\tif response.IsProgressNotify() && response.Header.Revision >= endRevision {\n\t\t\t\tcompletedOnce.Do(func() { close(completed) })\n\t\t\t}\n\t\t}\n\t\tcompletedOnce.Do(func() { close(completed) })\n\t}()\n\tdefer watchCancel()\n\n\tselect {\n\tcase <-created:\n\tcase <-time.After(5 * time.Second):\n\t\treturn nil, errors.New(\"ETCD_HISTORY_TIMEOUT: watcher was not created\")\n\t}\n\t// For an existing exact key, its latest mod revision is an explicit\n\t// replay boundary. This avoids relying on progress notifications,\n\t// which older etcd/jetcd combinations do not consistently emit.\n\tprogressCtx, progressCancel := context.WithTimeout(context.Background(), rpcTimeoutSeconds*time.Second)\n\t_ = client.RequestProgress(progressCtx)\n\tprogressCancel()\n\tselect {\n\tcase <-completed:\n\tcase <-time.After(15 * time.Second):\n\t\treturn nil, errors.New(\"ETCD_HISTORY_TIMEOUT: history replay did not reach the requested revision\")\n\t}\n\n\tfailureMu.Lock()\n\thistoryFailure := failure\n\tfailureMu.Unlock()\n\tif historyFailure != nil {\n\t\tif errors.Is(historyFailure, rpctypes.ErrCompacted) {","sourceCodeStart":159,"sourceCodeEnd":195,"githubUrl":"https://github.com/t8y2/dbx/blob/c0390bff16418b651f4728520d99adf8ce48829a/agents/drivers/etcd-go/history.go#L159-L195","documentation":"After issuing client.Watch with WithCreatedNotify, the library waits up to 5 seconds for the watcher-created signal. If etcd never confirms the watcher (channel stays silent), it aborts with this timeout rather than hanging. This indicates the watch stream could not even be established, typically a connectivity or server-capacity problem, not a data problem.","triggerScenarios":"history() called on a session whose gRPC watch stream cannot be established within 5s: unreachable endpoint after connect, watch API disabled/blocked (proxy), exhausted watch leader capacity, or network stall.","commonSituations":"etcd behind an LB/proxy dropping streaming RPCs; firewall idle-timeout killing gRPC streams; overloaded cluster refusing new watchers; network partition between agent and etcd; TLS handshake stalls.","solutions":["Verify network connectivity and that gRPC streams (not just HTTP) reach the etcd endpoints.","Retry the history call; transient stream-creation failures often resolve.","Check etcd server logs for watcher/stream errors and capacity (grpc keepalive settings).","If behind a proxy, ensure it supports long-lived gRPC streaming and idle timeouts above 5s+."],"exampleFix":"// before\nresult, err := session.history(params) // fails behind stream-hostile proxy\n// after\n// fix proxy/firewall gRPC streaming, then retry with a connect check\nif err := session.connect(); err != nil { return err }\nresult, err := session.history(params)","handlingStrategy":"retry","validationCode":"ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)\ndefer cancel()\nif _, err := client.Status(ctx, endpoint); err != nil {\n\treturn fmt.Errorf(\"etcd unreachable before history query: %w\", err)\n}","typeGuard":null,"tryCatchPattern":"result, err := session.history(params)\nif err != nil && strings.HasPrefix(err.Error(), \"ETCD_HISTORY_TIMEOUT: watcher was not created\") {\n\tif cerr := session.connect(); cerr != nil { return nil, cerr }\n\tresult, err = session.history(params) // single retry after reconnect\n}\nif err != nil { return nil, err }","preventionTips":["Health-check etcd (Status RPC) before long watch-based operations.","Ensure proxies/firewalls permit long-lived gRPC streams; raise idle timeouts.","Retry transient watch-creation failures with backoff.","Watch etcd server logs for stream/watch capacity errors."],"tags":["etcd","timeout","watch","network"],"backgroundTag":"watch-timeout","analyzedSha":"c0390bff16418b651f4728520d99adf8ce48829a","analyzedAt":"2026-09-05T23:05:10.900Z","contentChangedAt":"2026-09-05T23:05:10.900Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}