t8y2/dbx · error

ETCD_HISTORY_TIMEOUT

ETCD_HISTORY_TIMEOUT

Error message

ETCD_HISTORY_TIMEOUT: watcher was not created

What it means

After issuing client.Watch with WithCreatedNotify, the library waits up to 5 seconds for the watcher-created signal. If etcd never confirms the watcher (channel stays silent), it aborts with this timeout rather than hanging. This indicates the watch stream could not even be established, typically a connectivity or server-capacity problem, not a data problem.

Source

Thrown at agents/drivers/etcd-go/history.go:177

					row["metadata"] = metadataMap(item)
				}
				collector.append(row)
				if revision >= targetKeyRevision {
					completedOnce.Do(func() { close(completed) })
				}
			}
			if response.IsProgressNotify() && response.Header.Revision >= endRevision {
				completedOnce.Do(func() { close(completed) })
			}
		}
		completedOnce.Do(func() { close(completed) })
	}()
	defer watchCancel()

	select {
	case <-created:
	case <-time.After(5 * time.Second):
		return nil, errors.New("ETCD_HISTORY_TIMEOUT: watcher was not created")
	}
	// For an existing exact key, its latest mod revision is an explicit
	// replay boundary. This avoids relying on progress notifications,
	// which older etcd/jetcd combinations do not consistently emit.
	progressCtx, progressCancel := context.WithTimeout(context.Background(), rpcTimeoutSeconds*time.Second)
	_ = client.RequestProgress(progressCtx)
	progressCancel()
	select {
	case <-completed:
	case <-time.After(15 * time.Second):
		return nil, errors.New("ETCD_HISTORY_TIMEOUT: history replay did not reach the requested revision")
	}

	failureMu.Lock()
	historyFailure := failure
	failureMu.Unlock()
	if historyFailure != nil {
		if errors.Is(historyFailure, rpctypes.ErrCompacted) {

View on GitHub (pinned to c0390bff16)

Solutions

1. Verify network connectivity and that gRPC streams (not just HTTP) reach the etcd endpoints.
2. Retry the history call; transient stream-creation failures often resolve.
3. Check etcd server logs for watcher/stream errors and capacity (grpc keepalive settings).
4. If behind a proxy, ensure it supports long-lived gRPC streaming and idle timeouts above 5s+.

Example fix

// before
result, err := session.history(params) // fails behind stream-hostile proxy
// after
// fix proxy/firewall gRPC streaming, then retry with a connect check
if err := session.connect(); err != nil { return err }
result, err := session.history(params)

ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
defer cancel()
if _, err := client.Status(ctx, endpoint); err != nil {
	return fmt.Errorf("etcd unreachable before history query: %w", err)
}

result, err := session.history(params)
if err != nil && strings.HasPrefix(err.Error(), "ETCD_HISTORY_TIMEOUT: watcher was not created") {
	if cerr := session.connect(); cerr != nil { return nil, cerr }
	result, err = session.history(params) // single retry after reconnect
}
if err != nil { return nil, err }

When it happens

Trigger: history() called on a session whose gRPC watch stream cannot be established within 5s: unreachable endpoint after connect, watch API disabled/blocked (proxy), exhausted watch leader capacity, or network stall.

Common situations: etcd behind an LB/proxy dropping streaming RPCs; firewall idle-timeout killing gRPC streams; overloaded cluster refusing new watchers; network partition between agent and etcd; TLS handshake stalls.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• IoTDB is not reachable at {host}:{port}
• ZooKeeper connection timed out before a session was establis
• ETCD_LEASE_LIST_TIMEOUT
• ETCD_WATCH_SCOPE_INVALID
• ETCD_WATCH_NOT_FOUND

AI-assisted analysis of t8y2/dbx@c0390bff16 (2026-09-05). Data as JSON: /api/errors/e420761f907cdb78. Report an issue: GitHub.