t8y2/dbx · error

ETCD_WATCH_LIMIT

ETCD_WATCH_LIMIT

Error message

ETCD_WATCH_LIMIT: at most %d watches are allowed per connection

What it means

The etcd2 agent enforces a hard cap (maxWatches) on the number of concurrent watches per session connection. When watchStart is called and the session already has maxWatches active watches, it refuses to start another and returns this error instead of silently degrading or leaking connections. It is a protocol-level resource limit mirroring how real etcd servers bound watch resources per client.

Source

Thrown at agents/drivers/etcd2-go/watch.go:214

	defer s.watchesMu.Unlock()
	if _, exists := s.watches[id]; exists {
		return false
	}
	s.watches[id] = state
	return true
}

func (s *etcd2Session) removeWatch(id string) *watchState {
	s.watchesMu.Lock()
	defer s.watchesMu.Unlock()
	state := s.watches[id]
	delete(s.watches, id)
	return state
}

func (s *etcd2Session) watchStart(params map[string]json.RawMessage) (any, error) {
	if s.watchCount() >= maxWatches {
		return nil, fmt.Errorf("ETCD_WATCH_LIMIT: at most %d watches are allowed per connection", maxWatches)
	}
	key, err := keyBytesParam(params)
	if err != nil {
		return nil, err
	}
	scope := stringOrDefault(params, "scope", "key")
	if scope != "key" && scope != "prefix" {
		return nil, errors.New("ETCD_WATCH_SCOPE_INVALID: scope must be key or prefix")
	}
	client, err := s.activeClient()
	if err != nil {
		return nil, err
	}

	// startRevision maps to the v2 waitIndex: default to the current index+1.
	requestedRevision := longOrNull(params, "startRevision")
	var waitIndex int64
	if requestedRevision != nil && *requestedRevision > 0 {

View on GitHub (pinned to c0390bff16)

Solutions

1. Stop unused watches with watchStop before starting new ones so s.watches entries are deleted
2. Reuse an existing watch on the same key/scope instead of starting a duplicate
3. Create a new session/connection if you genuinely need more concurrent watches
4. Raise the maxWatches constant if your workload legitimately needs more watches per connection

Example fix

// before
session.watchStart(map[string]json.RawMessage{"key": raw("/cfg/a")}) // ... repeatedly, never stopping
// after
for id := range session.watches {
    if isStale(id) { session.watchStop(id) }
}
session.watchStart(map[string]json.RawMessage{"key": raw("/cfg/a")})

if session.watchCount() >= maxWatches {
    // stop stale watches or open a new session before calling watchStart
}

id, err := session.watchStart(params)
if err != nil {
    var limitErr bool
    limitErr = strings.Contains(err.Error(), "ETCD_WATCH_LIMIT")
    if limitErr {
        session.watchStop(oldestWatchID)
        id, err = session.watchStart(params)
    }
}

When it happens

Trigger: Calling watchStart (via the watch JSON-RPC handle) on an etcd2-go session that already has maxWatches watches registered; tests TestWatchLimits and TestLiveEtcd2Agent exercise this path. Watches are only freed when watchStop deletes them from s.watches.

Common situations: An agent creates a watch per table/record and never stops old watches; a long-lived session accumulates watches over many queries until the cap is reached; a reconnect loop that re-issues watches without cancelling the previous ones.

Related errors

• agent operation capacity is temporarily exhausted
• user is required
• Not connected
• Client certificate and key must be provided together
• No etcd endpoint configured

AI-assisted analysis of t8y2/dbx@c0390bff16 (2026-09-05). Data as JSON: /api/errors/7d882ca74045a7cb. Report an issue: GitHub.