ETCD_WATCH_LIMIT

ETCD_WATCH_LIMIT

Error message

ETCD_WATCH_LIMIT: at most %d watches are allowed per connection

What it means

Each etcdSession allows at most maxWatches concurrently registered watches. watchStart checks s.watchCount() before creating a new watch and returns this coded error (ETCD_WATCH_LIMIT) when the cap is reached, protecting the agent connection from unbounded watch fan-out. Existing watches keep working; only new registrations are refused.

Source

Thrown at agents/drivers/etcd-go/watch.go:219

	defer s.watchesMu.Unlock()
	if _, exists := s.watches[id]; exists {
		return false
	}
	s.watches[id] = state
	return true
}

func (s *etcdSession) removeWatch(id string) *watchState {
	s.watchesMu.Lock()
	defer s.watchesMu.Unlock()
	state := s.watches[id]
	delete(s.watches, id)
	return state
}

func (s *etcdSession) watchStart(params map[string]json.RawMessage) (any, error) {
	if s.watchCount() >= maxWatches {
		return nil, fmt.Errorf("ETCD_WATCH_LIMIT: at most %d watches are allowed per connection", maxWatches)
	}
	key, err := keyBytesParam(params)
	if err != nil {
		return nil, err
	}
	scope := stringOrDefault(params, "scope", "key")
	if scope != "key" && scope != "prefix" {
		return nil, errors.New("ETCD_WATCH_SCOPE_INVALID: scope must be key or prefix")
	}
	client, err := s.activeClient()
	if err != nil {
		return nil, err
	}
	requestedRevision := longOrNull(params, "startRevision")
	var startedRevision int64
	if requestedRevision != nil && *requestedRevision > 0 {
		startedRevision = *requestedRevision
	} else {

View on GitHub (pinned to c0390bff16)

Solutions

Cancel/stop unused watches first to free slots, then retry watchStart
Consolidate to fewer watches by watching key prefixes (scope=prefix) instead of many individual keys
Raise maxWatches if the workload legitimately requires more concurrent watches

Example fix

// before
for _, key := range keys {
    session.watch(map[string]any{"key": key}) // exhausts maxWatches
}

// after
session.watch(map[string]any{"key": prefix, "scope": "prefix"}) // one watch for all keys

Defensive patterns

Strategy: fallback

Validate before calling

// track active watches client-side and stop before exceeding the cap
if activeWatchCount >= maxWatches {
    stopOldestWatch()
}

Type guard

func isWatchLimit(err error) bool {
    return err != nil && strings.Contains(err.Error(), "ETCD_WATCH_LIMIT")
}

Try / catch

w, err := session.Call("watch", params)
if isWatchLimit(err) {
    // cancel stale watches and retry once, or consolidate to a prefix watch
    cancelUnusedWatches()
    w, err = session.Call("watch", params)
}

Prevention

Cancel watches when no longer needed
Prefer prefix/scope watches over one watch per key
Track watch ids and reuse them instead of re-registering

When it happens

Trigger: Calling the watch method on a session that already has maxWatches active watches (i.e. s.watchCount() >= maxWatches).

Common situations: Watch loops that create a new watch per key instead of reusing one; watches leaked because their ids were never used with watch cancel/stop; long-running services watching an ever-growing key set; low maxWatches relative to workload.

Related errors

AI-assisted analysis of t8y2/dbx@c0390bff16 (2026-09-05). Data as JSON: /api/errors/b182187d66aa1ff5. Report an issue: GitHub.