What causes "requested index not in buffer" in hashicorp/nomad?

Reconnect without StartExactlyAtIndex (or with Index=0) to subscribe from the current head and re-sync.

requested index not in buffer

Error message

requested index not in buffer

What it means

The event broker's Subscribe honors req.Index when the caller wants to resume from a specific Raft index. StartAtClosest returns the closest buffered head and an offset; if offset > 0 the requested index has already been discarded from the ring buffer, and when the caller demanded StartExactlyAtIndex the subscription cannot be established, so this exact error is returned.

Source

Thrown at nomad/stream/event_broker.go:123

// the requested index if it is no longer in the buffer. If StartExactlyAtIndex is
// set and the index is no longer in the buffer or not yet in the buffer an error
// will be returned.
//
// When a caller is finished with the subscription it must call Subscription.Unsubscribe
// to free ACL tracking resources.
func (e *EventBroker) Subscribe(req *SubscribeRequest) (*Subscription, error) {
	e.mu.Lock()
	defer e.mu.Unlock()

	var head *bufferItem
	var offset int
	if req.Index != 0 {
		head, offset = e.eventBuf.StartAtClosest(req.Index)
	} else {
		head = e.eventBuf.Head()
	}
	if offset > 0 && req.StartExactlyAtIndex {
		return nil, fmt.Errorf("requested index not in buffer")
	} else if offset > 0 {
		metrics.SetGauge([]string{"nomad", "event_broker", "subscription", "request_offset"}, float32(offset))
		e.logger.Debug("requested index no longer in buffer", "requsted", int(req.Index), "closest", int(head.Events.Index))
	}

	// Empty head so that calling Next on sub
	start := newBufferItem(&structs.Events{Index: req.Index})
	start.link.next.Store(head)
	close(start.link.nextCh)

	if req.Authenticate == nil {
		req.Authenticate = func() error {
			return nil
		}
	} else if err := req.Authenticate(); err != nil {
		return nil, err
	}

View on GitHub (pinned to 482b49bf1a)

Solutions

Reconnect without StartExactlyAtIndex (or with Index=0) to subscribe from the current head and re-sync.
Treat this as 'state lost' and re-fetch the authoritative resource list (job/deployments) after resubscribing.
Reduce reconnect latency / keep the stream alive so the requested index stays within the buffer window.
Server-side: increase event buffer size or shorten the stale-index window exposed to clients.

Example fix

// before
sub, err := broker.Subscribe(&stream.SubscribeRequest{Index: lastSeen, StartExactlyAtIndex: true})
// after
sub, err := broker.Subscribe(&stream.SubscribeRequest{Index: lastSeen, StartExactlyAtIndex: true})
if err != nil && err.Error() == "requested index not in buffer" {
  syncFullState()
  sub, err = broker.Subscribe(&stream.SubscribeRequest{}) // resume from head
}

Defensive patterns

Strategy: fallback

Validate before calling

// Only request an exact index if it is recent enough to plausibly be buffered
if lastSeen > 0 && time.Since(lastSeenTime) > bufferWindow {
  req.Index = 0 // resume from head and re-sync state instead
}

Type guard

func isIndexNotInBuffer(err error) bool { return err != nil && err.Error() == "requested index not in buffer" }

Try / catch

sub, err := broker.Subscribe(req)
if isIndexNotInBuffer(err) {
  syncFullState() // rebuild from authoritative API
  sub, err = broker.Subscribe(&stream.SubscribeRequest{})
}

Prevention

Bound the staleness of indexes you try to resume from.
Keep event stream connections alive (heartbeats) to avoid long reconnect gaps.
Always implement a full-state resync path for event consumers.

When it happens

Trigger: Creating a subscription (e.g. job plan/event stream RPC with Index > 0 and StartExactlyAtIndex=true) where the requested index is older than the oldest event still held by the broker's event buffer (buffer rollover).

Common situations: Slow or disconnected consumer reconnecting after long downtime (e.g. ACL-authenticated event stream UI resume); very chatty topics (deployment/evaluation events) flushing the small default buffer quickly; network partitions causing stale-index resumes.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/e381ead56da657ab. Report an issue: GitHub.