hashicorp/nomad · error

requested index not in buffer

Error message

requested index not in buffer

What it means

The event broker's Subscribe honors req.Index when the caller wants to resume from a specific Raft index. StartAtClosest returns the closest buffered head and an offset; if offset > 0 the requested index has already been discarded from the ring buffer, and when the caller demanded StartExactlyAtIndex the subscription cannot be established, so this exact error is returned.

Solutions

1. Reconnect without StartExactlyAtIndex (or with Index=0) to subscribe from the current head and re-sync.
2. Treat this as 'state lost' and re-fetch the authoritative resource list (job/deployments) after resubscribing.
3. Reduce reconnect latency / keep the stream alive so the requested index stays within the buffer window.
4. Server-side: increase event buffer size or shorten the stale-index window exposed to clients.

Example fix

// before
sub, err := broker.Subscribe(&stream.SubscribeRequest{Index: lastSeen, StartExactlyAtIndex: true})
// after
sub, err := broker.Subscribe(&stream.SubscribeRequest{Index: lastSeen, StartExactlyAtIndex: true})
if err != nil && err.Error() == "requested index not in buffer" {
  syncFullState()
  sub, err = broker.Subscribe(&stream.SubscribeRequest{}) // resume from head
}

// Only request an exact index if it is recent enough to plausibly be buffered
if lastSeen > 0 && time.Since(lastSeenTime) > bufferWindow {
  req.Index = 0 // resume from head and re-sync state instead
}

func isIndexNotInBuffer(err error) bool { return err != nil && err.Error() == "requested index not in buffer" }

sub, err := broker.Subscribe(req)
if isIndexNotInBuffer(err) {
  syncFullState() // rebuild from authoritative API
  sub, err = broker.Subscribe(&stream.SubscribeRequest{})
}

When it happens

Trigger: Creating a subscription (e.g. job plan/event stream RPC with Index > 0 and StartExactlyAtIndex=true) where the requested index is older than the oldest event still held by the broker's event buffer (buffer rollover).

Common situations: Slow or disconnected consumer reconnecting after long downtime (e.g. ACL-authenticated event stream UI resume); very chatty topics (deployment/evaluation events) flushing the small default buffer quickly; network partitions causing stale-index resumes.

Related errors

• event dropped from buffer
• subscription closed
• error marshaling json for stream
• failed to stream
• missing AllocID

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/e381ead56da657ab. Report an issue: GitHub.

Appendix: source

Thrown at nomad/stream/event_broker.go:123

// the requested index if it is no longer in the buffer. If StartExactlyAtIndex is
// set and the index is no longer in the buffer or not yet in the buffer an error
// will be returned.
//
// When a caller is finished with the subscription it must call Subscription.Unsubscribe
// to free ACL tracking resources.
func (e *EventBroker) Subscribe(req *SubscribeRequest) (*Subscription, error) {
	e.mu.Lock()
	defer e.mu.Unlock()

	var head *bufferItem
	var offset int
	if req.Index != 0 {
		head, offset = e.eventBuf.StartAtClosest(req.Index)
	} else {
		head = e.eventBuf.Head()
	}
	if offset > 0 && req.StartExactlyAtIndex {
		return nil, fmt.Errorf("requested index not in buffer")
	} else if offset > 0 {
		metrics.SetGauge([]string{"nomad", "event_broker", "subscription", "request_offset"}, float32(offset))
		e.logger.Debug("requested index no longer in buffer", "requsted", int(req.Index), "closest", int(head.Events.Index))
	}

	// Empty head so that calling Next on sub
	start := newBufferItem(&structs.Events{Index: req.Index})
	start.link.next.Store(head)
	close(start.link.nextCh)

	if req.Authenticate == nil {
		req.Authenticate = func() error {
			return nil
		}
	} else if err := req.Authenticate(); err != nil {
		return nil, err
	}

View on GitHub (pinned to 482b49bf1a)