What causes "event dropped from buffer" in hashicorp/nomad?

On this error, re-fetch the current state of the tracked resources and continue consuming (resync-on-gap pattern) — do not treat the stream as dead.

hashicorp/nomad · warning

event dropped from buffer

Error message

event dropped from buffer

What it means

Next() checks link.droppedCh right after waking on nextCh: if the buffer advanced past events the reader never consumed (reader too slow), it returns "event dropped from buffer". This is a liveness signal — the subscription stays valid, but a gap exists between the last delivered event and the next one, so consumers must not assume contiguous state.

Source

Thrown at nomad/stream/event_buffer.go:273

// Next return the next buffer item in the buffer. It may block until ctx is
// cancelled or until the next item is published.
func (i *bufferItem) Next(ctx context.Context, forceClose <-chan struct{}) (*bufferItem, error) {
	// See if there is already a next value, block if so. Note we don't rely on
	// state change (chan nil) as that's not threadsafe but detecting close is.
	select {
	case <-ctx.Done():
		return nil, ctx.Err()
	case <-forceClose:
		return nil, fmt.Errorf("subscription closed")
	case <-i.link.nextCh:
	}

	// Check if the reader is too slow and the event buffer as discarded the event
	// This must happen after the above select to prevent a random selection
	// between linkCh and droppedCh
	select {
	case <-i.link.droppedCh:
		return nil, fmt.Errorf("event dropped from buffer")
	default:
	}

	// If channel closed, there must be a next item to read
	nextRaw := i.link.next.Load()
	if nextRaw == nil {
		// shouldn't be possible
		return nil, errors.New("invalid next item")
	}
	next := nextRaw.(*bufferItem)
	if next.Err != nil {
		return nil, next.Err
	}
	return next, nil
}

// NextNoBlock returns the next item in the buffer without blocking. If it
// reaches the most recent item it will return nil.

View on GitHub (pinned to 482b49bf1a)

Solutions

On this error, re-fetch the current state of the tracked resources and continue consuming (resync-on-gap pattern) — do not treat the stream as dead.
Make per-event handling faster/offloaded so Next() is called promptly.
Reduce consumed topic set or filter events server-side to lower throughput.
Increase buffer capacity if you control the server configuration.

Example fix

// before
if err.Error() == "event dropped from buffer" { return err }
// after
if err.Error() == "event dropped from buffer" {
  if err := resyncState(); err != nil { return err } // gap detected: rebuild from API
  continue // subscription still valid
}

Defensive patterns

Strategy: fallback

Type guard

func isEventDropped(err error) bool { return err != nil && err.Error() == "event dropped from buffer" }

Try / catch

ev, err := sub.Next(ctx)
if isEventDropped(err) {
  if rerr := resyncTrackedState(); rerr != nil { return rerr }
  continue // subscription remains usable
}

Prevention

Make per-event processing fast; never block long between Next() calls.
Implement resync-on-gap so dropped events never cause silent divergence.
Filter topics/subjects to reduce event throughput to what the consumer can sustain.

When it happens

Trigger: Consuming events slower than the producer publishes; the ring buffer wraps and drops events before Next() reads them, firing droppedCh, on any active event stream subscription (job/evaluation/deployment topics).

Common situations: UIs or controllers processing heavy event bursts (many concurrent deployments) with per-event work or slow RPCs; long blocking handlers between Next() calls; under-provisioned clients on busy clusters.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/964d73d186f15560. Report an issue: GitHub.