thanos-io/thanos · error

add exemplars

Error message

add %d exemplars

What it means

Thanos Receive wraps storage.ErrOutOfOrderExemplar ('out of order exemplar') reporting N dropped exemplars. Exemplars are rejected when their timestamp is older than the last exemplar already stored for that series; the exemplar ring buffer enforces per-series ordering.

Solutions

1. Ensure exemplar timestamps are monotonically increasing per series at the producer.
2. Remove duplicate scrapers of the same target that emit interleaved exemplar streams.
3. Sync producer clocks (NTP) to avoid skewed timestamps across replicas.
4. Drop or reorder stale exemplar buffers before re-sending after failures.

Example fix

// before: buffered exemplars replayed in arbitrary order after retry
sendExemplars(shuffledBuffer)
// after: sort by timestamp before sending
sort.SliceStable(buf, func(i, j int) bool { return buf[i].Ts < buf[j].Ts })
sendExemplars(buf)

if ex.Ts <= lastExemplarTsPerSeries[seriesKey] {
    return fmt.Errorf("exemplar ts %d not newer than last %d", ex.Ts, lastExemplarTsPerSeries[seriesKey])
}

if errors.Is(err, storage.ErrOutOfOrderExemplar) { // sort/drop stale exemplars; never hot-retry or the same drop repeats }

When it happens

Trigger: A remote-write request includes exemplars whose timestamps precede previously accepted exemplars for the same series; appendExemplar returns storage.ErrOutOfOrderExemplar, the tracker counts it, collectErrors emits 'add N exemplars: out of order exemplar'.

Common situations: Sender retrying exemplar batches after a partial failure; multiple producers scraping the same instrumented target and delivering exemplars at different rates; clock skew on the application emitting OpenTelemetry/Prometheus exemplars.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• add samples
• failed to create matchers cache
• failed to parse remote write config
• applying config to remote storage
• start remote write agent db

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/d1635cfb2437acd1. Report an issue: GitHub.

Appendix: source

Thrown at pkg/receive/writer_errors.go:152

		level.Warn(tLogger).Log("msg", "Error on ingesting out-of-order samples", "numDropped", a.numSamplesOutOfOrder)
		errs.Add(errors.Wrapf(storage.ErrOutOfOrderSample, "add %d samples", a.numSamplesOutOfOrder))
	}
	if a.numSamplesDuplicates > 0 {
		level.Warn(tLogger).Log("msg", "Error on ingesting samples with different value but same timestamp", "numDropped", a.numSamplesDuplicates)
		errs.Add(errors.Wrapf(storage.ErrDuplicateSampleForTimestamp, "add %d samples", a.numSamplesDuplicates))
	}
	if a.numSamplesOutOfBounds > 0 {
		level.Warn(tLogger).Log("msg", "Error on ingesting samples that are too old or are too far into the future", "numDropped", a.numSamplesOutOfBounds)
		errs.Add(errors.Wrapf(storage.ErrOutOfBounds, "add %d samples", a.numSamplesOutOfBounds))
	}
	if a.numSamplesTooOld > 0 {
		level.Warn(tLogger).Log("msg", "Error on ingesting samples that are outside of the allowed out-of-order time window", "numDropped", a.numSamplesTooOld)
		errs.Add(errors.Wrapf(storage.ErrTooOldSample, "add %d samples", a.numSamplesTooOld))
	}

	if a.numExemplarsOutOfOrder > 0 {
		level.Warn(tLogger).Log("msg", "Error on ingesting out-of-order exemplars", "numDropped", a.numExemplarsOutOfOrder)
		errs.Add(errors.Wrapf(storage.ErrOutOfOrderExemplar, "add %d exemplars", a.numExemplarsOutOfOrder))
	}
	if a.numExemplarsDuplicate > 0 {
		level.Warn(tLogger).Log("msg", "Error on ingesting duplicate exemplars", "numDropped", a.numExemplarsDuplicate)
		errs.Add(errors.Wrapf(storage.ErrDuplicateExemplar, "add %d exemplars", a.numExemplarsDuplicate))
	}
	if a.numExemplarsLabelLength > 0 {
		level.Warn(tLogger).Log("msg", "Error on ingesting exemplars with label length exceeding maximum limit", "numDropped", a.numExemplarsLabelLength)
		errs.Add(errors.Wrapf(storage.ErrExemplarLabelLength, "add %d exemplars", a.numExemplarsLabelLength))
	}
	return errs
}

View on GitHub (pinned to 35b8b99117)