thanos-io/thanos · error

add samples

Error message

add %d samples

What it means

Thanos Receive wraps storage.ErrOutOfOrderSample ('out of order sample') reporting N dropped samples. The TSDB head rejected a sample because its timestamp is older than the most recent sample already appended for that series (outside the allowed out-of-order window, if configured).

Solutions

1. Enable and size the out-of-order window (--tsdb.out-of-order-time-window) on the receiver if delayed data is expected.
2. Fix producer-side clock synchronization (NTP) or the ordering bug in the sender.
3. Use backfilling (tsdb block tooling) for historical data instead of live remote-write.
4. Check prometheus_tsdb_head_max_time on the receiver to confirm which samples fall outside the head.

Example fix

// before: receiver rejects all late samples
// after: allow a 5m out-of-order window
r := receive.New(...)
opts.OutOfOrderTimeWindow = 5 * time.Minute

if sample.Ts < lastKnownHeadMaxTimeMs {
    return fmt.Errorf("sample ts %d older than head max %d", sample.Ts, lastKnownHeadMaxTimeMs)
}

if errors.Is(err, storage.ErrOutOfOrderSample) { // buffer for backfill or drop with metric increment; do not hot-retry }

When it happens

Trigger: A remote-write request contains a sample for a series whose timestamp is earlier than the series' current head max time and beyond the out-of-order acceptance window; append returns ErrOutOfOrderSample, the tracker counts it, collectErrors emits 'add N samples: out of order sample'.

Common situations: Replaying old data after a receiver outage or failover; clock skew between producers; multiple scrapers of the same target ingesting with different delays; backfilled historical data sent through live remote-write.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• add exemplars
• start remote write agent db
• create compactor
• open block
• failed to create matchers cache

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/b0fa6a84f3047a26. Report an issue: GitHub.

Appendix: source

Thrown at pkg/receive/writer_errors.go:135

func (a *writeErrorTracker) collectErrors(tLogger log.Logger) writeErrors {
	var errs writeErrors
	if a.numLabelsOutOfOrder > 0 {
		level.Warn(tLogger).Log("msg", "Error on series with out-of-order labels", "numDropped", a.numLabelsOutOfOrder)
		errs.Add(errors.Wrapf(labelpb.ErrOutOfOrderLabels, "add %d series", a.numLabelsOutOfOrder))
	}
	if a.numLabelsDuplicates > 0 {
		level.Warn(tLogger).Log("msg", "Error on series with duplicate labels", "numDropped", a.numLabelsDuplicates)
		errs.Add(errors.Wrapf(labelpb.ErrDuplicateLabels, "add %d series", a.numLabelsDuplicates))
	}
	if a.numLabelsEmpty > 0 {
		level.Warn(tLogger).Log("msg", "Error on series with empty label name or value", "numDropped", a.numLabelsEmpty)
		errs.Add(errors.Wrapf(labelpb.ErrEmptyLabels, "add %d series", a.numLabelsEmpty))
	}

	if a.numSamplesOutOfOrder > 0 {
		level.Warn(tLogger).Log("msg", "Error on ingesting out-of-order samples", "numDropped", a.numSamplesOutOfOrder)
		errs.Add(errors.Wrapf(storage.ErrOutOfOrderSample, "add %d samples", a.numSamplesOutOfOrder))
	}
	if a.numSamplesDuplicates > 0 {
		level.Warn(tLogger).Log("msg", "Error on ingesting samples with different value but same timestamp", "numDropped", a.numSamplesDuplicates)
		errs.Add(errors.Wrapf(storage.ErrDuplicateSampleForTimestamp, "add %d samples", a.numSamplesDuplicates))
	}
	if a.numSamplesOutOfBounds > 0 {
		level.Warn(tLogger).Log("msg", "Error on ingesting samples that are too old or are too far into the future", "numDropped", a.numSamplesOutOfBounds)
		errs.Add(errors.Wrapf(storage.ErrOutOfBounds, "add %d samples", a.numSamplesOutOfBounds))
	}
	if a.numSamplesTooOld > 0 {
		level.Warn(tLogger).Log("msg", "Error on ingesting samples that are outside of the allowed out-of-order time window", "numDropped", a.numSamplesTooOld)
		errs.Add(errors.Wrapf(storage.ErrTooOldSample, "add %d samples", a.numSamplesTooOld))
	}

	if a.numExemplarsOutOfOrder > 0 {
		level.Warn(tLogger).Log("msg", "Error on ingesting out-of-order exemplars", "numDropped", a.numExemplarsOutOfOrder)
		errs.Add(errors.Wrapf(storage.ErrOutOfOrderExemplar, "add %d exemplars", a.numExemplarsOutOfOrder))
	}

View on GitHub (pinned to 35b8b99117)