thanos-io/thanos · critical

commit samples

Error message

commit samples

What it means

Error added by Writer.Write when app.Commit() fails after appending all samples/histograms/exemplars. Commit persists the appended data into the TSDB head; failure means none of this batch is durable and the whole write should be considered failed. The raw storage error is wrapped as 'commit samples'.

Solutions

1. Check the wrapped cause and receiver logs for TSDB/WAL errors (disk full, corruption).
2. Free disk space / fix WAL directory permissions on the receive volume.
3. Have the sender retry the remote-write request - the failed commit did not persist data, so retry is safe.
4. Restart the receiver if the head is in an unrecoverable state and restore from block storage.

Example fix

// sender-side retry on commit failure
for attempt := 0; attempt < 3; attempt++ {
    if err := writer.Write(ctx, tenant, batch); err == nil { break }
    time.Sleep(backoff(attempt))
}

if err := writer.Write(ctx, tenant, series); err != nil {
    if strings.Contains(err.Error(), "commit samples") {
        // safe to retry: commit failed so nothing was persisted
        return retryWithBackoff(ctx, tenant, series)
    }
}

When it happens

Trigger: app.Commit() returning an error at the end of Write - typically because the TSDB head is closing/closed, a rollback happened internally, or the underlying WAL write failed.

Common situations: Receiver shutting down while writes are in flight; WAL directory full or on failing disk; TSDB head corruption; context-cancellation paths in the appender.

Related errors

• get tenant appendable
• get appender
• add series
• start remote write agent db
• open TSDB

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/89be6bcccec5a541. Report an issue: GitHub.

Appendix: source

Thrown at pkg/receive/writer.go:168

				labelpb.ReAllocZLabelsStrings(&ex.Labels)
				exLset := labelpb.ZLabelsToPromLabels(ex.Labels)
				exLogger := log.With(tLogger, "exemplarLset", exLset, "exemplar", ex.String())

				if _, err = app.AppendExemplar(ref, lset, exemplar.Exemplar{
					Labels: exLset,
					Value:  ex.Value,
					Ts:     ex.Timestamp,
					HasTs:  true,
				}); err != nil {
					errorTracker.addExemplarError(err, exLogger)
				}
			}
		}
	}

	errs := errorTracker.collectErrors(tLogger)
	if err := app.Commit(); err != nil {
		errs.Add(errors.Wrap(err, "commit samples"))
	}
	return errs.ErrOrNil()
}

View on GitHub (pinned to 35b8b99117)