thanos-io/thanos · critical

commit samples

Error message

commit samples

What it means

This error wraps a failure from app.Commit() at the end of CapNProtoWriter.Write. Commit flushes all appended series into the TSDB head; a failure here means none of the batch's samples were durably accepted. It is collected into the writeErrorTracker along with any per-series validation errors and returned to the caller.

Solutions

1. Inspect the wrapped cause for timestamp-out-of-bounds vs storage I/O errors
2. Align sender clocks and enforce ingestion time windowing (out-of-order/time-window flags) so samples fall within the head's bounds
3. Free disk space / fix storage backend health
4. Retry the replication; commit failures are batch-level so the sender must resend

Example fix

// before
if err := app.Commit(); err != nil {
	return err
}
// after
if err := app.Commit(); err != nil {
	errs.Add(errors.Wrap(err, "commit samples"))
	logger.Error(errs, "commit failed, requester should resend batch")
}
return errs.ErrOrNil()

// sender: clamp timestamps to the receiver's acceptance window before writing
now := time.Now().UnixMilli()
window := int64((13 * 24 * time.Hour) / time.Millisecond)
for i := range samples {
	if samples[i].T > now || samples[i].T < now-window { drop(i) }
}

err := writer.Write(ctx, wreq)
if err != nil {
	if strings.Contains(err.Error(), "commit samples") {
		// batch-level failure: re-queue the entire batch
	}
}

When it happens

Trigger: app.Commit() returns an error — head storage failure, too old/out-of-bounds sample timestamps rejected at commit, or the appender was invalidated mid-batch (e.g. head truncation).

Common situations: Samples outside the allowed time window (min/max block duration) arriving from misaligned senders; disk full or I/O errors on the TSDB directory; head compaction racing with writes.

Related errors

• commit samples
• create compactor
• open block
• start remote write agent db
• open TSDB

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/75f620c36b0a5c2f. Report an issue: GitHub.

Appendix: source

Thrown at pkg/receive/capnproto_writer.go:146

					exlset := &labelpb.ZLabelSet{Labels: labelpb.ZLabelsFromPromLabels(copiedLabels)}
					errorTracker.addLabelsError(err, exlset, exLogger)
					continue
				}
				if _, err = app.AppendExemplar(ref, lset, exemplar.Exemplar{
					Labels: copiedLabels,
					Value:  ex.Value,
					Ts:     ex.Ts,
					HasTs:  true,
				}); err != nil {
					errorTracker.addExemplarError(err, exLogger)
				}
			}
		}
	}

	errs := errorTracker.collectErrors(tLogger)
	if err := app.Commit(); err != nil {
		errs.Add(errors.Wrap(err, "commit samples"))
	}
	return errs.ErrOrNil()
}

// ValidateLabels validates label names and values (checks for empty
// names and values, out of order labels and duplicate label names)
// Returns appropriate error if validation fails on a label.
func validateLabels(lbls labels.Labels) error {
	if lbls.Len() == 0 {
		return labelpb.ErrEmptyLabels
	}

	var (
		isFirst  = true
		prevName string
	)
	return lbls.Validate(func(l labels.Label) error {
		if l.Name == "" || l.Value == "" {

View on GitHub (pinned to 35b8b99117)