thanos-io/thanos · error

writing data to local TSDB

Error message

writing %s data to local TSDB: %w

What it means

The receive handler collects per-tenant failures from h.writer.Write (local TSDB append) into an errs slice, formatting each as "writing <tenant> data to local TSDB: <err>". It is an aggregate wrapper: the underlying cause (TSDB head full, storage error, etc.) is preserved via %w.

Solutions

1. Read the wrapped cause (%w) of the first error to find the real TSDB failure
2. Check disk space and local TSDB data-dir health on the receive node
3. Fix the root cause per-tenant (e.g. rejected samples) and replay/republish the data — replication should cover it if --receive.replication-factor > 1
4. Monitor writer.Write errors and alert on storage pressure

Example fix

// before: ignore storage health until writes fail
// after: preflight check before accepting traffic
if err := syscall.Access(dataDir, os.O_RDWR); err != nil {
    logger.Warn("TSDB data dir not writable, draining receive", "err", err)
}

if free, err := diskFree(dataDir); err != nil || free < minFreeBytes {
    // refuse or shed writes before TSDB write fails
}

err := h.writer.Write(ctx, tenant, series)
if err != nil {
    var cause error
    for e := err; e != nil; e = errors.Unwrap(e) { cause = e }
    logger.Error("local TSDB write failed", "tenant", tenant, "cause", cause)
    // fall back to a healthy receive node if replicated
}

When it happens

Trigger: Any data plane write to the local TSDB fails inside handler.go:1418, e.g. storage unavailable, out-of-order sample rejected, head block corrupted, or disk full; one entry is appended per failing tenant batch.

Common situations: Disk full on receive node; TSDB compaction failing; tenant allowed but samples rejected; multiple tenants failing simultaneously after a storage outage.

Related errors

• start remote write agent db
• open TSDB
• write chunks
• write
• get appender

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/77482d1fd2295a55. Report an issue: GitHub.

Appendix: source

Thrown at pkg/receive/handler.go:1418

		data = append(data, wreqTenantTuple{
			wreq: &prompb.WriteRequest{
				Timeseries: r.Timeseries,
			},
			tenant: r.Tenant,
		})
	}

	// Fast path for IngestorOnly mode: write directly to local TSDB.
	// This skips distributeTimeseriesToReplicas and sendLocalWrite since
	// the Router already determined this data belongs to this node.
	if h.receiverMode == IngestorOnly {
		var errs = make([]error, 0, len(data))
		for _, di := range data {
			err := h.writer.Write(ctx, di.tenant, di.wreq.Timeseries)
			if err != nil {
				level.Debug(h.logger).Log("msg", "failed to write to local TSDB", "err", err, "tenant", di.tenant)

				errs = append(errs, fmt.Errorf("writing %s data to local TSDB: %w", di.tenant, err))
			}
		}

		if len(errs) > 0 {
			returnErr := errs[0]
			err := errors.Unwrap(returnErr)

			if len(errs) > 1 {
				returnErr = fmt.Errorf("got %d errors while writing to multiple tenants, first one: %w", len(errs), returnErr)
			}

			switch cause := errors.Cause(err); cause {
			case nil:
				panic("BUG: errors.Cause returned nil on a non-nil error")
			default:
				if isNotReady(cause) {
					return nil, status.Error(codes.Unavailable, returnErr.Error())
				}

View on GitHub (pinned to 35b8b99117)