thanos-io/thanos · error

writing locally

Error message

writing locally

What it means

This error wraps a failure from the local TSDB append path: localAsyncWriter.RemoteWrite iterates TimeseriesTenantData and calls lw.w.Write(ctx, tenant, timeseries) for each entry. If the head write for a tenant fails (e.g. storage error, out-of-order samples, out-of-bounds timestamp), the error is wrapped with 'writing locally' and propagated back to the forwarder.

Solutions

1. Check the wrapped cause in the log to distinguish out-of-order/out-of-bounds from storage-level failures.
2. Sync sender clocks and ensure no senders emit samples with timestamps in the past (or increase out-of-order window if configured).
3. Free disk space / increase head chunk limits on the receive node.
4. Verify the tenant's TSDB is not in a corrupted state; if it is, restart the instance to reopen storage.
5. Update senders (Prometheus remote_write config) to match the receive's allowed timestamp bounds.

Example fix

// before
if err := lw.w.Write(ctx, ts.Tenant, ts.Timeseries); err != nil {
  return nil, errors.Wrap(err, "writing locally")
}
// after (drop single unapplicable samples instead of failing whole request)
if err := lw.w.Write(ctx, ts.Tenant, ts.Timeseries); err != nil {
  if errors.Is(err, storage.ErrOutOfBounds) || errors.Is(err, storage.ErrOutOfOrderSample) {
    level.Warn(lw.logger).Log("msg", "dropping unapplicable samples", "tenant", ts.Tenant, "err", err)
    continue
  }
  return nil, errors.Wrap(err, "writing locally")
}

// pre-check sample applicability where possible
if ts.Timeseries != nil && len(ts.Timeseries.Samples) > 0 {
  for _, s := range ts.Timeseries.Samples {
    if s.Timestamp <= headMaxTime {
      return fmt.Errorf("sample timestamp %d is out of bounds for tenant %s", s.Timestamp, ts.Tenant)
    }
  }
}

// Go: classify storage errors before retrying
err := errors.Cause(respErr)
switch {
case errors.Is(err, storage.ErrOutOfOrderSample), errors.Is(err, storage.ErrOutOfBounds), errors.Is(err, storage.ErrDuplicateSampleForTimestamp):
  // do NOT retry; sample is unapplicable
default:
  // transient storage issue: retry with backoff
}

When it happens

Trigger: lw.w.Write returns non-nil error while appending the given timeseries to the tenant's local TSDB head inside localAsyncWriter.RemoteWrite.

Common situations: Out-of-order or too-old sample timestamps relative to the head; duplicate sample for the same timestamp with a different value; TSDB head reaching its size limit and refusing appends; disk full on the receive node; series label churn exceeding active-series limits.

Related errors

• start remote write agent db
• open TSDB
• get appender
• open index reader
• open chunk reader

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/6ca89ba24429249c. Report an issue: GitHub.

Appendix: source

Thrown at pkg/receive/handler.go:1937

	return nil
}

type localAsyncWriter struct {
	w *Writer
}

func (lw *localAsyncWriter) Close() error {
	return nil
}

func (lw *localAsyncWriter) RemoteWrite(ctx context.Context, in *storepb.WriteRequest, opts ...grpc.CallOption) (*storepb.WriteResponse, error) {
	if len(in.TimeseriesTenantData) == 0 {
		panic("BUG: localAsyncWriter.RemoteWrite called without TimeseriesTenantData")
	}

	for _, ts := range in.TimeseriesTenantData {
		if err := lw.w.Write(ctx, ts.Tenant, ts.Timeseries); err != nil {
			return nil, errors.Wrap(err, "writing locally")
		}
	}

	return &storepb.WriteResponse{}, nil
}

func (p *peerGroup) getConnection(ctx context.Context, endpoint Endpoint) (WriteableStoreAsyncClient, error) {
	if !p.isPeerUp(endpoint) {
		return nil, errUnavailable
	}

	// use a RLock first to prevent blocking if we don't need to.
	p.m.RLock()
	c, ok := p.connections[endpoint]
	p.m.RUnlock()
	if ok {
		return c, nil
	}

View on GitHub (pinned to 35b8b99117)