What causes "target not ready" in thanos-io/thanos?

Retry the write with backoff; the condition is transient by design

target not ready

Error message

target not ready

What it means

errNotReady is the sentinel returned by Write and receiveOTLPHTTP when the target receiver is not ready to accept writes (e.g. it is still starting up or refreshing its hashring). The HTTP handler maps it to http.StatusServiceUnavailable (503) and capnp maps it to WriteError_unavailable. Clients are expected to retry, ideally honoring Retry-After once supported.

Solutions

Retry the write with backoff; the condition is transient by design
Wait for the receiver to report ready before routing traffic (readiness probes / load balancer health checks)
Check why the target is not ready: hashring file load failures or slow startup in its logs
Harden the sender with bounded retries on 503 so temporary unavailability doesn't drop samples

Example fix

// before
err := r.Write(ctx, req)
// after
err := r.Write(ctx, req)
if err != nil && errors.Is(errors.Cause(err), errNotReady) {
    time.Sleep(backoff); goto retry // bounded retry loop
}

Defensive patterns

Strategy: retry

Type guard

func isNotReady(err error) bool {
    return errors.Is(errors.Cause(err), errNotReady)
}

Try / catch

err := r.Write(ctx, req)
if err != nil && errors.Is(errors.Cause(err), errNotReady) {
    select {
    case <-time.After(backoff):
        // retry with jitter, bounded attempts
    case <-ctx.Done():
        return ctx.Err()
    }
}

Prevention

Gate traffic on receiver readiness probes before routing writes
Wait for hashring loading to complete at startup before accepting writes
Use bounded exponential backoff with jitter for transient unavailability
Scale receivers ahead of traffic surges so they are ready when needed

When it happens

Trigger: Write or receiveOTLPHTTP forwards a write to a target whose receiver responds/behaves as not ready — the root cause of the returned error is errNotReady per errors.Cause.

Common situations: Writes hitting receivers during startup before the hashring is loaded, a receiver that hasn't joined the ring yet during scaling events, load balancers routing to pods not yet fully initialized.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

Related errors

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/dbf02f4a265f6e6b. Report an issue: GitHub.

Appendix: source

Thrown at pkg/receive/handler.go:92

	LimitStatsQueryParam = "limit"
	// Labels for metrics.
	labelSuccess = "success"
	labelError   = "error"
)

type ReplicationProtocol string

const (
	ProtobufReplication  ReplicationProtocol = "protobuf"
	CapNProtoReplication ReplicationProtocol = "capnproto"
)

var (
	// errConflict is returned whenever an operation fails due to any conflict-type error.
	errConflict = errors.New("conflict")

	errBadReplica  = errors.New("request replica exceeds receiver replication factor")
	errNotReady    = errors.New("target not ready")
	errUnavailable = errors.New("target not available")

	errValidation = errors.New("validation error")
)

type WriteableStoreAsyncClient interface {
	storepb.WriteableStoreClient
	RemoteWriteAsync(context.Context, *storepb.WriteRequest, endpointReplica, []int, chan writeResponse, func(error))
	// TryRemoteWriteAsync submits the request without blocking. Returns false if the peer's
	// worker pool is at capacity; the caller should fall back to RemoteWriteAsync.
	TryRemoteWriteAsync(context.Context, *storepb.WriteRequest, endpointReplica, []int, chan writeResponse, func(error)) bool
}

// Options for the web Handler.
type Options struct {
	Writer                  *Writer
	ListenAddress           string
	Registry                *prometheus.Registry

View on GitHub (pinned to 35b8b99117)