thanos-io/thanos · warning

target not available

Error message

target not available

What it means

errUnavailable is the sentinel returned by Write and receiveOTLPHTTP when the target receiver is unavailable (unreachable or not accepting connections). The HTTP handler and capnp server both map it to an unavailable status (http.StatusServiceUnavailable / WriteError_unavailable). Like errNotReady it is transient and matched via errors.Cause.

Solutions

1. Verify the target endpoint is running and reachable (ping/DNS/port check on the given host:port)
2. Update the hashring configuration to remove dead endpoints and confirm all receivers use the same config
3. Retry the write; the receiver will re-route per the current hashring
4. Check network policies / service discovery between the receive nodes

Example fix

// before (hashring endpoint pointing at removed instance)
- endpoints: ['thanos-receive-9.thanos-receive:10901']
// after
- endpoints: ['thanos-receive-0.thanos-receive:10901']

conn, err := net.DialTimeout("tcp", targetAddr, 2*time.Second)
if err != nil { return fmt.Errorf("target %s unreachable before write: %w", targetAddr, err) }
conn.Close()

func isUnavailable(err error) bool {
    return errors.Is(errors.Cause(err), errUnavailable)
}

err := r.Write(ctx, req)
if err != nil && errors.Is(errors.Cause(err), errUnavailable) {
    // re-route via hashring or retry after checking target health
    return http.StatusServiceUnavailable
}

When it happens

Trigger: Write or receiveOTLPHTTP attempts to replicate/forward a write to a target that is down, unreachable, or refuses the connection; the resulting error's root cause is errUnavailable.

Common situations: A receiver pod is crash-looping or being restarted during a deploy, network partition between receive nodes, DNS/name resolution failure for a hashring endpoint, endpoint removed from the ring but still cached.

Understand the failure class

Background: ECONNREFUSED and "connection refused" / "could not connect to server" errors: what they mean and how to fix them — this error's family across 44 libraries.

Related errors

• target not ready
• backing off forward request for endpoint
• forwarding request to endpoint
• failed to dial peer
• target not available: failed to dial peer

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/04ca0c526e08a03a. Report an issue: GitHub.

Appendix: source

Thrown at pkg/receive/handler.go:93

	// Labels for metrics.
	labelSuccess = "success"
	labelError   = "error"
)

type ReplicationProtocol string

const (
	ProtobufReplication  ReplicationProtocol = "protobuf"
	CapNProtoReplication ReplicationProtocol = "capnproto"
)

var (
	// errConflict is returned whenever an operation fails due to any conflict-type error.
	errConflict = errors.New("conflict")

	errBadReplica  = errors.New("request replica exceeds receiver replication factor")
	errNotReady    = errors.New("target not ready")
	errUnavailable = errors.New("target not available")

	errValidation = errors.New("validation error")
)

type WriteableStoreAsyncClient interface {
	storepb.WriteableStoreClient
	RemoteWriteAsync(context.Context, *storepb.WriteRequest, endpointReplica, []int, chan writeResponse, func(error))
	// TryRemoteWriteAsync submits the request without blocking. Returns false if the peer's
	// worker pool is at capacity; the caller should fall back to RemoteWriteAsync.
	TryRemoteWriteAsync(context.Context, *storepb.WriteRequest, endpointReplica, []int, chan writeResponse, func(error)) bool
}

// Options for the web Handler.
type Options struct {
	Writer                  *Writer
	ListenAddress           string
	Registry                *prometheus.Registry
	TenantHeader            string

View on GitHub (pinned to 35b8b99117)