thanos-io/thanos · error

target not available: failed to dial peer

Error message

target not available: failed to dial peer: %v

What it means

This is the fully-wrapped error returned by peerGroup.getConnection when a Protobuf replication dial fails: the dial error is wrapped as "failed to dial peer" and then again with the errUnavailable prefix, yielding "target not available: failed to dial peer: <cause>". It tells callers of the receive handler that the target peer could not be reached, so the write could not be forwarded for replication. The failing endpoint is simultaneously marked unavailable so later calls fail fast with errUnavailable until a backoff re-allow.

Solutions

1. Check the root cause in the inner error (%v suffix) — usually connection refused/timeout — and fix the peer's availability first.
2. Validate and correct hashring endpoint addresses/ports in --receive.hashrings.
3. Confirm Kubernetes Services/NetworkPolicies allow the receiver to reach peer gRPC ports.
4. Align gRPC/TLS client dial options with the peer's server TLS configuration.
5. Retry the write; replication failures can be retried once the peer's backoff state recovers.

Example fix

// caller side: don't hard-fail one unreplicable peer, log and continue
if err != nil {
	level.Error(logger).Log("msg", "failed to forward to peer", "err", err)
	return // or retry with backoff
}

// Go: check inner cause before retrying
if err != nil {
	var netErr net.Error
	if errors.As(err, &netErr) && netErr.Timeout() {
		// retryable
	}
}

// Go: inspect the wrapped chain
if err != nil {
	if strings.Contains(err.Error(), "target not available") {
		// peer marked unavailable — schedule retry with backoff
	}
}

When it happens

Trigger: Produced only in the ProtobufReplication branch of getConnection: p.dialer(endpoint.Address, p.dialOpts...) returns an error, causing return nil, errors.Wrap(dialError, errUnavailable.Error()). Callers hitting this include the receive Handler's forwarding path during replication to a remote endpoint.

Common situations: Rolling restarts or node failures take a peer down mid-write; misconfigured hashring addresses; NetworkPolicies dropping gRPC traffic; the peer's TCP port not exposed by the service; TLS handshake mismatch between receiver dialer options and peer server options.

Understand the failure class

Background: ECONNREFUSED and "connection refused" / "could not connect to server" errors: what they mean and how to fix them — this error's family across 44 libraries.

Related errors

• forwarding request to endpoint
• failed to dial peer
• no query API server reachable
• failed to get tsdb status from prometheus
• retrieving targets

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/af3e67f641a5139e. Report an issue: GitHub.

Appendix: source

Thrown at pkg/receive/handler.go:1982

	p.conns.Inc()

	var client peerClient
	if isLocalEndpoint(endpoint, p.localEndpoint) {
		client = &localAsyncWriter{
			w: p.writer,
		}
	} else {
		switch p.replicationProtocol {
		case CapNProtoReplication:
			client = writecapnp.NewRemoteWriteClient(writecapnp.NewTCPDialer(endpoint.CapNProtoAddress), p.logger)

		case ProtobufReplication:
			conn, err := p.dialer(endpoint.Address, p.dialOpts...)
			if err != nil {
				p.markPeerUnavailableUnlocked(endpoint)
				dialError := errors.Wrap(err, "failed to dial peer")
				return nil, errors.Wrap(dialError, errUnavailable.Error())
			}
			client = newProtobufPeer(conn)
		default:
			return nil, errors.Errorf("unknown replication protocol %v", p.replicationProtocol)
		}
	}

	var delay time.Duration
	if p.conns.Load() == 2 {
		delay = p.maxArtificialDelay
	}

	p.connections[endpoint] = newPeerWorker(client, p.forwardDelay.WithLabelValues(endpoint.Address), p.asyncForwardWorkersCount, delay)
	return p.connections[endpoint], nil
}

func (p *peerGroup) markPeerUnavailable(addr Endpoint) {
	p.m.Lock()

View on GitHub (pinned to 35b8b99117)