thanos-io/thanos · error

failed to dial peer

Error message

failed to dial peer

What it means

This error is produced in Thanos receive's peerGroup.getConnection when establishing a Protobuf replication (gRPC-style) connection to a remote receive peer fails. The underlying dial error is wrapped with pkg/errors.Wrap as "failed to dial peer" and returned to the caller so the querier/receiver knows the peer connection could not be created. It is a transient infrastructure error: the peer endpoint is also marked unavailable so subsequent attempts short-circuit with errUnavailable until the retry backoff expires.

Solutions

1. Verify the peer endpoint is running and listening: kubectl get pods / curl or nc to the peer's address:port.
2. Check the hashring configuration (--receive.hashrings) for incorrect addresses or ports and correct them.
3. Inspect network policy, firewall, and DNS so the receiver can reach peer addresses on the gRPC port.
4. Check TLS settings: if the peer requires mTLS, ensure the dialer options (--receive.grpc-* client TLS flags) match the server cert.
5. Retry after the peer backoff clears; the endpoint is marked unavailable and reconnection is attempted later.

Example fix

// before: hashring entry with wrong port
{"endpoints": ["receive-0.receive.default.svc:9999"]}
// after: correct gRPC port
{"endpoints": ["receive-0.receive.default.svc:10901"]}

// Go: probe reachability before dialing
conn, err := net.DialTimeout("tcp", endpoint.Address, 3*time.Second)
if err != nil {
	return nil, fmt.Errorf("peer %s unreachable: %w", endpoint.Address, err)
}
conn.Close()

// Go
client, err := peerGroup.getConnection(ctx, endpoint)
if err != nil {
	if errors.Is(err, errUnavailable) || strings.Contains(err.Error(), "failed to dial peer") {
		// backoff and retry; peer is marked unavailable
		select {
		case <-time.After(backoff):
			return p.getConnection(ctx, endpoint)
		case <-ctx.Done():
			return nil, ctx.Err()
		}
	}
	return nil, err
}

When it happens

Trigger: Occurs when p.dialer(endpoint.Address, p.dialOpts...) returns a non-nil error while creating a connection for a non-local endpoint under the ProtobufReplication protocol — i.e. the TCP/gRPC connection to the peer's address could not be established (unreachable host, refused port, TLS failure, timeout).

Common situations: Peer receive pod is down or restarting in Kubernetes; --receive-split or replication hashring config points at a stale IP/hostname; wrong port in the hashring's address; network policy or firewall blocks gRPC traffic; DNS resolution failure; peer uses TLS/mTLS but the dialer lacks matching credentials.

Understand the failure class

Background: ECONNREFUSED and "connection refused" / "could not connect to server" errors: what they mean and how to fix them — this error's family across 44 libraries.

Related errors

• forwarding request to endpoint
• target not available: failed to dial peer
• dialing connection
• no query API server reachable
• failed to get tsdb status from prometheus

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/5055f22194d4cff4. Report an issue: GitHub.

Appendix: source

Thrown at pkg/receive/handler.go:1981

	}

	p.conns.Inc()

	var client peerClient
	if isLocalEndpoint(endpoint, p.localEndpoint) {
		client = &localAsyncWriter{
			w: p.writer,
		}
	} else {
		switch p.replicationProtocol {
		case CapNProtoReplication:
			client = writecapnp.NewRemoteWriteClient(writecapnp.NewTCPDialer(endpoint.CapNProtoAddress), p.logger)

		case ProtobufReplication:
			conn, err := p.dialer(endpoint.Address, p.dialOpts...)
			if err != nil {
				p.markPeerUnavailableUnlocked(endpoint)
				dialError := errors.Wrap(err, "failed to dial peer")
				return nil, errors.Wrap(dialError, errUnavailable.Error())
			}
			client = newProtobufPeer(conn)
		default:
			return nil, errors.Errorf("unknown replication protocol %v", p.replicationProtocol)
		}
	}

	var delay time.Duration
	if p.conns.Load() == 2 {
		delay = p.maxArtificialDelay
	}

	p.connections[endpoint] = newPeerWorker(client, p.forwardDelay.WithLabelValues(endpoint.Address), p.asyncForwardWorkersCount, delay)
	return p.connections[endpoint], nil
}

func (p *peerGroup) markPeerUnavailable(addr Endpoint) {

View on GitHub (pinned to 35b8b99117)