thanos-io/thanos · error

failed to dial peer

Error message

failed to dial peer %s

What it means

TCPDialer.DialContext wraps the net.Dialer error when establishing the TCP connection to the capnp remote-write peer fails, adding the peer address for diagnosis. The connection never opened, so no capnp RPC was attempted.

Solutions

1. Verify the configured address (host:port) is correct and the peer is listening (nc/teleport test)
2. Check DNS resolution and Kubernetes service/endpoints
3. Inspect network policies/firewalls between sender and receiver
4. Increase dial timeout or check the context deadline; examine the wrapped underlying cause (connection refused vs timeout vs no such host)

Example fix

// before
client := writecapnp.NewRemoteWriteClient(...address: "store-gw:19391"...) // typo in port
// after
// use the correct capnp receive port from the peer's config
client := writecapnp.NewRemoteWriteClient(...address: "store-gw:19390"...)

// pre-flight connectivity check before dialing through the client:
conn, err := net.DialTimeout("tcp", host, 2*time.Second)
if err != nil {
    return fmt.Errorf("peer %s unreachable before write: %w", host, err)
}
conn.Close()

func isDialError(err error) bool {
    return strings.Contains(err.Error(), "failed to dial peer")
}

_, err := client.RemoteWrite(ctx, in)
if err != nil && strings.Contains(err.Error(), "failed to dial peer") {
    if nerr, ok := err.(net.Error); ok && nerr.Timeout() { /* backoff longer */ }
    return retryWithBackoff(ctx, writeFn)
}

When it happens

Trigger: Any call chain reaching RemoteWriteClient that must (re)connect: the target address is unreachable, refuses connections, DNS fails, or the context is canceled/deadline exceeded during dial.

Common situations: Wrong store-gateway/receive address in config; peer down or restarted; firewall or NetworkPolicy blocking the port; DNS misconfiguration in Kubernetes; dial timeout too small.

Understand the failure class

Background: ECONNREFUSED and "connection refused" / "could not connect to server" errors: what they mean and how to fix them — this error's family across 44 libraries.

Related errors

• failed to dial peer
• dialing connection
• failed to dial peer
• failed writing to peer
• failed to bootstrap capnp writer

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/bbe378a3ec3662b6. Report an issue: GitHub.

Appendix: source

Thrown at pkg/receive/writecapnp/client.go:41

)

type Dialer interface {
	DialContext(ctx context.Context) (net.Conn, error)
}

type TCPDialer struct {
	address string
}

func NewTCPDialer(address string) *TCPDialer {
	return &TCPDialer{address: address}
}

func (t TCPDialer) DialContext(ctx context.Context) (net.Conn, error) {
	var d net.Dialer
	conn, err := d.DialContext(ctx, "tcp", t.address)
	if err != nil {
		return nil, errors.Wrapf(err, "failed to dial peer %s", t.address)
	}
	return conn, nil
}

type RemoteWriteClient struct {
	mu sync.Mutex

	dialer Dialer
	conn   *rpc.Conn

	writer Writer
	logger log.Logger
}

func NewRemoteWriteClient(dialer Dialer, logger log.Logger) *RemoteWriteClient {
	return &RemoteWriteClient{
		dialer: dialer,
		logger: logger,

View on GitHub (pinned to 35b8b99117)