What causes "failed to dial peer" in thanos-io/thanos?

Confirm the peer address and that the capnp receiver is up and listening

failed to dial peer

Error message

failed to dial peer

What it means

RemoteWriteClient.connect wraps the dialer error when establishing the capnp connection fails. It is the connect-layer sibling of the TCPDialer error: the underlying DialContext error is preserved and wrapped with 'failed to dial peer'. connect is called from writeWithReconnect, so this surfaces when reconnecting for a write.

Solutions

Confirm the peer address and that the capnp receiver is up and listening
Check the wrapped cause (refused/timeout/DNS) to pick the right fix — DNS vs connectivity vs timeout
Increase the context deadline / write timeout if the peer is slow to accept
Add retry with backoff around RemoteWrite; the client reconnects automatically on the next attempt

Example fix

// before
conn, err := d.DialContext(ctx, "tcp", "old-pod-ip:19390") // pod rescheduled
// after
// resolve via service DNS instead of a pinned pod IP
conn, err := d.DialContext(ctx, "tcp", "receive.service:19390")

Defensive patterns

Strategy: retry

Validate before calling

// verify reachability before triggering a reconnect:
if _, err := net.DialTimeout("tcp", addr, 2*time.Second); err != nil {
    return fmt.Errorf("cannot reach peer %s, deferring write", addr)
}

Type guard

func isConnectDialError(err error) bool {
    return strings.Contains(err.Error(), "failed to dial peer")
}

Try / catch

_, err := client.RemoteWrite(ctx, in)
if err != nil && strings.Contains(err.Error(), "failed to dial peer") {
    if errors.Is(context.DeadlineExceeded, err) || errors.Is(context.Canceled, err) {
        return err // do not retry canceled contexts
    }
    time.Sleep(backoff)
    return client.RemoteWrite(ctx, in)
}

Prevention

Use stable service endpoints instead of pod IPs
Set context deadlines longer than dial timeout
Retry with exponential backoff — the client reconnects on next write
Monitor DNS and NetworkPolicy changes affecting the peer

When it happens

Trigger: writeWithReconnect detects r.conn == nil (or a prior write failed) and calls connect; the underlying TCP dial fails — peer down, bad address, DNS failure, or context cancellation.

Common situations: Peer restarted between writes; stale DNS after pod reschedule; network policy changes; context deadline exceeded while the peer is overloaded.

Understand the failure class

Background: ECONNREFUSED and "connection refused" / "could not connect to server" errors: what they mean and how to fix them — this error's family across 44 libraries.

Related errors

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/13103b1e26be823f. Report an issue: GitHub.

Appendix: source

Thrown at pkg/receive/writecapnp/client.go:206

		return nil, 0, fmt.Errorf("rpc failed%s", extraContext)
	case WriteError_none:
		return &storepb.WriteResponse{}, 0, nil
	default:
		panic("BUG: unhandled WriteError")
	}
}

func (r *RemoteWriteClient) connect(ctx context.Context) error {
	r.mu.Lock()
	defer r.mu.Unlock()
	if r.conn != nil {
		return nil
	}

	conn, err := r.dialer.DialContext(ctx)
	if err != nil {
		return errors.Wrap(err, "failed to dial peer")
	}
	r.conn = rpc.NewConn(rpc.NewPackedStreamTransport(conn), nil)
	writer := Writer(r.conn.Bootstrap(ctx))
	if err := writer.Resolve(ctx); err != nil {
		level.Warn(r.logger).Log("msg", "failed to bootstrap capnp writer, closing connection", "err", err)
		r.closeUnlocked()
		return errors.Wrap(err, "failed to bootstrap capnp writer")
	}

	r.writer = writer
	return nil
}

func (r *RemoteWriteClient) Close() error {
	r.mu.Lock()
	r.closeUnlocked()
	r.mu.Unlock()
	return nil

View on GitHub (pinned to 35b8b99117)