What causes "failed writing to peer" in thanos-io/thanos?

Retry the write — the client already reconnects; if this surfaces, reconnects were exhausted, so check peer health during the write window

thanos-io/thanos · error

failed writing to peer

Error message

failed writing to peer

What it means

During writeWithReconnect, the capnp RPC call returned a transport-level error (not a structured WriteError), so the client wraps it as 'failed writing to peer'. This indicates the connection broke mid-write or the RPC transport failed, distinct from an application-level WriteError like unavailable.

Solutions

Retry the write — the client already reconnects; if this surfaces, reconnects were exhausted, so check peer health during the write window
Enable keep-alives or reduce write batch size/idle time to avoid connection reaping
Check peer logs for restarts/crashes at the timestamp
Align capnp protocol versions between sender and receiver

Example fix

// before
resp, err := client.RemoteWrite(ctx, in) // err: failed writing to peer
// after
resp, err := client.RemoteWrite(ctx, in)
if err != nil && errors.Is(err, capnpErrDisconnected) {
    time.Sleep(backoff); continue // retry with fresh connection
}

Defensive patterns

Strategy: retry

Type guard

func isPeerWriteFailure(err error) bool {
    return strings.Contains(err.Error(), "failed writing to peer")
}

Try / catch

resp, werr, err := client.RemoteWrite(ctx, in)
if err != nil {
    if strings.Contains(err.Error(), "failed writing to peer") {
        select {
        case <-time.After(backoff):
            return client.RemoteWrite(ctx, in) // fresh connection on retry
        case <-ctx.Done():
            return ctx.Err()
        }
    }
    return err
}

Prevention

Enable TCP keep-alives to avoid half-open connections
Keep write batches modest so connections don't idle
Monitor peer restarts and OOM kills
Retry idempotent writes with backoff

When it happens

Trigger: writeWithReconnect invokes the capnp Write RPC and the underlying transport returns err != nil with no structured response; numReconnects is exhausted so no retry happens, and the raw error is wrapped.

Common situations: Peer closed the TCP connection mid-request (restart, OOM kill); network partition during large batch writes; idle connection reaped by a load balancer; capnp protocol mismatch between versions.

Related errors

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/08838403df18fb06. Report an issue: GitHub.

Appendix: source

Thrown at pkg/receive/writecapnp/client.go:160

		if err := params.SetWr(wr); err != nil {
			return err
		}

		return nil
	})
	defer release()

	s, err := result.Struct()
	if err != nil {
		if numReconnects > 0 && capnp.IsDisconnected(err) {
			level.Warn(r.logger).Log("msg", "rpc failed, reconnecting")
			if err := r.Close(); err != nil {
				return nil, 0, err
			}
			numReconnects--
			return r.writeWithReconnect(ctx, numReconnects, in)
		}
		return nil, 0, errors.Wrap(err, "failed writing to peer")
	}
	switch s.Error() {
	case WriteError_unavailable:
		return nil, WriteError_unavailable, nil
	case WriteError_alreadyExists:
		return nil, WriteError_alreadyExists, nil
	case WriteError_invalidArgument:
		return nil, WriteError_invalidArgument, nil
	case WriteError_internal:
		extraContext, err := s.ExtraErrorContext()
		if err != nil {
			if numReconnects > 0 && capnp.IsDisconnected(err) {
				level.Warn(r.logger).Log("msg", "rpc failed, reconnecting")
				if err := r.Close(); err != nil {
					return nil, 0, err
				}
				numReconnects--
				return r.writeWithReconnect(ctx, numReconnects, in)

View on GitHub (pinned to 35b8b99117)