thanos-io/thanos · error

rpc failed

Error message

rpc failed%s

What it means

When the capnp Write response reports an application error, writeWithReconnect formats 'rpc failed' followed by extraContext (the peer-provided error text or ' (no additional context provided)'). This is a structured, peer-reported RPC failure rather than a transport break.

Solutions

1. Read the extraContext suffix in the message — it states the peer's own failure reason
2. Check the receiving store/receive node logs for the matching error
3. Retry if the underlying cause is transient (e.g. unavailable is already soft-handled; other errors may be too)
4. Upgrade both sides if the WriteError enum may be newer on the peer

Example fix

// before
// error: rpc failed: storage full
// after
// act on context: free storage on the receiver, then retry RemoteWrite
if strings.Contains(err.Error(), "storage") { triggerReceiverMaintenance() }

func isRPCFailed(err error) bool {
    return strings.Contains(err.Error(), "rpc failed")
}

_, werr, err := client.RemoteWrite(ctx, in)
switch {
case err != nil && strings.Contains(err.Error(), "rpc failed"):
    ctx := extractExtraContext(err.Error()) // peer's stated reason
    log.Error("peer rejected write", "context", ctx)
    return handlePeerReportedFailure(ctx)
case werr == writecapnp.WriteError_unavailable:
    return retryLater()
}

When it happens

Trigger: RemoteWrite RPC succeeds at the transport level but the response's Error field is a WriteError value other than unavailable/alreadyExists/none, and the peer attached context text (or none).

Common situations: Receiver-side failures during ingestion (storage errors, tenant limits) reported with context; protocol mismatch producing an unrecognized WriteError enum; receiver returning an unnamed internal error.

Related errors

• failed writing to peer
• failed to bootstrap capnp writer
• failed to parse remote write config
• applying config to remote storage
• start remote write agent db

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/cf29cd9972299ec8. Report an issue: GitHub.

Appendix: source

Thrown at pkg/receive/writecapnp/client.go:189

		if err != nil {
			if numReconnects > 0 && capnp.IsDisconnected(err) {
				level.Warn(r.logger).Log("msg", "rpc failed, reconnecting")
				if err := r.Close(); err != nil {
					return nil, 0, err
				}
				numReconnects--
				return r.writeWithReconnect(ctx, numReconnects, in)
			}
			return nil, 0, errors.Wrap(err, "failed writing to peer")
		}

		if extraContext == "" {
			extraContext = " (no additional context provided)"
		} else {
			extraContext = ": " + extraContext
		}

		return nil, 0, fmt.Errorf("rpc failed%s", extraContext)
	case WriteError_none:
		return &storepb.WriteResponse{}, 0, nil
	default:
		panic("BUG: unhandled WriteError")
	}
}

func (r *RemoteWriteClient) connect(ctx context.Context) error {
	r.mu.Lock()
	defer r.mu.Unlock()
	if r.conn != nil {
		return nil
	}

	conn, err := r.dialer.DialContext(ctx)
	if err != nil {
		return errors.Wrap(err, "failed to dial peer")
	}

View on GitHub (pinned to 35b8b99117)