{"record":{"id":"cf29cd9972299ec8","repo":"thanos-io/thanos","slug":"rpc-failed-s","errorCode":null,"errorMessage":"rpc failed%s","messagePattern":"rpc failed(.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/receive/writecapnp/client.go","lineNumber":189,"sourceCode":"\t\tif err != nil {\n\t\t\tif numReconnects > 0 && capnp.IsDisconnected(err) {\n\t\t\t\tlevel.Warn(r.logger).Log(\"msg\", \"rpc failed, reconnecting\")\n\t\t\t\tif err := r.Close(); err != nil {\n\t\t\t\t\treturn nil, 0, err\n\t\t\t\t}\n\t\t\t\tnumReconnects--\n\t\t\t\treturn r.writeWithReconnect(ctx, numReconnects, in)\n\t\t\t}\n\t\t\treturn nil, 0, errors.Wrap(err, \"failed writing to peer\")\n\t\t}\n\n\t\tif extraContext == \"\" {\n\t\t\textraContext = \" (no additional context provided)\"\n\t\t} else {\n\t\t\textraContext = \": \" + extraContext\n\t\t}\n\n\t\treturn nil, 0, fmt.Errorf(\"rpc failed%s\", extraContext)\n\tcase WriteError_none:\n\t\treturn &storepb.WriteResponse{}, 0, nil\n\tdefault:\n\t\tpanic(\"BUG: unhandled WriteError\")\n\t}\n}\n\nfunc (r *RemoteWriteClient) connect(ctx context.Context) error {\n\tr.mu.Lock()\n\tdefer r.mu.Unlock()\n\tif r.conn != nil {\n\t\treturn nil\n\t}\n\n\tconn, err := r.dialer.DialContext(ctx)\n\tif err != nil {\n\t\treturn errors.Wrap(err, \"failed to dial peer\")\n\t}","sourceCodeStart":171,"sourceCodeEnd":207,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/receive/writecapnp/client.go#L171-L207","documentation":"When the capnp Write response reports an application error, writeWithReconnect formats 'rpc failed' followed by extraContext (the peer-provided error text or ' (no additional context provided)'). This is a structured, peer-reported RPC failure rather than a transport break.","triggerScenarios":"RemoteWrite RPC succeeds at the transport level but the response's Error field is a WriteError value other than unavailable/alreadyExists/none, and the peer attached context text (or none).","commonSituations":"Receiver-side failures during ingestion (storage errors, tenant limits) reported with context; protocol mismatch producing an unrecognized WriteError enum; receiver returning an unnamed internal error.","solutions":["Read the extraContext suffix in the message — it states the peer's own failure reason","Check the receiving store/receive node logs for the matching error","Retry if the underlying cause is transient (e.g. unavailable is already soft-handled; other errors may be too)","Upgrade both sides if the WriteError enum may be newer on the peer"],"exampleFix":"// before\n// error: rpc failed: storage full\n// after\n// act on context: free storage on the receiver, then retry RemoteWrite\nif strings.Contains(err.Error(), \"storage\") { triggerReceiverMaintenance() }","handlingStrategy":"try-catch","validationCode":null,"typeGuard":"func isRPCFailed(err error) bool {\n    return strings.Contains(err.Error(), \"rpc failed\")\n}","tryCatchPattern":"_, werr, err := client.RemoteWrite(ctx, in)\nswitch {\ncase err != nil && strings.Contains(err.Error(), \"rpc failed\"):\n    ctx := extractExtraContext(err.Error()) // peer's stated reason\n    log.Error(\"peer rejected write\", \"context\", ctx)\n    return handlePeerReportedFailure(ctx)\ncase werr == writecapnp.WriteError_unavailable:\n    return retryLater()\n}","preventionTips":["Always parse the extraContext suffix — it names the peer-side cause","Alert on receiver-side storage/limit conditions","Keep WriteError enum definitions in sync across versions"],"tags":["capnp","rpc","remote-write","peer-error"],"backgroundTag":"rpc-call-failed","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}