{"record":{"id":"08838403df18fb06","repo":"thanos-io/thanos","slug":"failed-writing-to-peer","errorCode":null,"errorMessage":"failed writing to peer","messagePattern":"failed writing to peer","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/receive/writecapnp/client.go","lineNumber":160,"sourceCode":"\t\tif err := params.SetWr(wr); err != nil {\n\t\t\treturn err\n\t\t}\n\n\t\treturn nil\n\t})\n\tdefer release()\n\n\ts, err := result.Struct()\n\tif err != nil {\n\t\tif numReconnects > 0 && capnp.IsDisconnected(err) {\n\t\t\tlevel.Warn(r.logger).Log(\"msg\", \"rpc failed, reconnecting\")\n\t\t\tif err := r.Close(); err != nil {\n\t\t\t\treturn nil, 0, err\n\t\t\t}\n\t\t\tnumReconnects--\n\t\t\treturn r.writeWithReconnect(ctx, numReconnects, in)\n\t\t}\n\t\treturn nil, 0, errors.Wrap(err, \"failed writing to peer\")\n\t}\n\tswitch s.Error() {\n\tcase WriteError_unavailable:\n\t\treturn nil, WriteError_unavailable, nil\n\tcase WriteError_alreadyExists:\n\t\treturn nil, WriteError_alreadyExists, nil\n\tcase WriteError_invalidArgument:\n\t\treturn nil, WriteError_invalidArgument, nil\n\tcase WriteError_internal:\n\t\textraContext, err := s.ExtraErrorContext()\n\t\tif err != nil {\n\t\t\tif numReconnects > 0 && capnp.IsDisconnected(err) {\n\t\t\t\tlevel.Warn(r.logger).Log(\"msg\", \"rpc failed, reconnecting\")\n\t\t\t\tif err := r.Close(); err != nil {\n\t\t\t\t\treturn nil, 0, err\n\t\t\t\t}\n\t\t\t\tnumReconnects--\n\t\t\t\treturn r.writeWithReconnect(ctx, numReconnects, in)","sourceCodeStart":142,"sourceCodeEnd":178,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/receive/writecapnp/client.go#L142-L178","documentation":"During writeWithReconnect, the capnp RPC call returned a transport-level error (not a structured WriteError), so the client wraps it as 'failed writing to peer'. This indicates the connection broke mid-write or the RPC transport failed, distinct from an application-level WriteError like unavailable.","triggerScenarios":"writeWithReconnect invokes the capnp Write RPC and the underlying transport returns err != nil with no structured response; numReconnects is exhausted so no retry happens, and the raw error is wrapped.","commonSituations":"Peer closed the TCP connection mid-request (restart, OOM kill); network partition during large batch writes; idle connection reaped by a load balancer; capnp protocol mismatch between versions.","solutions":["Retry the write — the client already reconnects; if this surfaces, reconnects were exhausted, so check peer health during the write window","Enable keep-alives or reduce write batch size/idle time to avoid connection reaping","Check peer logs for restarts/crashes at the timestamp","Align capnp protocol versions between sender and receiver"],"exampleFix":"// before\nresp, err := client.RemoteWrite(ctx, in) // err: failed writing to peer\n// after\nresp, err := client.RemoteWrite(ctx, in)\nif err != nil && errors.Is(err, capnpErrDisconnected) {\n    time.Sleep(backoff); continue // retry with fresh connection\n}","handlingStrategy":"retry","validationCode":null,"typeGuard":"func isPeerWriteFailure(err error) bool {\n    return strings.Contains(err.Error(), \"failed writing to peer\")\n}","tryCatchPattern":"resp, werr, err := client.RemoteWrite(ctx, in)\nif err != nil {\n    if strings.Contains(err.Error(), \"failed writing to peer\") {\n        select {\n        case <-time.After(backoff):\n            return client.RemoteWrite(ctx, in) // fresh connection on retry\n        case <-ctx.Done():\n            return ctx.Err()\n        }\n    }\n    return err\n}","preventionTips":["Enable TCP keep-alives to avoid half-open connections","Keep write batches modest so connections don't idle","Monitor peer restarts and OOM kills","Retry idempotent writes with backoff"],"tags":["network","capnp","rpc","remote-write"],"backgroundTag":"broken-pipe","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}