{"record":{"id":"13103b1e26be823f","repo":"thanos-io/thanos","slug":"failed-to-dial-peer-13103b","errorCode":null,"errorMessage":"failed to dial peer","messagePattern":"failed to dial peer","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/receive/writecapnp/client.go","lineNumber":206,"sourceCode":"\n\t\treturn nil, 0, fmt.Errorf(\"rpc failed%s\", extraContext)\n\tcase WriteError_none:\n\t\treturn &storepb.WriteResponse{}, 0, nil\n\tdefault:\n\t\tpanic(\"BUG: unhandled WriteError\")\n\t}\n}\n\nfunc (r *RemoteWriteClient) connect(ctx context.Context) error {\n\tr.mu.Lock()\n\tdefer r.mu.Unlock()\n\tif r.conn != nil {\n\t\treturn nil\n\t}\n\n\tconn, err := r.dialer.DialContext(ctx)\n\tif err != nil {\n\t\treturn errors.Wrap(err, \"failed to dial peer\")\n\t}\n\tr.conn = rpc.NewConn(rpc.NewPackedStreamTransport(conn), nil)\n\twriter := Writer(r.conn.Bootstrap(ctx))\n\tif err := writer.Resolve(ctx); err != nil {\n\t\tlevel.Warn(r.logger).Log(\"msg\", \"failed to bootstrap capnp writer, closing connection\", \"err\", err)\n\t\tr.closeUnlocked()\n\t\treturn errors.Wrap(err, \"failed to bootstrap capnp writer\")\n\t}\n\n\tr.writer = writer\n\treturn nil\n}\n\nfunc (r *RemoteWriteClient) Close() error {\n\tr.mu.Lock()\n\tr.closeUnlocked()\n\tr.mu.Unlock()\n\treturn nil","sourceCodeStart":188,"sourceCodeEnd":224,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/receive/writecapnp/client.go#L188-L224","documentation":"RemoteWriteClient.connect wraps the dialer error when establishing the capnp connection fails. It is the connect-layer sibling of the TCPDialer error: the underlying DialContext error is preserved and wrapped with 'failed to dial peer'. connect is called from writeWithReconnect, so this surfaces when reconnecting for a write.","triggerScenarios":"writeWithReconnect detects r.conn == nil (or a prior write failed) and calls connect; the underlying TCP dial fails — peer down, bad address, DNS failure, or context cancellation.","commonSituations":"Peer restarted between writes; stale DNS after pod reschedule; network policy changes; context deadline exceeded while the peer is overloaded.","solutions":["Confirm the peer address and that the capnp receiver is up and listening","Check the wrapped cause (refused/timeout/DNS) to pick the right fix — DNS vs connectivity vs timeout","Increase the context deadline / write timeout if the peer is slow to accept","Add retry with backoff around RemoteWrite; the client reconnects automatically on the next attempt"],"exampleFix":"// before\nconn, err := d.DialContext(ctx, \"tcp\", \"old-pod-ip:19390\") // pod rescheduled\n// after\n// resolve via service DNS instead of a pinned pod IP\nconn, err := d.DialContext(ctx, \"tcp\", \"receive.service:19390\")","handlingStrategy":"retry","validationCode":"// verify reachability before triggering a reconnect:\nif _, err := net.DialTimeout(\"tcp\", addr, 2*time.Second); err != nil {\n    return fmt.Errorf(\"cannot reach peer %s, deferring write\", addr)\n}","typeGuard":"func isConnectDialError(err error) bool {\n    return strings.Contains(err.Error(), \"failed to dial peer\")\n}","tryCatchPattern":"_, err := client.RemoteWrite(ctx, in)\nif err != nil && strings.Contains(err.Error(), \"failed to dial peer\") {\n    if errors.Is(context.DeadlineExceeded, err) || errors.Is(context.Canceled, err) {\n        return err // do not retry canceled contexts\n    }\n    time.Sleep(backoff)\n    return client.RemoteWrite(ctx, in)\n}","preventionTips":["Use stable service endpoints instead of pod IPs","Set context deadlines longer than dial timeout","Retry with exponential backoff — the client reconnects on next write","Monitor DNS and NetworkPolicy changes affecting the peer"],"tags":["network","capnp","dial","reconnect"],"backgroundTag":"connection-refused","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}