{"record":{"id":"5055f22194d4cff4","repo":"thanos-io/thanos","slug":"failed-to-dial-peer","errorCode":null,"errorMessage":"failed to dial peer","messagePattern":"failed to dial peer","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/receive/handler.go","lineNumber":1981,"sourceCode":"\t}\n\n\tp.conns.Inc()\n\n\tvar client peerClient\n\tif isLocalEndpoint(endpoint, p.localEndpoint) {\n\t\tclient = &localAsyncWriter{\n\t\t\tw: p.writer,\n\t\t}\n\t} else {\n\t\tswitch p.replicationProtocol {\n\t\tcase CapNProtoReplication:\n\t\t\tclient = writecapnp.NewRemoteWriteClient(writecapnp.NewTCPDialer(endpoint.CapNProtoAddress), p.logger)\n\n\t\tcase ProtobufReplication:\n\t\t\tconn, err := p.dialer(endpoint.Address, p.dialOpts...)\n\t\t\tif err != nil {\n\t\t\t\tp.markPeerUnavailableUnlocked(endpoint)\n\t\t\t\tdialError := errors.Wrap(err, \"failed to dial peer\")\n\t\t\t\treturn nil, errors.Wrap(dialError, errUnavailable.Error())\n\t\t\t}\n\t\t\tclient = newProtobufPeer(conn)\n\t\tdefault:\n\t\t\treturn nil, errors.Errorf(\"unknown replication protocol %v\", p.replicationProtocol)\n\t\t}\n\t}\n\n\tvar delay time.Duration\n\tif p.conns.Load() == 2 {\n\t\tdelay = p.maxArtificialDelay\n\t}\n\n\tp.connections[endpoint] = newPeerWorker(client, p.forwardDelay.WithLabelValues(endpoint.Address), p.asyncForwardWorkersCount, delay)\n\treturn p.connections[endpoint], nil\n}\n\nfunc (p *peerGroup) markPeerUnavailable(addr Endpoint) {","sourceCodeStart":1963,"sourceCodeEnd":1999,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/receive/handler.go#L1963-L1999","documentation":"This error is produced in Thanos receive's peerGroup.getConnection when establishing a Protobuf replication (gRPC-style) connection to a remote receive peer fails. The underlying dial error is wrapped with pkg/errors.Wrap as \"failed to dial peer\" and returned to the caller so the querier/receiver knows the peer connection could not be created. It is a transient infrastructure error: the peer endpoint is also marked unavailable so subsequent attempts short-circuit with errUnavailable until the retry backoff expires.","triggerScenarios":"Occurs when p.dialer(endpoint.Address, p.dialOpts...) returns a non-nil error while creating a connection for a non-local endpoint under the ProtobufReplication protocol — i.e. the TCP/gRPC connection to the peer's address could not be established (unreachable host, refused port, TLS failure, timeout).","commonSituations":"Peer receive pod is down or restarting in Kubernetes; --receive-split or replication hashring config points at a stale IP/hostname; wrong port in the hashring's address; network policy or firewall blocks gRPC traffic; DNS resolution failure; peer uses TLS/mTLS but the dialer lacks matching credentials.","solutions":["Verify the peer endpoint is running and listening: kubectl get pods / curl or nc to the peer's address:port.","Check the hashring configuration (--receive.hashrings) for incorrect addresses or ports and correct them.","Inspect network policy, firewall, and DNS so the receiver can reach peer addresses on the gRPC port.","Check TLS settings: if the peer requires mTLS, ensure the dialer options (--receive.grpc-* client TLS flags) match the server cert.","Retry after the peer backoff clears; the endpoint is marked unavailable and reconnection is attempted later."],"exampleFix":"// before: hashring entry with wrong port\n{\"endpoints\": [\"receive-0.receive.default.svc:9999\"]}\n// after: correct gRPC port\n{\"endpoints\": [\"receive-0.receive.default.svc:10901\"]}","handlingStrategy":"retry","validationCode":"// Go: probe reachability before dialing\nconn, err := net.DialTimeout(\"tcp\", endpoint.Address, 3*time.Second)\nif err != nil {\n\treturn nil, fmt.Errorf(\"peer %s unreachable: %w\", endpoint.Address, err)\n}\nconn.Close()","typeGuard":null,"tryCatchPattern":"// Go\nclient, err := peerGroup.getConnection(ctx, endpoint)\nif err != nil {\n\tif errors.Is(err, errUnavailable) || strings.Contains(err.Error(), \"failed to dial peer\") {\n\t\t// backoff and retry; peer is marked unavailable\n\t\tselect {\n\t\tcase <-time.After(backoff):\n\t\t\treturn p.getConnection(ctx, endpoint)\n\t\tcase <-ctx.Done():\n\t\t\treturn nil, ctx.Err()\n\t\t}\n\t}\n\treturn nil, err\n}","preventionTips":["Monitor peer endpoint health with readiness probes so dead peers are removed from the hashring.","Pin hashring addresses to stable DNS names (Service FQDNs), not pod IPs.","Test network policy/firewall rules between receiver pods in staging.","Keep TLS flags symmetric across receiver peers."],"tags":["network","grpc","dial","thanos-receive","replication"],"backgroundTag":"connection-refused","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}