{"record":{"id":"af3e67f641a5139e","repo":"thanos-io/thanos","slug":"target-not-available-failed-to-dial-peer-v","errorCode":null,"errorMessage":"target not available: failed to dial peer: %v","messagePattern":"target not available: failed to dial peer: (.+?)","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/receive/handler.go","lineNumber":1982,"sourceCode":"\n\tp.conns.Inc()\n\n\tvar client peerClient\n\tif isLocalEndpoint(endpoint, p.localEndpoint) {\n\t\tclient = &localAsyncWriter{\n\t\t\tw: p.writer,\n\t\t}\n\t} else {\n\t\tswitch p.replicationProtocol {\n\t\tcase CapNProtoReplication:\n\t\t\tclient = writecapnp.NewRemoteWriteClient(writecapnp.NewTCPDialer(endpoint.CapNProtoAddress), p.logger)\n\n\t\tcase ProtobufReplication:\n\t\t\tconn, err := p.dialer(endpoint.Address, p.dialOpts...)\n\t\t\tif err != nil {\n\t\t\t\tp.markPeerUnavailableUnlocked(endpoint)\n\t\t\t\tdialError := errors.Wrap(err, \"failed to dial peer\")\n\t\t\t\treturn nil, errors.Wrap(dialError, errUnavailable.Error())\n\t\t\t}\n\t\t\tclient = newProtobufPeer(conn)\n\t\tdefault:\n\t\t\treturn nil, errors.Errorf(\"unknown replication protocol %v\", p.replicationProtocol)\n\t\t}\n\t}\n\n\tvar delay time.Duration\n\tif p.conns.Load() == 2 {\n\t\tdelay = p.maxArtificialDelay\n\t}\n\n\tp.connections[endpoint] = newPeerWorker(client, p.forwardDelay.WithLabelValues(endpoint.Address), p.asyncForwardWorkersCount, delay)\n\treturn p.connections[endpoint], nil\n}\n\nfunc (p *peerGroup) markPeerUnavailable(addr Endpoint) {\n\tp.m.Lock()","sourceCodeStart":1964,"sourceCodeEnd":2000,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/receive/handler.go#L1964-L2000","documentation":"This is the fully-wrapped error returned by peerGroup.getConnection when a Protobuf replication dial fails: the dial error is wrapped as \"failed to dial peer\" and then again with the errUnavailable prefix, yielding \"target not available: failed to dial peer: <cause>\". It tells callers of the receive handler that the target peer could not be reached, so the write could not be forwarded for replication. The failing endpoint is simultaneously marked unavailable so later calls fail fast with errUnavailable until a backoff re-allow.","triggerScenarios":"Produced only in the ProtobufReplication branch of getConnection: p.dialer(endpoint.Address, p.dialOpts...) returns an error, causing return nil, errors.Wrap(dialError, errUnavailable.Error()). Callers hitting this include the receive Handler's forwarding path during replication to a remote endpoint.","commonSituations":"Rolling restarts or node failures take a peer down mid-write; misconfigured hashring addresses; NetworkPolicies dropping gRPC traffic; the peer's TCP port not exposed by the service; TLS handshake mismatch between receiver dialer options and peer server options.","solutions":["Check the root cause in the inner error (%v suffix) — usually connection refused/timeout — and fix the peer's availability first.","Validate and correct hashring endpoint addresses/ports in --receive.hashrings.","Confirm Kubernetes Services/NetworkPolicies allow the receiver to reach peer gRPC ports.","Align gRPC/TLS client dial options with the peer's server TLS configuration.","Retry the write; replication failures can be retried once the peer's backoff state recovers."],"exampleFix":"// caller side: don't hard-fail one unreplicable peer, log and continue\nif err != nil {\n\tlevel.Error(logger).Log(\"msg\", \"failed to forward to peer\", \"err\", err)\n\treturn // or retry with backoff\n}","handlingStrategy":"retry","validationCode":"// Go: check inner cause before retrying\nif err != nil {\n\tvar netErr net.Error\n\tif errors.As(err, &netErr) && netErr.Timeout() {\n\t\t// retryable\n\t}\n}","typeGuard":null,"tryCatchPattern":"// Go: inspect the wrapped chain\nif err != nil {\n\tif strings.Contains(err.Error(), \"target not available\") {\n\t\t// peer marked unavailable — schedule retry with backoff\n\t}\n}","preventionTips":["Use retries with jittered backoff around receive writes.","Keep hashring config in sync across all receiver instances (configmaps/helm).","Alert on replication failure metrics instead of silently dropping writes.","Ensure graceful shutdown drains replication writes before peers go away."],"tags":["network","grpc","peer-unavailable","thanos-receive","replication"],"backgroundTag":"connection-refused","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}