{"record":{"id":"f2cf392d7ba040f1","repo":"thanos-io/thanos","slug":"backing-off-forward-request-for-endpoint-v-targe","errorCode":null,"errorMessage":"backing off forward request for endpoint %v: target not available","messagePattern":"backing off forward request for endpoint (.+?): target not available","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"pkg/receive/handler.go","lineNumber":1247,"sourceCode":"}\n\n// prepareRemoteWrite resolves the peer connection, builds the WriteRequest, and constructs the\n// completion callback. Returns (nil, nil, nil) when a connection error has already been written to\n// responses and wg.Done called — callers must check for nil before proceeding.\nfunc (h *Handler) prepareRemoteWrite(\n\tctx context.Context,\n\twrites map[string]trackedSeries,\n\ter endpointReplica,\n\talreadyReplicated bool,\n\tresponses chan writeResponse,\n\twg *sync.WaitGroup,\n\tallIDs []int,\n) (WriteableStoreAsyncClient, *storepb.WriteRequest, func(error)) {\n\tendpoint := er.endpoint\n\tcl, err := h.peers.getConnection(ctx, endpoint)\n\tif err != nil {\n\t\tif errors.Is(err, errUnavailable) {\n\t\t\terr = errors.Wrapf(errUnavailable, \"backing off forward request for endpoint %v\", er)\n\t\t}\n\n\t\tresponses <- newWriteResponse(allIDs, err, er)\n\t\twg.Done()\n\t\treturn nil, nil, nil\n\t}\n\n\tdataTuples := make([]storepb.TimeSeriesTenantTuple, 0, len(writes))\n\tfor wTenant, ts := range writes {\n\t\tdataTuples = append(dataTuples, storepb.TimeSeriesTenantTuple{\n\t\t\tTimeseries: ts.timeSeries,\n\t\t\tTenant:     wTenant,\n\t\t})\n\t}\n\n\t// Replica is 1-indexed on the wire; 0 indicates un-replicated.\n\treq := &storepb.WriteRequest{\n\t\tTimeseriesTenantData: dataTuples,","sourceCodeStart":1229,"sourceCodeEnd":1265,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/receive/handler.go#L1229-L1265","documentation":"In the receive fan-out path, when the peer connection to a remote endpoint fails with errUnavailable, the error is re-wrapped as \"backing off forward request for endpoint %v\" and delivered back to the client as an Unavailable write response; the write should be retried later.","triggerScenarios":"h.peers.getConnection(ctx, endpoint) at handler.go:1247 fails because the target receive/querier endpoint is down, DNS-unresolvable, or the gossip/ring has not yet propagated the node's removal.","commonSituations":"A receive node crashed or is restarting while hashring still routes to it; Kubernetes service endpoints stale; split/replication configured with a dead peer; network partition.","solutions":["Retry the write with backoff — the error is intentionally Unavailable/transient","Check the target endpoint's health (is the receive pod/process up and listening?)","Update the hashring/peers config to remove or replace the dead endpoint","Verify DNS/service discovery resolves the endpoint address"],"exampleFix":"// before: hashring.json still lists a removed node\n{\"endpoints\":[\"receive-0:10901\",\"receive-2:10901\"]}\n// after: remove the unavailable endpoint\n{\"endpoints\":[\"receive-0:10901\"]}","handlingStrategy":"retry","validationCode":"// before writing, probe the endpoint\nconn, err := net.DialTimeout(\"tcp\", endpointAddr, 2*time.Second)\nif err != nil { /* endpoint unavailable — back off or re-route */ }","typeGuard":null,"tryCatchPattern":"err := client.Write(ctx, req)\nvar st *status.Status\nif errors.Is(err, errUnavailable) || status.Code(err) == codes.Unavailable {\n    time.Sleep(backoff.Next())\n    return retry(ctx, req) // idempotent retry with backoff\n}\nreturn err","preventionTips":["Use replication-factor > 1 so one unavailable endpoint does not lose data","Keep hashring config in sync with service discovery (auto-update on pod changes)","Health-check receive endpoints before routing and remove unhealthy ones","Retry writes with exponential backoff and jitter"],"tags":["network","availability","retry"],"backgroundTag":"connection-refused","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}