thanos-io/thanos · warning

backing off forward request for endpoint

Error message

backing off forward request for endpoint %v: target not available

What it means

In the receive fan-out path, when the peer connection to a remote endpoint fails with errUnavailable, the error is re-wrapped as "backing off forward request for endpoint %v" and delivered back to the client as an Unavailable write response; the write should be retried later.

Solutions

1. Retry the write with backoff — the error is intentionally Unavailable/transient
2. Check the target endpoint's health (is the receive pod/process up and listening?)
3. Update the hashring/peers config to remove or replace the dead endpoint
4. Verify DNS/service discovery resolves the endpoint address

Example fix

// before: hashring.json still lists a removed node
{"endpoints":["receive-0:10901","receive-2:10901"]}
// after: remove the unavailable endpoint
{"endpoints":["receive-0:10901"]}

// before writing, probe the endpoint
conn, err := net.DialTimeout("tcp", endpointAddr, 2*time.Second)
if err != nil { /* endpoint unavailable — back off or re-route */ }

err := client.Write(ctx, req)
var st *status.Status
if errors.Is(err, errUnavailable) || status.Code(err) == codes.Unavailable {
    time.Sleep(backoff.Next())
    return retry(ctx, req) // idempotent retry with backoff
}
return err

When it happens

Trigger: h.peers.getConnection(ctx, endpoint) at handler.go:1247 fails because the target receive/querier endpoint is down, DNS-unresolvable, or the gossip/ring has not yet propagated the node's removal.

Common situations: A receive node crashed or is restarting while hashring still routes to it; Kubernetes service endpoints stale; split/replication configured with a dead peer; network partition.

Understand the failure class

Background: ECONNREFUSED and "connection refused" / "could not connect to server" errors: what they mean and how to fix them — this error's family across 44 libraries.

Related errors

• fetch Prometheus flags
• target not ready
• target not available
• sync before first pass of downsampling
• sync before second pass of downsampling

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/f2cf392d7ba040f1. Report an issue: GitHub.

Appendix: source

Thrown at pkg/receive/handler.go:1247

}

// prepareRemoteWrite resolves the peer connection, builds the WriteRequest, and constructs the
// completion callback. Returns (nil, nil, nil) when a connection error has already been written to
// responses and wg.Done called — callers must check for nil before proceeding.
func (h *Handler) prepareRemoteWrite(
	ctx context.Context,
	writes map[string]trackedSeries,
	er endpointReplica,
	alreadyReplicated bool,
	responses chan writeResponse,
	wg *sync.WaitGroup,
	allIDs []int,
) (WriteableStoreAsyncClient, *storepb.WriteRequest, func(error)) {
	endpoint := er.endpoint
	cl, err := h.peers.getConnection(ctx, endpoint)
	if err != nil {
		if errors.Is(err, errUnavailable) {
			err = errors.Wrapf(errUnavailable, "backing off forward request for endpoint %v", er)
		}

		responses <- newWriteResponse(allIDs, err, er)
		wg.Done()
		return nil, nil, nil
	}

	dataTuples := make([]storepb.TimeSeriesTenantTuple, 0, len(writes))
	for wTenant, ts := range writes {
		dataTuples = append(dataTuples, storepb.TimeSeriesTenantTuple{
			Timeseries: ts.timeSeries,
			Tenant:     wTenant,
		})
	}

	// Replica is 1-indexed on the wire; 0 indicates un-replicated.
	req := &storepb.WriteRequest{
		TimeseriesTenantData: dataTuples,

View on GitHub (pinned to 35b8b99117)