thanos-io/thanos · error

forwarding request to endpoint

Error message

forwarding request to endpoint %v: %v

What it means

This error wraps any failure returned by the distributed-tenant RemoteWrite client when a receive handler forwards an incoming write request to a peer endpoint in the hashring. It is produced with github.com/pkg/errors.Wrapf, so the original client error is preserved as the cause and includes the target endpoint address. It signals that the request could not be successfully written to the remote endpoint during the synchronous forwarding path.

Solutions

1. Verify the target endpoint is healthy: check its /-/ready and that its gRPC port is reachable (curl/grpcurl) from this node.
2. Check inter-peer TLS configuration on both sides (cert paths, CA, SANs) since mismatch makes RemoteWrite fail immediately.
3. Inspect peer logs at the target endpoint for the underlying cause (unavailable, deadline exceeded, resource exhaustion).
4. Retry the write; receive fans out to other replicas so quorum may still be achieved, and the head may eventually be trimmed/repaired.
5. If persistent, scale or replace the unhealthy endpoint and re-check the hashring config matches across all nodes.

Example fix

// before (handler returns raw wrap)
_, err := p.client.RemoteWrite(ctx, req)
responseWriter <- newWriteResponse(seriesIDs, errors.Wrapf(err, "forwarding request to endpoint %v", er.endpoint), er)
// after (add timeout + typed logging before wrapping)
ctx, cancel := context.WithTimeout(ctx, p.forwardTimeout)
defer cancel()
_, err := p.client.RemoteWrite(ctx, req)
if err != nil {
  level.Error(p.logger).Log("msg", "forward failed", "endpoint", er.endpoint, "err", err)
}
responseWriter <- newWriteResponse(seriesIDs, errors.Wrapf(err, "forwarding request to endpoint %v", er.endpoint), er)

When it happens

Trigger: p.client.RemoteWrite(ctx, req) returns a non-nil error inside the receive_forward span; the wrapped error plus the EndpointRequest (er) are sent to the responseWriter for aggregation of quorum responses.

Common situations: A peer endpoint is down or restarting; DNS resolves but the peer refuses or times out on the gRPC storepb Write call; network partition between receive replicas; TLS certificate mismatch between peers; the peer's store API is overloaded and returns an unavailable/code=5xx gRPC status.

Understand the failure class

Background: 'Something went wrong' / 'Request failed (500)' / 'HTTP error! status: 404' — what failed HTTP requests actually mean and how to find the real cause — this error's family across 28 libraries.

Related errors

• failed to dial peer
• target not available: failed to dial peer
• no query API server reachable
• failed to get tsdb status from prometheus
• retrieving targets

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/f4aaeb579a96a541. Report an issue: GitHub.

Appendix: source

Thrown at pkg/receive/handler.go:1834

	return func() {
		if p.maxArtificialDelay > 0 {
			var randDuration = time.Duration(rand.Int63n(int64(p.maxArtificialDelay)))
			if randDuration < 1*time.Second {
				randDuration = 1 * time.Second
			}

			select {
			case <-time.After(randDuration):
			case <-ctx.Done():
			}
		}
		p.forwardDelay.Observe(time.Since(now).Seconds())

		tracing.DoInSpan(ctx, "receive_forward", func(ctx context.Context) {
			_, err := p.client.RemoteWrite(ctx, req)
			responseWriter <- newWriteResponse(
				seriesIDs,
				errors.Wrapf(err, "forwarding request to endpoint %v", er.endpoint),
				er,
			)
			if err != nil {
				sp := trace.SpanFromContext(ctx)
				sp.SetAttributes(attribute.Bool("error", true))
				sp.SetAttributes(attribute.String("error.msg", err.Error()))
			}
			cb(err)
		}, opentracing.Tags{
			"endpoint": er.endpoint,
			"replica":  er.replica,
		})
	}
}

func (p *peerWorker) RemoteWriteAsync(ctx context.Context, req *storepb.WriteRequest, er endpointReplica, seriesIDs []int, responseWriter chan writeResponse, cb func(error)) {
	if err := p.wp.Go(ctx, p.buildWork(ctx, req, er, seriesIDs, responseWriter, cb)); err != nil {
		tracing.DoInSpan(ctx, "receive_forward", func(ctx context.Context) {

View on GitHub (pinned to 35b8b99117)