thanos-io/thanos · error

scheduling forward request for endpoint

Error message

scheduling forward request for endpoint %v: %v

What it means

This error wraps a failure from the asynchronous forwarding path of the receive handler: when the endpoint request could not even be scheduled/enqueued for the worker (or the schedule call returned an error), the error is wrapped with the endpoint address and delivered to the response writer. Unlike error 770 it happens before/independent of the actual remote write result, typically at queueing time.

Solutions

1. Check whether the worker for that endpoint is alive and its queue is not saturated (increase concurrency/workers if persistently full).
2. Verify the endpoint is healthy; a slow peer causes backpressure that surfaces as scheduling failures on other requests.
3. Increase queue capacity or add replicas so the quorum can still succeed despite one failing endpoint.
4. Review circuit-breaker state: after repeated failures, cb(err) may open the breaker and cause further scheduling errors.
5. Confirm graceful shutdown ordering: ensure ingesters are not draining while traffic still routes to them.

Example fix

// before
if err := p.schedule(er); err != nil {
  tracing.DoInSpan(ctx, "receive_forward", func(ctx context.Context) {
    responseWriter <- newWriteResponse(seriesIDs, errors.Wrapf(err, "scheduling forward request for endpoint %v", er.endpoint), er)
  ...)
// after (guard against breaker-open with retry)
if err := p.schedule(er); err != nil {
  if errors.Is(err, errCircuitBreakerOpen) {
    // let quorum decide; count as unavailable, not fatal
    responseWriter <- newWriteResponse(seriesIDs, errors.Wrapf(err, "scheduling forward request for endpoint %v", er.endpoint), er)
    return
  }
  ...
}

// before forwarding, probe endpoint reachability
conn, err := net.DialTimeout("tcp", strings.TrimPrefix(endpoint, "http://"), 2*time.Second)
if err != nil { return fmt.Errorf("endpoint %v unreachable: %w", endpoint, err) }
conn.Close()

// Go: inspect wrapped cause and retry with backoff
resp := <-responseWriter
if resp.err != nil {
  if st, ok := status.FromError(errors.Cause(resp.err)); ok && st.Code() == codes.Unavailable {
    // retry with backoff or rely on quorum from other replicas
  }
}

When it happens

Trigger: The async forward scheduler returns an error for the EndpointRequest (e.g. the worker queue for the endpoint is full, the worker is shutting down, or an initial error was set), and errors.Wrapf is applied inside the receive_forward span before invoking the circuit breaker cb(err).

Common situations: An endpoint is slow or down so its bounded forwarding queue fills up; too many tenants/series causes backpressure; receive workers were not started or were closed during shutdown; misconfigured hashring producing very large fan-out per request.

Understand the failure class

Background: 'Something went wrong' / 'Request failed (500)' / 'HTTP error! status: 404' — what failed HTTP requests actually mean and how to find the real cause — this error's family across 28 libraries.

Related errors

• forwarding request to endpoint
• failed to dial peer
• target not available: failed to dial peer
• sync before first pass of downsampling
• sync before second pass of downsampling

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/27c8b24480ee21df. Report an issue: GitHub.

Appendix: source

Thrown at pkg/receive/handler.go:1858

				sp.SetAttributes(attribute.String("error.msg", err.Error()))
			}
			cb(err)
		}, opentracing.Tags{
			"endpoint": er.endpoint,
			"replica":  er.replica,
		})
	}
}

func (p *peerWorker) RemoteWriteAsync(ctx context.Context, req *storepb.WriteRequest, er endpointReplica, seriesIDs []int, responseWriter chan writeResponse, cb func(error)) {
	if err := p.wp.Go(ctx, p.buildWork(ctx, req, er, seriesIDs, responseWriter, cb)); err != nil {
		tracing.DoInSpan(ctx, "receive_forward", func(ctx context.Context) {
			sp := trace.SpanFromContext(ctx)
			sp.SetAttributes(attribute.Bool("error", true))
			sp.SetAttributes(attribute.String("error.msg", err.Error()))
			responseWriter <- newWriteResponse(
				seriesIDs,
				errors.Wrapf(err, "scheduling forward request for endpoint %v", er.endpoint),
				er,
			)
			cb(err)
		}, opentracing.Tags{
			"endpoint": er.endpoint,
			"replica":  er.replica,
		})
	}
}

func (p *peerWorker) TryRemoteWriteAsync(ctx context.Context, req *storepb.WriteRequest, er endpointReplica, seriesIDs []int, responseWriter chan writeResponse, cb func(error)) bool {
	return p.wp.TryGo(p.buildWork(ctx, req, er, seriesIDs, responseWriter, cb))
}

type peerGroup struct {
	logger                   log.Logger
	dialOpts                 []grpc.DialOption
	connections              map[Endpoint]*peerWorker

View on GitHub (pinned to 35b8b99117)