{"record":{"id":"27c8b24480ee21df","repo":"thanos-io/thanos","slug":"scheduling-forward-request-for-endpoint-v-v","errorCode":null,"errorMessage":"scheduling forward request for endpoint %v: %v","messagePattern":"scheduling forward request for endpoint (.+?): (.+?)","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/receive/handler.go","lineNumber":1858,"sourceCode":"\t\t\t\tsp.SetAttributes(attribute.String(\"error.msg\", err.Error()))\n\t\t\t}\n\t\t\tcb(err)\n\t\t}, opentracing.Tags{\n\t\t\t\"endpoint\": er.endpoint,\n\t\t\t\"replica\":  er.replica,\n\t\t})\n\t}\n}\n\nfunc (p *peerWorker) RemoteWriteAsync(ctx context.Context, req *storepb.WriteRequest, er endpointReplica, seriesIDs []int, responseWriter chan writeResponse, cb func(error)) {\n\tif err := p.wp.Go(ctx, p.buildWork(ctx, req, er, seriesIDs, responseWriter, cb)); err != nil {\n\t\ttracing.DoInSpan(ctx, \"receive_forward\", func(ctx context.Context) {\n\t\t\tsp := trace.SpanFromContext(ctx)\n\t\t\tsp.SetAttributes(attribute.Bool(\"error\", true))\n\t\t\tsp.SetAttributes(attribute.String(\"error.msg\", err.Error()))\n\t\t\tresponseWriter <- newWriteResponse(\n\t\t\t\tseriesIDs,\n\t\t\t\terrors.Wrapf(err, \"scheduling forward request for endpoint %v\", er.endpoint),\n\t\t\t\ter,\n\t\t\t)\n\t\t\tcb(err)\n\t\t}, opentracing.Tags{\n\t\t\t\"endpoint\": er.endpoint,\n\t\t\t\"replica\":  er.replica,\n\t\t})\n\t}\n}\n\nfunc (p *peerWorker) TryRemoteWriteAsync(ctx context.Context, req *storepb.WriteRequest, er endpointReplica, seriesIDs []int, responseWriter chan writeResponse, cb func(error)) bool {\n\treturn p.wp.TryGo(p.buildWork(ctx, req, er, seriesIDs, responseWriter, cb))\n}\n\ntype peerGroup struct {\n\tlogger                   log.Logger\n\tdialOpts                 []grpc.DialOption\n\tconnections              map[Endpoint]*peerWorker","sourceCodeStart":1840,"sourceCodeEnd":1876,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/receive/handler.go#L1840-L1876","documentation":"This error wraps a failure from the asynchronous forwarding path of the receive handler: when the endpoint request could not even be scheduled/enqueued for the worker (or the schedule call returned an error), the error is wrapped with the endpoint address and delivered to the response writer. Unlike error 770 it happens before/independent of the actual remote write result, typically at queueing time.","triggerScenarios":"The async forward scheduler returns an error for the EndpointRequest (e.g. the worker queue for the endpoint is full, the worker is shutting down, or an initial error was set), and errors.Wrapf is applied inside the receive_forward span before invoking the circuit breaker cb(err).","commonSituations":"An endpoint is slow or down so its bounded forwarding queue fills up; too many tenants/series causes backpressure; receive workers were not started or were closed during shutdown; misconfigured hashring producing very large fan-out per request.","solutions":["Check whether the worker for that endpoint is alive and its queue is not saturated (increase concurrency/workers if persistently full).","Verify the endpoint is healthy; a slow peer causes backpressure that surfaces as scheduling failures on other requests.","Increase queue capacity or add replicas so the quorum can still succeed despite one failing endpoint.","Review circuit-breaker state: after repeated failures, cb(err) may open the breaker and cause further scheduling errors.","Confirm graceful shutdown ordering: ensure ingesters are not draining while traffic still routes to them."],"exampleFix":"// before\nif err := p.schedule(er); err != nil {\n  tracing.DoInSpan(ctx, \"receive_forward\", func(ctx context.Context) {\n    responseWriter <- newWriteResponse(seriesIDs, errors.Wrapf(err, \"scheduling forward request for endpoint %v\", er.endpoint), er)\n  ...)\n// after (guard against breaker-open with retry)\nif err := p.schedule(er); err != nil {\n  if errors.Is(err, errCircuitBreakerOpen) {\n    // let quorum decide; count as unavailable, not fatal\n    responseWriter <- newWriteResponse(seriesIDs, errors.Wrapf(err, \"scheduling forward request for endpoint %v\", er.endpoint), er)\n    return\n  }\n  ...\n}","handlingStrategy":"retry","validationCode":"// before forwarding, probe endpoint reachability\nconn, err := net.DialTimeout(\"tcp\", strings.TrimPrefix(endpoint, \"http://\"), 2*time.Second)\nif err != nil { return fmt.Errorf(\"endpoint %v unreachable: %w\", endpoint, err) }\nconn.Close()","typeGuard":null,"tryCatchPattern":"// Go: inspect wrapped cause and retry with backoff\nresp := <-responseWriter\nif resp.err != nil {\n  if st, ok := status.FromError(errors.Cause(resp.err)); ok && st.Code() == codes.Unavailable {\n    // retry with backoff or rely on quorum from other replicas\n  }\n}","preventionTips":["Run at least 3 receive replicas so a single endpoint failure still meets quorum.","Monitor the forward latency/duration metrics per endpoint and alert on saturation.","Keep hashring configs identical across all receive nodes.","Use readiness probes so dead peers are removed from load-balancing quickly."],"tags":["network","async","thanos-receive","backpressure"],"backgroundTag":"http-request-failed","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}