{"record":{"id":"f4aaeb579a96a541","repo":"thanos-io/thanos","slug":"forwarding-request-to-endpoint-v-v","errorCode":null,"errorMessage":"forwarding request to endpoint %v: %v","messagePattern":"forwarding request to endpoint (.+?): (.+?)","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/receive/handler.go","lineNumber":1834,"sourceCode":"\treturn func() {\n\t\tif p.maxArtificialDelay > 0 {\n\t\t\tvar randDuration = time.Duration(rand.Int63n(int64(p.maxArtificialDelay)))\n\t\t\tif randDuration < 1*time.Second {\n\t\t\t\trandDuration = 1 * time.Second\n\t\t\t}\n\n\t\t\tselect {\n\t\t\tcase <-time.After(randDuration):\n\t\t\tcase <-ctx.Done():\n\t\t\t}\n\t\t}\n\t\tp.forwardDelay.Observe(time.Since(now).Seconds())\n\n\t\ttracing.DoInSpan(ctx, \"receive_forward\", func(ctx context.Context) {\n\t\t\t_, err := p.client.RemoteWrite(ctx, req)\n\t\t\tresponseWriter <- newWriteResponse(\n\t\t\t\tseriesIDs,\n\t\t\t\terrors.Wrapf(err, \"forwarding request to endpoint %v\", er.endpoint),\n\t\t\t\ter,\n\t\t\t)\n\t\t\tif err != nil {\n\t\t\t\tsp := trace.SpanFromContext(ctx)\n\t\t\t\tsp.SetAttributes(attribute.Bool(\"error\", true))\n\t\t\t\tsp.SetAttributes(attribute.String(\"error.msg\", err.Error()))\n\t\t\t}\n\t\t\tcb(err)\n\t\t}, opentracing.Tags{\n\t\t\t\"endpoint\": er.endpoint,\n\t\t\t\"replica\":  er.replica,\n\t\t})\n\t}\n}\n\nfunc (p *peerWorker) RemoteWriteAsync(ctx context.Context, req *storepb.WriteRequest, er endpointReplica, seriesIDs []int, responseWriter chan writeResponse, cb func(error)) {\n\tif err := p.wp.Go(ctx, p.buildWork(ctx, req, er, seriesIDs, responseWriter, cb)); err != nil {\n\t\ttracing.DoInSpan(ctx, \"receive_forward\", func(ctx context.Context) {","sourceCodeStart":1816,"sourceCodeEnd":1852,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/receive/handler.go#L1816-L1852","documentation":"This error wraps any failure returned by the distributed-tenant RemoteWrite client when a receive handler forwards an incoming write request to a peer endpoint in the hashring. It is produced with github.com/pkg/errors.Wrapf, so the original client error is preserved as the cause and includes the target endpoint address. It signals that the request could not be successfully written to the remote endpoint during the synchronous forwarding path.","triggerScenarios":"p.client.RemoteWrite(ctx, req) returns a non-nil error inside the receive_forward span; the wrapped error plus the EndpointRequest (er) are sent to the responseWriter for aggregation of quorum responses.","commonSituations":"A peer endpoint is down or restarting; DNS resolves but the peer refuses or times out on the gRPC storepb Write call; network partition between receive replicas; TLS certificate mismatch between peers; the peer's store API is overloaded and returns an unavailable/code=5xx gRPC status.","solutions":["Verify the target endpoint is healthy: check its /-/ready and that its gRPC port is reachable (curl/grpcurl) from this node.","Check inter-peer TLS configuration on both sides (cert paths, CA, SANs) since mismatch makes RemoteWrite fail immediately.","Inspect peer logs at the target endpoint for the underlying cause (unavailable, deadline exceeded, resource exhaustion).","Retry the write; receive fans out to other replicas so quorum may still be achieved, and the head may eventually be trimmed/repaired.","If persistent, scale or replace the unhealthy endpoint and re-check the hashring config matches across all nodes."],"exampleFix":"// before (handler returns raw wrap)\n_, err := p.client.RemoteWrite(ctx, req)\nresponseWriter <- newWriteResponse(seriesIDs, errors.Wrapf(err, \"forwarding request to endpoint %v\", er.endpoint), er)\n// after (add timeout + typed logging before wrapping)\nctx, cancel := context.WithTimeout(ctx, p.forwardTimeout)\ndefer cancel()\n_, err := p.client.RemoteWrite(ctx, req)\nif err != nil {\n  level.Error(p.logger).Log(\"msg\", \"forward failed\", \"endpoint\", er.endpoint, \"err\", err)\n}\nresponseWriter <- newWriteResponse(seriesIDs, errors.Wrapf(err, \"forwarding request to endpoint %v\", er.endpoint), er)","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":null,"preventionTips":[],"tags":["network","grpc","thanos-receive","replication"],"backgroundTag":"http-request-failed","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}