{"record":{"id":"8628e7185ca0904d","repo":"thanos-io/thanos","slug":"sending-tsdb-statistics-error-to-server-v","errorCode":null,"errorMessage":"sending tsdb statistics error to server %v","messagePattern":"sending tsdb statistics error to server (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/status/proxy.go","lineNumber":130,"sourceCode":"\t\t// Not an error if response strategy is warning.\n\t\treturn nil\n\t}\n\n\tfor {\n\t\tresp, err := statsClient.Recv()\n\t\tif err == io.EOF {\n\t\t\treturn nil\n\t\t}\n\n\t\tif err != nil {\n\t\t\terr = errors.Wrapf(err, \"receiving tsdb statistics from status client %v\", stream.client)\n\n\t\t\tif stream.request.PartialResponseStrategy == storepb.PartialResponseStrategy_ABORT {\n\t\t\t\treturn err\n\t\t\t}\n\n\t\t\tif err := stream.server.Send(statuspb.NewWarningTSDBStatisticsResponse(err)); err != nil {\n\t\t\t\treturn errors.Wrapf(err, \"sending tsdb statistics error to server %v\", stream.server)\n\t\t\t}\n\n\t\t\t// Not an error if response strategy is warning.\n\t\t\treturn nil\n\t\t}\n\n\t\tif w := resp.GetWarning(); w != \"\" {\n\t\t\tif err := stream.server.Send(statuspb.NewWarningTSDBStatisticsResponse(errors.New(w))); err != nil {\n\t\t\t\treturn errors.Wrapf(err, \"sending tsdb statistics warning to server %v\", stream.server)\n\t\t\t}\n\t\t\tcontinue\n\t\t}\n\n\t\tselect {\n\t\tcase stream.channel <- resp.GetStatistics():\n\t\tcase <-ctx.Done():\n\t\t\treturn ctx.Err()\n\t\t}","sourceCodeStart":112,"sourceCodeEnd":148,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/status/proxy.go#L112-L148","documentation":"When an upstream stream error occurs under the non-ABORT (warning) strategy, the proxy forwards it to the caller as a warning response. If that Send itself fails, the error is wrapped as 'sending tsdb statistics error to server %v'. This is a secondary failure — the proxy could not even deliver the warning.","triggerScenarios":"In receive(), after wrapping an upstream statsClient error and strategy is WARNINGS, `stream.server.Send(NewWarningTSDBStatisticsResponse(err))` fails because the downstream client disconnected or its stream is broken.","commonSituations":"Client canceled the request while the proxy was handling a store failure; downstream gRPC transport died concurrently; load balancer closed the client connection.","solutions":["Investigate why the downstream client connection terminated (client timeout, cancel, crash).","Check proxy-to-client network path and idle timeout settings.","Ensure clients keep the stream open until receiving all responses or explicitly cancel.","Treat as benign if the client merely gave up early; fix the original upstream error at the store."],"exampleFix":"// before\nctx, cancel := context.WithTimeout(ctx, 2*time.Second) // client gives up before warnings flush\n// after\nctx, cancel := context.WithTimeout(ctx, 30*time.Second) // allow full stream + warnings to be delivered","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"if err := call(ctx); err != nil {\n    if strings.Contains(err.Error(), \"sending tsdb statistics error to server\") {\n        // downstream client already gone; log and stop\n        logger.Debug(\"client disconnected during stats proxying\", \"err\", err)\n        return nil\n    }\n    return err\n}","preventionTips":["Clients must not cancel mid-request unless intentionally abandoning results.","Ensure client timeouts exceed proxy fan-out time.","Log client cancellations separately from real failures.","Keep network paths between client and proxy stable."],"tags":["grpc","streaming","proxy","double-failure"],"backgroundTag":"broken-pipe","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}