{"record":{"id":"23e9578782f9ba8c","repo":"thanos-io/thanos","slug":"closing-connection-for-s","errorCode":null,"errorMessage":"closing connection for %s","messagePattern":"closing connection for (.+?)","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"pkg/receive/handler.go","lineNumber":1916,"sourceCode":"\treturn nil\n}\n\nfunc (p *peerGroup) close(endpoint Endpoint) error {\n\tp.m.Lock()\n\tdefer p.m.Unlock()\n\n\tc, ok := p.connections[endpoint]\n\tif !ok {\n\t\t// NOTE(GiedriusS): this could be valid case when the connection\n\t\t// was never established.\n\t\treturn nil\n\t}\n\n\tp.forwardDelay.Delete(prometheus.Labels{\"worker\": endpoint.Address})\n\tp.connections[endpoint].wp.Close()\n\tdelete(p.connections, endpoint)\n\tif err := c.client.Close(); err != nil {\n\t\treturn fmt.Errorf(\"closing connection for %s\", endpoint)\n\t}\n\n\treturn nil\n}\n\ntype localAsyncWriter struct {\n\tw *Writer\n}\n\nfunc (lw *localAsyncWriter) Close() error {\n\treturn nil\n}\n\nfunc (lw *localAsyncWriter) RemoteWrite(ctx context.Context, in *storepb.WriteRequest, opts ...grpc.CallOption) (*storepb.WriteResponse, error) {\n\tif len(in.TimeseriesTenantData) == 0 {\n\t\tpanic(\"BUG: localAsyncWriter.RemoteWrite called without TimeseriesTenantData\")\n\t}\n","sourceCodeStart":1898,"sourceCodeEnd":1934,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/receive/handler.go#L1898-L1934","documentation":"This error is returned when closing the local gRPC client connection to an endpoint during teardown of the peer connections in the receive handler. The connection's Close() call failed, so the handler cannot cleanly close the per-endpoint connection; state (map entry, worker metric) was already removed. It uses fmt.Errorf without wrapping, so the underlying cause is only described textually.","triggerScenarios":"p.connections[endpoint] is being torn down (endpoint removed from hashring or handler shutting down) and c.client.Close() returns a non-nil error for the connection backing endpoint.Address.","commonSituations":"Connection already broken or in an error state after repeated network failures to the peer; gRPC client Close returning an error during concurrent shutdown; process exiting with in-flight RPCs to the removed endpoint.","solutions":["Treat as non-fatal: connection state is already deleted; confirm the endpoint was intentionally removed from the hashring.","Check for goroutine leaks or in-flight RPCs preventing clean close; allow draining before shutdown.","If it recurs, restart the receive instance to clear wedged gRPC connections.","Verify gRPC library version; upgrade if Close() errors spuriously under concurrency (known upstream issues in older grpc-go)."],"exampleFix":"// before\nif err := c.client.Close(); err != nil {\n  return fmt.Errorf(\"closing connection for %s\", endpoint)\n}\n// after (log instead of failing teardown)\nif err := c.client.Close(); err != nil {\n  level.Warn(p.logger).Log(\"msg\", \"closing connection\", \"endpoint\", endpoint, \"err\", err)\n}","handlingStrategy":"retry","validationCode":"// ensure workers/queues are initialized before scheduling writes\nif p.closed.Load() {\n  return errors.New(\"receive handler is shutting down; not scheduling forward\")\n}","typeGuard":null,"tryCatchPattern":"// Go: treat breaker-open/scheduling errors distinctly\nerr := errors.Cause(resp.err)\nswitch {\ncase errors.Is(err, errCircuitBreakerOpen):\n  // back off, don't hammer the endpoint\ncase isNetError(err):\n  // retry against another replica\n}","preventionTips":["Size forwarding queues and worker counts relative to peak write throughput.","Alert on endpoint forwarding queue utilization before it fills.","Use circuit breakers with a bounded open duration rather than permanent open.","Gracefully drain endpoints during shutdowns and scaling events."],"tags":["grpc","connection-close","thanos-receive","shutdown"],"backgroundTag":"connection-refused","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}