thanos-io/thanos · warning

closing connection for

Error message

closing connection for %s

What it means

This error is returned when closing the local gRPC client connection to an endpoint during teardown of the peer connections in the receive handler. The connection's Close() call failed, so the handler cannot cleanly close the per-endpoint connection; state (map entry, worker metric) was already removed. It uses fmt.Errorf without wrapping, so the underlying cause is only described textually.

Solutions

1. Treat as non-fatal: connection state is already deleted; confirm the endpoint was intentionally removed from the hashring.
2. Check for goroutine leaks or in-flight RPCs preventing clean close; allow draining before shutdown.
3. If it recurs, restart the receive instance to clear wedged gRPC connections.
4. Verify gRPC library version; upgrade if Close() errors spuriously under concurrency (known upstream issues in older grpc-go).

Example fix

// before
if err := c.client.Close(); err != nil {
  return fmt.Errorf("closing connection for %s", endpoint)
}
// after (log instead of failing teardown)
if err := c.client.Close(); err != nil {
  level.Warn(p.logger).Log("msg", "closing connection", "endpoint", endpoint, "err", err)
}

// ensure workers/queues are initialized before scheduling writes
if p.closed.Load() {
  return errors.New("receive handler is shutting down; not scheduling forward")
}

// Go: treat breaker-open/scheduling errors distinctly
err := errors.Cause(resp.err)
switch {
case errors.Is(err, errCircuitBreakerOpen):
  // back off, don't hammer the endpoint
case isNetError(err):
  // retry against another replica
}

When it happens

Trigger: p.connections[endpoint] is being torn down (endpoint removed from hashring or handler shutting down) and c.client.Close() returns a non-nil error for the connection backing endpoint.Address.

Common situations: Connection already broken or in an error state after repeated network failures to the peer; gRPC client Close returning an error during concurrent shutdown; process exiting with in-flight RPCs to the removed endpoint.

Understand the failure class

Background: ECONNREFUSED and "connection refused" / "could not connect to server" errors: what they mean and how to fix them — this error's family across 44 libraries.

Related errors

• forwarding request to endpoint
• failed to dial peer
• target not available: failed to dial peer
• building gRPC client
• endpoint

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/23e9578782f9ba8c. Report an issue: GitHub.

Appendix: source

Thrown at pkg/receive/handler.go:1916

	return nil
}

func (p *peerGroup) close(endpoint Endpoint) error {
	p.m.Lock()
	defer p.m.Unlock()

	c, ok := p.connections[endpoint]
	if !ok {
		// NOTE(GiedriusS): this could be valid case when the connection
		// was never established.
		return nil
	}

	p.forwardDelay.Delete(prometheus.Labels{"worker": endpoint.Address})
	p.connections[endpoint].wp.Close()
	delete(p.connections, endpoint)
	if err := c.client.Close(); err != nil {
		return fmt.Errorf("closing connection for %s", endpoint)
	}

	return nil
}

type localAsyncWriter struct {
	w *Writer
}

func (lw *localAsyncWriter) Close() error {
	return nil
}

func (lw *localAsyncWriter) RemoteWrite(ctx context.Context, in *storepb.WriteRequest, opts ...grpc.CallOption) (*storepb.WriteResponse, error) {
	if len(in.TimeseriesTenantData) == 0 {
		panic("BUG: localAsyncWriter.RemoteWrite called without TimeseriesTenantData")
	}

View on GitHub (pinned to 35b8b99117)