BUG: localAsyncWriter.RemoteWrite called without…

Error message

BUG: localAsyncWriter.RemoteWrite called without TimeseriesTenantData

What it means

This is a panic raised by localAsyncWriter.RemoteWrite as an internal invariant check: the local write path requires that incoming storepb.WriteRequest messages carry per-tenant series in TimeseriesTenantData. An empty TimeseriesTenantData means the request cannot be routed to any tenant's local TSDB writer, which indicates a programming bug upstream, not an operator error.

Solutions

Fix the caller to populate TimeseriesTenantData (tenant + timeseries) before invoking RemoteWrite on the local writer.
Audit any code path that builds storepb.WriteRequest to guarantee at least one TimeseriesTenantData entry.
If a legitimately empty write can occur, drop it early in the caller instead of calling the writer.
Update tests that construct empty requests to reflect the required shape.

Example fix

// before
p.client.RemoteWrite(ctx, &storepb.WriteRequest{})
// after
if len(req.TimeseriesTenantData) == 0 {
  return &storepb.WriteResponse{}, nil // nothing to write locally
}
p.client.RemoteWrite(ctx, req)

Defensive patterns

Strategy: try-catch

Try / catch

// Go: close errors during teardown should be logged, not propagated
if err := c.client.Close(); err != nil {
  level.Warn(logger).Log("msg", "connection close failed", "endpoint", endpoint, "err", err)
}

Prevention

Drain in-flight RPCs before closing connections during shutdown.
Avoid closing connections concurrently from multiple goroutines.
Track removed endpoints so close failures can be correlated with hashring changes.
Keep grpc-go updated to benefit from connection-close fixes.

When it happens

Trigger: RemoteWrite is called on a *localAsyncWriter with a *storepb.WriteRequest whose TimeseriesTenantData slice has length 0.

Common situations: A code change introduced a code path constructing WriteRequest without populating TimeseriesTenantData; a caller swapped the local writer in where the standard distributed client was expected with a differently-shaped request; tests invoking RemoteWrite with hand-built empty requests.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/ad49e441118ff61e. Report an issue: GitHub.

Appendix: source

Thrown at pkg/receive/handler.go:1932

	delete(p.connections, endpoint)
	if err := c.client.Close(); err != nil {
		return fmt.Errorf("closing connection for %s", endpoint)
	}

	return nil
}

type localAsyncWriter struct {
	w *Writer
}

func (lw *localAsyncWriter) Close() error {
	return nil
}

func (lw *localAsyncWriter) RemoteWrite(ctx context.Context, in *storepb.WriteRequest, opts ...grpc.CallOption) (*storepb.WriteResponse, error) {
	if len(in.TimeseriesTenantData) == 0 {
		panic("BUG: localAsyncWriter.RemoteWrite called without TimeseriesTenantData")
	}

	for _, ts := range in.TimeseriesTenantData {
		if err := lw.w.Write(ctx, ts.Tenant, ts.Timeseries); err != nil {
			return nil, errors.Wrap(err, "writing locally")
		}
	}

	return &storepb.WriteResponse{}, nil
}

func (p *peerGroup) getConnection(ctx context.Context, endpoint Endpoint) (WriteableStoreAsyncClient, error) {
	if !p.isPeerUp(endpoint) {
		return nil, errUnavailable
	}

	// use a RLock first to prevent blocking if we don't need to.
	p.m.RLock()

View on GitHub (pinned to 35b8b99117)