{"record":{"id":"ad49e441118ff61e","repo":"thanos-io/thanos","slug":"bug-localasyncwriter-remotewrite-called-without-t","errorCode":null,"errorMessage":"BUG: localAsyncWriter.RemoteWrite called without TimeseriesTenantData","messagePattern":"BUG: localAsyncWriter\\.RemoteWrite called without TimeseriesTenantData","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/receive/handler.go","lineNumber":1932,"sourceCode":"\tdelete(p.connections, endpoint)\n\tif err := c.client.Close(); err != nil {\n\t\treturn fmt.Errorf(\"closing connection for %s\", endpoint)\n\t}\n\n\treturn nil\n}\n\ntype localAsyncWriter struct {\n\tw *Writer\n}\n\nfunc (lw *localAsyncWriter) Close() error {\n\treturn nil\n}\n\nfunc (lw *localAsyncWriter) RemoteWrite(ctx context.Context, in *storepb.WriteRequest, opts ...grpc.CallOption) (*storepb.WriteResponse, error) {\n\tif len(in.TimeseriesTenantData) == 0 {\n\t\tpanic(\"BUG: localAsyncWriter.RemoteWrite called without TimeseriesTenantData\")\n\t}\n\n\tfor _, ts := range in.TimeseriesTenantData {\n\t\tif err := lw.w.Write(ctx, ts.Tenant, ts.Timeseries); err != nil {\n\t\t\treturn nil, errors.Wrap(err, \"writing locally\")\n\t\t}\n\t}\n\n\treturn &storepb.WriteResponse{}, nil\n}\n\nfunc (p *peerGroup) getConnection(ctx context.Context, endpoint Endpoint) (WriteableStoreAsyncClient, error) {\n\tif !p.isPeerUp(endpoint) {\n\t\treturn nil, errUnavailable\n\t}\n\n\t// use a RLock first to prevent blocking if we don't need to.\n\tp.m.RLock()","sourceCodeStart":1914,"sourceCodeEnd":1950,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/receive/handler.go#L1914-L1950","documentation":"This is a panic raised by localAsyncWriter.RemoteWrite as an internal invariant check: the local write path requires that incoming storepb.WriteRequest messages carry per-tenant series in TimeseriesTenantData. An empty TimeseriesTenantData means the request cannot be routed to any tenant's local TSDB writer, which indicates a programming bug upstream, not an operator error.","triggerScenarios":"RemoteWrite is called on a *localAsyncWriter with a *storepb.WriteRequest whose TimeseriesTenantData slice has length 0.","commonSituations":"A code change introduced a code path constructing WriteRequest without populating TimeseriesTenantData; a caller swapped the local writer in where the standard distributed client was expected with a differently-shaped request; tests invoking RemoteWrite with hand-built empty requests.","solutions":["Fix the caller to populate TimeseriesTenantData (tenant + timeseries) before invoking RemoteWrite on the local writer.","Audit any code path that builds storepb.WriteRequest to guarantee at least one TimeseriesTenantData entry.","If a legitimately empty write can occur, drop it early in the caller instead of calling the writer.","Update tests that construct empty requests to reflect the required shape."],"exampleFix":"// before\np.client.RemoteWrite(ctx, &storepb.WriteRequest{})\n// after\nif len(req.TimeseriesTenantData) == 0 {\n  return &storepb.WriteResponse{}, nil // nothing to write locally\n}\np.client.RemoteWrite(ctx, req)","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"// Go: close errors during teardown should be logged, not propagated\nif err := c.client.Close(); err != nil {\n  level.Warn(logger).Log(\"msg\", \"connection close failed\", \"endpoint\", endpoint, \"err\", err)\n}","preventionTips":["Drain in-flight RPCs before closing connections during shutdown.","Avoid closing connections concurrently from multiple goroutines.","Track removed endpoints so close failures can be correlated with hashring changes.","Keep grpc-go updated to benefit from connection-close fixes."],"tags":["panic","invariant","thanos-receive","protobuf"],"backgroundTag":"internal-invariant-violation","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}