{"record":{"id":"6ca89ba24429249c","repo":"thanos-io/thanos","slug":"writing-locally","errorCode":null,"errorMessage":"writing locally","messagePattern":"writing locally","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/receive/handler.go","lineNumber":1937,"sourceCode":"\treturn nil\n}\n\ntype localAsyncWriter struct {\n\tw *Writer\n}\n\nfunc (lw *localAsyncWriter) Close() error {\n\treturn nil\n}\n\nfunc (lw *localAsyncWriter) RemoteWrite(ctx context.Context, in *storepb.WriteRequest, opts ...grpc.CallOption) (*storepb.WriteResponse, error) {\n\tif len(in.TimeseriesTenantData) == 0 {\n\t\tpanic(\"BUG: localAsyncWriter.RemoteWrite called without TimeseriesTenantData\")\n\t}\n\n\tfor _, ts := range in.TimeseriesTenantData {\n\t\tif err := lw.w.Write(ctx, ts.Tenant, ts.Timeseries); err != nil {\n\t\t\treturn nil, errors.Wrap(err, \"writing locally\")\n\t\t}\n\t}\n\n\treturn &storepb.WriteResponse{}, nil\n}\n\nfunc (p *peerGroup) getConnection(ctx context.Context, endpoint Endpoint) (WriteableStoreAsyncClient, error) {\n\tif !p.isPeerUp(endpoint) {\n\t\treturn nil, errUnavailable\n\t}\n\n\t// use a RLock first to prevent blocking if we don't need to.\n\tp.m.RLock()\n\tc, ok := p.connections[endpoint]\n\tp.m.RUnlock()\n\tif ok {\n\t\treturn c, nil\n\t}","sourceCodeStart":1919,"sourceCodeEnd":1955,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/receive/handler.go#L1919-L1955","documentation":"This error wraps a failure from the local TSDB append path: localAsyncWriter.RemoteWrite iterates TimeseriesTenantData and calls lw.w.Write(ctx, tenant, timeseries) for each entry. If the head write for a tenant fails (e.g. storage error, out-of-order samples, out-of-bounds timestamp), the error is wrapped with 'writing locally' and propagated back to the forwarder.","triggerScenarios":"lw.w.Write returns non-nil error while appending the given timeseries to the tenant's local TSDB head inside localAsyncWriter.RemoteWrite.","commonSituations":"Out-of-order or too-old sample timestamps relative to the head; duplicate sample for the same timestamp with a different value; TSDB head reaching its size limit and refusing appends; disk full on the receive node; series label churn exceeding active-series limits.","solutions":["Check the wrapped cause in the log to distinguish out-of-order/out-of-bounds from storage-level failures.","Sync sender clocks and ensure no senders emit samples with timestamps in the past (or increase out-of-order window if configured).","Free disk space / increase head chunk limits on the receive node.","Verify the tenant's TSDB is not in a corrupted state; if it is, restart the instance to reopen storage.","Update senders (Prometheus remote_write config) to match the receive's allowed timestamp bounds."],"exampleFix":"// before\nif err := lw.w.Write(ctx, ts.Tenant, ts.Timeseries); err != nil {\n  return nil, errors.Wrap(err, \"writing locally\")\n}\n// after (drop single unapplicable samples instead of failing whole request)\nif err := lw.w.Write(ctx, ts.Tenant, ts.Timeseries); err != nil {\n  if errors.Is(err, storage.ErrOutOfBounds) || errors.Is(err, storage.ErrOutOfOrderSample) {\n    level.Warn(lw.logger).Log(\"msg\", \"dropping unapplicable samples\", \"tenant\", ts.Tenant, \"err\", err)\n    continue\n  }\n  return nil, errors.Wrap(err, \"writing locally\")\n}","handlingStrategy":"retry","validationCode":"// pre-check sample applicability where possible\nif ts.Timeseries != nil && len(ts.Timeseries.Samples) > 0 {\n  for _, s := range ts.Timeseries.Samples {\n    if s.Timestamp <= headMaxTime {\n      return fmt.Errorf(\"sample timestamp %d is out of bounds for tenant %s\", s.Timestamp, ts.Tenant)\n    }\n  }\n}","typeGuard":null,"tryCatchPattern":"// Go: classify storage errors before retrying\nerr := errors.Cause(respErr)\nswitch {\ncase errors.Is(err, storage.ErrOutOfOrderSample), errors.Is(err, storage.ErrOutOfBounds), errors.Is(err, storage.ErrDuplicateSampleForTimestamp):\n  // do NOT retry; sample is unapplicable\ndefault:\n  // transient storage issue: retry with backoff\n}","preventionTips":["Keep sender clocks NTP-synchronized to avoid out-of-order and out-of-bounds samples.","Monitor disk usage and TSDB head size on receive nodes.","Alert on 'writing locally' occurrences grouped by cause.","Set sensible out-of-order/out-of-limit windows for tenants that need them."],"tags":["tsdb","storage","thanos-receive","local-write"],"backgroundTag":"database-write-failed","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}