{"record":{"id":"77482d1fd2295a55","repo":"thanos-io/thanos","slug":"writing-s-data-to-local-tsdb-w","errorCode":null,"errorMessage":"writing %s data to local TSDB: %w","messagePattern":"writing (.+?) data to local TSDB: %w","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/receive/handler.go","lineNumber":1418,"sourceCode":"\t\tdata = append(data, wreqTenantTuple{\n\t\t\twreq: &prompb.WriteRequest{\n\t\t\t\tTimeseries: r.Timeseries,\n\t\t\t},\n\t\t\ttenant: r.Tenant,\n\t\t})\n\t}\n\n\t// Fast path for IngestorOnly mode: write directly to local TSDB.\n\t// This skips distributeTimeseriesToReplicas and sendLocalWrite since\n\t// the Router already determined this data belongs to this node.\n\tif h.receiverMode == IngestorOnly {\n\t\tvar errs = make([]error, 0, len(data))\n\t\tfor _, di := range data {\n\t\t\terr := h.writer.Write(ctx, di.tenant, di.wreq.Timeseries)\n\t\t\tif err != nil {\n\t\t\t\tlevel.Debug(h.logger).Log(\"msg\", \"failed to write to local TSDB\", \"err\", err, \"tenant\", di.tenant)\n\n\t\t\t\terrs = append(errs, fmt.Errorf(\"writing %s data to local TSDB: %w\", di.tenant, err))\n\t\t\t}\n\t\t}\n\n\t\tif len(errs) > 0 {\n\t\t\treturnErr := errs[0]\n\t\t\terr := errors.Unwrap(returnErr)\n\n\t\t\tif len(errs) > 1 {\n\t\t\t\treturnErr = fmt.Errorf(\"got %d errors while writing to multiple tenants, first one: %w\", len(errs), returnErr)\n\t\t\t}\n\n\t\t\tswitch cause := errors.Cause(err); cause {\n\t\t\tcase nil:\n\t\t\t\tpanic(\"BUG: errors.Cause returned nil on a non-nil error\")\n\t\t\tdefault:\n\t\t\t\tif isNotReady(cause) {\n\t\t\t\t\treturn nil, status.Error(codes.Unavailable, returnErr.Error())\n\t\t\t\t}","sourceCodeStart":1400,"sourceCodeEnd":1436,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/receive/handler.go#L1400-L1436","documentation":"The receive handler collects per-tenant failures from h.writer.Write (local TSDB append) into an errs slice, formatting each as \"writing <tenant> data to local TSDB: <err>\". It is an aggregate wrapper: the underlying cause (TSDB head full, storage error, etc.) is preserved via %w.","triggerScenarios":"Any data plane write to the local TSDB fails inside handler.go:1418, e.g. storage unavailable, out-of-order sample rejected, head block corrupted, or disk full; one entry is appended per failing tenant batch.","commonSituations":"Disk full on receive node; TSDB compaction failing; tenant allowed but samples rejected; multiple tenants failing simultaneously after a storage outage.","solutions":["Read the wrapped cause (%w) of the first error to find the real TSDB failure","Check disk space and local TSDB data-dir health on the receive node","Fix the root cause per-tenant (e.g. rejected samples) and replay/republish the data — replication should cover it if --receive.replication-factor > 1","Monitor writer.Write errors and alert on storage pressure"],"exampleFix":"// before: ignore storage health until writes fail\n// after: preflight check before accepting traffic\nif err := syscall.Access(dataDir, os.O_RDWR); err != nil {\n    logger.Warn(\"TSDB data dir not writable, draining receive\", \"err\", err)\n}\n","handlingStrategy":"try-catch","validationCode":"if free, err := diskFree(dataDir); err != nil || free < minFreeBytes {\n    // refuse or shed writes before TSDB write fails\n}","typeGuard":null,"tryCatchPattern":"err := h.writer.Write(ctx, tenant, series)\nif err != nil {\n    var cause error\n    for e := err; e != nil; e = errors.Unwrap(e) { cause = e }\n    logger.Error(\"local TSDB write failed\", \"tenant\", tenant, \"cause\", cause)\n    // fall back to a healthy receive node if replicated\n}","preventionTips":["Alert on receive disk usage and TSDB storage errors before writes fail","Ensure replication-factor >= 2 so client retries land on healthy nodes","Keep the TSDB data dir on adequately sized, monitored storage","Watch out-of-order/duplicate sample metrics per tenant and fix producers"],"tags":["storage","tsdb","write"],"backgroundTag":"database-write-failed","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}