{"record":{"id":"3937db48fa2aadcd","repo":"thanos-io/thanos","slug":"got-d-errors-while-writing-to-multiple-tenants-f","errorCode":null,"errorMessage":"got %d errors while writing to multiple tenants, first one: %w","messagePattern":"got (.+?) errors while writing to multiple tenants, first one: %w","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/receive/handler.go","lineNumber":1427,"sourceCode":"\t// This skips distributeTimeseriesToReplicas and sendLocalWrite since\n\t// the Router already determined this data belongs to this node.\n\tif h.receiverMode == IngestorOnly {\n\t\tvar errs = make([]error, 0, len(data))\n\t\tfor _, di := range data {\n\t\t\terr := h.writer.Write(ctx, di.tenant, di.wreq.Timeseries)\n\t\t\tif err != nil {\n\t\t\t\tlevel.Debug(h.logger).Log(\"msg\", \"failed to write to local TSDB\", \"err\", err, \"tenant\", di.tenant)\n\n\t\t\t\terrs = append(errs, fmt.Errorf(\"writing %s data to local TSDB: %w\", di.tenant, err))\n\t\t\t}\n\t\t}\n\n\t\tif len(errs) > 0 {\n\t\t\treturnErr := errs[0]\n\t\t\terr := errors.Unwrap(returnErr)\n\n\t\t\tif len(errs) > 1 {\n\t\t\t\treturnErr = fmt.Errorf(\"got %d errors while writing to multiple tenants, first one: %w\", len(errs), returnErr)\n\t\t\t}\n\n\t\t\tswitch cause := errors.Cause(err); cause {\n\t\t\tcase nil:\n\t\t\t\tpanic(\"BUG: errors.Cause returned nil on a non-nil error\")\n\t\t\tdefault:\n\t\t\t\tif isNotReady(cause) {\n\t\t\t\t\treturn nil, status.Error(codes.Unavailable, returnErr.Error())\n\t\t\t\t}\n\t\t\t\tif isConflict(cause) {\n\t\t\t\t\treturn nil, status.Error(codes.AlreadyExists, returnErr.Error())\n\t\t\t\t}\n\t\t\t\treturn nil, status.Error(codes.Internal, returnErr.Error())\n\t\t\t}\n\t\t}\n\n\t\treturn &storepb.WriteResponse{}, nil\n","sourceCodeStart":1409,"sourceCodeEnd":1445,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/receive/handler.go#L1409-L1445","documentation":"When writes for multiple tenants fail in one fan-out batch, the handler reports \"got %d errors while writing to multiple tenants, first one: %w\", keeping only the first error as the representative cause. The same wrapper name (returnErr) also appears in unrelated code (LazyBinaryReader.load), but the throw site is the receive handler.","triggerScenarios":"len(errs) > 1 in handler.go:1427: at least two tenant writes to the local TSDB failed in the same batch, e.g. a storage-wide outage (disk full, TSDB closed) affecting several tenants at once.","commonSituations":"Shared-storage failure hitting all tenants; simultaneous out-of-order rejections from several producers; head compaction stall delaying many writes.","solutions":["Inspect the first wrapped error's cause for the shared root failure (storage/disk)","Check receive node disk, memory, and TSDB health — multi-tenant failure usually means a node-level problem","Rely on replication: clients should retry; verify replication-factor > 1 covered the data","Fix the per-tenant causes (e.g. invalid samples) if logs show tenant-specific rejection reasons"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"if free, err := diskFree(dataDir); err != nil || free < minFreeBytes {\n    // node-level storage problem will hit all tenants; shed writes early\n}","typeGuard":null,"tryCatchPattern":"err := handler.Write(ctx, req)\nif err != nil && strings.Contains(err.Error(), \"errors while writing to multiple tenants\") {\n    // node-wide failure: retry against a different receive node after backoff\n    return retryOtherEndpoint(ctx, req)\n}","preventionTips":["Monitor node-level disk/memory — multi-tenant write failures indicate a node problem, not per-tenant config","Use replication so batch failures are survivable via client retries","Check the first error's wrapped cause, since only the first failure is reported","Avoid sending batches spanning many tenants to a node approaching resource limits"],"tags":["storage","multitenancy","aggregate-error"],"backgroundTag":"database-write-failed","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}