got errors while writing to multiple tenants, first one

Error message

got %d errors while writing to multiple tenants, first one: %w

What it means

When writes for multiple tenants fail in one fan-out batch, the handler reports "got %d errors while writing to multiple tenants, first one: %w", keeping only the first error as the representative cause. The same wrapper name (returnErr) also appears in unrelated code (LazyBinaryReader.load), but the throw site is the receive handler.

Solutions

Inspect the first wrapped error's cause for the shared root failure (storage/disk)
Check receive node disk, memory, and TSDB health — multi-tenant failure usually means a node-level problem
Rely on replication: clients should retry; verify replication-factor > 1 covered the data
Fix the per-tenant causes (e.g. invalid samples) if logs show tenant-specific rejection reasons

Defensive patterns

Strategy: try-catch

Validate before calling

if free, err := diskFree(dataDir); err != nil || free < minFreeBytes {
    // node-level storage problem will hit all tenants; shed writes early
}

Try / catch

err := handler.Write(ctx, req)
if err != nil && strings.Contains(err.Error(), "errors while writing to multiple tenants") {
    // node-wide failure: retry against a different receive node after backoff
    return retryOtherEndpoint(ctx, req)
}

Prevention

Monitor node-level disk/memory — multi-tenant write failures indicate a node problem, not per-tenant config
Use replication so batch failures are survivable via client retries
Check the first error's wrapped cause, since only the first failure is reported
Avoid sending batches spanning many tenants to a node approaching resource limits

When it happens

Trigger: len(errs) > 1 in handler.go:1427: at least two tenant writes to the local TSDB failed in the same batch, e.g. a storage-wide outage (disk full, TSDB closed) affecting several tenants at once.

Common situations: Shared-storage failure hitting all tenants; simultaneous out-of-order rejections from several producers; head compaction stall delaying many writes.

Related errors

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/3937db48fa2aadcd. Report an issue: GitHub.

Appendix: source

Thrown at pkg/receive/handler.go:1427

	// This skips distributeTimeseriesToReplicas and sendLocalWrite since
	// the Router already determined this data belongs to this node.
	if h.receiverMode == IngestorOnly {
		var errs = make([]error, 0, len(data))
		for _, di := range data {
			err := h.writer.Write(ctx, di.tenant, di.wreq.Timeseries)
			if err != nil {
				level.Debug(h.logger).Log("msg", "failed to write to local TSDB", "err", err, "tenant", di.tenant)

				errs = append(errs, fmt.Errorf("writing %s data to local TSDB: %w", di.tenant, err))
			}
		}

		if len(errs) > 0 {
			returnErr := errs[0]
			err := errors.Unwrap(returnErr)

			if len(errs) > 1 {
				returnErr = fmt.Errorf("got %d errors while writing to multiple tenants, first one: %w", len(errs), returnErr)
			}

			switch cause := errors.Cause(err); cause {
			case nil:
				panic("BUG: errors.Cause returned nil on a non-nil error")
			default:
				if isNotReady(cause) {
					return nil, status.Error(codes.Unavailable, returnErr.Error())
				}
				if isConflict(cause) {
					return nil, status.Error(codes.AlreadyExists, returnErr.Error())
				}
				return nil, status.Error(codes.Internal, returnErr.Error())
			}
		}

		return &storepb.WriteResponse{}, nil

View on GitHub (pinned to 35b8b99117)