thanos-io/thanos · error

add series

Error message

add series

What it means

During rewrite(), each sorted series is added to the new index writer via indexw.AddSeries. A failure wrapped as "add series" means the TSDB index writer rejected adding series i with its labels and chunk metas — usually a validation failure in the new index being built.

Solutions

1. Inspect the wrapped cause to see which series/chunk validation failed (labelset is in the rewrite log for duplicates)
2. Delete the corrupted block from the bucket so the Thanos compactor repairs or removes it, instead of repeatedly rewriting
3. Verify the source block (`thanos tools bucket verify`) and confirm the Prometheus version that produced it is supported
4. Retry after fixing underlying I/O/permission errors if the cause is disk-level

for _, s := range series {
	if labels.HasDuplicateLabelNames(s.lset) {
		return errors.Errorf("series %s has duplicate label names, block corrupt", s.lset)
	}
	for _, c := range s.chks {
		if c.MaxTime <= c.MinTime { return errors.Errorf("invalid chunk in series %s", s.lset) }
	}
}

if err := rewrite(...); err != nil {
	var idxErr *index.ValidationError
	if goerrors.As(errors.Cause(err), &idxErr) {
		// writer rejected a series/chunk: treat block as corrupt, delete & re-sync
		return markBlockCorrupt(ctx, bkt, id)
	}
	return err
}

When it happens

Trigger: indexw.AddSeries(i, s.lset, s.chks...) errors during rewrite/Repair: typically labels invalid (duplicate label names), series out of order relative to the writer's expectations, chunk metas invalid (e.g. empty chunks, wrong ordering), or an I/O error writing symbol/series entries.

Common situations: Repairing blocks that contain malformed series created by an older Prometheus version or by a buggy compaction; blocks with duplicate/out-of-order series discovered during verification and repair; corrupted index producing invalid chunk metadata that fails writer validation.

Related errors

• iterate series
• write chunks
• open block
• open block
• open chunks

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/2e288ec80ec39c9d. Report an issue: GitHub.

Appendix: source

Thrown at pkg/block/index.go:673

	for _, s := range series {
		// The TSDB library will throw an error if we add a series with
		// identical labels as the last series. This means that we have
		// discovered a duplicate time series in the old block. We drop
		// all duplicate series preserving the first one.
		// TODO: Add metric to count dropped series if repair becomes a daemon
		// rather than a batch job.
		if labels.Compare(lastSet, s.lset) == 0 {
			level.Warn(logger).Log("msg",
				"dropping duplicate series in tsdb block found",
				"labelset", s.lset.String(),
			)
			continue
		}
		if err := chunkw.WriteChunks(s.chks...); err != nil {
			return errors.Wrap(err, "write chunks")
		}
		if err := indexw.AddSeries(i, s.lset, s.chks...); err != nil {
			return errors.Wrap(err, "add series")
		}

		meta.Stats.NumChunks += uint64(len(s.chks))
		meta.Stats.NumSeries++

		for _, chk := range s.chks {
			meta.Stats.NumSamples += uint64(chk.Chunk.NumSamples())
		}

		s.lset.Range(func(l labels.Label) {
			valset, ok := values[l.Name]
			if !ok {
				valset = stringset{}
				values[l.Name] = valset
			}
			valset.set(l.Value)
		})
		postings.Add(i, s.lset)

View on GitHub (pinned to 35b8b99117)