thanos-io/thanos · error

read series

Error message

read series

What it means

For each posting ID, GatherIndexHealthStats calls r.Series(id, ...) to decode a series' labels and chunks. If the series entry cannot be read/decoded from the index, the error is wrapped as "read series". This means the series section of the index references data that is missing or malformed.

Solutions

1. Re-obtain the block from object storage or backup
2. Verify with VerifyIndex / tsdb tooling to pinpoint the bad series offset
3. Rebuild the block index via offline compaction of its chunks if the data is intact
4. Drop the corrupt block

Example fix

// before
stats, err := block.GatherIndexHealthStats(ctx, bdir) // read series: invalid series ref
// after
resid, err := block.Repair(ctx, logger, filepath.Dir(bdir), id, meta.Thanos.Source)
if err != nil { logger.Log("msg", "repair failed, dropping block", "err", err) }

meta, err := metadata.ReadFromDir(bdir)
if err != nil { return err }
if meta.Thanos.Downsample.Resolution < 0 { return fmt.Errorf("invalid metadata in %s", bdir) }

stats, err := block.GatherIndexHealthStats(ctx, bdir)
if err != nil && strings.Contains(err.Error(), "read series") {
    return fmt.Errorf("series section corrupt in %s, rebuild via compaction: %w", bdir, err)
}

When it happens

Trigger: GatherIndexHealthStats iterating postings of a block whose series entries are truncated or misaligned (e.g. wrong offsetMultiplier handling for index format, damaged index file), or ctx cancellation.

Common situations: Index files produced by buggy or downgraded TSDB writers, hand-copied blocks missing parts of the index, storage bit-rot, corrupted uploads.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• series
• reading series length failed
• invalid remaining size, even after refetch, remaining
• input block index not valid
• get all postings

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/a5fa26e1184b42da. Report an issue: GitHub.

Appendix: source

Thrown at pkg/block/index.go:279

	if version >= 2 {
		offsetMultiplier = 16
	}

	// Per series.
	var prevId storage.SeriesRef
	for p.Next() {
		prevLset.CopyFrom(lset)

		id := p.At()
		if prevId != 0 {
			// Approximate size.
			seriesSize.Add(int64(id-prevId) * int64(offsetMultiplier))
		}
		prevId = id
		stats.TotalSeries++

		if err := r.Series(id, &builder, &chks); err != nil {
			return stats, errors.Wrap(err, "read series")
		}
		lset = builder.Labels()
		if lset.IsEmpty() {
			return stats, errors.Errorf("empty label set detected for series %d", id)
		}
		if !prevLset.IsEmpty() && labels.Compare(prevLset, lset) >= 0 {
			return stats, errors.Errorf("series %v out of order; previous %v", lset, prevLset)
		}
		var l0 *labels.Label
		lset.Range(func(l labels.Label) {
			if l0 != nil {
				if l.Name < l0.Name {
					stats.OutOfOrderLabels++
					level.Warn(logger).Log("msg",
						"out-of-order label set: known bug in Prometheus 2.8.0 and below",
						"labelset", lset.String(),
						"series", fmt.Sprintf("%d", id),
					)

View on GitHub (pinned to 35b8b99117)