thanos-io/thanos · error

empty chunks for series

Error message

empty chunks for series %d

What it means

Every series must reference at least one chunk in the index. GatherIndexHealthStats errors with "empty chunks for series %d" when a decoded series has zero chunks. Such a series would be invisible to queries, so the index is considered corrupt.

Solutions

1. Restore the block from object storage or backup
2. Run offline compaction to rebuild the index from the chunk data if chunks are intact
3. Use block.Repair to drop incomplete/outside data where applicable
4. Delete the corrupt block and let it re-sync or be tombstoned

Example fix

// before
stats, err := block.GatherIndexHealthStats(ctx, bdir) // empty chunks for series 987
// after
if err := block.VerifyIndex(ctx, bdir, 1); err != nil {
    logger.Log("msg", "dropping corrupt block", "err", err)
    os.RemoveAll(bdir)
}

if err := block.VerifyIndex(ctx, bdir, 1); err != nil {
    return fmt.Errorf("unhealthy block %s, refusing to use: %w", bdir, err)
}

stats, err := block.GatherIndexHealthStats(ctx, bdir)
if err != nil && strings.Contains(err.Error(), "empty chunks for series") {
    return fmt.Errorf("index chunk section truncated in %s: %w", bdir, err)
}

When it happens

Trigger: r.Series returns successfully but the chunk meta slice is empty — caused by a damaged index chunk section, truncation cutting chunk metas off, or buggy index writers.

Common situations: Blocks truncated during upload/download, storage corruption cutting the index short, blocks produced by experimental/broken tooling, verification after incidents.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• input block index not valid
• get all postings
• label names
• metric label values
• read series

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/1c2c3c2991b89fb2. Report an issue: GitHub.

Appendix: source

Thrown at pkg/block/index.go:304

			return stats, errors.Errorf("series %v out of order; previous %v", lset, prevLset)
		}
		var l0 *labels.Label
		lset.Range(func(l labels.Label) {
			if l0 != nil {
				if l.Name < l0.Name {
					stats.OutOfOrderLabels++
					level.Warn(logger).Log("msg",
						"out-of-order label set: known bug in Prometheus 2.8.0 and below",
						"labelset", lset.String(),
						"series", fmt.Sprintf("%d", id),
					)
				}
			}
			l0 = &l
		})

		if len(chks) == 0 {
			return stats, errors.Errorf("empty chunks for series %d", id)
		}

		ooo := 0
		seriesLifeTimeMs := int64(0)
		// Per chunk in series.
		for i, c := range chks {
			stats.TotalChunks++

			chkDur := c.MaxTime - c.MinTime
			seriesLifeTimeMs += chkDur
			chunkDuration.Add(chkDur)
			if chkDur == 0 {
				stats.SingleSampleChunks++
			}

			// Approximate size.
			if i < len(chks)-2 {
				sgmIndex, chkStart := chunks.BlockChunkRef(c.Ref).Unpack()

View on GitHub (pinned to 35b8b99117)