thanos-io/thanos · error

empty label set detected for series

Error message

empty label set detected for series %d

What it means

GatherIndexHealthStats detects a series whose decoded label set is empty (no labels at all). Since every TSDB series must have at least a __name__ label, this is a hard invariant violation indicating a corrupt or malformed index. The error message includes the offending series reference ID.

Solutions

1. Treat the block as corrupt; restore it from object storage or backup
2. Identify the offending series ref from the error message to confirm offset corruption
3. Re-run offline compaction to rebuild the index if chunk data is intact
4. Delete the block so it is re-fetched or tombstoned

Example fix

// before
stats, err := block.GatherIndexHealthStats(ctx, bdir) // empty label set detected for series 12345
// after
meta, err := metadata.ReadFromDir(bdir)
if err != nil { return err }
if _, err := block.Repair(ctx, logger, filepath.Dir(bdir), meta.ULID, meta.Thanos.Source); err != nil {
    return fmt.Errorf("block %s unusable, remove manually: %w", meta.ULID, err)
}

if err := block.VerifyIndex(ctx, bdir, 1); err != nil {
    return fmt.Errorf("block %s failed index verification: %w", bdir, err)
}

stats, err := block.GatherIndexHealthStats(ctx, bdir)
var emptyLsetErr *errors.errorString
if err != nil && strings.Contains(err.Error(), "empty label set detected") {
    return block.Repair(ctx, logger, filepath.Dir(bdir), id, source, block.IgnoreIssue347OutsideChunks)
}

When it happens

Trigger: r.Series(id, ...) decodes successfully but yields an empty label set — typically a damaged series section, mispadded offsets (v2 index 16-byte padding), or an index written by buggy code.

Common situations: Blocks affected by historical TSDB index bugs, corrupted uploads, blocks whose index was partially overwritten, forensic verification after storage incidents.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• input block index not valid
• get all postings
• label names
• metric label values
• read series

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/ddb7027f72df73e8. Report an issue: GitHub.

Appendix: source

Thrown at pkg/block/index.go:283

	// Per series.
	var prevId storage.SeriesRef
	for p.Next() {
		prevLset.CopyFrom(lset)

		id := p.At()
		if prevId != 0 {
			// Approximate size.
			seriesSize.Add(int64(id-prevId) * int64(offsetMultiplier))
		}
		prevId = id
		stats.TotalSeries++

		if err := r.Series(id, &builder, &chks); err != nil {
			return stats, errors.Wrap(err, "read series")
		}
		lset = builder.Labels()
		if lset.IsEmpty() {
			return stats, errors.Errorf("empty label set detected for series %d", id)
		}
		if !prevLset.IsEmpty() && labels.Compare(prevLset, lset) >= 0 {
			return stats, errors.Errorf("series %v out of order; previous %v", lset, prevLset)
		}
		var l0 *labels.Label
		lset.Range(func(l labels.Label) {
			if l0 != nil {
				if l.Name < l0.Name {
					stats.OutOfOrderLabels++
					level.Warn(logger).Log("msg",
						"out-of-order label set: known bug in Prometheus 2.8.0 and below",
						"labelset", lset.String(),
						"series", fmt.Sprintf("%d", id),
					)
				}
			}
			l0 = &l
		})

View on GitHub (pinned to 35b8b99117)