thanos-io/thanos · error

get all postings

Error message

get all postings

What it means

GatherIndexHealthStats reads a block's index via Postings(ctx, index.AllPostingsKey(), "") to iterate every series. If the postings list for the all-postings key cannot be read (corrupt/truncated index file, missing postings table section, or context cancellation), the underlying error is wrapped with "get all postings". It signals the block index is unreadable at the postings level.

Solutions

1. Re-download or restore the block from object storage or a backup; the index is likely corrupt
2. Run VerifyIndex on the block to confirm and localize index corruption
3. Check ctx timeouts/deadlines are long enough when verifying large blocks
4. Delete the corrupt block (it will typically be re-synced/re-compacted) if it cannot be repaired

Example fix

// before
stats, err := block.GatherIndexHealthStats(ctx, bdir) // fails: get all postings
// after
if err := block.VerifyIndex(ctx, bdir, defaultVerifyIndexConcurrency); err != nil {
    logger.Log("msg", "corrupt block, removing", "dir", bdir, "err", err)
    os.RemoveAll(bdir)
}

f, err := os.Open(filepath.Join(bdir, "index"))
if err != nil { return err }
fi, _ := f.Stat(); f.Close()
if fi.Size() == 0 { return fmt.Errorf("index file empty in %s", bdir) }

func hasIndexFile(bdir string) bool {
    fi, err := os.Stat(filepath.Join(bdir, "index"))
    return err == nil && !fi.IsDir() && fi.Size() > 0
}

stats, err := block.GatherIndexHealthStats(ctx, bdir)
if err != nil && strings.Contains(err.Error(), "get all postings") {
    return fmt.Errorf("block %s index unreadable, re-sync from object storage: %w", bdir, err)
}

When it happens

Trigger: Calling GatherIndexHealthStats (directly or via VerifyIndex or processDownsampling) on a block whose index file has a damaged or missing postings section, or whose context is cancelled/times out while reading postings.

Common situations: Blocks corrupted by incomplete uploads to object storage, disk corruption on compacted blocks, interrupted block downloads, or cancelled/short-timeout contexts during verification of many blocks.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• input block index not valid
• label names
• metric label values
• read series
• empty label set detected for series

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/2e293d256b04ce2f. Report an issue: GitHub.

Appendix: source

Thrown at pkg/block/index.go:228

	}
	return n.sum / n.cnt
}

// GatherIndexHealthStats returns useful counters as well as outsider chunks (chunks outside of block time range) that
// helps to assess index health.
// It considers https://github.com/prometheus/tsdb/issues/347 as something that Thanos can handle.
// See HealthStats.Issue347OutsideChunks for details.
func GatherIndexHealthStats(ctx context.Context, logger log.Logger, fn string, minTime, maxTime int64) (stats HealthStats, err error) {
	r, err := index.NewFileReader(fn, index.DecodePostingsRaw)
	if err != nil {
		return stats, errors.Wrap(err, "open index file")
	}
	defer runutil.CloseWithErrCapture(&err, r, "gather index issue file reader")

	key, value := index.AllPostingsKey()
	p, err := r.Postings(ctx, key, value)
	if err != nil {
		return stats, errors.Wrap(err, "get all postings")
	}
	var (
		lset     labels.Labels
		prevLset labels.Labels
		builder  labels.ScratchBuilder

		chks []chunks.Meta

		seriesLifeDuration                          = newMinMaxSumInt64()
		seriesLifeDurationWithoutSingleSampleSeries = newMinMaxSumInt64()
		seriesChunks                                = newMinMaxSumInt64()
		chunkDuration                               = newMinMaxSumInt64()
		chunkSize                                   = newMinMaxSumInt64()
		seriesSize                                  = newMinMaxSumInt64()
	)

	lnames, err := r.LabelNames(ctx)
	if err != nil {

View on GitHub (pinned to 35b8b99117)