thanos-io/thanos · error

gather index issues for block

Error message

gather index issues for block %s

What it means

After downloading, Thanos validates each block's index with block.GatherIndexHealthStats. If scanning the index file for structural issues fails (e.g. file unreadable, truncated, or the health-stat walker errors), the error is wrapped as 'gather index issues for block <dir>'. This happens before any data is modified.

Solutions

1. Delete the block's local directory under --data-dir and let compaction re-download it
2. Verify the block in the bucket (thanos tools bucket verify) — if corrupt remotely, run bucket repair or delete the bad block
3. Check local disk health and free space on the compactor node
4. Upgrade Thanos if the index was produced by a newer Thanos/Prometheus with format changes

Example fix

// before
// corrupt block dir cached locally
# rm -rf /data/compact/0ULID.../
// after
// re-run compactor; block is re-downloaded and re-validated

if err != nil && strings.Contains(err.Error(), "gather index issues") {
    // purge local cache dir for that block and retry once
    os.RemoveAll(blockLocalDir)
}

When it happens

Trigger: block.GatherIndexHealthStats on filepath.Join(bdir, block.IndexFilename) returns an error: truncated/corrupt index file on disk, I/O failure reading the downloaded index, or internal validation walk failing.

Common situations: Incomplete block downloads (index.toml present but index truncated); disk corruption; version mismatch where index format is not understood; out-of-disk conditions during concurrent fetches.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• block with not healthy index found
• blocks with out-of-order chunks are dropped from compaction
• invalid, but reparable block
• block id , try running with --debug.accept-malformed-index
• create bucket compactor

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/8cfb53365722bc92. Report an issue: GitHub.

Appendix: source

Thrown at pkg/compact/compact.go:1230

		bdir := filepath.Join(dir, m.ULID.String())
		func(ctx context.Context, meta *metadata.Meta) {
			g.Go(func() error {
				start := time.Now()
				if err := tracing.DoInSpanWithErr(ctx, "compaction_block_download", func(ctx context.Context) error {
					return block.Download(ctx, cg.logger, cg.bkt, meta.ULID, bdir, objstore.WithFetchConcurrency(cg.blockFilesConcurrency))
				}, opentracing.Tags{"block.id": meta.ULID}); err != nil {
					return retry(errors.Wrapf(err, "download block %s", meta.ULID))
				}
				level.Debug(cg.logger).Log("msg", "downloaded block", "block", meta.ULID.String(), "duration", time.Since(start), "duration_ms", time.Since(start).Milliseconds())

				start = time.Now()
				// Ensure all input blocks are valid.
				var stats block.HealthStats
				if err := tracing.DoInSpanWithErr(ctx, "compaction_block_health_stats", func(ctx context.Context) (e error) {
					stats, e = block.GatherIndexHealthStats(ctx, cg.logger, filepath.Join(bdir, block.IndexFilename), meta.MinTime, meta.MaxTime)
					return e
				}, opentracing.Tags{"block.id": meta.ULID}); err != nil {
					return errors.Wrapf(err, "gather index issues for block %s", bdir)
				}

				if err := stats.CriticalErr(); err != nil {
					return halt(errors.Wrapf(err, "block with not healthy index found %s; Compaction level %v; Labels: %v", bdir, meta.Compaction.Level, meta.Thanos.Labels))
				}

				if err := stats.OutOfOrderChunksErr(); err != nil {
					return outOfOrderChunkError(errors.Wrapf(err, "blocks with out-of-order chunks are dropped from compaction:  %s", bdir), meta.ULID)
				}

				if err := stats.Issue347OutsideChunksErr(); err != nil {
					return issue347Error(errors.Wrapf(err, "invalid, but reparable block %s", bdir), meta.ULID)
				}

				if err := stats.OutOfOrderLabelsErr(); !cg.acceptMalformedIndex && err != nil {
					return errors.Wrapf(err,
						"block id %s, try running with --debug.accept-malformed-index", meta.ULID)
				}

View on GitHub (pinned to 35b8b99117)