thanos-io/thanos · error

gather index issues

Error message

gather index issues %s

What it means

After downloading, verifyIndex calls block.GatherIndexHealthStats to scan the index for corruptions (series outside chunk bounds, duplicate postings, issue-347 style defects); any error is wrapped as "gather index issues %s". This means the index health analysis itself could not complete, distinct from stats.AnyErr() which reports the found defects.

Solutions

1. Read the wrapped cause to see whether it's a parse, format, or I/O error.
2. Re-download with a fresh work directory to rule out local corruption.
3. If the index is truly unreadable, treat the block as corrupt: back it up and delete it, or restore from backup.
4. Upgrade Thanos if the index format is newer than the tool supports.

Example fix

// before: reusing a work dir with a truncated index file
workDir := "./work"
// after: fresh temp dir per verification run
workDir, _ := os.MkdirTemp("", "verifier-")
defer os.RemoveAll(workDir)

// ensure the downloaded index is non-empty and readable before analysis
fi, err := os.Stat(filepath.Join(dir, "index"))
if err != nil || fi.Size() == 0 { return fmt.Errorf("index file missing/empty locally") }

if err := verify.VerifyIssue(...); err != nil {
    if strings.Contains(err.Error(), "gather index issues") {
        // index unparseable: re-download with fresh work dir or treat block as corrupt
    }
}

When it happens

Trigger: GatherIndexHealthStats errors while parsing the downloaded index: truncated or unreadable index file on local disk, unsupported/invalid index format, or an I/O error reading the local file.

Common situations: Index file corrupted in the bucket so it cannot even be parsed; disk full in the work dir corrupting the download; extremely old index format not understood by this Thanos version.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• iterate series
• repaired block is invalid
• block with not healthy index found
• block id , try running with --debug.accept-malformed-index
• get series

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/e4d279df44aeb8f6. Report an issue: GitHub.

Appendix: source

Thrown at pkg/verifier/index_issue.go:139

		return errors.Wrapf(err, "upload of %s failed", resid)
	}

	level.Info(ctx.Logger).Log("msg", "safe deleting broken block", "id", id, "issue")
	if err := BackupAndDeleteDownloaded(ctx, filepath.Join(dir, id.String()), id); err != nil {
		return errors.Wrapf(err, "safe deleting old block %s failed", id)
	}

	return nil
}

func verifyIndex(ctx Context, id ulid.ULID, dir string, meta *metadata.Meta) (stats block.HealthStats, err error) {
	if err := objstore.DownloadFile(ctx, ctx.Logger, ctx.Bkt, path.Join(id.String(), block.IndexFilename), filepath.Join(dir, block.IndexFilename)); err != nil {
		return stats, errors.Wrapf(err, "download index file %s", path.Join(id.String(), block.IndexFilename))
	}

	stats, err = block.GatherIndexHealthStats(ctx, ctx.Logger, filepath.Join(dir, block.IndexFilename), meta.MinTime, meta.MaxTime)
	if err != nil {
		return stats, errors.Wrapf(err, "gather index issues %s", id)
	}

	level.Debug(ctx.Logger).Log("stats", fmt.Sprintf("%+v", stats), "id", id)

	return stats, stats.AnyErr()
}

View on GitHub (pinned to 35b8b99117)