thanos-io/thanos · critical

open index

Error message

open index

What it means

block.Repair wraps the error returned by b.Index(), which opens a read handle over the block's index file after the block was successfully opened. Failure means the index file exists but cannot be read/parsed into an index reader — typically decoding errors from corruption. Wrapped as 'open index: <cause>'.

Solutions

1. Read the wrapped cause; if it indicates index decoding corruption, verify with `thanos tools bucket verify` on the bucket or a local index inspection.
2. Re-fetch the block from object storage and compare checksums; retry Repair on the intact copy.
3. If no intact copy exists, delete the block and regenerate data from the source Prometheus (re-upload or re-compact).
4. Keep Repair as a last resort: prefer preventing issue-347 corruption by upgrading Prometheus past the bug.

Example fix

// before
_, err = block.RepairIssue347(ctx, logger, dir, id, source)
// treat as transient and retry forever
// after
_, err = block.RepairIssue347(ctx, logger, dir, id, source)
if err != nil && strings.Contains(err.Error(), "open index") {
    // index unreadable: re-download block once, then fail permanently
    if derr := redownloadBlock(ctx, bucket, dir, id); derr != nil {
        return derr
    }
    _, err = block.RepairIssue347(ctx, logger, dir, id, source)
}

resid, err := block.RepairIssue347(ctx, logger, dir, id, source)
if err != nil && strings.Contains(err.Error(), "open index") {
    // index unreadable: quarantine the block instead of retrying
    if qerr := quarantineBlock(dir, id); qerr != nil {
        return resid, qerr
    }
    return resid, fmt.Errorf("block %s quarantined: index unreadable", id)
}

When it happens

Trigger: Repair on a block whose index file is corrupt (e.g. the tsdb issue-347 out-of-order-chunk corruption that Repair targets), truncated during transfer, or whose TOC/symbols sections are damaged; disk read errors mid-open.

Common situations: Blocks affected by the Prometheus tsdb#347 bug where the index is malformed enough that even opening a reader fails; blocks uploaded while Prometheus was still writing them (rare, but seen with unsafe tooling); partial restore from backup.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• start remote write agent db
• open TSDB
• open block
• read dir
• stat

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/03fa8375c90cdd31. Report an issue: GitHub.

Appendix: source

Thrown at pkg/block/index.go:440

	resid = ulid.MustNew(ulid.Now(), entropy)

	meta, err := metadata.ReadFromDir(bdir)
	if err != nil {
		return resid, errors.Wrap(err, "read meta file")
	}
	if meta.Thanos.Downsample.Resolution > 0 {
		return resid, errors.New("cannot repair downsampled block")
	}

	b, err := tsdb.OpenBlock(logutil.GoKitLogToSlog(logger), bdir, nil, nil)
	if err != nil {
		return resid, errors.Wrap(err, "open block")
	}
	defer runutil.CloseWithErrCapture(&err, b, "repair block reader")

	indexr, err := b.Index()
	if err != nil {
		return resid, errors.Wrap(err, "open index")
	}
	defer runutil.CloseWithErrCapture(&err, indexr, "repair index reader")

	chunkr, err := b.Chunks()
	if err != nil {
		return resid, errors.Wrap(err, "open chunks")
	}
	defer runutil.CloseWithErrCapture(&err, chunkr, "repair chunk reader")

	resdir := filepath.Join(dir, resid.String())

	chunkw, err := chunks.NewWriter(filepath.Join(resdir, ChunksDirname))
	if err != nil {
		return resid, errors.Wrap(err, "open chunk writer")
	}
	defer runutil.CloseWithErrCapture(&err, chunkw, "repair chunk writer")

	indexw, err := index.NewWriter(context.TODO(), filepath.Join(resdir, IndexFilename))

View on GitHub (pinned to 35b8b99117)