thanos-io/thanos · error

repair failed for block

Error message

repair failed for block %s

What it means

This error wraps a failure from block.Repair during Thanos' overlapping-block repair flow (compact.go repairBucketBlock). block.Repair downloads the broken block, rewrites/removes out-of-order chunks related to the Prometheus issue 347, and produces a new block under tmpdir; any failure in that pipeline (download, rewrite, index creation) is surfaced wrapped with the original block id. It indicates the broken block could not be automatically fixed.

Solutions

1. Check compactor disk usage and increase the volume backing tmpdir; repair writes a full copy of the block
2. Run 'thanos tools bucket verify' to identify and isolate the corrupt block, then remove it from the bucket if it is not needed (data duplication is acceptable, data loss is not)
3. Look at the inner error in the logs: it distinguishes download failure vs index rewrite failure; fix the specific layer (network vs corruption)
4. Ensure only one compactor group operates per bucket/prefix to avoid races during repair
5. As a last resort, delete the broken block (MarkForDeletion path) and restore data for that window from a backup or re-scrape

// pre-check free disk space before enabling repair in the compactor
import "syscall"
var st syscall.Statfs_t
if err := syscall.Statfs(tmpdir, &st); err != nil || st.Bavail*uint64(st.Bsize) < minFreeBytes {
    return fmt.Errorf("insufficient disk in %s for repair", tmpdir)
}

if err := block.Repair(ctx, logger, tmpdir, ie.id, metadata.CompactorRepairSource, block.IgnoreIssue347OutsideChunk); err != nil {
    logger.Error("block repair failed; quarantining block", "id", ie.id, "err", err)
    // decide: mark source block for deletion or alert an operator instead of hot-looping
    return errors.Wrapf(err, "repair failed for block %s", ie.id)
}

When it happens

Trigger: block.Repair(ctx, logger, tmpdir, ie.id, metadata.CompactorRepairSource, block.IgnoreIssue347OutsideChunk) fails: the source block's index is unrecoverably corrupt, chunks outside the expected overlap window cannot be handled, disk full in tmpdir during repair, object storage download failure mid-repair, or the block cannot be opened as a TSDB block at all (beyond just bad meta).

Common situations: Bucket contains blocks from Prometheus instances with clock skew producing overlapping samples that the repair cannot partition cleanly; disk space exhausted on the compactor's local volume holding tmpdir; blocks affected by TSDB index corruption that even the issue-347 repair path cannot rebuild; concurrent compactor instances racing on the same block in the bucket.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• open block
• found chunks non-completely outside the block time range…
• open block
• open chunks
• next symbol

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/40e294e31de5af48. Report an issue: GitHub.

Appendix: source

Thrown at pkg/compact/compact.go:1141

	defer func() {
		if err := os.RemoveAll(tmpdir); err != nil {
			level.Warn(logger).Log("msg", "failed to remote tmpdir", "err", err, "tmpdir", tmpdir)
		}
	}()

	bdir := filepath.Join(tmpdir, ie.id.String())
	if err := block.Download(ctx, logger, bkt, ie.id, bdir); err != nil {
		return retry(errors.Wrapf(err, "download block %s", ie.id))
	}

	meta, err := metadata.ReadFromDir(bdir)
	if err != nil {
		return errors.Wrapf(err, "read meta from %s", bdir)
	}

	resid, err := block.Repair(ctx, logger, tmpdir, ie.id, metadata.CompactorRepairSource, block.IgnoreIssue347OutsideChunk)
	if err != nil {
		return errors.Wrapf(err, "repair failed for block %s", ie.id)
	}

	// Verify repaired id before uploading it.
	if err := block.VerifyIndex(ctx, logger, filepath.Join(tmpdir, resid.String(), block.IndexFilename), meta.MinTime, meta.MaxTime); err != nil {
		return errors.Wrapf(err, "repaired block is invalid %s", resid)
	}

	level.Info(logger).Log("msg", "uploading repaired block", "newID", resid)
	if err = block.Upload(ctx, logger, bkt, filepath.Join(tmpdir, resid.String()), metadata.NoneFunc); err != nil {
		return retry(errors.Wrapf(err, "upload of %s failed", resid))
	}

	level.Info(logger).Log("msg", "deleting broken block", "id", ie.id)

	// Spawn a new context so we always mark a block for deletion in full on shutdown.
	delCtx, cancel := context.WithTimeout(context.Background(), 5*time.Minute)
	defer cancel()

View on GitHub (pinned to 35b8b99117)