thanos-io/thanos · error

repaired block is invalid

Error message

repaired block is invalid %s

What it means

This error wraps a failure from block.VerifyIndex after the repair step produced a new block (resid) in tmpdir (compact.go repairBucketBlock). VerifyIndex opens the repaired block's index and checks postings/series against the expected [meta.MinTime, meta.MaxTime] window. If the freshly repaired block does not pass index validation, the compactor refuses to upload it, preventing corrupt data from entering the bucket.

Solutions

1. Read the inner VerifyIndex error: it names the failing index component (postings, symbols, series) and ULID range mismatch
2. Re-run the repair with free disk space and a clean tmpdir (stale tmp files can produce partial blocks)
3. Upgrade Thanos to the latest patch version; several repair/VerifyIndex bugs were fixed over releases
4. If repair repeatedly fails verification, delete the broken source block from the bucket (it is unusable anyway) rather than retrying indefinitely
5. Verify the source block with 'thanos tools bucket verify' to confirm whether the input was already unrecoverable

// sanity-check the repaired index before verification-time surprises
if _, err := os.Stat(filepath.Join(tmpdir, resid.String(), block.IndexFilename)); err != nil {
    return fmt.Errorf("repaired index missing: %w", err)
}

if err := block.VerifyIndex(ctx, logger, idxPath, meta.MinTime, meta.MaxTime); err != nil {
    logger.Error("repaired block failed verification", "newID", resid, "err", err)
    os.RemoveAll(filepath.Join(tmpdir, resid.String())) // clean partial repair output
    return errors.Wrapf(err, "repaired block is invalid %s", resid)
}

When it happens

Trigger: block.VerifyIndex(ctx, logger, filepath.Join(tmpdir, resid.String(), block.IndexFilename), meta.MinTime, meta.MaxTime) returns error: the repaired index has postings outside or inconsistent with the original block's minTime/maxTime, the index file is missing at tmpdir/<resid>/index, index binary format is unreadable, or the repair rewrote series incorrectly producing lookup failures.

Common situations: Repair produced a block whose time bounds shifted relative to the source meta.json; disk corruption or truncation in tmpdir; a Thanos bug in the issue-347 repair rewriting chunks producing an inconsistent index; running an older Thanos compactor repairing blocks written by newer Prometheus with index format changes.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• iterate series
• read meta from
• repair failed for block
• block with not healthy index found
• block id , try running with --debug.accept-malformed-index

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/1528b04ef6dd1deb. Report an issue: GitHub.

Appendix: source

Thrown at pkg/compact/compact.go:1146

	bdir := filepath.Join(tmpdir, ie.id.String())
	if err := block.Download(ctx, logger, bkt, ie.id, bdir); err != nil {
		return retry(errors.Wrapf(err, "download block %s", ie.id))
	}

	meta, err := metadata.ReadFromDir(bdir)
	if err != nil {
		return errors.Wrapf(err, "read meta from %s", bdir)
	}

	resid, err := block.Repair(ctx, logger, tmpdir, ie.id, metadata.CompactorRepairSource, block.IgnoreIssue347OutsideChunk)
	if err != nil {
		return errors.Wrapf(err, "repair failed for block %s", ie.id)
	}

	// Verify repaired id before uploading it.
	if err := block.VerifyIndex(ctx, logger, filepath.Join(tmpdir, resid.String(), block.IndexFilename), meta.MinTime, meta.MaxTime); err != nil {
		return errors.Wrapf(err, "repaired block is invalid %s", resid)
	}

	level.Info(logger).Log("msg", "uploading repaired block", "newID", resid)
	if err = block.Upload(ctx, logger, bkt, filepath.Join(tmpdir, resid.String()), metadata.NoneFunc); err != nil {
		return retry(errors.Wrapf(err, "upload of %s failed", resid))
	}

	level.Info(logger).Log("msg", "deleting broken block", "id", ie.id)

	// Spawn a new context so we always mark a block for deletion in full on shutdown.
	delCtx, cancel := context.WithTimeout(context.Background(), 5*time.Minute)
	defer cancel()

	// TODO(bplotka): Issue with this will introduce overlap that will halt compactor. Automate that (fix duplicate overlaps caused by this).
	if err := block.MarkForDeletion(delCtx, logger, bkt, ie.id, "source of repaired block", blocksMarkedForDeletion); err != nil {
		return errors.Wrapf(err, "marking old block %s for deletion has failed", ie.id)
	}
	return nil

View on GitHub (pinned to 35b8b99117)