thanos-io/thanos · error

cannot repair downsampled blocks

Error message

cannot repair downsampled blocks

What it means

During index-issue repair in pkg/verifier, repairIndex refuses to repair any block whose Thanos metadata Downsample.Resolution is greater than 0, returning 'cannot repair downsampled blocks'. Notably it wraps err (which is nil at this point), so the message is emitted with no underlying cause. Downsampled blocks cannot be rebuilt from raw index repair logic.

Solutions

1. Run the repair only on raw (resolution 0) blocks/buckets
2. Skip or exclude downsampled blocks from the repair set
3. If a downsampled block is corrupted, let the compactor regenerate it from raw data or delete it
4. Note the error wraps a nil err — treat the message alone as the cause

Example fix

// before
thanos tools bucket verify --objstore.bucket=metrics --repair --issues=index-issue
// after
# target the raw bucket only
thanos tools bucket verify --objstore.bucket=metrics-raw --repair --issues=index-issue

// Read block meta and skip downsampled blocks before repair
meta, err := block.ReadMeta(ctx, logger, bkt, id)
if err != nil {
    return err
}
if meta.Thanos.Downsample.Resolution > 0 {
    logger.Info("skipping downsampled block", "id", id)
    return nil
}

if err := repairIndex(ctx, id, dir); err != nil {
    if strings.Contains(err.Error(), "cannot repair downsampled blocks") {
        logger.Warn("skipping downsampled block; repair raw blocks instead", "id", id)
        return nil
    }
    return err
}

When it happens

Trigger: Running index-issue repair (e.g. --repair with index issues) over a bucket that contains downsampled blocks; repairIndex encounters a block with meta.Thanos.Downsample.Resolution > 0.

Common situations: Buckets where the compactor downsampling is enabled and the verifier iterates all blocks including 5m/1h downsampling resolutions, or pointing the verifier at a downsampling output bucket.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• raw resolution must be higher than the minimum block size…
• 5m resolution retention must be higher than the minimum…
• sync before first pass of downsampling
• first pass of downsampling failed
• sync before second pass of downsampling

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/98ed64a49568cee0. Report an issue: GitHub.

Appendix: source

Thrown at pkg/verifier/index_issue.go:90

		}
		level.Info(ctx.Logger).Log("msg", "all good, continuing", "id", id)
	}

	level.Info(ctx.Logger).Log("msg", "verified issue", "with-repair", repair)
	return nil
}

func repairIndex(stats block.HealthStats, ctx Context, id ulid.ULID, meta *metadata.Meta, dir string) (err error) {
	if stats.OutOfOrderChunks > stats.DuplicatedChunks {
		level.Warn(ctx.Logger).Log("msg", "detected overlaps are not entirely by duplicated chunks. We are able to repair only duplicates", "id", id)
	}

	if stats.OutsideChunks > (stats.CompleteOutsideChunks + stats.Issue347OutsideChunks) {
		level.Warn(ctx.Logger).Log("msg", "detected outsiders are not all 'complete' outsiders or outsiders from https://github.com/prometheus/tsdb/issues/347. We can safely delete only these outsiders", "id", id)
	}

	if meta.Thanos.Downsample.Resolution > 0 {
		return errors.Wrap(err, "cannot repair downsampled blocks")
	}

	level.Info(ctx.Logger).Log("msg", "downloading block for repair", "id", id)
	if err = block.Download(ctx, ctx.Logger, ctx.Bkt, id, path.Join(dir, id.String())); err != nil {
		return errors.Wrapf(err, "download block %s", id)
	}
	level.Info(ctx.Logger).Log("msg", "downloaded block to be repaired", "id", id, "issue")

	level.Info(ctx.Logger).Log("msg", "repairing block", "id", id, "issue")
	resid, err := block.Repair(
		ctx,
		ctx.Logger,
		dir,
		id,
		metadata.BucketRepairSource,
		block.IgnoreCompleteOutsideChunk,
		block.IgnoreDuplicateOutsideChunk,
		block.IgnoreIssue347OutsideChunk,

View on GitHub (pinned to 35b8b99117)