thanos-io/thanos · error

cannot repair downsampled block

Error message

cannot repair downsampled block

What it means

block.Repair refuses to run on a block whose metadata declares a non-zero Thanos downsample resolution. Repair rewrites raw (resolution=0) blocks to fix tsdb#347 index corruption; downsampled blocks have aggregated data, so the repair invariants and ignore functions do not apply to them. The check is done right after reading meta.json from the block directory.

Solutions

1. Check the block's meta.json before calling Repair and skip blocks with thanos.downsample.resolution > 0.
2. Only invoke repair on raw blocks uploaded by the Prometheus sidecar (source=sidecar/compactor with resolution 0).
3. If the downsampled block is actually broken, delete it and let the Thanos compactor regenerate it from the raw block instead of repairing it.

Example fix

// before
for _, id := range ids {
    if _, err := block.Repair(ctx, logger, dir, id, source, block.IgnoreCompleteOutsideChunk); err != nil {
        return err // fails on downsampled blocks
    }
}
// after
for _, id := range ids {
    meta, err := metadata.ReadFromDir(filepath.Join(dir, id.String()))
    if err != nil {
        return err
    }
    if meta.Thanos.Downsample.Resolution > 0 {
        continue // skip downsampled blocks
    }
    if _, err := block.Repair(ctx, logger, dir, id, source, block.IgnoreCompleteOutsideChunk); err != nil {
        return err
    }
}

meta, err := metadata.ReadFromDir(filepath.Join(dir, id.String()))
if err != nil {
    return err
}
if meta.Thanos.Downsample.Resolution > 0 {
    return fmt.Errorf("block %s is downsampled (resolution=%d); skip repair", id, meta.Thanos.Downsample.Resolution)
}

func isRawBlock(meta *metadata.Meta) bool {
    return meta != nil && meta.Thanos.Downsample.Resolution == 0
}

resid, err := block.RepairIssue347(ctx, logger, dir, id, source)
if err != nil {
    if strings.Contains(err.Error(), "cannot repair downsampled block") {
        logger.Log("msg", "skipping downsampled block", "block", id)
        return id, nil // expected condition, not a failure
    }
    return id, err
}

When it happens

Trigger: Calling block.Repair (directly or via RepairIssue347/repairIndex) on a block directory whose meta.json has thanos.downsample.resolution > 0 — i.e. any 5m or 1h downsampled block.

Common situations: Running the issue-347 repair tooling against a whole bucket/dir without filtering by resolution; pointing repair at a compacted/downsampled block produced by the thanos compact downscoping; scripting repair over every ULID in a data dir that contains both raw and downsampled blocks.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

• raw resolution must be higher than the minimum block size…
• 5m resolution retention must be higher than the minimum…
• sync before first pass of downsampling
• first pass of downsampling failed
• sync before second pass of downsampling

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/05e96465b90ee3da. Report an issue: GitHub.

Appendix: source

Thrown at pkg/block/index.go:429

// - all "complete" outsiders (they will not accessed anyway)
// - removes all near "complete" outside chunks introduced by https://github.com/prometheus/tsdb/issues/347.
// Fixable inconsistencies are resolved in the new block.
// TODO(bplotka): https://github.com/thanos-io/thanos/issues/378.
func Repair(ctx context.Context, logger log.Logger, dir string, id ulid.ULID, source metadata.SourceType, ignoreChkFns ...ignoreFnType) (resid ulid.ULID, err error) {
	if len(ignoreChkFns) == 0 {
		return resid, errors.New("no ignore chunk function specified")
	}

	bdir := filepath.Join(dir, id.String())
	entropy := rand.New(rand.NewSource(time.Now().UnixNano()))
	resid = ulid.MustNew(ulid.Now(), entropy)

	meta, err := metadata.ReadFromDir(bdir)
	if err != nil {
		return resid, errors.Wrap(err, "read meta file")
	}
	if meta.Thanos.Downsample.Resolution > 0 {
		return resid, errors.New("cannot repair downsampled block")
	}

	b, err := tsdb.OpenBlock(logutil.GoKitLogToSlog(logger), bdir, nil, nil)
	if err != nil {
		return resid, errors.Wrap(err, "open block")
	}
	defer runutil.CloseWithErrCapture(&err, b, "repair block reader")

	indexr, err := b.Index()
	if err != nil {
		return resid, errors.Wrap(err, "open index")
	}
	defer runutil.CloseWithErrCapture(&err, indexr, "repair index reader")

	chunkr, err := b.Chunks()
	if err != nil {
		return resid, errors.Wrap(err, "open chunks")
	}

View on GitHub (pinned to 35b8b99117)