thanos-io/thanos · critical · halt

compact blocks

Error message

compact blocks %v

What it means

The actual merge of source blocks via comp.CompactWithBlockPopulator (TSDB compaction) failed. Thanos wraps this in a halt error including the source block directories, because a failed compaction may have left partial state that requires operator attention before retrying safely.

Solutions

1. Check disk space and memory on the compactor; enlarge --data-dir and re-run
2. Delete the partially written compaction output/ULID directories under the group's data dir and let compaction retry
3. Inspect the wrapped TSDB error; upgrade Thanos/Prometheus if it matches a known merge bug (e.g. histogram or symbol merge panics)
4. Reduce compaction concurrency (--compact.concurrency) and compaction level sizes if OOM/timeout is the cause

Example fix

// before
// halt: disk full during merge
// after
# df -h /data && rm -rf /data/compact/<partial-ULID>
# thanos compact --data-dir /data --objstore.bucket prom

// Ensure enough headroom before a big compaction:
df -h /data  # keep > 2x largest level block size free

if err := compactor.Compact(ctx); err != nil {
    if strings.Contains(err.Error(), "compact blocks") {
        // halt error: clean partial output dirs, free disk, then re-run
        cleanupPartialCompactions(dataDir)
    }
}

When it happens

Trigger: comp.CompactWithBlockPopulator(dir, toCompactDirs, nil, populateBlockFunc) returns error: merged-OOO histogram issues, disk full, corrupted local block data, panics inside TSDB merge, or index writing failures during merge.

Common situations: Insufficient disk space in --data-dir for the merged output; very large compactions exhausting memory (mmap'd index) or hitting TSDB merge bugs; corrupted downloaded data that passed health checks; OOM-killed compactor leaving partial temp dirs.

Related errors

• found chunks outside the block time range introduced by…
• create compactor
• create bucket compactor
• compaction
• critical error detected

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/f6c3fdb62211e0c0. Report an issue: GitHub.

Appendix: source

Thrown at pkg/compact/compact.go:1274

	sourceBlockStr := fmt.Sprintf("%v", toCompactDirs)

	if err := g.Wait(); err != nil {
		return false, nil, err
	}

	level.Info(cg.logger).Log("msg", "downloaded and verified blocks; compacting blocks", "duration", time.Since(begin), "duration_ms", time.Since(begin).Milliseconds(), "plan", sourceBlockStr)

	begin = time.Now()
	var compIDs []ulid.ULID
	if err := tracing.DoInSpanWithErr(ctx, "compaction", func(ctx context.Context) (e error) {
		populateBlockFunc, e := compactionLifecycleCallback.GetBlockPopulator(ctx, cg.logger, cg)
		if e != nil {
			return e
		}
		compIDs, e = comp.CompactWithBlockPopulator(dir, toCompactDirs, nil, populateBlockFunc)
		return e
	}); err != nil {
		return false, nil, halt(errors.Wrapf(err, "compact blocks %v", toCompactDirs))
	}
	if len(compIDs) == 0 {
		// No compacted blocks means all compacted blocks are of no sample.
		level.Info(cg.logger).Log("msg", "no compacted blocks, deleting source blocks", "blocks", sourceBlockStr)
		for _, meta := range toCompact {
			if meta.Stats.NumSamples == 0 {
				if err := cg.deleteBlock(meta.ULID, filepath.Join(dir, meta.ULID.String()), blockDeletableChecker); err != nil {
					level.Warn(cg.logger).Log("msg", "failed to mark for deletion an empty block found during compaction", "block", meta.ULID)
				}
			}
		}
		// Even though no compacted blocks, there may be more work to do.
		return true, nil, nil
	}
	cg.compactions.Inc()
	if overlappingBlocks {
		cg.verticalCompactions.Inc()
	}

View on GitHub (pinned to 35b8b99117)