thanos-io/thanos · critical

critical error detected

Error message

critical error detected

What it means

Thanos Compactor wraps errors it classifies as HaltError (non-retriable, potentially data-corrupting, e.g. duplicate compaction, corrupt block index) with the message 'critical error detected'. With --no-halt-on-error, instead of blocking forever (select{}), the compactor returns this wrapped error and the process exits. It signals a serious problem that requires manual investigation before rerunning.

Solutions

1. Inspect the full wrapped error and the block IDs mentioned; run 'thanos tools bucket verify' on the affected block/group.
2. Ensure only one compactor instance runs per bucket/resolution (check for duplicate deployments or lost lock).
3. Remove or quarantine the corrupt/overlapping blocks using 'thanos tools bucket mark --marker=deletion-mark.json' after backup, then rerun the compactor.
4. If you intentionally use --no-halt-on-error, add alerting on the thanos_compact_halted metric and on this error so the process failing is noticed and investigated.

Example fix

// before
thanos compact --data-dir=/var/thanos/compact --objstore.bucket=thanos --no-halt-on-error
// after
thanos compact --data-dir=/var/thanos/compact --objstore.bucket=thanos
# keep halt-on-error default (blocks for investigation) and alert on thanos_compact_halted==1

// Before/while running compactor, ensure single ownership + verify bucket
# thanos tools bucket verify --objstore.bucket=thanos --objstore-backup.bucket=thanos-backup
if os.Getenv("THANOS_SINGLE_COMPACTOR") != "1" {
    log.Fatal("refusing to start: only one compactor per bucket is allowed")
}

// In custom runners, treat halt errors as fatal, never retry
if err := runCompaction(ctx); err != nil {
    if compact.IsHaltError(err) {
        alertHalting(err) // page on-call; do NOT restart-loop
        os.Exit(1)
    }
}

When it happens

Trigger: compact.Compact (or runCompaction) returns an err for which compact.IsHaltError(err) is true — most commonly ErrCompactionGroupMerge, ErrUnitAppend, ErrDuplicate_compaction or ErrBlocksOverlapping, or an index issue in a TSDB block during compaction. Only reached when conf.haltOnError is false (--no-halt-on-error flag was set).

Common situations: Operators running with --no-halt-on-error hit it when compactor races with another compactor instance (overlapping blocks), when a block's index.json is corrupted after an interrupted upload, or when metadata/labels are inconsistent in the bucket. Running two compactors for the same bucket without proper ownership is the classic cause.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• create bucket compactor
• compaction
• error executing compaction
• could not group metadata for compaction
• could not calculate compaction progress

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/c0adb6e6c09ec77d. Report an issue: GitHub.

Appendix: source

Thrown at cmd/thanos/compact.go:552

		}

		// --wait=true is specified.
		return runutil.Repeat(conf.waitInterval, ctx.Done(), func() error {
			err := compactMainFn()
			if err == nil {
				compactMetrics.iterations.Inc()
				return nil
			}

			// The HaltError type signals that we hit a critical bug and should block
			// for investigation. You should alert on this being halted.
			if compact.IsHaltError(err) {
				if conf.haltOnError {
					level.Error(logger).Log("msg", "critical error detected; halting", "err", err)
					compactMetrics.halted.Set(1)
					select {}
				} else {
					return errors.Wrap(err, "critical error detected")
				}
			}

			// The RetryError signals that we hit an retriable error (transient error, no connection).
			// You should alert on this being triggered too frequently.
			if compact.IsRetryError(err) {
				level.Error(logger).Log("msg", "retriable error", "err", err)
				compactMetrics.retried.Inc()
				// TODO(bplotka): use actual "retry()" here instead of waiting 5 minutes?
				return nil
			}

			return errors.Wrap(err, "error executing compaction")
		})
	}, func(error) {
		cancel()
	})

View on GitHub (pinned to 35b8b99117)