thanos-io/thanos · critical

panicked while compacting

Error message

panicked while compacting %s: %v

What it means

pkg/compact/compact.go recovers from a panic inside a compaction group goroutine and converts it into a returned error, listing the block IDs that were being compacted. The panic (nil map access, index out of range, etc. inside tsdb compaction) is translated to a wrapped error rerr that propagates out of runCompact and kills the compactor via cmd/thanos/compact.go. It signals an unexpected bug, not a data problem per se.

Solutions

1. Read the block IDs in the message and inspect/mark those blocks: run `thanos tools bucket verify` on them
2. Upgrade Thanos (and its bundled Prometheus tsdb) to the latest patch release — the panic is usually a known fixed bug
3. Quarantine the offending block by marking it for no-compact (thanos tools bucket mark --marker=no-compact-mark.json) and rerun
4. Check the full stack trace (the panic value %v plus logs) and file an issue if reproducible
5. Restart the compactor; ensure only one compactor instance runs to avoid concurrent writes corrupting state

Example fix

// before: old version panics on block
thanos compact --data-dir=/data (panics on block 01FX...)
// after: mark the bad block, then rerun
thanos tools bucket mark --marker=no-compact-mark.json --id=01FX... 
thanos compact --data-dir=/data

// before compacting, verify blocks
err := tools.BucketVerify(ctx, logger, bkt, idWhitelist)
if err != nil {
    // quarantine bad blocks before compaction
    tools.MarkNoCompact(ctx, bkt, badIDs)
}

// Go
err := runCompact(...)
var perr *panicCompactionError
if errors.As(err, &perr) {
    logger.Error("compaction panic, quarantining blocks", "blocks", perr.BlockIDs)
    markNoCompact(perr.BlockIDs)
    os.Exit(1) // compactor should restart cleanly
}

When it happens

Trigger: A panic (nil pointer, slice out of range, assertion) occurs inside cg.compact's compaction execution; the deferred recover in runCompact catches it and sets rerr = fmt.Errorf("panicked while compacting %s: %v", ids, p).

Common situations: Upstream Prometheus tsdb bugs triggered by malformed or legacy block indexes; corrupted snappy/segment data; OOM-adjacent nil derefs; running an older Thanos against blocks produced by a newer Prometheus version.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• level is bigger then default set of
• get compaction levels
• create compactor
• create bucket compactor
• compaction

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/7ab47636b9ebef7b. Report an issue: GitHub.

Appendix: source

Thrown at pkg/compact/compact.go:945

		}
	}()

	if err := dir.MkdirAll(subPath, 0750); err != nil {
		return false, nil, errors.Wrap(err, "create compaction group dir")
	}

	defer func() {
		if p := recover(); p != nil {
			var sb strings.Builder

			cgIDs := cg.IDs()
			for i, blid := range cgIDs {
				_, _ = sb.WriteString(blid.String())
				if i < len(cgIDs)-1 {
					_, _ = sb.WriteString(",")
				}
			}
			rerr = fmt.Errorf("panicked while compacting %s: %v", sb.String(), p)
		}
	}()

	errChan := make(chan error, 1)
	err := tracing.DoInSpanWithErr(ctx, "compaction_group", func(ctx context.Context) (err error) {
		shouldRerun, compIDs, err = cg.compact(ctx, subDir, planner, comp, blockDeletableChecker, compactionLifecycleCallback, errChan)
		return err
	}, opentracing.Tags{"group.key": cg.Key()})
	errChan <- err
	close(errChan)
	if err != nil {
		cg.compactionFailures.Inc()
		return false, nil, err
	}
	cg.compactionRunsCompleted.Inc()
	return shouldRerun, compIDs, nil
}

View on GitHub (pinned to 35b8b99117)