{"record":{"id":"7ab47636b9ebef7b","repo":"thanos-io/thanos","slug":"panicked-while-compacting-s-v","errorCode":null,"errorMessage":"panicked while compacting %s: %v","messagePattern":"panicked while compacting (.+?): (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"pkg/compact/compact.go","lineNumber":945,"sourceCode":"\t\t}\n\t}()\n\n\tif err := dir.MkdirAll(subPath, 0750); err != nil {\n\t\treturn false, nil, errors.Wrap(err, \"create compaction group dir\")\n\t}\n\n\tdefer func() {\n\t\tif p := recover(); p != nil {\n\t\t\tvar sb strings.Builder\n\n\t\t\tcgIDs := cg.IDs()\n\t\t\tfor i, blid := range cgIDs {\n\t\t\t\t_, _ = sb.WriteString(blid.String())\n\t\t\t\tif i < len(cgIDs)-1 {\n\t\t\t\t\t_, _ = sb.WriteString(\",\")\n\t\t\t\t}\n\t\t\t}\n\t\t\trerr = fmt.Errorf(\"panicked while compacting %s: %v\", sb.String(), p)\n\t\t}\n\t}()\n\n\terrChan := make(chan error, 1)\n\terr := tracing.DoInSpanWithErr(ctx, \"compaction_group\", func(ctx context.Context) (err error) {\n\t\tshouldRerun, compIDs, err = cg.compact(ctx, subDir, planner, comp, blockDeletableChecker, compactionLifecycleCallback, errChan)\n\t\treturn err\n\t}, opentracing.Tags{\"group.key\": cg.Key()})\n\terrChan <- err\n\tclose(errChan)\n\tif err != nil {\n\t\tcg.compactionFailures.Inc()\n\t\treturn false, nil, err\n\t}\n\tcg.compactionRunsCompleted.Inc()\n\treturn shouldRerun, compIDs, nil\n}\n","sourceCodeStart":927,"sourceCodeEnd":963,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/compact/compact.go#L927-L963","documentation":"pkg/compact/compact.go recovers from a panic inside a compaction group goroutine and converts it into a returned error, listing the block IDs that were being compacted. The panic (nil map access, index out of range, etc. inside tsdb compaction) is translated to a wrapped error rerr that propagates out of runCompact and kills the compactor via cmd/thanos/compact.go. It signals an unexpected bug, not a data problem per se.","triggerScenarios":"A panic (nil pointer, slice out of range, assertion) occurs inside cg.compact's compaction execution; the deferred recover in runCompact catches it and sets rerr = fmt.Errorf(\"panicked while compacting %s: %v\", ids, p).","commonSituations":"Upstream Prometheus tsdb bugs triggered by malformed or legacy block indexes; corrupted snappy/segment data; OOM-adjacent nil derefs; running an older Thanos against blocks produced by a newer Prometheus version.","solutions":["Read the block IDs in the message and inspect/mark those blocks: run `thanos tools bucket verify` on them","Upgrade Thanos (and its bundled Prometheus tsdb) to the latest patch release — the panic is usually a known fixed bug","Quarantine the offending block by marking it for no-compact (thanos tools bucket mark --marker=no-compact-mark.json) and rerun","Check the full stack trace (the panic value %v plus logs) and file an issue if reproducible","Restart the compactor; ensure only one compactor instance runs to avoid concurrent writes corrupting state"],"exampleFix":"// before: old version panics on block\nthanos compact --data-dir=/data (panics on block 01FX...)\n// after: mark the bad block, then rerun\nthanos tools bucket mark --marker=no-compact-mark.json --id=01FX... \nthanos compact --data-dir=/data","handlingStrategy":"try-catch","validationCode":"// before compacting, verify blocks\nerr := tools.BucketVerify(ctx, logger, bkt, idWhitelist)\nif err != nil {\n    // quarantine bad blocks before compaction\n    tools.MarkNoCompact(ctx, bkt, badIDs)\n}","typeGuard":null,"tryCatchPattern":"// Go\nerr := runCompact(...)\nvar perr *panicCompactionError\nif errors.As(err, &perr) {\n    logger.Error(\"compaction panic, quarantining blocks\", \"blocks\", perr.BlockIDs)\n    markNoCompact(perr.BlockIDs)\n    os.Exit(1) // compactor should restart cleanly\n}","preventionTips":["Run a recent Thanos release (panic bugs are fixed upstream)","Verify bucket blocks before compaction with `thanos tools bucket verify`","Run exactly one compactor instance per bucket","Watch logs for panic stack traces and file upstream issues"],"tags":["panic","compaction","goroutine-recover"],"backgroundTag":"internal-invariant-violation","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}