{"record":{"id":"c0adb6e6c09ec77d","repo":"thanos-io/thanos","slug":"critical-error-detected","errorCode":null,"errorMessage":"critical error detected","messagePattern":"critical error detected","errorType":"console","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"cmd/thanos/compact.go","lineNumber":552,"sourceCode":"\t\t}\n\n\t\t// --wait=true is specified.\n\t\treturn runutil.Repeat(conf.waitInterval, ctx.Done(), func() error {\n\t\t\terr := compactMainFn()\n\t\t\tif err == nil {\n\t\t\t\tcompactMetrics.iterations.Inc()\n\t\t\t\treturn nil\n\t\t\t}\n\n\t\t\t// The HaltError type signals that we hit a critical bug and should block\n\t\t\t// for investigation. You should alert on this being halted.\n\t\t\tif compact.IsHaltError(err) {\n\t\t\t\tif conf.haltOnError {\n\t\t\t\t\tlevel.Error(logger).Log(\"msg\", \"critical error detected; halting\", \"err\", err)\n\t\t\t\t\tcompactMetrics.halted.Set(1)\n\t\t\t\t\tselect {}\n\t\t\t\t} else {\n\t\t\t\t\treturn errors.Wrap(err, \"critical error detected\")\n\t\t\t\t}\n\t\t\t}\n\n\t\t\t// The RetryError signals that we hit an retriable error (transient error, no connection).\n\t\t\t// You should alert on this being triggered too frequently.\n\t\t\tif compact.IsRetryError(err) {\n\t\t\t\tlevel.Error(logger).Log(\"msg\", \"retriable error\", \"err\", err)\n\t\t\t\tcompactMetrics.retried.Inc()\n\t\t\t\t// TODO(bplotka): use actual \"retry()\" here instead of waiting 5 minutes?\n\t\t\t\treturn nil\n\t\t\t}\n\n\t\t\treturn errors.Wrap(err, \"error executing compaction\")\n\t\t})\n\t}, func(error) {\n\t\tcancel()\n\t})\n","sourceCodeStart":534,"sourceCodeEnd":570,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/cmd/thanos/compact.go#L534-L570","documentation":"Thanos Compactor wraps errors it classifies as HaltError (non-retriable, potentially data-corrupting, e.g. duplicate compaction, corrupt block index) with the message 'critical error detected'. With --no-halt-on-error, instead of blocking forever (select{}), the compactor returns this wrapped error and the process exits. It signals a serious problem that requires manual investigation before rerunning.","triggerScenarios":"compact.Compact (or runCompaction) returns an err for which compact.IsHaltError(err) is true — most commonly ErrCompactionGroupMerge, ErrUnitAppend, ErrDuplicate_compaction or ErrBlocksOverlapping, or an index issue in a TSDB block during compaction. Only reached when conf.haltOnError is false (--no-halt-on-error flag was set).","commonSituations":"Operators running with --no-halt-on-error hit it when compactor races with another compactor instance (overlapping blocks), when a block's index.json is corrupted after an interrupted upload, or when metadata/labels are inconsistent in the bucket. Running two compactors for the same bucket without proper ownership is the classic cause.","solutions":["Inspect the full wrapped error and the block IDs mentioned; run 'thanos tools bucket verify' on the affected block/group.","Ensure only one compactor instance runs per bucket/resolution (check for duplicate deployments or lost lock).","Remove or quarantine the corrupt/overlapping blocks using 'thanos tools bucket mark --marker=deletion-mark.json' after backup, then rerun the compactor.","If you intentionally use --no-halt-on-error, add alerting on the thanos_compact_halted metric and on this error so the process failing is noticed and investigated."],"exampleFix":"// before\nthanos compact --data-dir=/var/thanos/compact --objstore.bucket=thanos --no-halt-on-error\n// after\nthanos compact --data-dir=/var/thanos/compact --objstore.bucket=thanos\n# keep halt-on-error default (blocks for investigation) and alert on thanos_compact_halted==1","handlingStrategy":"validation","validationCode":"// Before/while running compactor, ensure single ownership + verify bucket\n# thanos tools bucket verify --objstore.bucket=thanos --objstore-backup.bucket=thanos-backup\nif os.Getenv(\"THANOS_SINGLE_COMPACTOR\") != \"1\" {\n    log.Fatal(\"refusing to start: only one compactor per bucket is allowed\")\n}","typeGuard":null,"tryCatchPattern":"// In custom runners, treat halt errors as fatal, never retry\nif err := runCompaction(ctx); err != nil {\n    if compact.IsHaltError(err) {\n        alertHalting(err) // page on-call; do NOT restart-loop\n        os.Exit(1)\n    }\n}","preventionTips":["Never run two compactors against the same bucket/resolution.","Keep the default --halt-on-error and alert on thanos_compact_halted instead of auto-restarting.","Run periodic 'thanos tools bucket verify' to catch corruption early.","Monitor thanos_compact_group_compactions_failures_total for rising failure rates."],"tags":["compaction","thanos","halt-error","storage"],"backgroundTag":"internal-invariant-violation","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}