{"record":{"id":"4e587a498b9009e8","repo":"thanos-io/thanos","slug":"error-executing-compaction","errorCode":null,"errorMessage":"error executing compaction","messagePattern":"error executing compaction","errorType":"console","errorClass":null,"httpStatus":null,"severity":"error","filePath":"cmd/thanos/compact.go","lineNumber":565,"sourceCode":"\t\t\t\tif conf.haltOnError {\n\t\t\t\t\tlevel.Error(logger).Log(\"msg\", \"critical error detected; halting\", \"err\", err)\n\t\t\t\t\tcompactMetrics.halted.Set(1)\n\t\t\t\t\tselect {}\n\t\t\t\t} else {\n\t\t\t\t\treturn errors.Wrap(err, \"critical error detected\")\n\t\t\t\t}\n\t\t\t}\n\n\t\t\t// The RetryError signals that we hit an retriable error (transient error, no connection).\n\t\t\t// You should alert on this being triggered too frequently.\n\t\t\tif compact.IsRetryError(err) {\n\t\t\t\tlevel.Error(logger).Log(\"msg\", \"retriable error\", \"err\", err)\n\t\t\t\tcompactMetrics.retried.Inc()\n\t\t\t\t// TODO(bplotka): use actual \"retry()\" here instead of waiting 5 minutes?\n\t\t\t\treturn nil\n\t\t\t}\n\n\t\t\treturn errors.Wrap(err, \"error executing compaction\")\n\t\t})\n\t}, func(error) {\n\t\tcancel()\n\t})\n\n\tif conf.wait {\n\t\tif !conf.disableWeb {\n\t\t\tr := route.New()\n\n\t\t\tins := extpromhttp.NewInstrumentationMiddleware(reg, nil)\n\n\t\t\tglobal := ui.NewBucketUI(logger, conf.webConf.externalPrefix, conf.webConf.prefixHeaderName, component)\n\t\t\tglobal.Register(r, ins)\n\n\t\t\t// Configure Request Logging for HTTP calls.\n\t\t\topts := []logging.Option{logging.WithDecider(func(_ string, _ error) logging.Decision {\n\t\t\t\treturn logging.NoLogCall\n\t\t\t})}","sourceCodeStart":547,"sourceCodeEnd":583,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/cmd/thanos/compact.go#L547-L583","documentation":"When the compaction iteration returns an error that is neither a HaltError nor a RetryError, the compactor wraps it as 'error executing compaction' and terminates. It is the catch-all fatal path for errors that cannot be retried nor classified as halting bugs (often context cancellation or unexpected I/O errors from the bucket store).","triggerScenarios":"Inside the runutil.Repeat compaction loop, compact.Compact returns an error that fails both compact.IsHaltError and compact.IsRetryError — e.g. a non-retryable bucketGetObject error, an object-store access/permission error not classified as retryable, or ctx cancellation mid-compaction.","commonSituations":"Misconfigured object storage credentials (403 not classified as retry), deleting the data-dir mid-run, network policies causing non-transient storage API errors, or a stale context being cancelled by an sibling service shutting down.","solutions":["Read the wrapped inner error to see the real cause (credentials, 404, timeout, etc.) and fix the object store configuration accordingly.","Verify bucket credentials and permissions (s3 access keys, GCS service account) using 'thanos tools bucket ls'.","Check whether the process is being cancelled (SIGTERM, k8s liveness kill) mid-compaction; give compaction enough grace period.","If the error is genuinely transient but unclassified, upgrade Thanos — error classification (retry vs halt) improves between releases."],"exampleFix":"// before\nreturn errors.Wrap(err, \"error executing compaction\") // context canceled\n// after\n# ensure parent ctx is long-lived; check runGroup signal handling\nctx, cancel = context.WithCancel(context.Background()) // wire to term signals via g.Add(...)","handlingStrategy":"try-catch","validationCode":"// Pre-flight: check object store reachable and credentials valid before starting compaction\n_, err := bkt.Iter(ctx, \"\", func(string) error { return nil })\nif err != nil {\n    return fmt.Errorf(\"bucket preflight failed: %w\", err)\n}","typeGuard":null,"tryCatchPattern":"if err := runCompaction(ctx); err != nil {\n    switch {\n    case compact.IsRetryError(err):\n        scheduleRetry() // transient\n    case compact.IsHaltError(err):\n        alertAndHalt(err)\n    default:\n        log.Fatalf(\"error executing compaction: %v\", err) // fatal, inspect inner cause\n    }\n}","preventionTips":["Validate object-store credentials and connectivity before launching the compactor.","Give the process adequate SIGTERM grace so ctx isn't cancelled mid-compaction.","Don't delete/rotate the data-dir while compaction runs.","Keep Thanos updated so more errors are classified retryable."],"tags":["compaction","thanos","object-storage"],"backgroundTag":"api-error-response","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}