{"record":{"id":"8bb69a51494c993f","repo":"thanos-io/thanos","slug":"retention-failed","errorCode":null,"errorMessage":"retention failed","messagePattern":"retention failed","errorType":"console","errorClass":null,"httpStatus":null,"severity":"error","filePath":"cmd/thanos/compact.go","lineNumber":523,"sourceCode":"\t\t\t\tconf.blockFilesConcurrency,\n\t\t\t\tmetadata.HashFunc(conf.hashFunc),\n\t\t\t\tconf.acceptMalformedIndex,\n\t\t\t); err != nil {\n\t\t\t\treturn errors.Wrap(err, \"second pass of downsampling failed\")\n\t\t\t}\n\n\t\t\tlevel.Info(logger).Log(\"msg\", \"downsampling iterations done\")\n\t\t} else {\n\t\t\tlevel.Info(logger).Log(\"msg\", \"downsampling was explicitly disabled\")\n\t\t}\n\n\t\t// TODO(bwplotka): Find a way to avoid syncing if no op was done.\n\t\tif err := sy.SyncMetas(ctx); err != nil {\n\t\t\treturn errors.Wrap(err, \"sync before retention\")\n\t\t}\n\n\t\tif err := compact.ApplyRetentionPolicyByResolution(ctx, logger, insBkt, sy.Metas(), retentionByResolution, compactMetrics.blocksMarked.WithLabelValues(metadata.DeletionMarkFilename, \"\")); err != nil {\n\t\t\treturn errors.Wrap(err, \"retention failed\")\n\t\t}\n\n\t\treturn cleanPartialMarked()\n\t}\n\n\tg.Add(func() error {\n\t\tdefer runutil.CloseWithLogOnErr(logger, insBkt, \"bucket client\")\n\n\t\tif !conf.wait {\n\t\t\treturn compactMainFn()\n\t\t}\n\n\t\t// --wait=true is specified.\n\t\treturn runutil.Repeat(conf.waitInterval, ctx.Done(), func() error {\n\t\t\terr := compactMainFn()\n\t\t\tif err == nil {\n\t\t\t\tcompactMetrics.iterations.Inc()\n\t\t\t\treturn nil","sourceCodeStart":505,"sourceCodeEnd":541,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/cmd/thanos/compact.go#L505-L541","documentation":"Wrapper error in runCompact (cmd/thanos/compact.go:523) around compact.ApplyRetentionPolicyByResolution. Marking expired blocks for deletion failed; the inner error identifies which block or store operation broke.","triggerScenarios":"ApplyRetentionPolicyByResolution failing while marking blocks past their resolution-specific retention for deletion: object store write/mark failures, meta read errors, or issues with the blocksMarked metric collector.","commonSituations":"Bucket write permissions missing (read-only credentials can sync but not write deletion marks), retention configs (e.g. retention.1h) deleting many blocks at once hitting store limits, corrupt metas on retention candidates.","solutions":["Check the wrapped inner error for the specific failing block/operation","Ensure bucket credentials have write permission (deletion marks are written to the bucket)","Verify retention flag values are sane and intended (durations with correct units)","Retry; retention marking is idempotent for already-marked blocks"],"exampleFix":"// before\nthanos compact --retention.1h=0s ... # with read-only bucket credentials\n// after\n# grant write access (PutObject/DeleteObject) to the compactor's service account\nthanos compact --retention.1h=336h ...","handlingStrategy":"validation","validationCode":"// before starting compactor, verify write access\niterCtx, cancel := context.WithTimeout(ctx, 30*time.Second)\ndefer cancel()\n// ensure the service account has PutObject/DeleteObject on the bucket\n_ = iterCtx","typeGuard":null,"tryCatchPattern":"if err := compact.ApplyRetentionPolicyByResolution(...); err != nil {\n\tlevel.Error(logger).Log(\"msg\", \"retention failed\", \"err\", err)\n\treturn errors.Wrap(err, \"retention failed\")\n}","preventionTips":["Give the compactor's service account write permissions (deletion marks)","Smoke-test retention in staging with small retention values","Review retention flags for unit mistakes (h vs d) before deploy"],"tags":["thanos","retention","object-storage","permissions"],"backgroundTag":"permission-denied","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}