{"record":{"id":"f00e64a905d27f59","repo":"thanos-io/thanos","slug":"retention-failed-f00e64","errorCode":null,"errorMessage":"retention failed","messagePattern":"retention failed","errorType":"console","errorClass":null,"httpStatus":null,"severity":"error","filePath":"cmd/thanos/tools_bucket.go","lineNumber":1451,"sourceCode":"\t\t\t\t0,\n\t\t\t)\n\t\t\tif err != nil {\n\t\t\t\treturn errors.Wrap(err, \"create syncer\")\n\t\t\t}\n\t\t}\n\n\t\tctx := context.Background()\n\t\tlevel.Info(logger).Log(\"msg\", \"syncing blocks metadata\")\n\t\tif err := sy.SyncMetas(ctx); err != nil {\n\t\t\treturn errors.Wrap(err, \"sync blocks\")\n\t\t}\n\n\t\tlevel.Info(logger).Log(\"msg\", \"synced blocks done\")\n\n\t\tlevel.Warn(logger).Log(\"msg\", \"GLOBAL COMPACTOR SHOULD __NOT__ BE RUNNING ON THE SAME BUCKET\")\n\n\t\tif err := compact.ApplyRetentionPolicyByResolution(ctx, logger, insBkt, sy.Metas(), retentionByResolution, stubCounter); err != nil {\n\t\t\treturn errors.Wrap(err, \"retention failed\")\n\t\t}\n\t\treturn nil\n\t})\n}\n\nfunc registerBucketUploadBlocks(app extkingpin.AppClause, objStoreConfig *extflag.PathOrContent) {\n\tcmd := app.Command(\"upload-blocks\", \"Upload blocks push blocks from the provided path to the object storage.\")\n\n\ttbc := &bucketUploadBlocksConfig{}\n\ttbc.registerBucketUploadBlocksFlag(cmd)\n\n\tcmd.Setup(func(g *run.Group, logger log.Logger, reg *prometheus.Registry, _ opentracing.Tracer, _ <-chan struct{}, _ bool) error {\n\t\tif len(tbc.labels) == 0 {\n\t\t\treturn errors.New(\"no external labels configured, uniquely identifying external labels must be configured; see https://thanos.io/tip/thanos/storage.md#external-labels for details.\")\n\t\t}\n\n\t\tlset, err := parseFlagLabels(tbc.labels)\n\t\tif err != nil {","sourceCodeStart":1433,"sourceCodeEnd":1469,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/cmd/thanos/tools_bucket.go#L1433-L1469","documentation":"Returned when compact.ApplyRetentionPolicyByResolution fails after the metadata sync. This step deletes blocks exceeding the retention configured per resolution (e.g. --retention-resolution-raw). Any failure during the retention delete pass is wrapped with 'retention failed'.","triggerScenarios":"Running a bucket tools command with retention flags set, when ApplyRetentionPolicyByResolution cannot mark/delete expired blocks — e.g. delete permission denied on the bucket, or failure re-syncing metas to decide deletions.","commonSituations":"Object storage credentials lack delete permissions (S3 s3:DeleteObject), read-only bucket policies, or object store errors while marking blocks for deletion.","solutions":["Read the wrapped root cause; most often it is a delete-permission error on the bucket.","Grant delete permissions to the credentials used by the tool.","Re-run the tool after fixing storage access; retention is idempotent.","If block-marking fails due to concurrent compaction, stop the global compactor before running retention."],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// Verify delete permission before running retention\nbucket, err := objstore.NewClient(logger, cfg)\nif err != nil { return err }\niterCtx, cancel := context.WithTimeout(ctx, 30*time.Second)\ndefer cancel()\nerr = bucket.Iter(iterCtx, \"\", func(string) error { return nil })\nif err != nil { return fmt.Errorf(\"bucket not listable: %w\", err) }\n// For S3 also assert s3:DeleteObject in the IAM policy out-of-band.","typeGuard":null,"tryCatchPattern":"if err := compact.ApplyRetentionPolicyByResolution(ctx, logger, bkt, metas, retention, deleter); err != nil {\n    log.Printf(\"retention failed (check delete permissions): %v\", err)\n    // retention is idempotent; safe to re-run after fixing access\n}","preventionTips":["Grant delete permissions (e.g. s3:DeleteObject, storage.objects.delete) to the tool's credentials.","Run retention during maintenance windows with no concurrent compactor.","Test the policy with a dry-run delete on a test object first."],"tags":["object-storage","retention","permissions"],"backgroundTag":"insufficient-permissions","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}