{"record":{"id":"d757b65a210da6cd","repo":"thanos-io/thanos","slug":"delete-block","errorCode":null,"errorMessage":"delete block","messagePattern":"delete block","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/compact/retention.go","lineNumber":44,"sourceCode":"\tctx context.Context,\n\tlogger log.Logger,\n\tbkt objstore.Bucket,\n\tmetas map[ulid.ULID]*metadata.Meta,\n\tretentionByResolution map[ResolutionLevel]time.Duration,\n\tblocksMarkedForDeletion prometheus.Counter,\n) error {\n\tlevel.Info(logger).Log(\"msg\", \"start optional retention\")\n\tfor id, m := range metas {\n\t\tretentionDuration := retentionByResolution[ResolutionLevel(m.Thanos.Downsample.Resolution)]\n\t\tif retentionDuration.Seconds() == 0 {\n\t\t\tcontinue\n\t\t}\n\n\t\tmaxTime := time.Unix(m.MaxTime/1000, 0)\n\t\tif time.Now().After(maxTime.Add(retentionDuration)) {\n\t\t\tlevel.Info(logger).Log(\"msg\", \"applying retention: marking block for deletion\", \"id\", id, \"maxTime\", maxTime.String())\n\t\t\tif err := block.MarkForDeletion(ctx, logger, bkt, id, fmt.Sprintf(\"block exceeding retention of %v\", retentionDuration), blocksMarkedForDeletion); err != nil {\n\t\t\t\treturn errors.Wrap(err, \"delete block\")\n\t\t\t}\n\t\t}\n\t}\n\tlevel.Info(logger).Log(\"msg\", \"optional retention apply done\")\n\treturn nil\n}\n","sourceCodeStart":26,"sourceCodeEnd":51,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/compact/retention.go#L26-L51","documentation":"ApplyRetentionPolicyByResolution wraps any failure from block.MarkForDeletion — the operation that writes a deletion marker (meta.json update/marker object) into object storage for blocks older than the retention period. The wrap means the block was identified as exceeding retention, but persisting its deletion mark failed, so retention was not applied for that block.","triggerScenarios":"During retention compaction (vertical/horizontal sync of blocks per resolution): maxTime = MaxTime/1000 is in the past beyond retentionDuration, and block.MarkForDeletion fails when writing the deletion marker to the bucket (bkt).","commonSituations":"Object store outages or throttling (S3/GCS 5xx, rate limits), expired/insufficient bucket credentials lacking write permission, corrupt block meta.json in the store, network partitions between Thanos compactor and object storage.","solutions":["Check object storage connectivity and credentials (bucket write permissions); retry the compactor — MarkForDeletion is retried safely on the next retention run.","Inspect the wrapped cause (errors.Cause) to see the bucket-level error (e.g. NoSuchKey, AccessDenied) and fix storage-side.","Verify the block's meta.json is readable/corrupt-free; delete or repair broken blocks manually if the store returns decode errors.","Reduce retention-scan pressure: ensure the compactor isn't racing a concurrent deletion from another instance (use a single retention owner or blocking)."],"exampleFix":"// before: retention loop aborts on first storage error\nif err := block.MarkForDeletion(ctx, logger, bkt, id, msg, marked); err != nil {\n    return errors.Wrap(err, \"delete block\")\n}\n// after: log and continue, letting the next run retry\nif err := block.MarkForDeletion(ctx, logger, bkt, id, msg, marked); err != nil {\n    level.Error(logger).Log(\"msg\", \"failed to mark block for deletion; will retry next run\", \"id\", id, \"err\", err)\n    continue\n}","handlingStrategy":"retry","validationCode":"// verify bucket write access before running retention\nconst maxTime = m.MaxTime / 1000\nif Date.now() / 1000 <= maxTime + retentionSeconds {\n    return null // block within retention, skip MarkForDeletion\n}\n// pre-flight: attempt a no-op object write to validate credentials/permissions","typeGuard":null,"tryCatchPattern":"for attempt := 0; attempt < 3; attempt++ {\n    err := ApplyRetentionPolicyByResolution(ctx, logger, bkt, cfg)\n    if err == nil { break }\n    if !isTransientStorageErr(errors.Cause(err)) { return err }\n    time.Sleep(backoff(attempt))\n}","preventionTips":["Run a pre-flight bucket connectivity/write check before retention cycles","Run a single retention owner per bucket prefix to avoid racing deletions","Monitor object storage error rates and alerts on compactor logs","Keep retention durations comfortably larger than the compaction interval"],"tags":["go","object-storage","retention","thanos"],"backgroundTag":"file-write-failed","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}