{"record":{"id":"ae20fd6a1582803e","repo":"thanos-io/thanos","slug":"marking-old-block-s-for-deletion-has-failed","errorCode":null,"errorMessage":"marking old block %s for deletion has failed","messagePattern":"marking old block (.+?) for deletion has failed","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/compact/compact.go","lineNumber":1162,"sourceCode":"\t// Verify repaired id before uploading it.\n\tif err := block.VerifyIndex(ctx, logger, filepath.Join(tmpdir, resid.String(), block.IndexFilename), meta.MinTime, meta.MaxTime); err != nil {\n\t\treturn errors.Wrapf(err, \"repaired block is invalid %s\", resid)\n\t}\n\n\tlevel.Info(logger).Log(\"msg\", \"uploading repaired block\", \"newID\", resid)\n\tif err = block.Upload(ctx, logger, bkt, filepath.Join(tmpdir, resid.String()), metadata.NoneFunc); err != nil {\n\t\treturn retry(errors.Wrapf(err, \"upload of %s failed\", resid))\n\t}\n\n\tlevel.Info(logger).Log(\"msg\", \"deleting broken block\", \"id\", ie.id)\n\n\t// Spawn a new context so we always mark a block for deletion in full on shutdown.\n\tdelCtx, cancel := context.WithTimeout(context.Background(), 5*time.Minute)\n\tdefer cancel()\n\n\t// TODO(bplotka): Issue with this will introduce overlap that will halt compactor. Automate that (fix duplicate overlaps caused by this).\n\tif err := block.MarkForDeletion(delCtx, logger, bkt, ie.id, \"source of repaired block\", blocksMarkedForDeletion); err != nil {\n\t\treturn errors.Wrapf(err, \"marking old block %s for deletion has failed\", ie.id)\n\t}\n\treturn nil\n}\n\nfunc (cg *Group) compact(ctx context.Context, dir string, planner Planner, comp Compactor, blockDeletableChecker BlockDeletableChecker, compactionLifecycleCallback CompactionLifecycleCallback, errChan chan error) (bool, []ulid.ULID, error) {\n\tcg.mtx.Lock()\n\tdefer cg.mtx.Unlock()\n\n\t// Check for overlapped blocks.\n\toverlappingBlocks := false\n\tif err := cg.areBlocksOverlapping(nil); err != nil {\n\t\t// TODO(bwplotka): It would really nice if we could still check for other overlaps than replica. In fact this should be checked\n\t\t// in syncer itself. Otherwise with vertical compaction enabled we will sacrifice this important check.\n\t\tif !cg.enableVerticalCompaction {\n\t\t\treturn false, nil, halt(errors.Wrap(err, \"pre compaction overlap check\"))\n\t\t}\n\n\t\toverlappingBlocks = true","sourceCodeStart":1144,"sourceCodeEnd":1180,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/compact/compact.go#L1144-L1180","documentation":"This error wraps a failure of block.MarkForDeletion in the final step of repairBucketBlock (compact.go). After the repaired replacement block was uploaded, the compactor marks the original broken block (ie.id) in the bucket with a deletion marker so it will be garbage-collected. If marking fails, the bucket is left with both the broken block and the repaired one, which can later cause overlap-halt of compaction (as the TODO in the source notes).","triggerScenarios":"block.MarkForDeletion(delCtx, logger, bkt, ie.id, \"source of repaired block\", blocksMarkedForDeletion) fails: object storage write of the deletion marker fails (permission, network, 5xx), the 5-minute delCtx timeout expires on a slow storage backend, or the block was already marked/deleted concurrently by another compactor.","commonSituations":"Slow or rate-limited object storage making the marker PUT exceed the 5m timeout; credentials allowing read but not write of marker objects; two compactor instances repairing the same overlap simultaneously; transient network blip exactly at the cleanup step after a long repair.","solutions":["Manually mark or delete the broken block: create the deletion-mark.json object for ie.id or run 'thanos tools bucket verify'/'bucket mark --marker=deletion-mark' so the overlap does not halt compaction","Fix bucket write permissions for the compactor (it needs write access for markers, not just block upload)","Ensure a single compactor instance/group per bucket prefix to avoid concurrent repair/marker races","If storage latency is the cause, the compactor retries the whole repair cycle; still clean the duplicate manually to avoid the overlap-halt described in the TODO","Check object storage availability/5xx errors at the time of failure and re-run compaction"],"exampleFix":"// before: broken block left in bucket, compaction halts on overlap\n// after: mark it for deletion manually\n$ thanos tools bucket mark --marker=deletion-mark \\\n    --objstore.config-file=bucket.yaml --id=<brokenBlockULID> \\\n    --details=\"source of repaired block\"","handlingStrategy":"retry","validationCode":"// ensure write access for deletion markers before starting the compactor\nif err := bkt.Upload(ctx, \"marker-probe\", strings.NewReader(\"probe\")); err != nil {\n    return fmt.Errorf(\"cannot write deletion markers: %w\", err)\n}","typeGuard":null,"tryCatchPattern":"if err := block.MarkForDeletion(delCtx, logger, bkt, ie.id, \"source of repaired block\", blocksMarkedForDeletion); err != nil {\n    logger.Error(\"failed to mark repaired-over block; clean it manually to avoid overlap halt\",\n        \"id\", ie.id, \"err\", err)\n    return errors.Wrapf(err, \"marking old block %s for deletion has failed\", ie.id)\n}","preventionTips":["Single compactor instance per bucket prefix to avoid concurrent repair/marker races","Give the compactor full write access — deletion markers are objects too","Alert on this error: leftover broken blocks cause overlap that halts compaction later","Know the manual remediation: 'thanos tools bucket mark --marker=deletion-mark --id=<ULID>'"],"tags":["thanos","compactor","object-storage","cleanup","garbage-collection"],"backgroundTag":"file-write-failed","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}