thanos-io/thanos · error

remove old block dir

Error message

remove old block dir %s

What it means

Wraps os.RemoveAll in Group.deleteBlock when deleting a compacted source block's local directory after it has been uploaded. Failure to remove the local dir (busy files, permissions) aborts deletion marking; the leftover dir is cleaned on a later run.

Solutions

1. Check filesystem permissions on the local block dir
2. Look for processes holding files open
3. Retry; the compactor cleans it on next run

When it happens

Trigger: Thrown at pkg/compact/compact.go:1387 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/75c44b801eeca09b. Report an issue: GitHub.

Appendix: source

Thrown at pkg/compact/compact.go:1387

	// into the next planning cycle.
	// Eventually the block we just uploaded should get synced into the group again (including sync-delay).
	for _, meta := range toCompact {
		if err := tracing.DoInSpanWithErr(ctx, "compaction_block_delete", func(ctx context.Context) error {
			return cg.deleteBlock(meta.ULID, filepath.Join(dir, meta.ULID.String()), blockDeletableChecker)
		}, opentracing.Tags{"block.id": meta.ULID}); err != nil {
			return false, nil, retry(errors.Wrapf(err, "mark old block for deletion from bucket"))
		}
		cg.groupGarbageCollectedBlocks.Inc()
	}

	level.Info(cg.logger).Log("msg", "finished compacting blocks", "duration", time.Since(groupCompactionBegin),
		"duration_ms", time.Since(groupCompactionBegin).Milliseconds(), "result_blocks", compIDStrs, "source_blocks", sourceBlockStr)
	return true, compIDs, nil
}

func (cg *Group) deleteBlock(id ulid.ULID, bdir string, blockDeletableChecker BlockDeletableChecker) error {
	if err := os.RemoveAll(bdir); err != nil {
		return errors.Wrapf(err, "remove old block dir %s", id)
	}

	if blockDeletableChecker.CanDelete(cg, id) {
		// Spawn a new context so we always mark a block for deletion in full on shutdown.
		delCtx, cancel := context.WithTimeout(context.Background(), 5*time.Minute)
		defer cancel()
		level.Info(cg.logger).Log("msg", "marking compacted block for deletion", "old_block", id)
		if err := block.MarkForDeletion(delCtx, cg.logger, cg.bkt, id, "source of compacted block", cg.blocksMarkedForDeletion); err != nil {
			return errors.Wrapf(err, "mark block %s for deletion from bucket", id)
		}
	}
	return nil
}

// BucketCompactor compacts blocks in a bucket.
type BucketCompactor struct {
	logger                         log.Logger
	sy                             *Syncer

View on GitHub (pinned to 35b8b99117)