thanos-io/thanos · error

delete

Error message

delete %s

What it means

Delete (block directory removal) failed deleting an object from the bucket during recursive block deletion. Deletion order matters (meta.json first, deletion-mark.json last) to avoid malformed partial blocks, so failures abort the deletion.

Solutions

1. Check bucket delete permissions
2. Retry after transient storage errors
3. Verify no retention policy blocks object deletion

Example fix

// before
err := block.Delete(ctx, logger, bkt, id)
// after
if err := block.Delete(ctx, logger, bkt, id); err != nil {
    if isAccessDenied(err) { return fmt.Errorf("grant s3:DeleteObject on bucket: %w", err) }
    return err
}

if err := block.Delete(ctx, logger, bkt, id); err != nil {
    var denied interface{ AccessDenied() bool }
    if strings.Contains(err.Error(), "delete ") && isAccessDenied(err) {
        return fmt.Errorf("bucket credentials lack delete permission: %w", err)
    }
    return err
}

When it happens

Trigger: bkt.Delete on <ULID>/meta.json fails due to insufficient delete permissions, object-store errors, versioning/retention policies, or network failure.

Common situations: IAM policy allows read but not delete (common on shared buckets); S3 Object Lock / retention or soft-delete policies; bucket configured read-only for compactor; transient backend errors.

Understand the failure class

Background: Permission denied / not authorized / 403 Forbidden: access-control rejections when the caller lacks the required role, grant, or ownership — this error's family across 18 libraries.

Related errors

• retention failed
• retention failed
• BaseFetcher: iter bucket
• list chunk files
• safe deleting old block

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/9d43cc2a2bbfe87c. Report an issue: GitHub.

Appendix: source

Thrown at pkg/block/block.go:215

// Delete removes directory that is meant to be block directory.
// NOTE: Always prefer this method for deleting blocks.
//   - We have to delete block's files in the certain order (meta.json first and deletion-mark.json last)
//     to ensure we don't end up with malformed partial blocks. Thanos system handles well partial blocks
//     only if they don't have meta.json. If meta.json is present Thanos assumes valid block.
//   - This avoids deleting empty dir (whole bucket) by mistake.
func Delete(ctx context.Context, logger log.Logger, bkt objstore.Bucket, id ulid.ULID) error {
	metaFile := path.Join(id.String(), MetaFilename)
	deletionMarkFile := path.Join(id.String(), metadata.DeletionMarkFilename)

	// Delete block meta file.
	ok, err := bkt.Exists(ctx, metaFile)
	if err != nil {
		return errors.Wrapf(err, "stat %s", metaFile)
	}

	if ok {
		if err := bkt.Delete(ctx, metaFile); err != nil {
			return errors.Wrapf(err, "delete %s", metaFile)
		}
		level.Debug(logger).Log("msg", "deleted file", "file", metaFile, "bucket", bkt.Name())
	}

	// Delete the block objects, but skip:
	// - The metaFile as we just deleted. This is required for eventual object storages (list after write).
	// - The deletionMarkFile as we'll delete it at last.
	err = deleteDirRec(ctx, logger, bkt, id.String(), func(name string) bool {
		return name == metaFile || name == deletionMarkFile
	})
	if err != nil {
		return err
	}

	// Delete block deletion mark.
	ok, err = bkt.Exists(ctx, deletionMarkFile)
	if err != nil {
		return errors.Wrapf(err, "stat %s", deletionMarkFile)

View on GitHub (pinned to 35b8b99117)