thanos-io/thanos · error

stat

Error message

stat %s

What it means

Delete removes a block's objects from the bucket. Before deleting meta.json it calls bkt.Exists; this error wraps a failure of that Exists (stat) call, meaning the storage backend could not report whether <ULID>/meta.json is present.

Solutions

1. Check object store connectivity, credentials, and bucket configuration
2. Inspect the wrapped backend error for the root cause (auth vs network vs throttling)
3. Retry with backoff; delete is resumable since it re-stats each file

Example fix

// before
if err := block.Delete(ctx, logger, bkt, id); err != nil { return err }
// after
err := block.Delete(ctx, logger, bkt, id)
if err != nil && strings.Contains(err.Error(), "stat ") {
    time.Sleep(backoff)
    err = block.Delete(ctx, logger, bkt, id)
}

if err := bkt.Iter(ctx, "", func(string) error { return nil }); err != nil {
    return fmt.Errorf("bucket client unhealthy: %w", err)
}

if err := block.Delete(ctx, logger, bkt, id); err != nil {
    if strings.Contains(err.Error(), "stat ") {
        return backoff.Retry(func() error { return block.Delete(ctx, logger, bkt, id) })
    }
    return err
}

When it happens

Trigger: bkt.Exists on <ULID>/meta.json fails due to network errors, auth failures, throttling, or unsupported/buggy bucket client during block.Delete.

Common situations: Expired object-store credentials; S3/GCS API outages; rate limiting; misconfigured bucket name; context deadline exceeded.

Understand the failure class

Background: 'Something went wrong' / 'Request failed (500)' / 'HTTP error! status: 404' — what failed HTTP requests actually mean and how to find the real cause — this error's family across 28 libraries.

Related errors

• sync before first pass of downsampling
• sync before second pass of downsampling
• upload file to bucket
• read meta.json for block
• check exists in bucket

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/58e71722f419f039. Report an issue: GitHub.

Appendix: source

Thrown at pkg/block/block.go:210

	markedForDeletion.Inc()
	level.Info(logger).Log("msg", "block has been marked for deletion", "block", id)
	return nil
}

// Delete removes directory that is meant to be block directory.
// NOTE: Always prefer this method for deleting blocks.
//   - We have to delete block's files in the certain order (meta.json first and deletion-mark.json last)
//     to ensure we don't end up with malformed partial blocks. Thanos system handles well partial blocks
//     only if they don't have meta.json. If meta.json is present Thanos assumes valid block.
//   - This avoids deleting empty dir (whole bucket) by mistake.
func Delete(ctx context.Context, logger log.Logger, bkt objstore.Bucket, id ulid.ULID) error {
	metaFile := path.Join(id.String(), MetaFilename)
	deletionMarkFile := path.Join(id.String(), metadata.DeletionMarkFilename)

	// Delete block meta file.
	ok, err := bkt.Exists(ctx, metaFile)
	if err != nil {
		return errors.Wrapf(err, "stat %s", metaFile)
	}

	if ok {
		if err := bkt.Delete(ctx, metaFile); err != nil {
			return errors.Wrapf(err, "delete %s", metaFile)
		}
		level.Debug(logger).Log("msg", "deleted file", "file", metaFile, "bucket", bkt.Name())
	}

	// Delete the block objects, but skip:
	// - The metaFile as we just deleted. This is required for eventual object storages (list after write).
	// - The deletionMarkFile as we'll delete it at last.
	err = deleteDirRec(ctx, logger, bkt, id.String(), func(name string) bool {
		return name == metaFile || name == deletionMarkFile
	})
	if err != nil {
		return err
	}

View on GitHub (pinned to 35b8b99117)