What causes "probe object still exists after purge" in JuliusBrussee/caveman?

Retry the probe after a short interval to distinguish lag from a real delete defect

probe object still exists after purge

Error message

probe object still exists after purge

What it means

At the end of objectstore.Probe, after PurgeObject removed the probe object (every version, for stores implementing VersionPurger), an Exists check still reports the key present. The design intent is that a successful probe leaves nothing behind; a surviving object means deletion did not fully take effect — typically versioning semantics or replication lag on the bucket rather than an application bug.

Source

Thrown at shared/platform/objectstore/objectstore.go:266

		}
	}()
	got, err := store.Get(ctx, key)
	if err != nil {
		return err
	}
	if !bytes.Equal(got, probe) {
		return errors.New("probe bytes did not round-trip")
	}
	if err := PurgeObject(ctx, store, key); err != nil {
		return err
	}
	cleaned = true
	exists, err := store.Exists(ctx, key)
	if err != nil {
		return err
	}
	if exists {
		return errors.New("probe object still exists after purge")
	}
	return nil
}

func (m *minioStore) Put(ctx context.Context, key string, body []byte, contentType string) error {
	if contentType == "" {
		contentType = "application/octet-stream"
	}
	_, err := m.client.PutObject(ctx, m.bucket, key, bytes.NewReader(body), int64(len(body)),
		minio.PutObjectOptions{ContentType: contentType})
	if err != nil {
		return fmt.Errorf("objectstore: put %q: %w", key, err)
	}
	return nil
}

func (m *minioStore) Get(ctx context.Context, key string) ([]byte, error) {
	obj, err := m.client.GetObject(ctx, m.bucket, key, minio.GetObjectOptions{})

View on GitHub (pinned to 766dce6b13)

Solutions

Retry the probe after a short interval to distinguish lag from a real delete defect
If it persists, inspect the bucket: list object versions for a recent _cave_health/* key and confirm the backend implements delete-all-versions the way VersionPurger expects
For replication-enabled buckets, wait for or disable replication of the health prefix, or accept a probe that tolerates eventual deletion only after verifying with the provider

Defensive patterns

Strategy: retry

Try / catch

if err := objectstore.Probe(ctx, store); err != nil {
	if strings.Contains(err.Error(), "still exists after purge") {
		// likely replication/versioning lag: retry once after a short delay before treating as a defect
	}
	return err
}

Prevention

Exclude the _cave_health prefix from replication rules where the provider allows it
For custom Store implementations, implement VersionPurger so deletes remove all versions, matching probe expectations

When it happens

Trigger: Probe's Exists call returning true after PurgeObject succeeded — e.g. the bucket has object versioning plus cross-region replication so delete markers/replica visibility lag, or a non-VersionPurger store where a plain delete left a version visible, or an S3-compatible backend whose stat-after-delete is eventually consistent.

Common situations: Production buckets with replication (CRR) where replica listings lag behind deletes; MinIO in distributed mode with eventual consistency tuned; a custom Store implementation whose Delete is a no-op for current versions; switching a deployment from a versioned to unversioned bucket mid-life so old versions remain.

Related errors

AI-assisted analysis of JuliusBrussee/caveman@766dce6b13 (2026-08-18). Data as JSON: /api/errors/50f5dec40cc95f40. Report an issue: GitHub.