{"record":{"id":"50f5dec40cc95f40","repo":"JuliusBrussee/caveman","slug":"probe-object-still-exists-after-purge","errorCode":null,"errorMessage":"probe object still exists after purge","messagePattern":"probe object still exists after purge","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"shared/platform/objectstore/objectstore.go","lineNumber":266,"sourceCode":"\t\t}\n\t}()\n\tgot, err := store.Get(ctx, key)\n\tif err != nil {\n\t\treturn err\n\t}\n\tif !bytes.Equal(got, probe) {\n\t\treturn errors.New(\"probe bytes did not round-trip\")\n\t}\n\tif err := PurgeObject(ctx, store, key); err != nil {\n\t\treturn err\n\t}\n\tcleaned = true\n\texists, err := store.Exists(ctx, key)\n\tif err != nil {\n\t\treturn err\n\t}\n\tif exists {\n\t\treturn errors.New(\"probe object still exists after purge\")\n\t}\n\treturn nil\n}\n\nfunc (m *minioStore) Put(ctx context.Context, key string, body []byte, contentType string) error {\n\tif contentType == \"\" {\n\t\tcontentType = \"application/octet-stream\"\n\t}\n\t_, err := m.client.PutObject(ctx, m.bucket, key, bytes.NewReader(body), int64(len(body)),\n\t\tminio.PutObjectOptions{ContentType: contentType})\n\tif err != nil {\n\t\treturn fmt.Errorf(\"objectstore: put %q: %w\", key, err)\n\t}\n\treturn nil\n}\n\nfunc (m *minioStore) Get(ctx context.Context, key string) ([]byte, error) {\n\tobj, err := m.client.GetObject(ctx, m.bucket, key, minio.GetObjectOptions{})","sourceCodeStart":248,"sourceCodeEnd":284,"githubUrl":"https://github.com/JuliusBrussee/caveman/blob/766dce6b1394ebb56a3090748d5a0240a5aefb36/shared/platform/objectstore/objectstore.go#L248-L284","documentation":"At the end of objectstore.Probe, after PurgeObject removed the probe object (every version, for stores implementing VersionPurger), an Exists check still reports the key present. The design intent is that a successful probe leaves nothing behind; a surviving object means deletion did not fully take effect — typically versioning semantics or replication lag on the bucket rather than an application bug.","triggerScenarios":"Probe's Exists call returning true after PurgeObject succeeded — e.g. the bucket has object versioning plus cross-region replication so delete markers/replica visibility lag, or a non-VersionPurger store where a plain delete left a version visible, or an S3-compatible backend whose stat-after-delete is eventually consistent.","commonSituations":"Production buckets with replication (CRR) where replica listings lag behind deletes; MinIO in distributed mode with eventual consistency tuned; a custom Store implementation whose Delete is a no-op for current versions; switching a deployment from a versioned to unversioned bucket mid-life so old versions remain.","solutions":["Retry the probe after a short interval to distinguish lag from a real delete defect","If it persists, inspect the bucket: list object versions for a recent _cave_health/* key and confirm the backend implements delete-all-versions the way VersionPurger expects","For replication-enabled buckets, wait for or disable replication of the health prefix, or accept a probe that tolerates eventual deletion only after verifying with the provider"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"if err := objectstore.Probe(ctx, store); err != nil {\n\tif strings.Contains(err.Error(), \"still exists after purge\") {\n\t\t// likely replication/versioning lag: retry once after a short delay before treating as a defect\n\t}\n\treturn err\n}","preventionTips":["Exclude the _cave_health prefix from replication rules where the provider allows it","For custom Store implementations, implement VersionPurger so deletes remove all versions, matching probe expectations"],"tags":["objectstore","s3","versioning","delete","eventual-consistency","probe","go"],"backgroundTag":"eventual-consistency","analyzedSha":"766dce6b1394ebb56a3090748d5a0240a5aefb36","analyzedAt":"2026-08-18T03:14:35.516Z","contentChangedAt":"2026-08-18T03:14:35.516Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}