{"record":{"id":"b8fa3ccb2ef22ce7","repo":"JuliusBrussee/caveman","slug":"probe-bytes-did-not-round-trip","errorCode":null,"errorMessage":"probe bytes did not round-trip","messagePattern":"probe bytes did not round-trip","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"shared/platform/objectstore/objectstore.go","lineNumber":255,"sourceCode":"\tkey := \"_cave_health/\" + base64.RawURLEncoding.EncodeToString(probe)\n\tif err := store.Put(ctx, key, probe, \"application/octet-stream\"); err != nil {\n\t\treturn err\n\t}\n\tcleaned := false\n\tdefer func() {\n\t\tif !cleaned {\n\t\t\t// Keep health checks within their caller deadline. A failed probe may\n\t\t\t// leave one random, tenant-free canary for lifecycle cleanup; readiness\n\t\t\t// must never hang on an unbounded background delete.\n\t\t\t_ = PurgeObject(ctx, store, key)\n\t\t}\n\t}()\n\tgot, err := store.Get(ctx, key)\n\tif err != nil {\n\t\treturn err\n\t}\n\tif !bytes.Equal(got, probe) {\n\t\treturn errors.New(\"probe bytes did not round-trip\")\n\t}\n\tif err := PurgeObject(ctx, store, key); err != nil {\n\t\treturn err\n\t}\n\tcleaned = true\n\texists, err := store.Exists(ctx, key)\n\tif err != nil {\n\t\treturn err\n\t}\n\tif exists {\n\t\treturn errors.New(\"probe object still exists after purge\")\n\t}\n\treturn nil\n}\n\nfunc (m *minioStore) Put(ctx context.Context, key string, body []byte, contentType string) error {\n\tif contentType == \"\" {\n\t\tcontentType = \"application/octet-stream\"","sourceCodeStart":237,"sourceCodeEnd":273,"githubUrl":"https://github.com/JuliusBrussee/caveman/blob/766dce6b1394ebb56a3090748d5a0240a5aefb36/shared/platform/objectstore/objectstore.go#L237-L273","documentation":"objectstore.Probe writes 32 random bytes to _cave_health/<random> under the configured bucket, reads them back, and requires byte equality. A mismatch means the store round-tripped different bytes than were written — a strong signal of corruption, an interfering proxy/cache, or a backend that is not honoring writes the way production retention needs. The probe body is random and tenant-free, so content is never the cause.","triggerScenarios":"Probe's Get after a successful Put returns bytes that differ — e.g. a misconfigured CDN or gateway in front of the bucket serving stale/other content, a bucket name collision where another writer overwrote the key, or storage corruption. Runs automatically from FromEnv in production, and from any explicit health check.","commonSituations":"S3_ENDPOINT pointed at a caching proxy that serves a stale copy of a just-written key; two environments sharing one bucket where a lifecycle or replication job mutates objects; exotic MinIO builds with broken read-after-write consistency; extremely rare bit-level corruption on long-lived nodes.","solutions":["Re-run the probe once to rule out a transient read-after-write window","If it persists, inspect what the endpoint frontends: bypass any CDN/proxy and point S3_ENDPOINT at the origin bucket service","Verify the bucket is dedicated to this service (no other writers of _cave_health keys) and check provider status dashboards for the region"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"if err := objectstore.Probe(ctx, store); err != nil {\n\tif strings.Contains(err.Error(), \"did not round-trip\") {\n\t\t// storage-path integrity failure: retry once, then escalate — do not serve traffic\n\t}\n\treturn err\n}","preventionTips":["Run Probe from FromEnv in production so misbehaving storage frontends are caught at boot, not mid-request","Keep the S3 endpoint pointed at the origin service, never at a cache/CDN that can serve stale writes"],"tags":["objectstore","s3","data-integrity","probe","health-check","go"],"backgroundTag":"data-integrity-check-failed","analyzedSha":"766dce6b1394ebb56a3090748d5a0240a5aefb36","analyzedAt":"2026-08-18T03:14:35.516Z","contentChangedAt":"2026-08-18T03:14:35.516Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}