JuliusBrussee/caveman · critical

probe bytes did not round-trip

Error message

probe bytes did not round-trip

What it means

objectstore.Probe writes 32 random bytes to _cave_health/<random> under the configured bucket, reads them back, and requires byte equality. A mismatch means the store round-tripped different bytes than were written — a strong signal of corruption, an interfering proxy/cache, or a backend that is not honoring writes the way production retention needs. The probe body is random and tenant-free, so content is never the cause.

Source

Thrown at shared/platform/objectstore/objectstore.go:255

	key := "_cave_health/" + base64.RawURLEncoding.EncodeToString(probe)
	if err := store.Put(ctx, key, probe, "application/octet-stream"); err != nil {
		return err
	}
	cleaned := false
	defer func() {
		if !cleaned {
			// Keep health checks within their caller deadline. A failed probe may
			// leave one random, tenant-free canary for lifecycle cleanup; readiness
			// must never hang on an unbounded background delete.
			_ = PurgeObject(ctx, store, key)
		}
	}()
	got, err := store.Get(ctx, key)
	if err != nil {
		return err
	}
	if !bytes.Equal(got, probe) {
		return errors.New("probe bytes did not round-trip")
	}
	if err := PurgeObject(ctx, store, key); err != nil {
		return err
	}
	cleaned = true
	exists, err := store.Exists(ctx, key)
	if err != nil {
		return err
	}
	if exists {
		return errors.New("probe object still exists after purge")
	}
	return nil
}

func (m *minioStore) Put(ctx context.Context, key string, body []byte, contentType string) error {
	if contentType == "" {
		contentType = "application/octet-stream"

View on GitHub (pinned to 766dce6b13)

Solutions

1. Re-run the probe once to rule out a transient read-after-write window
2. If it persists, inspect what the endpoint frontends: bypass any CDN/proxy and point S3_ENDPOINT at the origin bucket service
3. Verify the bucket is dedicated to this service (no other writers of _cave_health keys) and check provider status dashboards for the region

if err := objectstore.Probe(ctx, store); err != nil {
	if strings.Contains(err.Error(), "did not round-trip") {
		// storage-path integrity failure: retry once, then escalate — do not serve traffic
	}
	return err
}

When it happens

Trigger: Probe's Get after a successful Put returns bytes that differ — e.g. a misconfigured CDN or gateway in front of the bucket serving stale/other content, a bucket name collision where another writer overwrote the key, or storage corruption. Runs automatically from FromEnv in production, and from any explicit health check.

Common situations: S3_ENDPOINT pointed at a caching proxy that serves a stale copy of a just-written key; two environments sharing one bucket where a lifecycle or replication job mutates objects; exotic MinIO builds with broken read-after-write consistency; extremely rare bit-level corruption on long-lived nodes.

Related errors

• probe object still exists after purge
• objectstore: S3_ENDPOINT is required in production
• objectstore: endpoint and bucket are required
• objectstore: access key and secret key are required
• objectstore: production requires TLS

AI-assisted analysis of JuliusBrussee/caveman@766dce6b13 (2026-08-18). Data as JSON: /api/errors/b8fa3ccb2ef22ce7. Report an issue: GitHub.