What causes "objectstore: production read/write/delete probe failed: %w" in JuliusBrussee/caveman?

Check the wrapped cause (%w) for the concrete failure: connection refused vs access denied vs missing bucket.

objectstore: production read/write/delete probe failed

Error message

objectstore: production read/write/delete probe failed: %w

What it means

FromEnv wraps a failed Probe with 'objectstore: production read/write/delete probe failed:'. In production mode the library deliberately proves the bucket actually accepts Put/Get/Delete (and version purge) at startup instead of trusting a syntactically valid config, so misconfigured buckets fail loudly at boot.

Solutions

Check the wrapped cause (%w) for the concrete failure: connection refused vs access denied vs missing bucket.
Create the configured bucket and enable versioning if the deployment requires it.
Fix credentials/IAM so the configured keys can Put, Get, and Delete objects.
Verify endpoint/SSL settings (CAVE_OBJECTSTORE_ENDPOINT, USE_SSL) and network reachability from the service.
Increase the probe timeout env if the endpoint is slow but reachable.

Example fix

// before
CAVE_OBJECTSTORE_ENDPOINT=minio.internal:9000  // unreachable; probe fails at startup
// after
CAVE_OBJECTSTORE_ENDPOINT=minio.default.svc.cluster.local:9000
CAVE_OBJECTSTORE_ACCESS_KEY=...  // with s3:PutObject/GetObject/DeleteObject on the bucket
CAVE_OBJECTSTORE_BUCKET=app-artifacts  // bucket exists and is versioned

Defensive patterns

Strategy: try-catch

Validate before calling

// pre-flight outside the library
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
mc, _ := minio.New(endpoint, &minio.Options{Creds: credentials.NewStaticV4(ak, sk, ""), Secure: useSSL})
if _, err := mc.BucketExists(ctx, bucket); err != nil {
    return fmt.Errorf("bucket %q unreachable before startup: %w", bucket, err)
}

Try / catch

store, err := objectstore.FromEnv(ctx)
if err != nil {
    var probeErr = err // wrapped Probe failure
    return fmt.Errorf("object store not usable (bucket/creds/endpoint): %w", probeErr)
}

Prevention

Create and version the bucket in infrastructure-as-code before deploying.
Grant the static keys Put/Get/Delete on the bucket via IAM policy.
Verify endpoint and UseSSL settings match the deployment (no scheme in endpoint).
Smoke-test storage connectivity in CI against the same env config.

When it happens

Trigger: Calling objectstore.FromEnv with production mode enabled (e.g. CAVE_ENV=production) where the bucket does not exist, credentials lack put/get/delete permissions, the endpoint is unreachable, or the bucket lacks required versioning — Probe returns an error which is wrapped.

Common situations: MinIO/S3 endpoint typo, wrong access/secret keys, IAM policy missing s3:PutObject or s3:DeleteObject, bucket created without versioning, DNS/firewall blocking the storage endpoint, CAVE_OBJECTSTORE_PROBE_TIMEOUT too small for a slow endpoint.

Understand the failure class

Background: "API error: {status}" and "HTTP 401/403/404/429/5xx" errors: non-2xx HTTP responses explained — this error's family across 27 libraries.

Related errors

AI-assisted analysis of JuliusBrussee/caveman@3ee70a1026 (2026-09-20). Data as JSON: /api/errors/2a5ebad63df495ab. Report an issue: GitHub.

Appendix: source

Thrown at shared/platform/objectstore/objectstore.go:182

		Region:    env.String("S3_REGION", "us-east-1"),
		UseSSL:    useSSL,
	}
	if err := validateConfig(cfg, production); err != nil {
		return nil, err
	}
	store, err := New(cfg)
	if err != nil {
		return nil, err
	}
	if production {
		timeout := time.Duration(env.Int("CAVE_OBJECTSTORE_PROBE_TIMEOUT_MS", 10000)) * time.Millisecond
		if timeout <= 0 {
			timeout = 10 * time.Second
		}
		ctx, cancel := context.WithTimeout(context.Background(), timeout)
		defer cancel()
		if err := Probe(ctx, store); err != nil {
			return nil, fmt.Errorf("objectstore: production read/write/delete probe failed: %w", err)
		}
	}
	return store, nil
}

func validateConfig(cfg Config, production bool) error {
	if strings.TrimSpace(cfg.Endpoint) == "" || strings.TrimSpace(cfg.Bucket) == "" {
		return errors.New("objectstore: endpoint and bucket are required")
	}
	if strings.TrimSpace(cfg.AccessKey) == "" || strings.TrimSpace(cfg.SecretKey) == "" {
		return errors.New("objectstore: access key and secret key are required")
	}
	if !production {
		return nil
	}
	if !cfg.UseSSL || strings.HasPrefix(strings.ToLower(strings.TrimSpace(cfg.Endpoint)), "http://") {
		return errors.New("objectstore: production requires TLS")
	}

View on GitHub (pinned to 3ee70a1026)