What causes "objectstore: read %q: %w" in JuliusBrussee/caveman?

Retry the Get with a bounded backoff — a single transient network hiccup is the most common cause.

objectstore: read

Error message

objectstore: read %q: %w

What it means

minioStore.Get wraps a failure from io.ReadAll while streaming the object body from MinIO/S3. The error surfaces only when the underlying read of the response stream fails mid-transfer (after a successful StatObject/fetch); 'NoSuchKey' responses are already translated to the sentinel ErrNotFound, and oversized payloads become ErrTooLarge, so anything wrapped here is a genuine transport/IO problem. It preserves the object key and the underlying error via %w.

Solutions

Retry the Get with a bounded backoff — a single transient network hiccup is the most common cause.
Inspect the wrapped error (errors.Unwrap / minio.ToErrorResponse) to check for timeout vs. connection reset vs. TLS.
Verify network path: LB/proxy idle timeouts, keep-alive settings between client and MinIO endpoint.
Confirm TLS configuration matches the endpoint (scheme http vs https, CA trust).
Reduce object size or maxBytes if very large bodies consistently fail, and stream instead of ReadAll.

Example fix

// before
body, err := store.Get(ctx, key, 10<<20)
if err != nil {
    return err
}
// after
body, err := store.Get(ctx, key, 10<<20)
if err != nil {
    if errors.Is(err, objectstore.ErrNotFound) || errors.Is(err, objectstore.ErrTooLarge) {
        return err
    }
    // transient read failure: retry once
    body, err = store.Get(ctx, key, 10<<20)
    if err != nil {
        return fmt.Errorf("get %s: %w", key, err)
    }
}

Defensive patterns

Strategy: retry

Try / catch

if _, err := store.Get(ctx, key, max); err != nil && !errors.Is(err, objectstore.ErrNotFound) && !errors.Is(err, objectstore.ErrTooLarge) {
    return retryOnTransient(ctx, func() error { _, err = store.Get(ctx, key, max); return err })
}

Prevention

Keep request timeouts generous enough for maxBytes-sized objects.
Distinguish sentinel errors (ErrNotFound, ErrTooLarge) from transport failures before retrying.
Monitor connection-reset rates between app and MinIO; they indicate LB/keep-alive issues.
Use keep-alive and sane idle timeouts on the MinIO client transport.

When it happens

Trigger: Calling Get(ctx, key, maxBytes) on a minioStore when the HTTP body read from the object store fails: connection reset mid-stream, TLS errors, timeouts during transfer, or proxy/network interruption after the object was found but before the body was fully read.

Common situations: Flaky networks between app and MinIO; large objects over slow links hitting request deadlines; MinIO gateway to an unhealthy backend; load balancer idle-timeout closing the connection mid-body; expired credentials causing mid-stream auth failures on some proxies.

Related errors

AI-assisted analysis of JuliusBrussee/caveman@3ee70a1026 (2026-09-20). Data as JSON: /api/errors/575af4404f9b1f39. Report an issue: GitHub.

Appendix: source

Thrown at shared/platform/objectstore/objectstore.go:298

	return nil
}

func (m *minioStore) Get(ctx context.Context, key string) ([]byte, error) {
	obj, err := m.client.GetObject(ctx, m.bucket, key, minio.GetObjectOptions{})
	if err != nil {
		return nil, fmt.Errorf("objectstore: get %q: %w", key, err)
	}
	defer obj.Close()
	maxBytes := int64(env.Int("CAVE_OBJECTSTORE_MAX_READ_BYTES", 64<<20))
	if maxBytes <= 0 {
		maxBytes = 64 << 20
	}
	body, err := io.ReadAll(io.LimitReader(obj, maxBytes+1))
	if err != nil {
		if minio.ToErrorResponse(err).Code == "NoSuchKey" {
			return nil, ErrNotFound
		}
		return nil, fmt.Errorf("objectstore: read %q: %w", key, err)
	}
	if int64(len(body)) > maxBytes {
		return nil, ErrTooLarge
	}
	return body, nil
}

func (m *minioStore) Delete(ctx context.Context, key string) error {
	if err := m.client.RemoveObject(ctx, m.bucket, key, minio.RemoveObjectOptions{}); err != nil {
		return fmt.Errorf("objectstore: delete %q: %w", key, err)
	}
	return nil
}

func (m *minioStore) DeleteAllVersions(ctx context.Context, key string) error {
	return m.deleteVersions(ctx, key, true)
}

View on GitHub (pinned to 3ee70a1026)