juicedata/juicefs · error

expect err is not nil

Error message

expect err is not nil

What it means

This error comes from the 'delete an object' objbench case when blob.Head(ctx, key) unexpectedly SUCCEEDS after the key was deleted. The test expects Head to return an error (object not found) once the object is deleted; a nil error means the delete did not actually take effect or the backend is not reporting absence. objbench throws it to flag a backend that violates the expected delete visibility semantics.

Source

Thrown at cmd/objbench.go:867

			return fmt.Errorf("failed to head object %s", err)
		} else {
			if h.Key() != key {
				return fmt.Errorf("expected key 'test' but got %s", h.Key())
			}
		}
		return nil
	})

	runCase("delete an object", func(blob object.ObjectStorage) error {
		br := []byte("hello")
		if err := blob.Put(ctx, key, bytes.NewReader(br)); err != nil {
			return fmt.Errorf("put object failed: %s", err)
		}
		if err := blob.Delete(ctx, key); err != nil {
			return fmt.Errorf("delete failed: %s", err)
		}
		if _, err := blob.Head(ctx, key); err == nil {
			return fmt.Errorf("expect err is not nil")
		}

		if err := blob.Delete(ctx, key); err != nil {
			return fmt.Errorf("delete not existed: %v", err)
		}
		return nil
	})

	runCase("delete non-exist", func(blob object.ObjectStorage) error {
		if err := blob.Delete(ctx, key); err != nil {
			return fmt.Errorf("deleting a non-existent object returns an error %v", err)
		}
		return nil
	})

	runCase("list objects", func(blob object.ObjectStorage) error {
		br := []byte("hello")
		if err := blob.Put(ctx, key, bytes.NewReader(br)); err != nil {

View on GitHub (pinned to c9a67b23e8)

Solutions

  1. Inspect the custom/underlying ObjectStorage implementation's Delete and Head for swallowed errors or stale caching.
  2. Disable or shorten lookup caches (e.g. --attr-cache, dir entry cache) when benchmarking filesystem-backed stores.
  3. Confirm the delete actually hits the backend (list the bucket after the run).
  4. If the backend has asynchronous delete semantics, that backend is non-conformant for JuiceFS; fix the implementation.
  5. Rerun objbench with a clean bucket/prefix to rule out leftover objects with the same key.

Example fix

// before: Head succeeds after Delete
if _, err := blob.Head(ctx, key); err == nil {
    return fmt.Errorf("expect err is not nil")
}
// after (in the backend implementation): propagate the removal
func (s *store) Delete(ctx context.Context, key string) error {
    return os.Remove(s.path(key)) // do NOT swallow ENOENT silently
}
Defensive patterns

Strategy: try-catch

Validate before calling

// after delete, confirm absence out-of-band before asserting Head fails
_, err := blob.Head(ctx, key)
deleted := err != nil

Try / catch

if _, err := blob.Head(ctx, key); err == nil {
    log.Printf("object still present after delete; check backend delete semantics/caches")
}

Prevention

When it happens

Trigger: After Put + successful Delete of the test key, blob.Head(ctx, key) returns a nil error and a valid object header — i.e. the object is still listable/headable. Causes include asynchronous delete semantics, a cached/virtual backend (e.g. mem or a buggy filesystem wrapper) that still reports the key, or the delete silently no-oping.

Common situations: Testing a custom ObjectStorage implementation that doesn't propagate deletes; a filesystem backend where unlink failed but Delete swallowed the error; eventual-consistency or caching layers (dir lookup cache) serving a stale entry; testing against a bucket with versioning where Head resolves to an old version.

Understand the failure class

Background: "invalid response format", "malformed payload", "missing data field": when an API returns 200 but the response shape is wrong — this error's family across 23 libraries.

Related errors

AI-assisted analysis of juicedata/juicefs@c9a67b23e8 (2026-09-06). Data as JSON: /api/errors/b2a557e8f5df04b8. Report an issue: GitHub.