What causes "expect key %v, but got %v" in juicedata/juicefs?

Check whether the key should exist; dump related keys with `etcdctl get --prefix` to see what etcd actually returned

expect key %v, but got %v

Error message

expect key %v, but got %v

What it means

`etcdTxn.get` fetched a key but etcd returned a KV whose key differs from the requested one (since Get with a key acts as a range from that key). The wrapper panics because the transaction observed the wrong key, which would break optimistic-concurrency (observed revision) bookkeeping.

Source

Thrown at pkg/meta/tkv_etcd.go:65

		return v
	}
	resp, err := tx.kv.Get(tx.ctx, k, etcd.WithLimit(1))
	if err != nil {
		panic(fmt.Errorf("get %v: %s", k, err))
	}
	if resp.Count == 0 {
		tx.observed[k] = 0
		return nil
	}
	if resp.Count > 1 {
		panic(fmt.Errorf("expect 1 keys but got %d", resp.Count))
	}
	for _, pair := range resp.Kvs {
		if bytes.Equal(pair.Key, key) {
			tx.observed[k] = pair.ModRevision
			return pair.Value
		} else {
			panic(fmt.Errorf("expect key %v, but got %v", k, string(pair.Key)))
		}
	}
	panic("unreachable")
}

func (tx *etcdTxn) gets(keys ...[]byte) [][]byte {
	if len(keys) > 128 {
		var rs = make([][]byte, 0, len(keys))
		for i := 0; i < len(keys); i += 128 {
			rs = append(rs, tx.gets(keys[i:min(i+128, len(keys))]...)...)
		}
		return rs
	}
	ops := make([]etcd.Op, len(keys))
	for i, key := range keys {
		ops[i] = etcd.OpGet(string(key))
	}
	r, err := tx.kv.Do(tx.ctx, etcd.OpTxn(nil, ops, nil))

View on GitHub (pinned to c9a67b23e8)

Solutions

Check whether the key should exist; dump related keys with `etcdctl get --prefix` to see what etcd actually returned
Verify you are running an unmodified JuiceFS build and a supported etcd version
If the key was deleted concurrently, retry the operation; if it reproduces, report to JuiceFS with versions and key bytes

Defensive patterns

Strategy: try-catch

Try / catch

if strings.Contains(err.Error(), "expect key ") {
    log.Fatalf("etcd txn observed wrong key, likely deleted concurrently or a bug: %v", err)
}

Prevention

Avoid sharing one metadata DB across unrelated tools that delete keys
Use a dedicated prefix for the volume
Report reproducible cases to JuiceFS with etcd version and stack trace

When it happens

Trigger: The etcd Get returns a successor key instead of the exact requested key — happens when the requested key does not exist but the Get was issued without proper range end (WithRange/WithPrefix), so the next key in order is returned.

Common situations: Reading a deleted/nonexistent key with a misconfigured txn implementation; a forked or modified tkv_etcd.go; extremely rare with stock code, points to implementation or etcd-server anomaly.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

AI-assisted analysis of juicedata/juicefs@c9a67b23e8 (2026-09-06). Data as JSON: /api/errors/ac527d429650f779. Report an issue: GitHub.