What causes "batch get with %d keys: %s" in juicedata/juicefs?

Read the wrapped error; if it is 'grpc: received message larger than max', reduce batch size or raise `--max-request-bytes`/etcd's max-request-bytes on the server

juicedata/juicefs · error

batch get with %d keys: %s

Error message

batch get with %d keys: %s

What it means

`etcdTxn.gets` performs a batched read using a single etcd Txn with one OpGet per key. If the overall Txn Do call fails (connection, auth, timeout, message too large), it panics with this message including the key count and error.

Source

Thrown at pkg/meta/tkv_etcd.go:85

	}
	panic("unreachable")
}

func (tx *etcdTxn) gets(keys ...[]byte) [][]byte {
	if len(keys) > 128 {
		var rs = make([][]byte, 0, len(keys))
		for i := 0; i < len(keys); i += 128 {
			rs = append(rs, tx.gets(keys[i:min(i+128, len(keys))]...)...)
		}
		return rs
	}
	ops := make([]etcd.Op, len(keys))
	for i, key := range keys {
		ops[i] = etcd.OpGet(string(key))
	}
	r, err := tx.kv.Do(tx.ctx, etcd.OpTxn(nil, ops, nil))
	if err != nil {
		panic(fmt.Errorf("batch get with %d keys: %s", len(keys), err))
	}
	rs := make(map[string][]byte)
	for _, res := range r.Txn().Responses {
		for _, p := range res.GetResponseRange().Kvs {
			k := string(p.Key)
			tx.observed[k] = p.ModRevision
			rs[k] = p.Value
		}
	}
	values := make([][]byte, len(keys))
	for i, key := range keys {
		k := string(key)
		if v, ok := tx.buffer[k]; ok {
			values[i] = v
			continue
		}
		values[i] = rs[k]
		if len(values[i]) == 0 {

View on GitHub (pinned to c9a67b23e8)

Solutions

Read the wrapped error; if it is 'grpc: received message larger than max', reduce batch size or raise `--max-request-bytes`/etcd's max-request-bytes on the server
Check etcd health (`etcdctl endpoint health`, disk latency); tune heartbeat/election timeouts
Retry the operation after transient failures; ensure stable network between client and etcd
Scale out keys across smaller transactions if a single txn regularly carries hundreds of keys

Example fix

// before (etcd server default)
max-request-bytes = 1572864
// after (etcd server config, if large batches are required)
max-request-bytes = 10485760

Defensive patterns

Strategy: retry

Validate before calling

// Pre-flight: etcd health and gRPC limits
etcdctl endpoint health
etcdctl get "" --prefix --keys-only --limit=1  # auth + connectivity check

Try / catch

err := doMetaOperation()
if err != nil && strings.Contains(err.Error(), "larger than max") {
    // split the batch or raise etcd max-request-bytes
} else if err != nil {
    retry.WithBackoff(err)
}

Prevention

Keep per-txn key counts modest; avoid huge batch metadata operations in custom tooling
Raise etcd server max-request-bytes only deliberately
Monitor etcd fsync/p99 latency; provision fast SSDs
Ensure network MTU and connectivity between client and etcd are stable

When it happens

Trigger: A transaction calling `gets(...)` where the combined etcd txn RPC fails — network interruption, context deadline, too many/too large keys exceeding gRPC message limits, or etcd unavailable.

Common situations: Very large batch reads (many big inode attribute keys) hitting etcd's default 1.5MB gRPC message size; etcd under load with slow fsync causing request timeouts; transient network errors during heavy metadata workloads.

Understand the failure class

Background: Database query failed: Internal Server Error 500s wrapping SQL, Prisma, and connection failures — what to check first — this error's family across 16 libraries.

Related errors

AI-assisted analysis of juicedata/juicefs@c9a67b23e8 (2026-09-06). Data as JSON: /api/errors/999dca0b5010d649. Report an issue: GitHub.