{"record":{"id":"999dca0b5010d649","repo":"juicedata/juicefs","slug":"batch-get-with-d-keys-s","errorCode":null,"errorMessage":"batch get with %d keys: %s","messagePattern":"batch get with (.+?) keys: (.+?)","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/meta/tkv_etcd.go","lineNumber":85,"sourceCode":"\t}\n\tpanic(\"unreachable\")\n}\n\nfunc (tx *etcdTxn) gets(keys ...[]byte) [][]byte {\n\tif len(keys) > 128 {\n\t\tvar rs = make([][]byte, 0, len(keys))\n\t\tfor i := 0; i < len(keys); i += 128 {\n\t\t\trs = append(rs, tx.gets(keys[i:min(i+128, len(keys))]...)...)\n\t\t}\n\t\treturn rs\n\t}\n\tops := make([]etcd.Op, len(keys))\n\tfor i, key := range keys {\n\t\tops[i] = etcd.OpGet(string(key))\n\t}\n\tr, err := tx.kv.Do(tx.ctx, etcd.OpTxn(nil, ops, nil))\n\tif err != nil {\n\t\tpanic(fmt.Errorf(\"batch get with %d keys: %s\", len(keys), err))\n\t}\n\trs := make(map[string][]byte)\n\tfor _, res := range r.Txn().Responses {\n\t\tfor _, p := range res.GetResponseRange().Kvs {\n\t\t\tk := string(p.Key)\n\t\t\ttx.observed[k] = p.ModRevision\n\t\t\trs[k] = p.Value\n\t\t}\n\t}\n\tvalues := make([][]byte, len(keys))\n\tfor i, key := range keys {\n\t\tk := string(key)\n\t\tif v, ok := tx.buffer[k]; ok {\n\t\t\tvalues[i] = v\n\t\t\tcontinue\n\t\t}\n\t\tvalues[i] = rs[k]\n\t\tif len(values[i]) == 0 {","sourceCodeStart":67,"sourceCodeEnd":103,"githubUrl":"https://github.com/juicedata/juicefs/blob/c9a67b23e8e08ec23ec331aa6f1675e2319e921c/pkg/meta/tkv_etcd.go#L67-L103","documentation":"`etcdTxn.gets` performs a batched read using a single etcd Txn with one OpGet per key. If the overall Txn Do call fails (connection, auth, timeout, message too large), it panics with this message including the key count and error.","triggerScenarios":"A transaction calling `gets(...)` where the combined etcd txn RPC fails — network interruption, context deadline, too many/too large keys exceeding gRPC message limits, or etcd unavailable.","commonSituations":"Very large batch reads (many big inode attribute keys) hitting etcd's default 1.5MB gRPC message size; etcd under load with slow fsync causing request timeouts; transient network errors during heavy metadata workloads.","solutions":["Read the wrapped error; if it is 'grpc: received message larger than max', reduce batch size or raise `--max-request-bytes`/etcd's max-request-bytes on the server","Check etcd health (`etcdctl endpoint health`, disk latency); tune heartbeat/election timeouts","Retry the operation after transient failures; ensure stable network between client and etcd","Scale out keys across smaller transactions if a single txn regularly carries hundreds of keys"],"exampleFix":"// before (etcd server default)\nmax-request-bytes = 1572864\n// after (etcd server config, if large batches are required)\nmax-request-bytes = 10485760","handlingStrategy":"retry","validationCode":"// Pre-flight: etcd health and gRPC limits\netcdctl endpoint health\netcdctl get \"\" --prefix --keys-only --limit=1  # auth + connectivity check","typeGuard":null,"tryCatchPattern":"err := doMetaOperation()\nif err != nil && strings.Contains(err.Error(), \"larger than max\") {\n    // split the batch or raise etcd max-request-bytes\n} else if err != nil {\n    retry.WithBackoff(err)\n}","preventionTips":["Keep per-txn key counts modest; avoid huge batch metadata operations in custom tooling","Raise etcd server max-request-bytes only deliberately","Monitor etcd fsync/p99 latency; provision fast SSDs","Ensure network MTU and connectivity between client and etcd are stable"],"tags":["etcd","grpc","batch","timeout","juicefs"],"backgroundTag":"database-query-failed","analyzedSha":"c9a67b23e8e08ec23ec331aa6f1675e2319e921c","analyzedAt":"2026-09-06T17:55:48.476Z","contentChangedAt":"2026-09-06T17:55:48.476Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}