{"record":{"id":"09152f3b5660214b","repo":"juicedata/juicefs","slug":"invalid-value-for-delayed-slices-q-v","errorCode":null,"errorMessage":"invalid value for delayed slices %q: %v","messagePattern":"invalid value for delayed slices %q: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/meta/tkv.go","lineNumber":3087,"sourceCode":"\t\t\treturn nil\n\t\t}, 0); err != nil {\n\t\t\tlogger.Warnf(\"Scan delayed slices: %s\", err)\n\t\t\treturn count, err\n\t\t}\n\n\t\tfor _, key := range keys {\n\t\t\tif ctx.Canceled() {\n\t\t\t\treturn count, ctx.Err()\n\t\t\t}\n\t\t\tif err := m.txn(ctx, func(tx *kvTxn) error {\n\t\t\t\tss, rs = ss[:0], rs[:0]\n\t\t\t\tbuf := tx.get(key)\n\t\t\t\tif len(buf) == 0 {\n\t\t\t\t\treturn nil\n\t\t\t\t}\n\t\t\t\tm.decodeDelayedSlices(buf, &ss)\n\t\t\t\tif len(ss) == 0 {\n\t\t\t\t\treturn fmt.Errorf(\"invalid value for delayed slices %q: %v\", key, buf)\n\t\t\t\t}\n\t\t\t\tfor _, s := range ss {\n\t\t\t\t\trs = append(rs, tx.incrBy(m.sliceKey(s.Id, s.Size), -1))\n\t\t\t\t}\n\t\t\t\ttx.delete(key)\n\t\t\t\tb := utils.ReadBuffer(key[1:])\n\t\t\t\tts := int64(b.Get64())\n\t\t\t\tid := b.Get64()\n\t\t\t\tm.genLog(tx, time.Now(), \"CLEANUP_DELAYED_SLICES(%d,%d)\", id, ts)\n\t\t\t\treturn nil\n\t\t\t}); err != nil {\n\t\t\t\tlogger.Warnf(\"Cleanup delayed slices %q: %s\", key, err)\n\t\t\t\tcontinue\n\t\t\t}\n\t\t\tfor i, s := range ss {\n\t\t\t\tif rs[i] < 0 {\n\t\t\t\t\tm.deleteSlice(s.Id, s.Size)\n\t\t\t\t\tcount++","sourceCodeStart":3069,"sourceCodeEnd":3105,"githubUrl":"https://github.com/juicedata/juicefs/blob/c9a67b23e8e08ec23ec331aa6f1675e2319e921c/pkg/meta/tkv.go#L3069-L3105","documentation":"kvMeta.deleteSustained (delayed-slice reference counting) decodes a 'delayed slices' metadata record with decodeDelayedSlices; the decode produced zero slices even though the raw buffer was non-empty, meaning the stored value is corrupt or written by an incompatible format. The client refuses to proceed rather than decrementing slice refcounts against garbage data.","triggerScenarios":"Running the 'gc' or 'fsck' style sweep (deleteSustained/delSustained path) over a TKV (TiKV/etcd/Badger/FDB) metadata engine whose DELAYED_SLICE ('S' prefix) record bytes are corrupt, truncated, or written by a different JuiceFS version with a different serialization.","commonSituations":"Metadata engine data partially restored from a backup or dump/load migration; concurrent writes from mixed JuiceFS client versions; disk-level corruption in a KV store.","solutions":["Verify all clients use the same JuiceFS version (check MinClientVersion compatibility; upgrade/downgrade to match the version that wrote the record)","Dump the offending key ('S'-prefixed delayed slice key) and inspect the raw bytes; if corrupt, restore metadata from a known-good backup (juicefs dump/load)","Run 'juicefs fsck' against the volume to detect and repair inconsistent metadata","If the key is genuinely orphaned garbage, delete the specific key directly in the KV store after backing it up"],"exampleFix":"// before (corrupt record blocks gc entirely)\nm.decodeDelayedSlices(buf, &ss)\nif len(ss) == 0 {\n\treturn fmt.Errorf(\"invalid value for delayed slices %q: %v\", key, buf)\n}\n// after (operational workaround: skip and log the bad key)\nm.decodeDelayedSlices(buf, &ss)\nif len(ss) == 0 {\n\tlogger.Warnf(\"skip corrupt delayed slices %q: %v\", key, buf)\n\treturn nil\n}","handlingStrategy":"validation","validationCode":"// before gc/fsck, verify metadata consistency\nout, _ := exec.Command(\"juicefs\", \"fsck\", metaURL).CombinedOutput()\nif strings.Contains(string(out), \"corrupt\") { /* repair or restore before gc */ }","typeGuard":null,"tryCatchPattern":"if err := runGc(); err != nil {\n\tif strings.Contains(err.Error(), \"invalid value for delayed slices\") {\n\t\t// restore metadata from backup, then retry\n\t}\n}","preventionTips":["Use a single, consistent JuiceFS client version across all mounts","Take regular 'juicefs dump' backups and validate them by test-loading","Never hand-edit keys in the metadata KV store","Run 'juicefs fsck' after any restore or migration"],"tags":["metadata","corruption","kv-store","gc"],"backgroundTag":"schema-validation-failed","analyzedSha":"c9a67b23e8e08ec23ec331aa6f1675e2319e921c","analyzedAt":"2026-09-06T17:55:48.476Z","contentChangedAt":"2026-09-06T17:55:48.476Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}