{"record":{"id":"60c36aaa86b5f951","repo":"hyperledger/fabric","slug":"error-while-unmarshalling-metadata-bytes-for-ledge","errorCode":null,"errorMessage":"error while unmarshalling metadata bytes for ledger [%s]","messagePattern":"error while unmarshalling metadata bytes for ledger \\[(.+?)\\]","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"core/ledger/kvledger/kv_ledger_provider.go","lineNumber":452,"sourceCode":"\tlogger.Debug(\"Removing ledgers in state UNDER_CONSTRUCTION or UNDER_DELETION\")\n\titr := p.idStore.db.GetIterator(metadataKeyPrefix, metadataKeyStop)\n\tdefer itr.Release()\n\tif err := itr.Error(); err != nil {\n\t\treturn errors.WithMessage(err, \"error obtaining iterator for incomplete ledger scans\")\n\t}\n\tfor {\n\t\thasMore := itr.Next()\n\t\terr := itr.Error()\n\t\tif err != nil {\n\t\t\treturn errors.WithMessage(err, \"error while iterating over ledger list while scanning for incomplete ledgers\")\n\t\t}\n\t\tif !hasMore {\n\t\t\treturn nil\n\t\t}\n\t\tledgerID := ledgerIDFromMetadataKey(itr.Key())\n\t\tmetadata := &msgs.LedgerMetadata{}\n\t\tif err := proto.Unmarshal(itr.Value(), metadata); err != nil {\n\t\t\treturn errors.Wrapf(err, \"error while unmarshalling metadata bytes for ledger [%s]\", ledgerID)\n\t\t}\n\t\tif metadata.Status == msgs.Status_UNDER_CONSTRUCTION || metadata.Status == msgs.Status_UNDER_DELETION {\n\t\t\tlogger.Infow(\n\t\t\t\t\"A partial ledger was identified at peer launch, indicating a peer stop/crash during creation or a failed channel unjoin.  The partial ledger wil be deleted.\",\n\t\t\t\t\"ledgerID\", ledgerID,\n\t\t\t\t\"Status\", metadata.Status,\n\t\t\t)\n\t\t\tif err := p.runCleanup(ledgerID); err != nil {\n\t\t\t\tlogger.Errorw(\n\t\t\t\t\t\"Error while deleting a partially created ledger at start\",\n\t\t\t\t\t\"ledgerID\", ledgerID,\n\t\t\t\t\t\"Status\", metadata.Status,\n\t\t\t\t\t\"error\", err,\n\t\t\t\t)\n\t\t\t\treturn errors.WithMessagef(err, \"error while deleting a partially constructed ledger with status [%s] at start for ledger = [%s]\", metadata.Status, ledgerID)\n\t\t\t}\n\t\t}\n\t}","sourceCodeStart":434,"sourceCodeEnd":470,"githubUrl":"https://github.com/hyperledger/fabric/blob/2736b63f8fd5932511d56fe68b7039d15977f7f6/core/ledger/kvledger/kv_ledger_provider.go#L434-L470","documentation":"During provider startup, deletePartialLedgers iterates all ledger metadata keys in the idStore and proto-unmarshals each value. If a stored metadata blob cannot be decoded into msgs.LedgerMetadata, the startup aborts with this wrapped error. It indicates idStore corruption or bytes written by an incompatible format/version.","triggerScenarios":"NewProvider -> deletePartialLedgers (also invoked directly by tests) when proto.Unmarshal(itr.Value(), metadata) fails for a ledger metadata key: corrupted idStore bytes, truncated LevelDB values after a crash, or metadata written by a different Fabric version/protobuf schema.","commonSituations":"LevelDB files damaged by power loss or an incomplete copy (rsync of a live DB); restoring an idStore from mismatched backup parts; upgrading/downgrading Fabric across incompatible id-store formats without running the format upgrade.","solutions":["Restore the idStore (and all ledger DBs together) from a known-good, consistent backup taken with the peer stopped.","Check disk health and copy procedures; never copy LevelDB directories while the peer is running.","If only one ledger is corrupt and its data is expendable, surgically remove that ledger's metadata/genesis keys from the idStore (offline) or restore and then unjoin/re-join the channel.","If format mismatch is suspected, run the id-store/DB format upgrade path (ledger.UpgradeIDStoreFormat / UpgradeDBs) appropriate for your Fabric version before starting."],"exampleFix":"// before: copying a live LevelDB while the peer runs yields corrupt metadata\nrsync -a /var/hyperledger/production/ledgersData/ backup/\n\n// after: stop the peer, then copy for a consistent idStore\nsystemctl stop peer\nrsync -a /var/hyperledger/production/ledgersData/ backup/\nsystemctl start peer","handlingStrategy":"validation","validationCode":"// Offline check before peer start: open the idStore read-only and verify metadata decodes\nfunc checkMetadataDecodable(dbPath string) error {\n\tdb, err := leveldb.OpenFile(dbPath, &opt.Options{ReadOnly: true})\n\tif err != nil {\n\t\treturn err\n\t}\n\tdefer db.Close()\n\titr := db.NewIterator(util.BytesPrefix([]byte(\"l\")), nil)\n\tdefer itr.Release()\n\tfor itr.Next() {\n\t\tm := &msgs.LedgerMetadata{}\n\t\tif err := proto.Unmarshal(itr.Value(), m); err != nil {\n\t\t\treturn fmt.Errorf(\"corrupt metadata for key %q: %w\", itr.Key(), err)\n\t\t}\n\t}\n\treturn itr.Error()\n}","typeGuard":null,"tryCatchPattern":"if err := kvledger.NewProvider(conf); err != nil {\n\tif strings.Contains(err.Error(), \"error while unmarshalling metadata bytes\") {\n\t\tlog.Fatalf(\"idStore is corrupt: restore ledgersData from a consistent offline backup: %v\", err)\n\t}\n\treturn err\n}","preventionTips":["Always stop the peer before copying or backing up LevelDB data directories.","Back up ledgersData as one atomic set (idStore + state + history + pvtdata + bookkeeping).","Monitor disk/filesystem health; use journaled filesystems and UPS to reduce corruption risk.","Pin Fabric versions and run official format upgrades when upgrading peers."],"tags":["corruption","idstore","protobuf","unmarshal","peer-startup","go"],"backgroundTag":"protobuf-unmarshal-failed","analyzedSha":"2736b63f8fd5932511d56fe68b7039d15977f7f6","analyzedAt":"2026-09-04T08:52:36.465Z","contentChangedAt":"2026-09-04T08:52:36.465Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}