hyperledger/fabric · critical

error while unmarshalling metadata bytes for ledger [%s]

Error message

error while unmarshalling metadata bytes for ledger [%s]

What it means

During provider startup, deletePartialLedgers iterates all ledger metadata keys in the idStore and proto-unmarshals each value. If a stored metadata blob cannot be decoded into msgs.LedgerMetadata, the startup aborts with this wrapped error. It indicates idStore corruption or bytes written by an incompatible format/version.

Source

Thrown at core/ledger/kvledger/kv_ledger_provider.go:452

	logger.Debug("Removing ledgers in state UNDER_CONSTRUCTION or UNDER_DELETION")
	itr := p.idStore.db.GetIterator(metadataKeyPrefix, metadataKeyStop)
	defer itr.Release()
	if err := itr.Error(); err != nil {
		return errors.WithMessage(err, "error obtaining iterator for incomplete ledger scans")
	}
	for {
		hasMore := itr.Next()
		err := itr.Error()
		if err != nil {
			return errors.WithMessage(err, "error while iterating over ledger list while scanning for incomplete ledgers")
		}
		if !hasMore {
			return nil
		}
		ledgerID := ledgerIDFromMetadataKey(itr.Key())
		metadata := &msgs.LedgerMetadata{}
		if err := proto.Unmarshal(itr.Value(), metadata); err != nil {
			return errors.Wrapf(err, "error while unmarshalling metadata bytes for ledger [%s]", ledgerID)
		}
		if metadata.Status == msgs.Status_UNDER_CONSTRUCTION || metadata.Status == msgs.Status_UNDER_DELETION {
			logger.Infow(
				"A partial ledger was identified at peer launch, indicating a peer stop/crash during creation or a failed channel unjoin.  The partial ledger wil be deleted.",
				"ledgerID", ledgerID,
				"Status", metadata.Status,
			)
			if err := p.runCleanup(ledgerID); err != nil {
				logger.Errorw(
					"Error while deleting a partially created ledger at start",
					"ledgerID", ledgerID,
					"Status", metadata.Status,
					"error", err,
				)
				return errors.WithMessagef(err, "error while deleting a partially constructed ledger with status [%s] at start for ledger = [%s]", metadata.Status, ledgerID)
			}
		}
	}

View on GitHub (pinned to 2736b63f8f)

Solutions

1. Restore the idStore (and all ledger DBs together) from a known-good, consistent backup taken with the peer stopped.
2. Check disk health and copy procedures; never copy LevelDB directories while the peer is running.
3. If only one ledger is corrupt and its data is expendable, surgically remove that ledger's metadata/genesis keys from the idStore (offline) or restore and then unjoin/re-join the channel.
4. If format mismatch is suspected, run the id-store/DB format upgrade path (ledger.UpgradeIDStoreFormat / UpgradeDBs) appropriate for your Fabric version before starting.

Example fix

// before: copying a live LevelDB while the peer runs yields corrupt metadata
rsync -a /var/hyperledger/production/ledgersData/ backup/

// after: stop the peer, then copy for a consistent idStore
systemctl stop peer
rsync -a /var/hyperledger/production/ledgersData/ backup/
systemctl start peer

// Offline check before peer start: open the idStore read-only and verify metadata decodes
func checkMetadataDecodable(dbPath string) error {
	db, err := leveldb.OpenFile(dbPath, &opt.Options{ReadOnly: true})
	if err != nil {
		return err
	}
	defer db.Close()
	itr := db.NewIterator(util.BytesPrefix([]byte("l")), nil)
	defer itr.Release()
	for itr.Next() {
		m := &msgs.LedgerMetadata{}
		if err := proto.Unmarshal(itr.Value(), m); err != nil {
			return fmt.Errorf("corrupt metadata for key %q: %w", itr.Key(), err)
		}
	}
	return itr.Error()
}

if err := kvledger.NewProvider(conf); err != nil {
	if strings.Contains(err.Error(), "error while unmarshalling metadata bytes") {
		log.Fatalf("idStore is corrupt: restore ledgersData from a consistent offline backup: %v", err)
	}
	return err
}

When it happens

Trigger: NewProvider -> deletePartialLedgers (also invoked directly by tests) when proto.Unmarshal(itr.Value(), metadata) fails for a ledger metadata key: corrupted idStore bytes, truncated LevelDB values after a crash, or metadata written by a different Fabric version/protobuf schema.

Common situations: LevelDB files damaged by power loss or an incomplete copy (rsync of a live DB); restoring an idStore from mismatched backup parts; upgrading/downgrading Fabric across incompatible id-store formats without running the format upgrade.

Understand the failure class

Background: "cannot parse invalid wire-format data", "cannot unmarshal", "failed unmarshalling": protobuf unmarshal errors explained — this error's family across 10 libraries.

Related errors

• error marshalling ledger metadata
• transaction %d is invalid: %v
• error unmarshalling
• error unmarshalling original config
• error unmarshalling updated config

AI-assisted analysis of hyperledger/fabric@2736b63f8f (2026-09-04). Data as JSON: /api/errors/60c36aaa86b5f951. Report an issue: GitHub.