hyperledger/fabric · critical

failed to repair WAL: %s

Error message

failed to repair WAL: %s

What it means

This error is thrown by RaftStorage.createOrReadWAL when, after detecting an io.ErrUnexpectedEOF while reading the Write-Ahead Log, the etcd/raft wal.Repair() call fails to truncate the corrupt WAL segment. It means the WAL is damaged in a way the single automatic repair pass cannot fix, so chain start-up aborts rather than risk losing raft state.

Source

Thrown at orderer/consensus/etcdraft/storage.go:221

	for {
		if w, err = wal.Open(lg.Zap(), walDir, walsnap); err != nil {
			return nil, st, nil, errors.Errorf("failed to open WAL: %s", err)
		}

		if _, st, ents, err = w.ReadAll(); err != nil {
			lg.Warnf("Failed to read WAL: %s", err)

			if errc := w.Close(); errc != nil {
				return nil, st, nil, errors.Errorf("failed to close erroneous WAL: %s", errc)
			}

			// only repair UnexpectedEOF and only repair once
			if repaired || err != io.ErrUnexpectedEOF {
				return nil, st, nil, errors.Errorf("failed to read WAL and cannot repair: %s", err)
			}

			if !wal.Repair(lg.Zap(), walDir) {
				return nil, st, nil, errors.Errorf("failed to repair WAL: %s", err)
			}

			repaired = true
			// next loop should be able to open WAL and return
			continue
		}

		// successfully opened WAL and read all entries, break
		break
	}

	return w, st, ents, nil
}

// Snapshot returns the latest snapshot stored in memory
func (rs *RaftStorage) Snapshot() *raftpb.Snapshot {
	sn, _ := rs.ram.Snapshot() // Snapshot always returns nil error
	return sn

View on GitHub (pinned to 2736b63f8f)

Solutions

  1. Inspect the WAL directory for corrupted/truncated segment files and check disk space and permissions
  2. If repair cannot fix it, take a recent on-disk snapshot, then move the corrupted WAL directory aside (backup it) and restart so the node replays from the snapshot
  3. Restore the node's data from a backup or re-join the node to the channel as a fresh consenter if no valid snapshot exists
  4. Ensure orderly shutdown handling to avoid repeated kill -9 during writes

Example fix

// before
if !wal.Repair(lg.Zap(), walDir) {
	return nil, st, nil, errors.Errorf("failed to repair WAL: %s", err)
}
// after (operator remediation)
// mv /var/hyperledger/production/orderer/channels/<channel>/wal /var/hyperledger/.../wal.corrupt
// ensure a valid snapshot exists in the snapshots dir, then restart the orderer
Defensive patterns

Strategy: validation

Validate before calling

// before starting the orderer
walDir := "/var/hyperledger/production/orderer/channels/<chan>/wal"
entries, err := os.ReadDir(walDir)
if err != nil || len(entries) == 0 {
	log.Fatalf("WAL dir missing/empty: %v", err)
}
for _, e := range entries {
	if info, err := e.Info(); err != nil || info.Size() == 0 {
		log.Printf("suspect WAL segment %s — verify disk health/backup", e.Name())
	}
}

Try / catch

// operator script (error surfaces at startup, not catchable in code)
if ! systemctl start orderer; then
	journalctl -u orderer | grep 'failed to repair WAL' && \
	echo 'Backup snapshot exists?'; ls /var/hyperledger/production/orderer/channels/<chan>/snapshots
fi

Prevention

When it happens

Trigger: Second CreateStorage call after the first repair pass already ran (repaired==true cannot re-enter), or wal.Repair returns false because it cannot identify/truncate the corrupted segment (e.g. malformed file beyond the last valid record, permission or I/O failure during truncation).

Common situations: Node crashed or was killed (kill -9, power loss) mid-WAL-write leaving a partially written tail; disk full during WAL writes; corrupted WAL files restored from an incomplete backup; running the node again after an earlier repair already consumed the one-shot repair attempt.

Related errors

AI-assisted analysis of hyperledger/fabric@2736b63f8f (2026-09-04). Data as JSON: /api/errors/8eab7c5d66ef76c0. Report an issue: GitHub.