{"record":{"id":"8eab7c5d66ef76c0","repo":"hyperledger/fabric","slug":"failed-to-repair-wal-s","errorCode":null,"errorMessage":"failed to repair WAL: %s","messagePattern":"failed to repair WAL: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"orderer/consensus/etcdraft/storage.go","lineNumber":221,"sourceCode":"\tfor {\n\t\tif w, err = wal.Open(lg.Zap(), walDir, walsnap); err != nil {\n\t\t\treturn nil, st, nil, errors.Errorf(\"failed to open WAL: %s\", err)\n\t\t}\n\n\t\tif _, st, ents, err = w.ReadAll(); err != nil {\n\t\t\tlg.Warnf(\"Failed to read WAL: %s\", err)\n\n\t\t\tif errc := w.Close(); errc != nil {\n\t\t\t\treturn nil, st, nil, errors.Errorf(\"failed to close erroneous WAL: %s\", errc)\n\t\t\t}\n\n\t\t\t// only repair UnexpectedEOF and only repair once\n\t\t\tif repaired || err != io.ErrUnexpectedEOF {\n\t\t\t\treturn nil, st, nil, errors.Errorf(\"failed to read WAL and cannot repair: %s\", err)\n\t\t\t}\n\n\t\t\tif !wal.Repair(lg.Zap(), walDir) {\n\t\t\t\treturn nil, st, nil, errors.Errorf(\"failed to repair WAL: %s\", err)\n\t\t\t}\n\n\t\t\trepaired = true\n\t\t\t// next loop should be able to open WAL and return\n\t\t\tcontinue\n\t\t}\n\n\t\t// successfully opened WAL and read all entries, break\n\t\tbreak\n\t}\n\n\treturn w, st, ents, nil\n}\n\n// Snapshot returns the latest snapshot stored in memory\nfunc (rs *RaftStorage) Snapshot() *raftpb.Snapshot {\n\tsn, _ := rs.ram.Snapshot() // Snapshot always returns nil error\n\treturn sn","sourceCodeStart":203,"sourceCodeEnd":239,"githubUrl":"https://github.com/hyperledger/fabric/blob/2736b63f8fd5932511d56fe68b7039d15977f7f6/orderer/consensus/etcdraft/storage.go#L203-L239","documentation":"This error is thrown by RaftStorage.createOrReadWAL when, after detecting an io.ErrUnexpectedEOF while reading the Write-Ahead Log, the etcd/raft wal.Repair() call fails to truncate the corrupt WAL segment. It means the WAL is damaged in a way the single automatic repair pass cannot fix, so chain start-up aborts rather than risk losing raft state.","triggerScenarios":"Second CreateStorage call after the first repair pass already ran (repaired==true cannot re-enter), or wal.Repair returns false because it cannot identify/truncate the corrupted segment (e.g. malformed file beyond the last valid record, permission or I/O failure during truncation).","commonSituations":"Node crashed or was killed (kill -9, power loss) mid-WAL-write leaving a partially written tail; disk full during WAL writes; corrupted WAL files restored from an incomplete backup; running the node again after an earlier repair already consumed the one-shot repair attempt.","solutions":["Inspect the WAL directory for corrupted/truncated segment files and check disk space and permissions","If repair cannot fix it, take a recent on-disk snapshot, then move the corrupted WAL directory aside (backup it) and restart so the node replays from the snapshot","Restore the node's data from a backup or re-join the node to the channel as a fresh consenter if no valid snapshot exists","Ensure orderly shutdown handling to avoid repeated kill -9 during writes"],"exampleFix":"// before\nif !wal.Repair(lg.Zap(), walDir) {\n\treturn nil, st, nil, errors.Errorf(\"failed to repair WAL: %s\", err)\n}\n// after (operator remediation)\n// mv /var/hyperledger/production/orderer/channels/<channel>/wal /var/hyperledger/.../wal.corrupt\n// ensure a valid snapshot exists in the snapshots dir, then restart the orderer","handlingStrategy":"validation","validationCode":"// before starting the orderer\nwalDir := \"/var/hyperledger/production/orderer/channels/<chan>/wal\"\nentries, err := os.ReadDir(walDir)\nif err != nil || len(entries) == 0 {\n\tlog.Fatalf(\"WAL dir missing/empty: %v\", err)\n}\nfor _, e := range entries {\n\tif info, err := e.Info(); err != nil || info.Size() == 0 {\n\t\tlog.Printf(\"suspect WAL segment %s — verify disk health/backup\", e.Name())\n\t}\n}","typeGuard":null,"tryCatchPattern":"// operator script (error surfaces at startup, not catchable in code)\nif ! systemctl start orderer; then\n\tjournalctl -u orderer | grep 'failed to repair WAL' && \\\n\techo 'Backup snapshot exists?'; ls /var/hyperledger/production/orderer/channels/<chan>/snapshots\nfi","preventionTips":["Always shut the orderer down gracefully instead of kill -9","Monitor disk space on the WAL volume","Keep regular backups of the orderer's ledger/snapshot data","Never delete WAL segment files manually while the node runs"],"tags":["raft","wal","corruption","storage","persistence"],"backgroundTag":"wal-corruption","analyzedSha":"2736b63f8fd5932511d56fe68b7039d15977f7f6","analyzedAt":"2026-09-04T08:52:36.465Z","contentChangedAt":"2026-09-04T08:52:36.465Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}