{"record":{"id":"d65a10925403248e","repo":"hyperledger/fabric","slug":"failed-to-save-snapshot-to-wal-s","errorCode":null,"errorMessage":"failed to save snapshot to WAL: %s","messagePattern":"failed to save snapshot to WAL: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"orderer/consensus/etcdraft/storage.go","lineNumber":283,"sourceCode":"\t}\n\n\treturn nil\n}\n\nfunc (rs *RaftStorage) saveSnap(snap *raftpb.Snapshot) error {\n\trs.lg.Infof(\"Persisting snapshot (term: %d, index: %d) to WAL and disk\", snap.GetMetadata().Term, snap.GetMetadata().Index)\n\n\t// must save the snapshot index to the WAL before saving the\n\t// snapshot to maintain the invariant that we only Open the\n\t// wal at previously-saved snapshot indexes.\n\twalsnap := &walpb.Snapshot{\n\t\tIndex:     snap.GetMetadata().Index,\n\t\tTerm:      snap.GetMetadata().Term,\n\t\tConfState: snap.GetMetadata().GetConfState(),\n\t}\n\n\tif err := rs.wal.SaveSnapshot(walsnap); err != nil {\n\t\treturn errors.Errorf(\"failed to save snapshot to WAL: %s\", err)\n\t}\n\n\tif err := rs.snap.SaveSnap(snap); err != nil {\n\t\treturn errors.Errorf(\"failed to save snapshot to disk: %s\", err)\n\t}\n\n\trs.lg.Debugf(\"Releasing lock to wal files prior to %d\", snap.GetMetadata().GetIndex())\n\tif err := rs.wal.ReleaseLockTo(snap.GetMetadata().GetIndex()); err != nil {\n\t\treturn err\n\t}\n\n\treturn nil\n}\n\n// TakeSnapshot takes a snapshot at index i from MemoryStorage, and persists it to wal and disk.\nfunc (rs *RaftStorage) TakeSnapshot(i uint64, cs *raftpb.ConfState, data []byte) error {\n\trs.lg.Debugf(\"Creating snapshot at index %d from MemoryStorage\", i)\n\tsnap, err := rs.ram.CreateSnapshot(i, cs, data)","sourceCodeStart":265,"sourceCodeEnd":301,"githubUrl":"https://github.com/hyperledger/fabric/blob/2736b63f8fd5932511d56fe68b7039d15977f7f6/orderer/consensus/etcdraft/storage.go#L265-L301","documentation":"saveSnap first records a snapshot entry inside the WAL (wal.SaveSnapshot with the snapshot's index/term/confstate) before writing the snapshot file itself. If the WAL write fails, the node cannot durably record where the snapshot was taken, so the snapshot operation is aborted with this wrapped error.","triggerScenarios":"rs.wal.SaveSnapshot(walsnap) returns an error — typically disk full, WAL file handle closed, I/O error, or the WAL directory being removed while the node runs during TakeSnapshot.","commonSituations":"Disk quota exceeded on the orderer's data volume; WAL directory deleted or unmounted; I/O errors from failing hardware; snapshot triggered concurrently with node shutdown closing the WAL.","solutions":["Check free disk space and filesystem health on the WAL volume","Verify the WAL directory exists and is writable by the orderer process","Check orderer logs for an earlier error that closed the WAL, and restart the node if storage state is inconsistent","Restore from backup / re-provision the node if the storage layer is unrecoverable"],"exampleFix":"// before\nif err := rs.wal.SaveSnapshot(walsnap); err != nil {\n\treturn errors.Errorf(\"failed to save snapshot to WAL: %s\", err)\n}\n// after (operator check before restart)\n// df -h /var/hyperledger/production/orderer\n// chown -R orderer:orderer /var/hyperledger/production/orderer","handlingStrategy":"validation","validationCode":"// operator preflight before starting the orderer\nwalDir := \"/var/hyperledger/production/orderer/channels/<chan>/wal\"\nif fi, err := os.Stat(walDir); err != nil || !fi.IsDir() {\n\tlog.Fatalf(\"WAL dir not accessible: %v\", err)\n}\nif err := syscall.Access(walDir, os.O_WRONLY); err != nil {\n\tlog.Fatalf(\"WAL dir not writable: %v\", err)\n}\nif free, err := freeDiskBytes(walDir); err != nil || free < 1<<30 {\n\tlog.Fatalf(\"insufficient free disk: %d bytes\", free)\n}","typeGuard":null,"tryCatchPattern":"// wrap TakeSnapshot-triggering path\nif err := storage.TakeSnapshot(i, cs, data); err != nil {\n\tif strings.Contains(err.Error(), \"failed to save snapshot to WAL\") {\n\t\tlg.Errorf(\"WAL snapshot write failed (disk full/IO?): %s\", err)\n\t\t// alert ops, do not retry blindly — snapshot must be re-triggered after fix\n\t}\n\treturn err\n}","preventionTips":["Monitor free disk space and alert well before exhaustion","Keep the WAL and snapshot volumes dedicated and mounted at boot","Avoid stopping the node mid-snapshot; check logs after restart","Use healthy storage; watch for SMART/IO errors"],"tags":["raft","wal","snapshot","storage","io"],"backgroundTag":"wal-write-failed","analyzedSha":"2736b63f8fd5932511d56fe68b7039d15977f7f6","analyzedAt":"2026-09-04T08:52:36.465Z","contentChangedAt":"2026-09-04T08:52:36.465Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}