{"record":{"id":"6a20c8a31099c994","repo":"hyperledger/fabric","slug":"failed-to-save-snapshot-to-disk-s","errorCode":null,"errorMessage":"failed to save snapshot to disk: %s","messagePattern":"failed to save snapshot to disk: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"orderer/consensus/etcdraft/storage.go","lineNumber":287,"sourceCode":"\nfunc (rs *RaftStorage) saveSnap(snap *raftpb.Snapshot) error {\n\trs.lg.Infof(\"Persisting snapshot (term: %d, index: %d) to WAL and disk\", snap.GetMetadata().Term, snap.GetMetadata().Index)\n\n\t// must save the snapshot index to the WAL before saving the\n\t// snapshot to maintain the invariant that we only Open the\n\t// wal at previously-saved snapshot indexes.\n\twalsnap := &walpb.Snapshot{\n\t\tIndex:     snap.GetMetadata().Index,\n\t\tTerm:      snap.GetMetadata().Term,\n\t\tConfState: snap.GetMetadata().GetConfState(),\n\t}\n\n\tif err := rs.wal.SaveSnapshot(walsnap); err != nil {\n\t\treturn errors.Errorf(\"failed to save snapshot to WAL: %s\", err)\n\t}\n\n\tif err := rs.snap.SaveSnap(snap); err != nil {\n\t\treturn errors.Errorf(\"failed to save snapshot to disk: %s\", err)\n\t}\n\n\trs.lg.Debugf(\"Releasing lock to wal files prior to %d\", snap.GetMetadata().GetIndex())\n\tif err := rs.wal.ReleaseLockTo(snap.GetMetadata().GetIndex()); err != nil {\n\t\treturn err\n\t}\n\n\treturn nil\n}\n\n// TakeSnapshot takes a snapshot at index i from MemoryStorage, and persists it to wal and disk.\nfunc (rs *RaftStorage) TakeSnapshot(i uint64, cs *raftpb.ConfState, data []byte) error {\n\trs.lg.Debugf(\"Creating snapshot at index %d from MemoryStorage\", i)\n\tsnap, err := rs.ram.CreateSnapshot(i, cs, data)\n\tif err != nil {\n\t\treturn errors.Errorf(\"failed to create snapshot from MemoryStorage: %s\", err)\n\t}\n","sourceCodeStart":269,"sourceCodeEnd":305,"githubUrl":"https://github.com/hyperledger/fabric/blob/2736b63f8fd5932511d56fe68b7039d15977f7f6/orderer/consensus/etcdraft/storage.go#L269-L305","documentation":"saveSnap writes the snapshot file to the snapshot directory via rs.snap.SaveSnap after the WAL entry succeeds. If persisting the snapshot file to disk fails, this error is returned, leaving the WAL snapshot entry present but the snapshot data missing on disk.","triggerScenarios":"rs.snap.SaveSnap(snap) fails — snapshot directory missing/unwritable, disk full, or an I/O error (bad sector, ENOSPC, permission denied) while writing the snapshot file.","commonSituations":"Volume filled between WAL write and snapshot write; snapshot directory removed by cleanup scripts or wrong file permissions after a container image change; failing disk.","solutions":["Check disk space on the snapshot data volume and free space if needed","Verify the snapshots directory exists and is writable by the orderer process","Inspect orderer logs for the underlying I/O error and fix storage hardware/mount issues","Restart the node; if state is inconsistent, restore from backup or re-join the consenter"],"exampleFix":"// before\nif err := rs.snap.SaveSnap(snap); err != nil {\n\treturn errors.Errorf(\"failed to save snapshot to disk: %s\", err)\n}\n// after (operator check)\n// df -h /var/hyperledger/production/orderer/snapshots\n// mkdir -p <snapshots-dir> && chown orderer:orderer <snapshots-dir>","handlingStrategy":"validation","validationCode":"// operator preflight on the snapshot directory\nsnapDir := \"/var/hyperledger/production/orderer/channels/<chan>/snapshots\"\nif err := os.MkdirAll(snapDir, 0o755); err != nil {\n\tlog.Fatalf(\"cannot create snapshot dir: %v\", err)\n}\nif err := syscall.Access(snapDir, os.O_WRONLY); err != nil {\n\tlog.Fatalf(\"snapshot dir not writable by orderer user: %v\", err)\n}","typeGuard":null,"tryCatchPattern":"if err := storage.TakeSnapshot(i, cs, data); err != nil {\n\tif strings.Contains(err.Error(), \"failed to save snapshot to disk\") {\n\t\tlg.Errorf(\"snapshot file write failed: %s\", err)\n\t\t// check ENOSPC / EACCES on snapshots dir before restarting\n\t}\n\treturn err\n}","preventionTips":["Ensure the snapshots directory exists with correct ownership after image/upgrade changes","Do not run aggressive cleanup scripts on the snapshots directory","Monitor disk usage on the snapshot volume","Watch I/O error counters on the storage device"],"tags":["raft","snapshot","disk","storage","io"],"backgroundTag":"snapshot-write-failed","analyzedSha":"2736b63f8fd5932511d56fe68b7039d15977f7f6","analyzedAt":"2026-09-04T08:52:36.465Z","contentChangedAt":"2026-09-04T08:52:36.465Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}